Synthesising 2D Video from 3D Motion Data for Machine Learning Applications

Sensors, vol. 22, pp. 6522

Abstract

To increase the utility of legacy, gold-standard, three-dimensional (3D) motion capture datasets for computer vision-based machine learning applications, this study proposed and validated a method to synthesise two-dimensional (2D) video image frames from historic 3D motion data. We applied the video-based human pose estimation model OpenPose to real (in situ) and synthesised 2D videos and compared anatomical landmark keypoint outputs, with trivial observed differences (2.11−3.49 mm). We further demonstrated the utility of the method in a downstream machine learning use-case in which we trained and then tested the validity of an artificial neural network (ANN) to estimate ground reaction forces (GRFs) using synthesised and real 2D videos. Training an ANN to estimate GRFs using eight OpenPose keypoints derived from synthesised 2D videos resulted in accurate waveform GRF estimations (r > 0.9; nRMSE < 14%). When compared with using the smaller number of real videos only, accuracy was improved by adding the synthetic views and enlarging the dataset. The results highlight the utility of the developed approach to enlarge small 2D video datasets, or to create 2D video images to accompany 3D motion capture datasets to make them accessible for machine learning applications.

Authors 8

1. Marion Mundt

   The University of Western Australia

   Affiliation as printed

   UWA Minderoo Tech & Policy Lab, Law School, The University of Western Australia, Crawley, WA 6009, Australia

2. Henrike Oberlack Aachen Institute of General Mechanics

   RWTH Aachen University

   Affiliation as printed

   Institute of General Mechanics, RWTH Aachen University, 52062 Aachen, Germany

3. Molly Goldacre

   The University of Western Australia

   Affiliation as printed

   UWA Minderoo Tech & Policy Lab, Law School, The University of Western Australia, Crawley, WA 6009, Australia

4. Julia Powles

   The University of Western Australia

   Affiliation as printed

   UWA Minderoo Tech & Policy Lab, Law School, The University of Western Australia, Crawley, WA 6009, Australia

5. Johannes Funken

   German Sport University Cologne

   Affiliation as printed

   Institute of Biomechanics and Orthopaedics, German Sport University Cologne, 50933 Cologne, Germany

6. Corey Morris

   The University of Western Australia

   Affiliation as printed

   School of Human Sciences, The University of Western Australia, Crawley, WA 6009, Australia

   UWA Minderoo Tech & Policy Lab, Law School, The University of Western Australia, Crawley, WA 6009, Australia

7. Wolfgang Potthast

   German Sport University Cologne

   Affiliation as printed

   Institute of Biomechanics and Orthopaedics, German Sport University Cologne, 50933 Cologne, Germany

8. Jacqueline A. Alderson corresponding

   The University of Western Australia · Auckland University of Technology

   Affiliation as printed

   Sports Performance Research Institute New Zealand (SPRINZ), Auckland University of Technology, Auckland 1010, New Zealand

   UWA Minderoo Tech & Policy Lab, Law School, The University of Western Australia, Crawley, WA 6009, Australia