Condor: Mobile Golf Swing Tracking via Sensor Fusion using Conditional Generative Adversarial Networks
Authors: 
Hong Jia, 
Jun Liu, Yuezhong Wu, Tomasz Bednarz, 
Lina Yao, 
Wen HuAuthors Info & Claims
Hong Jia, Jun Liu, Yuezhong Wu, Tomasz Bednarz, Lina Yao, Wen HuAuthors Info & ClaimsPages 31 - 42
Abstract
This paper explores the possibility of incorporating sensor-rich and ubiquitously deployed mobile devices into sports analytics, particularly to the game of golf. We develop a novel solution to track a player's swing in threedimensional (3D) space using inexpensive tools such as depth sensors and Inertial Measurement Units (IMUs). Existing solutions based on these devices cannot produce consistent and accurate swing-tracking. This is due to commonly known issues with occlusion and low sampling rates generated by depth sensors and complex IMU noise models. To overcome these limitations, we introduce Condor, a tailored deep neural networks to make use of sensor fusion to combine the advantages of these two sensor modalities, where IMUs are not affected by occlusion and can support high sampling rates and depth sensors produce more accurate motion measurements than those produced by IMU. Condor could be implemented with edge devices such as a smart wristband and a smartphone, which are ubiquitously available, for accurate golf swing analytics (e.g. tracking, analysis and assessment) in the wild. Our comprehensive experiment shows that proposed method outperforms other solutions and reaches 6.57 cm error in subject-dependent model and less than 10 cm error for unknow-subjects via a tailored conditional Generative Adversarial Networks (cGAN).
References
[1]
R. Alp Güler, N. Neverova, and I. Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7297-7306, 2018.
[2]
M. Andriluka, U. Iqbal, E. Insafutdinov, L. Pishchulin, A. Milan, J. Gall, and B. Schiele. Posetrack: A benchmark for human pose estimation and tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5167-5176, 2018.
[3]
D. Arsenault. A quaternion-based motion tracking and gesture recognition system using wireless inertial sensors. PhD thesis, Carleton University, 2014.
[4]
P. Blank, B. H. Groh, and B. M. Eskofier. Ball speed and spin estimation in table tennis using a racket-mounted inertial sensor. In Proceedings of the 2017 ACM International Symposium on Wearable Computers, ISWC '17, pages 2-9, New York, NY, USA, 2017. ACM.
[5]
A. Brennan, J. Zhang, K. Deluzio, and Q. Li. Quantification of inertial sensor-based 3d joint angle measurement accuracy using an instrumented gimbal. Gait & posture, 34(3):320-323, 2011.
[6]
Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh. Openpose: realtime multi-person 2d pose estimation using part affinity fields. arXiv preprint arXiv:1812.08008, 2018.
[7]
C. Chen, X. Lu, A. Markham, and N. Trigoni. Ionet: Learning to cure the curse of drift in inertial odometry. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[8]
M. Chesser, L. Chea, and D. C. Ranasinghe. Field deployable realtime indoor spatial tracking system for human behavior observations. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, pages 369-370. ACM, 2018.
[9]
S. Choppin, S. Albrecht, J. Spurr, and J. Capel-Davies. The effect of ball wear on ball aerodynamics: An investigation using hawk-eye data. In Multidisciplinary Digital Publishing Institute Proceedings, volume 2, page 265, 2018.
[10]
S. Chun, D. Kang, H.-R. Choi, A. Park, K.-K. Lee, and J. Kim. A sensor-aided self coaching model for uncocking improvement in golf swing. Multimedia Tools and Applications, 72(1):253-279, 2014.
[11]
D. Comotti, M. Ermidoro, M. Galizzi, and A. Vitali. Development of a wireless low-power multi-sensor network for motion tracking applications. In 2013 IEEE International Conference on Body Sensor Networks, pages 1-6. IEEE, 2013.
[12]
R. Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
[13]
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS'14, pages 2672-2680, Cambridge, MA, USA, 2014. MIT Press.
[14]
M. Gowda, A. Dhekne, S. Shen, R. R. Choudhury, L. Yang, S. Golwalkar, and A. Essanian. Bringing iot to sports analytics. In 14th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 17), pages 499-513, 2017.
[15]
K. Greff, R. K. Srivastava, J. Koutnı́k, B. R. Steunebrink, and J. Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10):2222-2232, 2016.
[16]
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
[17]
S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
[18]
L. Jiang, S. Xiao, and C. He. Kinect depth map inpainting using a multi-scale deep convolutional neural network. In Proceedings of the 2018 International Conference on Image and Graphics Processing, pages 91-95. ACM, 2018.
[19]
W. Jiang, C. Miao, F. Ma, S. Yao, Y. Wang, Y. Yuan, H. Xue, C. Song, X. Ma, D. Koutsonikolas, W. Xu, and L. Su. Towards environment independent device free human activity recognition. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, MobiCom '18, page 289-304, New York, NY, USA, 2018. Association for Computing Machinery.
[20]
R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network architectures. In International Conference on Machine Learning, pages 2342-2350, 2015.
[21]
F. Karim, S. Majumdar, and H. Darabi. Insights into lstm fully convolutional networks for time series classification. arXiv preprint arXiv:1902.10756, 2019.
[22]
A. Khan, J. Nicholson, and T. Plötz. Activity recognition for quality assessment of batting shots in cricket using a hierarchical representation. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 1(3):62:1-62:31, Sept. 2017.
[23]
K. Kumada, Y. Usui, and K. Kondo. Golf swing tracking and evaluation using kinect sensor and particle filter. In 2013 International Symposium on Intelligent Signal Processing and Communication Systems, pages 698-703. IEEE, 2013.
[24]
C. Li, Z. Zhao, and X. Guo. Articulatedfusion: Real-time reconstruction of motion, geometry and segmentation using a single depth camera. In Proceedings of the European Conference on Computer Vision (ECCV), pages 317-332, 2018.
[25]
M. Li, H. Yu, X. Zheng, and A. I. Mourikis. High-fidelity sensor modeling and self-calibration in vision-aided inertial navigation. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 409-416. IEEE, 2014.
[26]
N. Ma, X. Zhang, H.-T. Zheng, and J. Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pages 116-131, 2018.
[27]
N. Nakano, T. Sakura, K. Ueda, L. Omura, A. Kimura, Y. Iino, S. Fukashiro, and S. Yoshioka. Evaluation of 3d markerless motion capture accuracy using openpose with multiple video cameras. bioRxiv, page 842492, 2019.
[28]
C. N. K. Nam, H. J. Kang, and Y. S. Suh. Golf swing motion tracking using inertial sensors and a stereo camera. IEEE Transactions on Instrumentation and measurement, 63(4):943-952, 2014.
[29]
A. Newell, K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In European Conference on Computer Vision, pages 483-499. Springer, 2016.
[30]
S. Park, J. Yong Chang, H. Jeong, J.-H. Lee, and J.-Y. Park. Accurate and efficient 3d human pose estimation algorithm using single depth images for pose analysis in golf. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 49-57, 2017.
[31]
A. Pfister, A. M. West, S. Bronner, and J. A. Noah. Comparative abilities of microsoft kinect and vicon 3d motion capture for gait analysis. Journal of medical engineering & technology, 38(5):274-280, 2014.
[32]
N. Point. Inc.: Optitrack-optical motion tracking solutions, 2009.
[33]
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018.
[34]
S. Shen, M. Gowda, and R. Roy Choudhury. Closing the gaps in inertial motion tracking. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, pages 429-444. ACM, 2018.
[35]
T. Yu, Z. Zheng, K. Guo, J. Zhao, Q. Dai, H. Li, G. Pons-Moll, and Y. Liu. Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7287-7296, 2018.
[36]
X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6848-6856, 2018.
[37]
H. Zhao, S. Wang, G. Zhou, and W. Jung. Tenniseye: Tennis ball speed estimation using a racket-mounted motion sensor. In 2019 18th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), pages 241-252, April 2019.
[38]
X. Zhou, M. Zhu, G. Pavlakos, S. Leonardos, K. G. Derpanis, and K. Daniilidis. Monocap: Monocular human motion capture using a cnn coupled with a geometric prior. IEEE transactions on pattern analysis and machine intelligence, 41(4):901-914, 2019.
Index Terms
- Condor: Mobile Golf Swing Tracking via Sensor Fusion using Conditional Generative Adversarial Networks
Index terms have been assigned to the content through auto-classification.
Recommendations
Mobile golf swing tracking using deep learning with data fusion: poster abstract
SenSys '19: Proceedings of the 17th Conference on Embedded Networked Sensor SystemsSwing tracking is one of the key information for many sports such as golf. One approach to track swing is to use IMU to measure linear acceleration then get position by two-time integration. However, the complex noise model of the IMU limit the accuracy ...
A sensor-aided self coaching model for uncocking improvement in golf swing
This paper describes an autonomous kinematic analysis platform for wrist angle measurement that is capable of evaluating a user's uncocking motion in his or her golf swing and providing instructional multimodal feedback to improve his or her skills. ...
Smart motion reconstruction system for golf swing: a DBN model based transportable, non-intrusive and inexpensive golf swing capture and reconstruction system
In the past decade, golf has stimulated people's great interest and the number of golf players has increased significantly. Therefore, how to train a golfer to make a perfect swing has attracted extensive research attentions. Among these researches, the ...
Comments
Information & Contributors
Information
Published In
February 2021
201 pages
- General Chairs:
- Polly Huang,
- Marco Zuniga,
- Tpc Co-chairss:
- Guoliang Xing,
- Chiara Petrioli
Sponsors
- EWSN: International Conference on Embedded Wireless Systems and Networks
- Huawei: Huawei
- CHIIoT: CHIIoT Workshop on Computer Human Interaction in IoT Applications
- sigbed: ACM Special Interest Group on Embedded Systems
- sigmobile: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing
Publisher
Junction Publishing
United States
Publication History
Published: 28 April 2021
Check for updates
Qualifiers
- Research-article
Conference
EWSN '21: Proceedings of the 2021 International Conference on Embedded Wireless Systems and Networks
February 17, 2021
Delft, The Netherlands
Acceptance Rates
EWSN '21 Paper Acceptance Rate 14 of 44 submissions, 32%;
Overall Acceptance Rate 81 of 195 submissions, 42%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 06 Aug 2024