Abstract
To diagnose, plan, and treat musculoskeletal pathologies, understanding and reproducing muscle recruitment for complex movements is essential. With muscle activations for movements often being highly redundant, nonlinear, and time dependent, machine learning can provide a solution for their modeling and control for anatomy-specific musculoskeletal simulations. Sophisticated biomechanical simulations often require specialized computational environments, being numerically complex and slow, hindering their integration with typical deep learning frameworks. In this work, a deep reinforcement learning (DRL) based inverse dynamics controller is trained to control muscle activations of a biomechanical model of the human shoulder. In a generalizable end-to-end fashion, muscle activations are learned given current and desired position-velocity pairs. A customized reward functions for trajectory control is introduced, enabling straightforward extension to additional muscles and higher degrees of freedom. Using the biomechanical model, multiple episodes are simulated on a cluster simultaneously using the evolving neural models of the DRL being trained. Results are presented for a single-axis motion control of shoulder abduction for the task of following randomly generated angular trajectories.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Abdi, A.H., Saha, P., Srungarapu, V.P., Fels, S.: Muscle excitation estimation in biomechanical simulation using NAF reinforcement learning. In: Computational Biomechanics for Medicine, pp. 133–141 (2020)
Artstein, Z.: Discrete and continuous bang-bang and facial spaces or: look for the extreme points. Siam Rev. 22(2), 172–185 (1980)
Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. 2. Athena Scientific, Belmont (2012)
Blemker, S.S., Pinsky, P.M., Delp, S.L.: A 3D model of muscle reveals the causes of nonuniform strains in the biceps brachii. J. Biomech. 38(4), 657–665 (2005)
Bolsterlee, B., Veeger, H.E.J., van der Helm, F.C.T.: Modelling clavicular and scapular kinematics: from measurement to simulation. Med. Biol. Eng. Comput. 52(3), 283–291 (2013). https://doi.org/10.1007/s11517-013-1065-2
Brown, J.M.M., Wickham, J.B., McAndrew, D.J., Huang, X.F.: Muscles within muscles: coordination of 19 muscle segments within three shoulder muscles during isometric motor tasks. J. Electromyogr. Kinesiol. 17(1), 57–73 (2007)
Contemori, S., Panichi, R., Biscarini, A.: Effects of scapular retraction/protraction position and scapular elevation on shoulder girdle muscle activity during glenohumeral abduction. Hum. Mov. Sci. 64, 55–66 (2019)
Craik, J.D., Mallina, R., Ramasamy, V., Little, N.J.: Human evolution and tears of the rotator cuff. Int. Orthop. 38(3), 547–552 (2013). https://doi.org/10.1007/s00264-013-2204-y
Dhariwal, P., et al.: OpenAI baselines (2017). https://github.com/openai/baselines
Di Giacomo, G., Pouliart, N., Costantini, A., De Vita, A.: Atlas of Functional Shoulder Anatomy. Springer, New York (2008)
Faure, F., et al.: SOFA: a multi-model framework for interactive physical simulation. In: Payan, Y. (ed.) Soft Tissue Biomechanical Modeling for Computer Assisted Surgery. SMTEB, vol. 11, pp. 283–321. Springer, Heidelberg (2012). https://doi.org/10.1007/8415_2012_125
Gerber, C., Snedeker, J.G., Baumgartner, D., Viehöfer, A.F.: Supraspinatus tendon load during abduction is dependent on the size of the critical shoulder angle: a biomechanical analysis. J. Orthop. Res. 32(7), 952–957 (2014)
Heess, N., et al.: Emergence of locomotion behaviours in rich environments. arXiv:1707.02286 (2017)
James, S., Johns, E.: 3D simulation for robot arm control with deep Q-learning. arXiv:1609.03759 (2016)
Kidziński, Ł., et al.: Learning to run challenge: synthesizing physiologically accurate motion using deep reinforcement learning. In: Escalera, S., Weimer, M. (eds.) The NIPS ’17 Competition: Building Intelligent Systems. TSSCML, pp. 101–120. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-94042-7_6
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv:1509.02971 (2015)
Lloyd, J.E., Stavness, I., Fels, S.: ArtiSynth: a fast interactive biomechanical modeling toolkit combining multibody and finite element simulation. Soft Tissue Biomechanical Modeling for Computer Assisted Surgery. SMTEB, vol. 11, pp. 355–394. Springer, Heidelberg (2012). https://doi.org/10.1007/8415_2012_126
Mitsuhashi, N., Fujieda, K., Tamura, T., Kawamoto, S., Takagi, T., Okubo, K.: Bodyparts3D: 3D structure database for anatomical concepts. Nucleic Acids Res. 37, D782–D785 (2008)
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32, pp. 8024–8035 (2019)
Pean, F., Goksel, O.: Surface-based modeling of muscles: functional simulation of the shoulder. Med. Eng. Phys. 82, 1–12 (2020)
Péan, F., Tanner, C., Gerber, C., Fürnstahl, P., Goksel, O.: A comprehensive and volumetric musculoskeletal model for the dynamic simulation of the shoulder function. Comput. Methods Biomech. Biomed. Eng. 22(7), 740–751 (2019)
Reed, D., Cathers, I., Halaki, M., Ginn, K.: Does supraspinatus initiate shoulder abduction? J. Electromyogr. Kinesiol. 23(2), 425–429 (2013)
Schulman, J., Levine, S., Moritz, P., Jordan, M., Abbeel, P.: Trust region policy optimization. In: International Conference on Machine Learning (ICML), vol. PMLR 37, pp. 1889–1897 (2015)
Schulman, J., Moritz, P., Levine, S., Jordan, M.I., Abbeel, P.: High-dimensional continuous control using generalized advantage estimation. In: International Conference on Learning Representations (ICLR) (2016)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv:1707.06347 (2017)
Silver, D., Huang, A., Maddison, C.J., Guez, A., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)
Spong, M.W., Hutchinson, S., Vidyasagar, M.: Robot Modeling and Control. Wiley, Hoboken (2020)
Stavness, I., Lloyd, J.E., Fels, S.: Automatic prediction of tongue muscle activations using a finite element model. J. Biomech. 45(16), 2841–2848 (2012)
Streit, J.J., et al.: Pectoralis major tendon transfer for the treatment of scapular winging due to long thoracic nerve palsy. J. Shoulder Elbow Surg. 21(5), 685–690 (2012)
Tsurumine, Y., Cui, Y., Uchibe, E., Matsubara, T.: Deep reinforcement learning with smooth policy update: application to robotic cloth manipulation. Robot. Autonom. Syst. 112, 72–83 (2019)
Van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double Q-learning. In: AAAI Conference on Artificial Intelligence, pp. 2094–2100 (2016)
Wickham, J., Pizzari, T., Stansfeld, K., Burnside, A., Watson, L.: Quantifying ‘normal’ shoulder muscle activity during abduction. J. Electromyogr. Kinesiol. 20(2), 212–222 (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 696 KB)
Supplementary material 2 (mp4 718 KB)
Supplementary material 3 (mp4 716 KB)
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Joos, E., Péan, F., Goksel, O. (2020). Reinforcement Learning of Musculoskeletal Control from Functional Simulations. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12263. Springer, Cham. https://doi.org/10.1007/978-3-030-59716-0_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-59716-0_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59715-3
Online ISBN: 978-3-030-59716-0
eBook Packages: Computer ScienceComputer Science (R0)
