research-article

Example-based Motion Synthesis via Generative Motion Matching

Authors:

Olga Sorkine-Hornung,

Baoquan ChenAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 42, Issue 4

Article No.: 94, Pages 1 - 12

https://doi.org/10.1145/3592395

Published: 26 July 2023 Publication History

Abstract

We present GenMM, a generative model that "mines" as many diverse motions as possible from a single or few example sequences. In stark contrast to existing data-driven methods, which typically require long offline training time, are prone to visual artifacts, and tend to fail on large and complex skeletons, GenMM inherits the training-free nature and the superior quality of the well-known Motion Matching method. GenMM can synthesize a high-quality motion within a fraction of a second, even with highly complex and large skeletal structures. At the heart of our generative framework lies the generative motion matching module, which utilizes the bidirectional visual similarity as a generative cost function to motion matching, and operates in a multi-stage framework to progressively refine a random guess using exemplar motion matches. In addition to diverse motion generation, we show the versatility of our generative framework by extending it to a number of scenarios that are not possible with motion matching alone, including motion completion, key frame-guided generation, infinite looping, and motion reassembly.

Supplementary Material

ZIP File (papers_485-supplemental.zip)

supplemental material

Download
380.08 MB

MP4 File (papers_485_VOD.mp4)

presentation

Download
568.29 MB

References

[1]

Adobe Systems Inc. 2022. Mixamo. https://www.mixamo.com Accessed: 2022-03-25.

[2]

Okan Arikan and David A Forsyth. 2002. Interactive motion generation from examples. ACM Transactions on Graphics (TOG) 21, 3 (2002), 483--490.

Digital Library

[3]

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (TOG) 28, 3 (2009), 24.

Digital Library

[4]

Connelly Barnes and Fang-Lue Zhang. 2017. A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media 3, 1 (2017), 3--20.

[5]

Blender Online Community. 2023. Blender - a 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam.

[6]

Richard Bowden. 2000. Learning statistical models of human motion. In IEEE Workshop on Human Modeling, Analysis and Synthesis, CVPR, Vol. 2000. Citeseer.

[7]

Matthew Brand and Aaron Hertzmann. 2000. Style machines. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 183--192.

Digital Library

[8]

Peter J Burt and Edward H Adelson. 1987. The Laplacian pyramid as a compact image code. In Readings in computer vision. Elsevier, 671--679.

[9]

Michael Buttner. 2019. Machine learning for motion synthesis and character control in games. Proc. of I3D 2019 (2019).

[10]

Michael Büttner and Simon Clavet. 2015. Motion Matching - The Road to Next Gen Animation. https://www.youtube.com/watch?v=z_wpgHFSWss&t=658s

[11]

Jinxiang Chai and Jessica K Hodgins. 2007. Constraint-based motion optimization using a statistical dynamic model. In ACM SIGGRAPH 2007 papers. 8--es.

Digital Library

[12]

Jeremy S De Bonet. 1997. Multiresolution sampling procedure for analysis and synthesis of texture images. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. 361--368.

Digital Library

[13]

Yinglin Duan, Yue Lin, Zhengxia Zou, Yi Yuan, Zhehui Qian, and Bohan Zhang. 2022. A Unified Framework for Real Time Motion Completion. (2022).

[14]

Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision. 4346--4354.

Digital Library

[15]

Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In NIPS.

[16]

Niv Granot, Ben Feinstein, Assaf Shocher, Shai Bagon, and Michal Irani. 2022. Drop the gan: In defense of patches nearest neighbors as single image generative models. In Conference on Computer Vision and Pattern Recognition (CVPR). 13460--13469.

[17]

Keith Grochow, Steven L Martin, Aaron Hertzmann, and Zoran Popović. 2004. Style-based inverse kinematics. In ACM SIGGRAPH 2004 Papers. 522--531.

Digital Library

[18]

Ikhsanul Habibie, Mohamed Elgharib, Kripasindhu Sarkar, Ahsan Abdullah, Simbarashe Nyatsanga, Michael Neff, and Christian Theobalt. 2022. A Motion Matching-based Framework for Controllable Gesture Synthesis from Speech. In ACM SIGGRAPH 2022 Conference Proceedings. 1--9.

Digital Library

[19]

Charles Han, Eric Risser, Ravi Ramamoorthi, and Eitan Grinspun. 2008. Multiscale texture synthesis. In ACM SIGGRAPH 2008 papers. 1--8.

Digital Library

[20]

Geof Harrower. 2018. Real player motion tech in'ea sports ufc 3'. Proc. of GDC 2018 (2018).

[21]

Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. 2020. Robust motion in-betweening. ACM Transactions on Graphics (TOG) 39, 4 (2020), 60--1.

Digital Library

[22]

Chengan He, Jun Saito, James Zachary, Holly Rushmeier, and Yi Zhou. 2022. NeMF: Neural Motion Fields for Kinematic Animation. In Advances in Neural Information Processing Systems.

[23]

David J Heeger and James R Bergen. 1995. Pyramid-based texture analysis/synthesis. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 229--238.

[24]

Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. 2020. Moglow: Probabilistic and controllable motion synthesis using normalising flows. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1--14.

Digital Library

[25]

Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa. 2020. Learned motion matching. ACM Transactions on Graphics (TOG) 39, 4 (2020), 53--1.

Digital Library

[26]

Daniel Holden, Taku Komura, and Jun Saito. 2017. Phase-functioned neural networks for character control. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--13.

Digital Library

[27]

Daniel Holden, Jun Saito, and Taku Komura. 2016. A deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--11.

Digital Library

[28]

Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. 2015. Learning motion manifolds with convolutional autoencoders. In SIGGRAPH Asia 2015 technical briefs. 1--4.

[29]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. CVPR (2017).

[30]

Deok-Kyeong Jang, Soomin Park, and Sung-Hee Lee. 2022. Motion Puzzle: Arbitrary Motion Style Transfer by Body Part. ACM Transactions on Graphics (TOG) (2022).

[31]

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In International Conference on Learning Representations.

[32]

Lucas Kovar, Michael Gleicher, and Frédéric Pighin. 2002. Motion Graphs. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (San Antonio, Texas) (SIGGRAPH '02). Association for Computing Machinery, New York, NY, USA, 473--482.

Digital Library

[33]

Jehee Lee, Jinxiang Chai, Paul SA Reitsma, Jessica K Hodgins, and Nancy S Pollard. 2002. Interactive control of avatars animated with human motion data. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 491--500.

Digital Library

[34]

Seyoung Lee, Jiye Lee, and Jehee Lee. 2022. Learning Virtual Chimeras by Dynamic Motion Reassembly. ACM Trans. Graph. 41, 6, Article 182 (2022).

Digital Library

[35]

Yongjoon Lee, Kevin Wampler, Gilbert Bernstein, Jovan Popović, and Zoran Popović. 2010. Motion fields for interactive character locomotion. In ACM Transactions on Graphics (TOG). 1--8.

[36]

Sergey Levine, Jack M Wang, Alexis Haraux, Zoran Popović, and Vladlen Koltun. 2012. Continuous character control with low-dimensional embeddings. ACM Transactions on Graphics (TOG) 31, 4 (2012), 1--10.

Digital Library

[37]

Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung. 2022. GANimator: Neural Motion Synthesis from a Single Sequence. ACM Transactions on Graphics (TOG) 41, 4 (2022), 138.

Digital Library

[38]

Yan Li, Tianshu Wang, and Heung-Yeung Shum. 2002. Motion texture: a two-level statistical model for character motion synthesis. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 465--472.

Digital Library

[39]

Dario Pavllo, David Grangier, and Michael Auli. 2018. Quaternet: A quaternion-based recurrent model for human motion. arXiv preprint arXiv:1805.06485 (2018).

[40]

Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) 37, 4 (2018), 1--14.

Digital Library

[41]

Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. 2021. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1--20.

Digital Library

[42]

Ken Perlin. 1985. An image synthesizer. ACM Siggraph Computer Graphics 19, 3 (1985), 287--296.

Digital Library

[43]

Ken Perlin and Athomas Goldberg. 1996. Improv: A system for scripting interactive actors in virtual worlds. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 205--216.

Digital Library

[44]

Katherine Pullen and Christoph Bregler. 2000. Animating by multi-level sampling. In Proceedings Computer Animation 2000. IEEE, 36--42.

Digital Library

[45]

Katherine Pullen and Christoph Bregler. 2002. Motion capture assisted animation: Texturing and synthesis. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 501--508.

Digital Library

[46]

Jia Qin, Youyi Zheng, and Kun Zhou. 2022. Motion In-betweening via Two-stage Transformers. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1--16.

Digital Library

[47]

Sigal Raab, Inbal Leibovitch, Peizhuo Li, Kfir Aberman, Olga Sorkine-Hornung, and Daniel Cohen-Or. 2023a. MoDi: Unconditional Motion Synthesis from Diverse Data. (2023).

[48]

Sigal Raab, Inbal Leibovitch, Guy Tevet, Moab Arar, Amit H Bermano, and Daniel Cohen-Or. 2023b. Single Motion Diffusion. arXiv preprint arXiv:2302.05905 (2023).

[49]

Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J Guibas. 2021. Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision. 11488--11499.

[50]

Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. 2019. Singan: Learning a generative model from a single natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4570--4580.

[51]

Mingyi Shi, Kfir Aberman, Andreas Aristidou, Taku Komura, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. 2020. Motionet: 3d human motion reconstruction from monocular video with skeleton consistency. ACM Transactions on Graphics (TOG) 40, 1 (2020), 1--15.

Digital Library

[52]

Denis Simakov, Yaron Caspi, Eli Shechtman, and Michal Irani. 2008. Summarizing visual data using bidirectional similarity. In Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1--8.

[53]

Sebastian Starke, Ian Mason, and Taku Komura. 2022. Deepphase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1--13.

Digital Library

[54]

Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. 2022a. MotionCLIP: Exposing Human Motion Generation to CLIP Space. arXiv preprint arXiv:2203.08063 (2022).

[55]

Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. 2022b. Human motion diffusion model. arXiv preprint arXiv:2209.14916 (2022).

[56]

Truebones Motions Animation Studios. 2022. Truebones. https://truebones.gumroad.com/ Accessed: 2022-9-2.

[57]

Jonathan Tseng, Rodrigo Castellon, and C Karen Liu. 2022. EDGE: Editable Dance Generation From Music. arXiv preprint arXiv:2211.10658 (2022).

[58]

Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. Advances in neural information processing systems 30 (2017).

[59]

Jack M Wang, David J Fleet, and Aaron Hertzmann. 2007. Gaussian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence 30, 2 (2007), 283--298.

[60]

Li-Yi Wei, Sylvain Lefebvre, Vivek Kwatra, and Greg Turk. 2009. State of the art in example-based texture synthesis. Eurographics 2009, State of the Art Report, EG-STAR (2009), 93--117.

[61]

Li-Yi Wei and Marc Levoy. 2000. Fast texture synthesis using tree-structured vector quantization. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 479--488.

Digital Library

[62]

Mengyi Zhao, Mengyuan Liu, Bin Ren, Shuling Dai, and Nicu Sebe. 2023. Modiff: Action-Conditioned 3D Motion Generation with Denoising Diffusion Probabilistic Models. arXiv preprint arXiv:2301.03949 (2023).

[63]

Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. 2019. On the continuity of rotation representations in neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR). 5745--5753.

[64]

Yi Zhou, Zimo Li, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li. 2018. Auto-conditioned recurrent networks for extended complex human motion synthesis. In International Conference on Learning Representations.

Cited By

Chai ZQin H(2025)Dynamic Motion Transition: A Hybrid Data-Driven and Model-Driven Method for Human Pose TransitionsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337242131:3(1848-1861)Online publication date: Mar-2025
https://doi.org/10.1109/TVCG.2024.3372421
Tao DRuizhen HLibin LLi YHao Z(2024)Research progress in human-like indoor scene interactionJournal of Image and Graphics10.11834/jig.24000429:6(1575-1606)Online publication date: 2024
https://doi.org/10.11834/jig.240004
Sui YMao WWang SWang XWu JHe XChua T(2024)Enhancing Out-of-distribution Generalization on Graphs via Causal Attention LearningACM Transactions on Knowledge Discovery from Data10.1145/364439218:5(1-24)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3644392
Show More Cited By

Index Terms

Example-based Motion Synthesis via Generative Motion Matching
1. Computing methodologies
  1. Computer graphics
    1. Animation
      1. Motion processing

Recommendations

Hybrid motion graph for character motion synthesis

Objective: This paper proposes a novel framework of Hybrid Motion Graph (HMG) for creating character animations, which enhances the graph-based structural control by motion field representations for efficient motion synthesis of diverse and interactive ...
Rhythmic-motion synthesis based on motion-beat analysis

Real-time animation of human-like characters is an active research area in computer graphics. The conventional approaches have, however, hardly dealt with the rhythmic patterns of motions, which are essential in handling rhythmic motions such as dancing ...
Rhythmic-motion synthesis based on motion-beat analysis
SIGGRAPH '03: ACM SIGGRAPH 2003 Papers

Real-time animation of human-like characters is an active research area in computer graphics. The conventional approaches have, however, hardly dealt with the rhythmic patterns of motions, which are essential in handling rhythmic motions such as dancing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 42, Issue 4

August 2023

1912 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3609020

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2023

Published in TOG Volume 42, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Ministry of Science and Technology of the People's Republic of China
European Research Council

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
345
Total Downloads

Downloads (Last 12 months)248
Downloads (Last 6 weeks)11

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chai ZQin H(2025)Dynamic Motion Transition: A Hybrid Data-Driven and Model-Driven Method for Human Pose TransitionsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337242131:3(1848-1861)Online publication date: Mar-2025
https://doi.org/10.1109/TVCG.2024.3372421
Tao DRuizhen HLibin LLi YHao Z(2024)Research progress in human-like indoor scene interactionJournal of Image and Graphics10.11834/jig.24000429:6(1575-1606)Online publication date: 2024
https://doi.org/10.11834/jig.240004
Sui YMao WWang SWang XWu JHe XChua T(2024)Enhancing Out-of-distribution Generalization on Graphs via Causal Attention LearningACM Transactions on Knowledge Discovery from Data10.1145/364439218:5(1-24)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3644392
Zhang SPeng SXu TYang YChen TXue NShen YBao HHu RZhou X(2024)MaPa: Text-driven Photorealistic Material Painting for 3D ShapesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657504(1-12)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657504
Goel PWang KLiu CFatahalian K(2024)Iterative Motion Editing with Natural LanguageACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657447(1-9)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657447
Hou YPark JShin WHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Collaborative Filtering Based on Diffusion Models: Unveiling the Potential of High-Order ConnectivityProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657742(1360-1369)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657742
Barquero GEscalera SPalmero C(2024)Seamless Human Motion Composition with Blended Positional Encodings2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00051(457-469)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00051
Yu HLiu ZTu HChen KLi A(2024)Generalizable inductive relation prediction with causal subgraphWorld Wide Web10.1007/s11280-024-01264-527:3Online publication date: 12-Apr-2024
https://dl.acm.org/doi/10.1007/s11280-024-01264-5
Ma YLiu TLiu Z(2024)Diverse Motions and Responses in Crowd SimulationComputer Animation and Virtual Worlds10.1002/cav.7000235:6Online publication date: 26-Nov-2024
https://doi.org/10.1002/cav.70002
Li ZXu QYe XRen BLiu L(2023)DiffFR: Differentiable SPH-Based Fluid-Rigid Coupling for Rigid Body ControlACM Transactions on Graphics10.1145/361831842:6(1-17)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618318
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents