research-article

Open access

MoSculp: Interactive Visualization of Shape and Time

Authors:

Stefanie Mueller,

William T. FreemanAuthors Info & Claims

UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Pages 275 - 285

https://doi.org/10.1145/3242587.3242592

Published: 11 October 2018 Publication History

Abstract

We present a system that visualizes complex human motion via 3D motion sculptures-a representation that conveys the 3D structure swept by a human body as it moves through space. Our system computes a motion sculpture from an input video, and then embeds it back into the scene in a 3D-aware fashion. The user may also explore the sculpture directly in 3D or physically print it. Our interactive interface allows users to customize the sculpture design, for example, by selecting materials and lighting conditions. To provide this end-to-end workflow, we introduce an algorithm that estimates a human's 3D geometry over time from a set of 2D images, and develop a 3D-aware image-based rendering approach that inserts the sculpture back into the original video. By automating the process, our system takes motion sculpture creation out of the realm of professional artists, and makes it applicable to a wide range of existing video material. By conveying 3D information to users, motion sculptures reveal space-time motion information that is difficult to perceive with the naked eye, and allow viewers to interpret how different parts of the object interact over time. We validate the effectiveness of motion sculptures with user studies, finding that our visualizations are more informative about motion than existing stroboscopic and space-time visualization methods.

Supplementary Material

suppl.mov (ufp1015.mp4)

Supplemental video

Download
113.69 MB

suppl.mov (ufp1015p.mp4)

Supplemental video

Download
7.90 MB

MP4 File (p275-zhang.mp4)

Download
320.04 MB

References

[1]

Ankur Agarwal and Bill Triggs. 2006. Recovering 3D Human Pose from Monocular Images. IEEE Transactions on Pattern Analysis and Machine Intelligence (2006).

Digital Library

[2]

Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, and Michael Cohen. 2004. Interactive Digital Photomontage. ACM Transactions on Graphics (2004).

Digital Library

[3]

Patrick Baudisch, Desney Tan, Maxime Collomb, Dan Robbins, Ken Hinckley, Maneesh Agrawala, Shengdong Zhao, and Gonzalo Ramos. 2006. Phosphor: Explaining Transitions in the User Interface Using Afterglow Effects. In ACM Symposium on User Interface Software and Technology.

Digital Library

[4]

Eric P. Bennett and Leonard McMillan. 2007. Computational Time-Lapse Video. ACM Transactions on Graphics (2007).

Digital Library

[5]

Moshe Blank, Lena Gorelick, Eli Shechtman, Michal Irani, and Ronen Basri. 2005. Actions as Space-Time Shapes. In IEEE International Conference on Computer Vision.

Digital Library

[6]

Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black. 2016. Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image. In European Conference on Computer Vision.

[7]

Simon Bouvier-Zappa, Victor Ostromoukhov, and Pierre Poulin. 2007. Motion Cues for Illustration of Skeletal Motion Capture Data. In International Symposium on Non-Photorealistic Animation and Rendering.

Digital Library

[8]

Marta Braun. 1992. Picturing Time: the Work of Etienne-Jules Marey.

[9]

Matthew Brown and David G. Lowe. 2007. Automatic Panoramic Image Stitching Using Invariant Features. International Journal of Computer Vision (2007).

Digital Library

[10]

Kevin Burg and Jamie Beck. 2015. The School of American Ballet. https://vimeo.com/121834884. (2015).

[11]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. In IEEE Conference on Computer Vision and Pattern Recognition.

[12]

Ching-Hang Chen and Deva Ramanan. 2017. 3D Human Pose Estimation = 2D Pose Estimation + Matching. In IEEE Conference on Computer Vision and Pattern Recognition.

[13]

Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. 2013. KNN Matting. IEEE Transactions on Pattern Analysis and Machine Intelligence (2013).

Digital Library

[14]

Pei-Yu (Peggy) Chi, Daniel Vogel, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2016. Authoring Illustrations of Human Movements by Iterative Physical Demonstration. In ACM Symposium on User Interface Software and Technology.

Digital Library

[15]

James E. Cutting. 2002. Representing Motion in a Static Image: Constraints and Parallels in Art, Science, and Popular Culture. Perception (2002).

[16]

JL Design. 2013. CCTV Documentary (Director's Cut). https://vimeo.com/69948148. (2013).

[17]

William T. Freeman and Hao Zhang. 2003. Shape-Time Photography. In IEEE Conference on Computer Vision and Pattern Recognition.

[18]

Eyal Gever. 2014. Kick Motion Sculpture Simulation and 3D Video Capture. https://vimeo.com/102911464. (2014).

[19]

Tobias Gremmler. 2016. Kung Fu Motion Visualization. https://vimeo.com/163153865. (2016).

[20]

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask R-CNN. In IEEE International Conference on Computer Vision.

[21]

Nicholas R. Howe, Michael E. Leventon, and William T. Freeman. 2000. Bayesian Reconstruction of 3D Human Motion from Single-Camera Video. In Advances in Neural Information Processing Systems.

Digital Library

[22]

Yinghao Huang, Federica Bogo, Christoph Lassner, Angjoo Kanazawa, Peter V. Gehler, Javier Romero, Ijaz Akhter, and Michael J. Black. 2017. Towards AccurateMarker-Less Human Shape and Pose Estimation Over Time. In International Conference on 3D Vision.

[23]

Arjun Jain, Thorsten Thormählen, Hans-Peter Seidel, and Christian Theobalt. 2010. MovieReshape: Tracking and Reshaping of Humans in Videos. ACM Transactions on Graphics (2010).

Digital Library

[24]

Peter Jansen. 2008. Human Motions. https://www.humanmotions.com/. (2008).

[25]

Yvonne Jansen, Pierre Dragicevic, and Jean-Daniel Fekete. 2013. Evaluating the Efficiency of Physical Visualizations. In ACM CHI Conference on Human Factors in Computing Systems.

Digital Library

[26]

Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. 2018. End-to-End Recovery of Human Shape and Pose. In IEEE Conference on Computer Vision and Pattern Recognition.

[27]

Yuya Kawagishi, Kazuhide Hatsuyama, and Kunio Kondo. 2003. Cartoon Blur: Nonphotorealistic Motion Blur. In Computer Graphics International.

[28]

Rubaiat Habib Kazi, Tovi Grossman, Cory Mogk, Ryan Schmidt, and George Fitzmaurice. 2016. ChronoFab: Fabricating Motion. In ACM CHI Conference on Human Factors in Computing Systems.

Digital Library

[29]

Rohit Ashok Khot, Larissa Hjorth, and Florian "Floyd" Mueller. 2014. Understanding Physical Activity Through 3D Printed Material Artifacts. In ACM CHI Conference on Human Factors in Computing Systems.

Digital Library

[30]

Felix Klose, Oliver Wang, Jean-Charles Bazin, Marcus Magnor, and Alexander Sorkine-Hornung. 2015. Sampling Based Scene-Space Video Processing. ACM Transactions on Graphics (2015).

Digital Library

[31]

Adam Lake, Carl Marshall, Mark Harris, and Marc Blackstein. 2000. Stylized Rendering Techniques for Scalable Real-Time 3D Animation. In International Symposium on Non-Photorealistic Animation and Rendering.

Digital Library

[32]

Ce Liu. 2009. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Ph.D. Dissertation. Massachusetts Institute of Technology.

Digital Library

[33]

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Transactions on Graphics (2015).

Digital Library

[34]

Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In European Conference on Computer Vision.

[35]

Maic Masuch, Stefan Schlechtweg, and Ronny Schulz. 1999. Speedlines: Depicting Motion in Motionless Pictures. ACM Transactions on Graphics (1999).

Digital Library

[36]

Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. 2017. VNect: Real-Time 3D Human Pose Estimation with a Single RGB Camera. ACM Transactions on Graphics (2017).

Digital Library

[37]

Eadweard Muybridge. 1985. Horses and Other Animals in Motion: 45 Classic Photographic Sequences.

[38]

Cuong Nguyen, Yuzhen Niu, and Feng Liu. 2012. Video Summagator: an Interface for Video Summarization and Navigation. In ACM CHI Conference on Human Factors in Computing Systems.

Digital Library

[39]

Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis. 2018. Ordinal Depth Supervision for 3D Human Pose Estimation. In IEEE Conference on Computer Vision and Pattern Recognition.

[40]

Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, and Kostas Daniilidis. 2017. Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose. In IEEE Conference on Computer Vision and Pattern Recognition.

[41]

Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Daniilidis. 2018. Learning to Estimate 3D Human Pose and Shape from a Single Color Image. In IEEE Conference on Computer Vision and Pattern Recognition.

[42]

Yael Pritch, Eitam Kav-Venaki, and Shmuel Peleg. 2009. Shift-Map Image Editing. In IEEE Conference on Computer Vision and Pattern Recognition.

[43]

Johannes Schmid, Robert W. Sumner, Huw Bowles, and Markus H. Gross. 2010. Programmable Motion Effects. ACM Transactions on Graphics (2010).

Digital Library

[44]

Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In IEEE Conference on Computer Vision and Pattern Recognition.

[45]

Kalyan Sunkavalli, Neel Joshi, Sing Bing Kang, Michael F. Cohen, and Hanspeter Pfister. 2012. Video Snapshots: Creating High-Quality Images from Video Clips. IEEE Transactions on Visualization and Computer Graphics (2012).

Digital Library

[46]

Saiganesh Swaminathan, Conglei Shi, Yvonne Jansen, Pierre Dragicevic, Lora A. Oehlberg, and Jean-Daniel Fekete. 2014. Supporting the Design and Fabrication of Physical Visualizations. In ACM CHI Conference on Human Factors in Computing Systems.

Digital Library

[47]

Okihide Teramoto, In Kyu Park, and Takeo Igarashi. 2010. Interactive Motion Photography from a Single Image. The Visual Computer (2010).

Digital Library

[48]

Denis Tome, Christopher Russell, and Lourdes Agapito. 2017. Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image. In IEEE Conference on Computer Vision and Pattern Recognition.

[49]

Cesar Torres, Wilmot Li, and Eric Paulos. 2016. ProxyPrint: Supporting Crafting Practice Through Physical Computational Proxies. In ACM Conference on Designing Interactive Systems.

Digital Library

[50]

Xiaowei Zhou, Menglong Zhu, Spyridon Leonardos, Konstantinos G. Derpanis, and Kostas Daniilidis. 2016. Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video. In IEEE Conference on Computer Vision and Pattern Recognition.

[51]

Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael J. Black. 2017. 3D Menagerie: Modeling the 3D Shape and Pose of Animals. In IEEE Conference on Computer Vision and Pattern Recognition.

Cited By

Basset JBénard PBarla P(2024)SMEAR: Stylized Motion Exaggeration with ARt-directionACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657457(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657457
Albanis GZioulis NKolomvatsos K(2024)BundleMoCap++: Efficient, robust and smooth motion capture from sparse multiview videosComputer Vision and Image Understanding10.1016/j.cviu.2024.104190(104190)Online publication date: Oct-2024
https://doi.org/10.1016/j.cviu.2024.104190
Albanis GZioulis NKolomvatsos K(2023)BundleMoCap: Efficient, Robust and Smooth Motion Capture from Sparse Multiview VideosProceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production10.1145/3626495.3626511(1-9)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3626495.3626511
Show More Cited By

Index Terms

MoSculp: Interactive Visualization of Shape and Time
1. Human-centered computing
  1. Visualization
    1. Visualization application domains
      1. Information visualization

Recommendations

Human video textures
I3D '09: Proceedings of the 2009 symposium on Interactive 3D graphics and games

This paper describes a data-driven approach for generating photorealistic animations of human motion. Each animation sequence follows a user-choreographed path and plays continuously by seamlessly transitioning between different segments of the captured ...
Opacity light fields: interactive rendering of surface light fields with view-dependent opacity
I3D '03: Proceedings of the 2003 symposium on Interactive 3D graphics

We present new hardware-accelerated techniques for rendering surface light fields with opacity hulls that allow for interactive visualization of objects that have complex reflectance properties and elaborate geometrical details. The opacity hull is a ...
Image-based motion blur for stop motion animation
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques

Stop motion animation is a well-established technique where still pictures of static scenes are taken and then played at film speeds to show motion. A major limitation of this method appears when fast motions are desired; most motion appears to have ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

October 2018

1016 pages

ISBN:9781450359481

DOI:10.1145/3242587

General Chairs:
Patrick Baudisch
Hasso-Plattner Institute, Germany
,
Albrecht Schmidt
LMU, Germany
,
Program Chair:
Andy Wilson
Microsoft Research, USA

Copyright © 2018 Owner/Author.

This work is licensed under a Creative Commons Attribution-ShareAlike International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Facebook Fellowship
DARPA MediFor
Shell Research

Conference

UIST '18

Sponsor:

UIST '18: The 31st Annual ACM Symposium on User Interface Software and Technology

October 14, 2018

Berlin, Germany

Acceptance Rates

UIST '18 Paper Acceptance Rate 80 of 375 submissions, 21%;

Overall Acceptance Rate 842 of 3,967 submissions, 21%

Upcoming Conference

UIST '24

Sponsor:
sigchi
sigchi

The 37th Annual ACM Symposium on User Interface Software and Technology

October 13 - 16, 2024

Pittsburgh , PA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
1,685
Total Downloads

Downloads (Last 12 months)200
Downloads (Last 6 weeks)25

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Basset JBénard PBarla P(2024)SMEAR: Stylized Motion Exaggeration with ARt-directionACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657457(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657457
Albanis GZioulis NKolomvatsos K(2024)BundleMoCap++: Efficient, robust and smooth motion capture from sparse multiview videosComputer Vision and Image Understanding10.1016/j.cviu.2024.104190(104190)Online publication date: Oct-2024
https://doi.org/10.1016/j.cviu.2024.104190
Albanis GZioulis NKolomvatsos K(2023)BundleMoCap: Efficient, Robust and Smooth Motion Capture from Sparse Multiview VideosProceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production10.1145/3626495.3626511(1-9)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3626495.3626511
Cho HKomar MLindlbauer D(2023)RealityReplayProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108887:3(1-25)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610888
Yu EBlackburn-Matzen KNguyen CWang OHabib Kazi RBousseau A(2023)VideoDoodles: Hand-Drawn Animations on Videos with Scene-Aware CanvasesACM Transactions on Graphics10.1145/359241342:4(1-12)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592413
Qiu WLegrady G(2023)Fencing Hallucination: An Interactive Installation for Fencing with AI and Synthesizing ChronophotographsExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3583929(1-5)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3583929
Luiz Ramos Barbosa CAugusto Soares de Sousa TCunha Monteiro WHortencio Dos Santos DDavi Oliveira de Araujo TSerique Meiguins B(2023)An Accuracy Assessment for Active Data Physicalization2023 27th International Conference Information Visualisation (IV)10.1109/IV60283.2023.00027(103-108)Online publication date: 25-Jul-2023
https://doi.org/10.1109/IV60283.2023.00027
Vialle MAlaoui SSkouras MVilvanathan VSchwartz ERonfard RMaiden NSas C(2022)Visualizing Isadora Duncan’s movements qualitiesProceedings of the 14th Conference on Creativity and Cognition10.1145/3527927.3532805(196-207)Online publication date: 20-Jun-2022
https://dl.acm.org/doi/10.1145/3527927.3532805
Engert SKlamka KPeetz ADachselt R(2022)STRAIDE: A Research Platform for Shape-Changing Spatial Displays based on Actuated StringsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517462(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517462
Hamanishi NRekimoto J(2021)Motion-specific browsing method by mapping to a circle for personal video Observation with Head-Mounted DisplaysProceedings of the Augmented Humans International Conference 202110.1145/3458709.3458959(240-250)Online publication date: 22-Feb-2021
https://dl.acm.org/doi/10.1145/3458709.3458959
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents