research-article

Open access

Cine-AI: Generating Video Game Cutscenes in the Style of Human Directors

Authors:

Perttu Hämäläinen,

Christian GuckelsbergerAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 6, Issue CHI PLAY

Article No.: 223, Pages 1 - 23

https://doi.org/10.1145/3549486

Published: 31 October 2022 Publication History

Abstract

Cutscenes form an integral part of many video games, but their creation is costly, time-consuming, and requires skills that many game developers lack. While AI has been leveraged to semi-automate cutscene production, the results typically lack the internal consistency and uniformity in style that is characteristic of professional human directors. We overcome this shortcoming with Cine-AI, an open-source procedural cinematography toolset capable of generating in-game cutscenes in the style of eminent human directors. Implemented in the popular game engine Unity, Cine-AI features a novel timeline and storyboard interface for design-time manipulation, combined with runtime cinematography automation. Via two user studies, each employing quantitative and qualitative measures, we demonstrate that Cine-AI generates cutscenes that people correctly associate with a target director, while providing above-average usability. Our director imitation dataset is publicly available, and can be extended by users and film enthusiasts.

Supplementary Material

ZIP File (v6chip223aux.zip)

The auxiliary materials comprise briefings, questionnaires and example stimuli from our two experiments which can be opened with any .pdf reader. We moreover include participant responses from our first and second experiment in .xlsx format, to be opened e.g. with Microsoft Excel.

Download
894.70 KB

MP4 File (v6chip223.mp4)

Supplemental video

Download
82.41 MB

References

[1]

Dan Ablan. 2002. Digital Cinematography & Directing. Pearson Education.

[2]

Daniel Arijon. 1991. Grammar of the Film Language. Silman-James Press.

[3]

Alok Baikadi, Julius Goth, Christopher Mitchell, Eun Ha, Bradford Mott, and James Lester. 2011. Towards a Computational Model of Narrative Visualization. In Proc. Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, 2--9.

[4]

Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer.

Digital Library

[5]

Rogerio Bonatti, Cherie Ho, Wenshan Wang, Sanjiban Choudhury, and Sebastian A. Scherer. 2019. Towards a robust aerial cinematography platform: Localizing and tracking moving targets in unstructured environments. In Proc. Conference on Intelligent Robots and Systems. IEEE, 229--236.

[6]

John Brooke. 1996. SUS: A Quick and Dirty Usability Scale. In Usability Evaluation in Industry. CRC Press, 189--195.

[7]

Ludovic Burg, Christophe Lino, and Marc Christie. 2020. Real-time Anticipation of Occlusions for Automated Camera Control in Toric Space. Computer Graphics Forum, Vol. 39, 2 (2020), 523--533.

[8]

Fred Charles, Jean-Luc Lugrin, Marc Cavazza, and Steven J. Mead. 2002. Real-Time Camera Control for Interactive Storytelling. In Proc. GAME-ON. 73--76.

[9]

David Christianson, Sean Anderson, Li-wei He, David Salesin, Daniel Weld, and Michael Cohen. 1996. Declarative Camera Control for Automatic Cinematography. Proc. National Conference on Artificial Intelligence, Vol. 1 (1996), 148--155.

[10]

Marc Christie and Patrick Olivier. 2009. Camera Control in Computer Graphics: Models, Techniques and Applications. In SIGGRAPH ASIA Courses. ACM, Article 3, 197 pages.

[11]

Lynne Cooke. 2010. Assessing Concurrent Think-Aloud Protocol as a Usability Test Method: A Technical Communication Approach. Trans. Professional Communication, Vol. 53, 3 (2010), 202--215.

[12]

Steven M. Drucker and David Zeltzer. 1994. Intelligent Camera Control in a Virtual Environment. In Proc. Graphics Interface. 1--9.

[13]

K. Anders Ericsson and Herbert A. Simon. 1984. Protocol Analysis: Verbal Reports as Data. MIT Press.

[14]

Joseph L. Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, Vol. 33, 3 (1973), 613--619.

[15]

Marco Furini, Filippo Geraci, Manuela Montangero, and Marco Pellegrini. 2010. STIMO: STIll and MOving video storyboard for the web scenario. Multimedia Tools And Applications, Vol. 46 (2010), 47--69.

Digital Library

[16]

Natasa Gisev, J. Simon Bell, and Timothy F. Chen. 2013. Interrater Agreement and Interrater Reliability: Key Concepts, Approaches, and Applications. Research in Social and Administrative Pharmacy, Vol. 9, 3 (2013), 330--338.

[17]

Mark Haigh-Hutchinson. 2009. Real-Time Cameras: A Guide for Game Designers and Developers. Chapter 9 - Motion and Collision, 313--364.

[18]

Shakil A. Hannan, Daisuke Endou, and Masanobu Yamamoto. 2005. Director-Oriented Shot Assessment and Evaluation in Virtual Cinematography. The Journal of the Institute of Image Information and Television Engineers, Vol. 59, 4 (2005), 592--603.

[19]

Li-wei He, Michael Cohen, and David Salesin. 1996. The Virtual Cinematographer: A Paradigm for Automatic Real-Time Camera Control and Directing. Proc. SIGGRAPH Conference on Computer Graphics, 217--224.

[20]

Timothy Heiderich. 2012. Cinematography Techniques: The Different Types of Shots in Film. Videomakers, Vol. 22 (2012).

[21]

Arnav Jhala and R. Young. 2005. A Discourse Planning Approach to Cinematic Camera Control for Narratives in Virtual Environments. Proc. National Conference on Artificial Intelligence, Vol. 1, 307--312.

[22]

Hongda Jiang, Bin Wang, Xi Wang, Marc Christie, and Baoquan Chen. 2020. Example-driven virtual cinematography by learning camera behaviors. Trans. on Graphics, Vol. 39, 4 (2020), 45--1.

Digital Library

[23]

Anna Kantosalo and Hannu Toivonen. 2016. Modes for creative human-computer collaboration: Alternating and task-divided co-creativity. In Proc. Conference on Computational Creativity. 77--84.

[24]

Peter Karp and Steven Feiner. 1993. Automated Presentation Planning of Animation Using Task Decomposition With Heuristic Reasoning. In Proc. Graphics Interface. 118--127.

[25]

Kevin Kennedy and Robert Mercer. 2002. Planning Animation Cinematography and Shot Structure to Communicate Theme and Mood. In Proc. Symposium on Smart Graphics. 1--8.

Digital Library

[26]

Terry K. Koo and Mae Y. Li. 2016. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine, Vol. 15, 2 (2016), 155--163.

[27]

J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics (1977), 159--174.

[28]

James R. Lewis. 2018. The System Usability Scale: Past, Present, and Future. International Journal of Human--Computer Interaction, Vol. 34, 7 (2018), 577--590.

[29]

Christophe Lino and Marc Christie. 2012. Efficient Composition for Virtual Camera Control. In SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, 65--70.

[30]

Christophe Lino and Marc Christie. 2015. Intuitive and Efficient Camera Control with the Toric Space. Trans. on Graphics, Vol. 34, 4 (2015), 1--12.

Digital Library

[31]

Christophe Lino, Marc Christie, Fabrice Lamarche, Schofield Guy, and Patrick Olivier. 2010. A Real-time Cinematography System for Interactive 3D Environments. In Proc. SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, 139--148.

[32]

Joseph V. Mascelli. 1965. The Five C's of Cinematography: Motion Picture Filming Techniques. Cine-Graphic Publications.

[33]

Thomas Oskam, Robert W Sumner, Nils Thuerey, and Markus Gross. 2009. Visibility Transition Planning for Dynamic Camera Control. In Proc. SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, 55--65.

Digital Library

[34]

David Pizzi, Marc Cavazza, Alex Whittaker, and Jean-Luc Lugrin. 2008. Automatic generation of game level solutions as storyboards. In Proc. Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, 96--101.

[35]

Rémi Ronfard, Vineet Gandhi, Laurent Boiron, and Vaishnavi Ameya Murukutla. 2013. The Prose Storyboard Language: A Tool for Annotating and Directing Movies. In Proc. Workshop on Intelligent Cinematography and Editing at the Foundations of Digital Games Conference. 1--8.

[36]

James Rucks and Nikolaos Katzakis. 2021. CamerAI: Chase Camera in a Dense Environment using a Proximal Policy Optimization-trained Neural Network. In Proc. Conference on Games. IEEE, 1--8.

Digital Library

[37]

Jeff Sauro. 2011. A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices. Measuring Usability LLC.

[38]

Jeff Sauro and James R. Lewis. 2012. Quantifying the User Experience. Morgan Kaufmann, Chapter 8 - Standardized Usability Questionnaires, 185--240.

Digital Library

[39]

Patrick E. Shrout and Joseph L. Fleiss. 1979. Intraclass Correlations: Uses in Assessing Rater Reliability. Psychological Bulletin, Vol. 86, 2 (1979), 420.

[40]

James Skorupski and Michael Mateas. 2009. Interactive Story Generation for Writers: Lessons Learned from the Wide Ruled Authoring Tool. In Proc. Digital Arts and Culture. 1--10.

[41]

Edirlei Soares de Lima, Cesar Pozzer, Marcos d'Ornellas, Angelo Ciarlini, Bruno Feijó, and Antonio Furtado. 2009. Virtual Cinematography Director for Interactive Storytelling. In Proc. Conference on Advances in Computer Entertainment Technology. 263--270.

Digital Library

[42]

The Editors of Encyclopaedia Britannica. 2021. “Cinematography". In Encyclopedia Britannica. https://www.britannica.com/topic/cinematography

[43]

Paul Wheeler. 2005. Practical Cinematography. Taylor & Francis.

[44]

Ke Xie, Hao Yang, Shengqiu Huang, Dani Lischinski, Marc Christie, Kai Xu, Minglun Gong, Daniel Cohen-Or, and Hui Huang. 2018. Creating and Chaining Camera Moves for Quadrotor Videography. Trans. on Graphics, Vol. 37 (2018), 1--14.

Digital Library

[45]

R. Young and Johanna Moore. 1994. DPOCL: A Principled Approach to Discourse Planning. In Proc. Workshop on Natural Language Generation. 1--8.

Cited By

Wang ZJia JSun SWu HHan RLi ZTang DZhou JLuo J(2024)DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00754(7892-7901)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00754
Kim HAli GHan BKim HKim JShin HKim GHwang J(2024)ASAP for multi-outputs: auto-generating storyboard and pre-visualization with virtual actors based on screenplayMultimedia Tools and Applications10.1007/s11042-024-19904-3Online publication date: 3-Aug-2024
https://doi.org/10.1007/s11042-024-19904-3
Manuri FSanna AScarzello MDe Pace F(2024)A Novel Approach to 3D StoryboardingIntelligent Technologies for Interactive Entertainment10.1007/978-3-031-55722-4_13(178-192)Online publication date: 23-Mar-2024
https://doi.org/10.1007/978-3-031-55722-4_13

Index Terms

Cine-AI: Generating Video Game Cutscenes in the Style of Human Directors
1. Human-centered computing
  1. Human computer interaction (HCI)
  2. Visualization
    1. Visualization techniques

Recommendations

Is cheating a human function? The roles of presence, state hostility, and enjoyment in an unfair video game

In sports and board games, when an opponent cheats, the other players typically greet it with disdain, anger, and disengagement. However, work has yet to fully address the role of the computer cheating in video games. In this study, participants played ...
The interplay between immersion and appeal in video games

A 2í 2 study on immersion and appeal of video games is presented.Immersion and appeal are found to be highly related.Immersion is found to be unaffected by playing experience.Appeal is found to be affected both by game played and playing experience. ...
The effects of video game realism on attention, retention and aggressive outcomes

This study used a between subjects, post-test only design to test the effects of video game realism (Doom 1 vs. Doom 3) and a control condition on attention, retention and aggressive outcomes. Overall, those who played Doom 3 perceived it as ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 6, Issue CHI PLAY

CHI PLAY

October 2022

986 pages

EISSN:2573-0142

DOI:10.1145/3570219

Editor:
Jeff Nichols
Apple Inc., United States

Issue’s Table of Contents

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 October 2022

Published in PACMHCI Volume 6, Issue CHI PLAY

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Academy of Finland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
1,239
Total Downloads

Downloads (Last 12 months)634
Downloads (Last 6 weeks)70

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZJia JSun SWu HHan RLi ZTang DZhou JLuo J(2024)DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00754(7892-7901)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00754
Kim HAli GHan BKim HKim JShin HKim GHwang J(2024)ASAP for multi-outputs: auto-generating storyboard and pre-visualization with virtual actors based on screenplayMultimedia Tools and Applications10.1007/s11042-024-19904-3Online publication date: 3-Aug-2024
https://doi.org/10.1007/s11042-024-19904-3
Manuri FSanna AScarzello MDe Pace F(2024)A Novel Approach to 3D StoryboardingIntelligent Technologies for Interactive Entertainment10.1007/978-3-031-55722-4_13(178-192)Online publication date: 23-Mar-2024
https://doi.org/10.1007/978-3-031-55722-4_13

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents