research-article

ShadowPlay2.5D: A 360-Degree Video Authoring Tool for Immersive Appreciation of Classical Chinese Poetry

Authors:

Xiaojuan MaAuthors Info & Claims

Journal on Computing and Cultural Heritage (JOCCH), Volume 13, Issue 1

Article No.: 5, Pages 1 - 20

https://doi.org/10.1145/3352590

Published: 05 February 2020 Publication History

Abstract

An immersive experience brought about by virtual reality can potentially enhance the appreciation of classical Chinese poetry, which is difficult to describe clearly in everyday language or ordinary media. However, making 3-dimensional illustrations for a 360-degree display in virtual reality is usually a labor-intensive and time-consuming procedure and hard to master for non-professional media creators, such as teachers. Motivated by the homology theory of classical Chinese poetry and painting, we propose an image-based approach of building 2.5-dimensional immersive stories to visualize classical Chinese poetry. Specifically, using Chinese shadow play as a metaphor, we have designed and implemented ShadowPlay2.5D, a sketch-based authoring tool to help novices create 360-degree videos of classical Chinese poetry easily. To ensure coverage of the diverse themes in Chinese poetry and preserve the sense of culture, we build a Chinese ink-painting style image repository of essential poetic elements identified via crowdsourcing. To facilitate construction of 2.5-dimensional scenes, we design features that support puppet-like animation, instancing, and camera organization in a 3-dimensional environment. Through two user studies, we show that ShadowPlay2.5D can help novices make a short 360-degree video in about 10--15 minutes, and the 2.5D stylized illustrations created can bring about a better immersive experience for poetry appreciation.

References

[1]

Fathima Assilmia, Yun Suen Pai, Keiko Okawa, and Kai Kunze. 2017. IN360: A 360-degree-video platform to change students preconceived notions on their career. In Proceedings of the CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHIEA’17). ACM, New York, NY, 2359--2365.

Digital Library

[2]

Autodesk. 2017. Autodesk Maya: Computer Animation and Modelling Software. Retrieved from https://www.autodesk.com.hk/products/maya.

[3]

Ruth Aylett and Sandy Louchart. 2003. Towards a narrative theory of virtual reality. Virtual Real. 7, 1 (2003), 2--9.

Digital Library

[4]

Sarang S. Bhadsavie, Xie Hunt Shannon Yap, Justin Segler, Rahul Jaisimha, Nishant Raman, Yengzhou Feng, Sierra J. Biggs, Micah Peoples, Robert B. Brenner, and Brian C. McCann. 2017. Immerj: A novel system for democratizing immersive storytelling. In Proceedings of the IEEE Conference on Virtual Reality (VR’17). 367--368.

[5]

David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res. 3 (Jan. 2003), 993--1022.

[6]

Blender. 2017. Blender: Open Source 3D Creation. Retrieved from https://www.blender.org.

[7]

Online Blog. 2019. From Meter to Meaning: Translations and Interpretations of Ancient Chinese Poetry. Retrieved from https://frommetertomeaning.wordpress.com/山居秋暝-王维-autumn-dusk-in-the-mountains-wang-wei.

[8]

Adrien Bousseau, Theophanis Tsandilas, Lora Oehlberg, and Wendy E. Mackay. 2016. How novices sketch and prototype hand-fabricated objects. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI’16). ACM, New York, NY, 397--408.

Digital Library

[9]

Mar Camacho, Vanessa Esteve-González, and Mercé Gisbert. 2016. From digital storytelling to 3D game-based storytelling: Fostering student’s creativity in initial teacher training. In Global Learn. Association for the Advancement of Computing in Education, 240--245.

[10]

Parmit K. Chilana, Nathaniel Hudson, Srinjita Bhaduri, Prashant Shashikumar, and Shaun Kane. 2018. Supporting remote real-time expert help: Opportunities and challenges for novice 3D modelers. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC’18). IEEE, 157--166.

[11]

Nelson S. H. Chu and Chiew-Lan Tai. 2004. Real-time painting with an expressive virtual chinese brush. IEEE Comput. Graph. Appl. 24, 5 (Sept. 2004), 76--85.

Digital Library

[12]

Chao Ding and Ligang Liu. 2016. A survey of sketch-based modeling systems. Front. Comput. Sci. 10, 6 (2016), 985--999.

Digital Library

[13]

Julie Dorsey, Songhua Xu, Gabe Smedresman, Holly Rushmeier, and Leonard McMillan. 2007. The mental canvas: A tool for conceptual architectural design and analysis. In Proceedings of the 15th Pacific Conference on Computer Graphics and Applications (PG’07). IEEE Computer Society, Washington, DC, 201--210.

[14]

Netease Education. 2019. Primary and Secondary School Textbooks Major Revision: Ancient Poetry Accounted for More Than 50%. Retrieved from http://edu.163.com/17/0804/07/CQVRN1T200297VGM.html.

[15]

Jinbo Feng and Zachary Wartell. 2014. Riding the plane: Bimanual, desktop 3D manipulation. In Proceedings of the Adjunct Publication of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST’14). ACM, New York, NY, 93--94.

Digital Library

[16]

Christopher Thompson Funkhouser and Sandy Baldwin. 2007. Prehistoric Digital Poetry: An Archaeology of Forms, 1959--1995. The University of Alabama Press.

[17]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. MIT Press, 2672--2680.

Digital Library

[18]

Bin Guan. 2017. A study on oriental aesthetics in pictorialism. In Proceedings of the 2nd International Conference on Contemporary Education, Social Sciences and Humanities. 466--472.

[19]

Rorik Henrikson, Bruno Araujo, Fanny Chevalier, Karan Singh, and Ravin Balakrishnan. 2016. Multi-device storyboards for cinematic narratives in VR. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST’16). ACM, New York, NY, 787--796.

Digital Library

[20]

Rorik Henrikson, Bruno De Araujo, Fanny Chevalier, Karan Singh, and Ravin Balakrishnan. 2016. Storeoboard: Sketching stereoscopic storyboards. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI’16). ACM, New York, NY, 4587--4598.

Digital Library

[21]

David Hinton. 2014. Classical Chinese Poetry: An Anthology. Farrar, Straus, and Giroux.

[22]

Shu-Wei Hsu and Tsai-Yen Li. 2005. Generating secondary motions in shadow play animations with motion planning techniques. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques: Sketches (ACM SIGGRAPH’05). ACM, New York, NY.

Digital Library

[23]

Chih-Fang Huang, Hsiang-Pin Lu, and Jenny Ren. 2012. Algorithmic approach to sonification of classical chinese poetry. Multimedia Tools Appl. 61, 2 (2012), 489--518.

[24]

Alec Jacobson. 2015. Breathing life into shapes. IEEE Comput. Graph. Appl. 35, 5 (Sept. 2015), 92--100.

Digital Library

[25]

Xuezhi Jin. 1984. 王维诗中的绘画美 [The Beauty of Paintings in Wang Wei’s Poems]. 文学遗产 [Literary Heritage] 4 (1984), 55--66.

[26]

Hannes Kaufmann, Dieter Schmalstieg, and Michael Wagner. 2000. Construct3D: A virtual reality application for mathematics and geometry education. Edu. Info. Technol. 5, 4 (Dec. 2000), 263--276.

Digital Library

[27]

Rubaiat Habib Kazi, Tovi Grossman, Nobuyuki Umetani, and George Fitzmaurice. 2016. Motion amplifiers: Sketching dynamic illustrations using the principles of 2D animation. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI’16). ACM, New York, NY, 4599--4609.

Digital Library

[28]

Rubaiat Habib Kazi, Takeo Igarashi, Shengdong Zhao, and Richard Davis. 2012. Vignette: Interactive texture design and manipulation with freeform gestures for pen-and-ink illustration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’12). ACM, New York, NY, 1727--1736.

Digital Library

[29]

Yu-Chi Lai, Bo-An Chen, Kuo-Wei Chen, Wei-Lin Si, Chih-Yuan Yao, and Eugene Zhang. 2017. Data-driven NPR illustrations of natural flows in chinese painting. IEEE Trans. Visual. Comput. Graph. 23, 12 (2017), 1--1.

[30]

Joshua Leung and Daniel M. Lara. 2015. Grease pencil: Integrating animated freehand drawings into 3D production environments. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques: SIGGRAPH Asia: Technical Briefs (SA’15). ACM, New York, NY.

Digital Library

[31]

Liang Li. 2004. 诗画同源与山水文化 [Poetry and Painting Homology and Landscape Culture]. 中华书局 [Zhonghua Book Company].

[32]

Hui Liang, Shujie Deng, Jian Chang, Jian Jun Zhang, Can Chen, and Ruofeng Tong. 2018. Semantic framework for interactive animation generation and its application in virtual shadow play performance. Virtual Real. 22, 2 (June 2018), 149--165.

Digital Library

[33]

Min Lin, Zhenzhen Hu, Si Liu, Meng Wang, Richang Hong, and Shuicheng Yan. 2013. eHeritage of shadow puppetry: Creation and manipulation. In Proceedings of the 21st ACM International Conference on Multimedia (MM’13). ACM, New York, NY, 183--192.

Digital Library

[34]

Fei Lu, Feng Tian, Yingying Jiang, Xiang Cao, Wencan Luo, Guang Li, Xiaolong Zhang, Guozhong Dai, and Hongan Wang. 2011. ShadowStory: Creative and collaborative digital storytelling inspired by cultural heritage. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). ACM, New York, NY, 1919--1928.

Digital Library

[35]

Sean Macdonald. 2015. Animation in China: History, Aesthetics, Media. Routledge.

[36]

Valeria Manera, Emmanuelle Chapoulie, Jérémy Bourgeois, Rachid Guerchouche, Renaud David, Jan Ondrej, George Drettakis, and Philippe Robert. 2016. A feasibility study with image-based rendered virtual reality in patients with mild cognitive impairment and dementia. PloS One 11, 3 (2016).

[37]

Alfreda Murck and Wen Fong. 1991. Words and Images: Chinese Poetry, Calligraphy, and Painting. Metropolitan Museum of Art.

[38]

The British Museum. 2018. Exhibition: A journey through China and South Asia in eight objects. Retrieved from http://ow.ly/LYM830hPDAY.

[39]

Randy Pausch, Jon Snoddy, Robert Taylor, Scott Watson, and Eric Haseltine. 1996. Disney’s aladdin: First steps toward storytelling in virtual reality. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. ACM, 193--203.

Digital Library

[40]

People.cn. 2019. Why is the Chinese Poetry Conference Widely Sought After? Retrieved from http://media.people.com.cn/n1/2017/0209/c40606-29067270.html.

[41]

Reva Ramlogan, Ahtsham U. Niazi, Rongyu Jin, James Johnson, Vincent W. Chan, and Anahi Perlas. 2017. A virtual reality simulation model of spinal ultrasound: Role in teaching spinal sonoanatomy. Region. Anesth. Pain Med. 42, 2 (2017), 217--222.

[42]

Alec Rivers, Takeo Igarashi, and Frédo Durand. 2010. 2.5D cartoon models. ACM Trans. Graph. 29, 4 (2010), 59:1--59:7.

Digital Library

[43]

Annie Katsura Rollins. 2017. Chinese Shadow Puppetry: A Comprehensive Informational Website. Retrieved from https://www.chineseshadowpuppetry.com.

[44]

Marie-Laure Ryan. 1999. Immersion vs. Interactivity: Virtual reality and literary theory. SubStance 28, 2 (1999), 110--137.

[45]

Scott Schaefer, Travis McPhail, and Joe Warren. 2006. Image deformation using moving least squares. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques: ACM SIGGRAPH: Papers (SIGGRAPH’06). ACM, New York, NY, 533--540.

Digital Library

[46]

Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R. Voss, and Jiawei Han. 2018. Automated phrase mining from massive text corpora. IEEE Trans. Knowl. Data Eng. 30, 10 (Oct. 2018), 1825--1837.

[47]

Ivan E. Sutherland. 1964. Sketchpad a man-machine graphical communication system. Simulation 2, 5 (1964).

[48]

Frank Thomas and Ollie Johnston. 1995. The Illusion of Life: Disney Animation. Hyperion, New York.

[49]

Audrey Tse, Charlene Jennett, Joanne Moore, Zillah Watson, Jacob Rigby, and Anna L. Cox. 2017. Was I there?: Impact of platform and headphones on 360 video immersion. In Proceedings of the CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA’17). ACM, New York, NY, 2967--2974.

Digital Library

[50]

Betty Tseng. 2010. English Translation of Chinese Poetry. Retrieved from https://28utscprojects.wordpress.com/2010/12/12/249.

[51]

Rich Wareham and Joan Lasenby. 2008. Bone glow: An improved method for the assignment of weights for mesh deformation. LNCS 5098 (2008).

[52]

David Weintrop, Afsoon Afzal, Jean Salac, Patrick Francis, Boyang Li, David C. Shepherd, and Diana Franklin. 2018. Evaluating CoBlox: A comparative study of robotics programming environments for adult novices. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI’18). ACM, New York, NY.

Digital Library

[53]

Jun Xing, Hsiang-Ting Chen, and Li-Yi Wei. 2014. Autocomplete painting repetitions. ACM Trans. Graph. 33, 6 (2014), 172:1--172:11.

Digital Library

[54]

Jun Xing, Rubaiat Habib Kazi, Tovi Grossman, Li-Yi Wei, Jos Stam, and George Fitzmaurice. 2016. Energy-Brushes: Interactive tools for illustrating stylized elemental dynamics. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST’16). ACM, New York, NY, 755--766.

Digital Library

[55]

Jun Xing, Li-Yi Wei, Takaaki Shiratori, and Koji Yatani. 2015. Autocomplete hand-drawn animations. ACM Trans. Graph. 34, 6 (Oct. 2015).

Digital Library

[56]

Songhua Xu, Yingqing Xu, Sing Bing Kang, David H. Salesin, Yunhe Pan, and Heung-Yeung Shum. 2006. Animating chinese paintings through stroke-based decomposition. ACM Trans. Graph. 25, 2 (2006), 239--267.

Digital Library

[57]

Yuanchong Xu and Ming Xu. 2019. Retrieved from http://www.en84.com/dianji/shi/201103/00005875.html.

[58]

Siyao Yan, Xuling Zheng, Xiaodong Shi, and Fakui Zheng. 2015. Study of intelligent multimedia display system for classic chinese poetry. ACTA Scientiarum Naturalium Universitatis Pekinensis 51, 2 (2015), 255--261.

[59]

Che-Ching Yang, Shian-Shyong Tseng, Anthony Y. H. Liao, and Tyne Liang. 2013. Situated poetry learning using multimedia resource sharing approach.J. Edu. Technol. Soc. 16, 2 (2013).

[60]

Satoshi Yonemoto. 2012. A sketch-based skeletal figure animation tool for novice users. In Proceedings of the 9th International Conference on Computer Graphics, Imaging and Visualization. 37--42.

Digital Library

[61]

Mingmin Zhang, Xuan Chen, and Zhigeng Pan. 2014. Interaction study of digital shadow play using the kinect. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques: SIGGRAPH Asia: Posters (SA’14). ACM, New York, NY, 17:1.

Digital Library

[62]

Yi-Bo Zhu, Chen-Jia Li, I. Fan Shen, Kwan-Liu Ma, and Aleksander Stompel. 2003. A new form of traditional art: Visual simulation of chinese shadow play. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques: Sketches and Applications (ACM SIGGRAPH’03). 1.

Digital Library

Cited By

Li Y(2024)Application of JSA, a Sentiment Analysis Model in Chinese Language and Literature, to Sentiment Recognition in Classical PoetryApplied Mathematics and Nonlinear Sciences10.2478/amns-2024-22059:1Online publication date: 5-Aug-2024
https://doi.org/10.2478/amns-2024-2205
Yu ZDuan P(2024)Meta‐analyses of anxiety, motivation, performance, satisfaction, and self‐efficacy in virtual reality‐assisted language educationForeign Language Annals10.1111/flan.1274857:2(550-580)Online publication date: 22-Jan-2024
https://doi.org/10.1111/flan.12748
Zhang WZhang JWong KWang YFeng YWang LChen W(2024)Computational Approaches for Traditional Chinese Painting: From the “Six Principles of Painting” PerspectiveJournal of Computer Science and Technology10.1007/s11390-024-3408-x39:2(269-285)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s11390-024-3408-x
Show More Cited By

Index Terms

ShadowPlay2.5D: A 360-Degree Video Authoring Tool for Immersive Appreciation of Classical Chinese Poetry
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools

Recommendations

Jigsaw: Authoring Immersive Storytelling Experiences with Augmented Reality and Internet of Things
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Augmented Reality (AR) presents new opportunities for immersive storytelling. However, this immersiveness faces two main hurdles. First, AR’s immersive quality is often confined to visual elements, such as pixels on a screen. Second, crafting immersive ...
Action Units: Directing User Attention in 360-degree Video based VR
VRST '19: Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology

A key challenge to effective storytelling using Virtual Reality (VR), such as with 360-degree videos, is how to direct user attention to important content without taking away user agency for free exploration. In this paper, we introduce the notion of an ...
The Geometry of Storytelling: Theatrical Use of Space for 360-degree Videos and Virtual Reality
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

360-degree filming and head-mounted displays (HMDs) give recorded media a new sense of space. Theatre practitioners' expertise in manipulating spatial interactions has much to contribute to immersive recorded content. Four theatre directors led teams of ...

Comments

Information & Contributors

Information

Published In

cover image Journal on Computing and Cultural Heritage

Journal on Computing and Cultural Heritage Volume 13, Issue 1

February 2020

143 pages

ISSN:1556-4673

EISSN:1556-4711

DOI:10.1145/3382043

Editor:
Franco Niccolucci
VAST-LAB at PIN, University of Florence, Italy

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 February 2020

Accepted: 01 July 2019

Revised: 01 July 2019

Received: 01 July 2018

Published in JOCCH Volume 13, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Research Grants Council of the Hong Kong Special Administrative Region, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
651
Total Downloads

Downloads (Last 12 months)105
Downloads (Last 6 weeks)6

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li Y(2024)Application of JSA, a Sentiment Analysis Model in Chinese Language and Literature, to Sentiment Recognition in Classical PoetryApplied Mathematics and Nonlinear Sciences10.2478/amns-2024-22059:1Online publication date: 5-Aug-2024
https://doi.org/10.2478/amns-2024-2205
Yu ZDuan P(2024)Meta‐analyses of anxiety, motivation, performance, satisfaction, and self‐efficacy in virtual reality‐assisted language educationForeign Language Annals10.1111/flan.1274857:2(550-580)Online publication date: 22-Jan-2024
https://doi.org/10.1111/flan.12748
Zhang WZhang JWong KWang YFeng YWang LChen W(2024)Computational Approaches for Traditional Chinese Painting: From the “Six Principles of Painting” PerspectiveJournal of Computer Science and Technology10.1007/s11390-024-3408-x39:2(269-285)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s11390-024-3408-x
Ching C(2024)Breaking Barriers for Classical Chinese: Tang Poetry in Virtual RealityVirtual, Augmented and Mixed Reality10.1007/978-3-031-61047-9_2(19-43)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-61047-9_2
Logothetis ISfyrakis MVidakis N(2023)EduARdo—Unity Components for Augmented Reality EnvironmentsInformation10.3390/info1404025214:4(252)Online publication date: 21-Apr-2023
https://doi.org/10.3390/info14040252
Hu C(2023)Painting Color Editing System Based on Virtual Reality Sensor TechnologyJournal of Sensors10.1155/2023/64618432023(1-6)Online publication date: 3-Apr-2023
https://doi.org/10.1155/2023/6461843
Tian JCao YFeng LFu DYuan LQu HWang YFan M(2023)PoeticAR: Reviving Traditional Poetry of the Heritage Site of Jichang Garden via Augmented RealityInternational Journal of Human–Computer Interaction10.1080/10447318.2023.217680640:6(1438-1454)Online publication date: 13-Feb-2023
https://doi.org/10.1080/10447318.2023.2176806
Boboc RBăutu EGîrbacia FPopovici NPopovici D(2022)Augmented Reality in Cultural Heritage: An Overview of the Last Decade of ApplicationsApplied Sciences10.3390/app1219985912:19(9859)Online publication date: 30-Sep-2022
https://doi.org/10.3390/app12199859
Lin SChen MChen NShao MMaiden NSas C(2022)Interactive Cartoon Animation through Collaborative Storytelling: A Field StudyProceedings of the 14th Conference on Creativity and Cognition10.1145/3527927.3535204(505-509)Online publication date: 20-Jun-2022
https://dl.acm.org/doi/10.1145/3527927.3535204
Coelho HMonteiro PGonçalves GMelo MBessa M(2022)Authoring tools for virtual reality experiences: a systematic reviewMultimedia Tools and Applications10.1007/s11042-022-12829-981:19(28037-28060)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1007/s11042-022-12829-9
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents