course

Hands-on: rapid interactive application prototyping for media arts and stage performance and beyond

Authors:

Serguei A. Mokhov,

Sudhir P. Mudur,

Peter GrogonoAuthors Info & Claims

SA '18: SIGGRAPH Asia 2018 Courses

Article No.: 9, Pages 1 - 32

https://doi.org/10.1145/3277644.3277760

Published: 04 December 2018 Publication History

Abstract

We complement the last three editions of the course at SIGGRAPH Asia (2015, 2016) and SIGGRAPH (2017) to make it more of a hands-on nature and include OpenISS. We explore a rapid prototyping of interactive graphical applications for stage and beyond using Jitter/Max and Processing with OpenGL, shaders, and featuring connectivity with various devices. Such rapid prototyping environment is ideal for entertainment computing, as well as for artists and live performances using real-time interactive graphics. We share the expertise we developed in connecting the real-time graphics with on-stage performance with the Illimitable Space System (ISS) v2 and its OpenISS core.

References

[1]

Edward A. Ashcroft, Anthony A. Faustini, Rangaswamy Jagannathan, and William W. Wadge. 1995. Multidimensional Programming. Oxford University Press, London. ISBN: 978-0195075977.

Digital Library

[2]

Edward A. Ashcroft and William W. Wadge. 1977. Lucid, a nonprocedural language with iteration. Commun. ACM 20, 7 (July 1977), 519--526.

Digital Library

[3]

Sebouh-Steve Bardakjian, Miao Song, Serguei A. Mokhov, and Sudhir P. Mudur. 2016. ISSv3: From Human Motion in the Real to the Interactive Documentary Film in AR/VR. In Proceedings of the SIGGRAPH ASIA 2016 Workshop on Virtual Reality Meets Physical Reality (VR Meets PR 2016). ACM, New York, NY, USA.

Digital Library

[4]

Greg Borenstein. 2013. OpenCV for Processing. {online}. (July 2013). https://github.com/atduskgreg/opencv-processing.

[5]

Niels Böttcher. 2007--2013. An introduction to Max/MSP. {online}, Medialogy, Aalborg University Copenhagen. (2007--2013). http://imi.aau.dk/~nib/maxmsp/introduction_to_MaxMsp.ppt.

[6]

Tom Butterworth and Anton Marini. 2013. Syphon for Jitter. {online}. (Nov. 2013). https://github.com/Syphon/Jitter/releases/.

[7]

Craig Caldwell. 2015. Bringing Story to Life: For Programmers, Animators, VFX Artists, and Interactive Designers. In ACM SIGGRAPH 2015 Courses (SIGGRAPH'15). ACM, New York, NY, USA, 6:1--6:10.

Digital Library

[8]

Andres Colubri. 2014. Syphon for Processing. {online}. (2014). https://github.com/Syphon/Processing/releases.

[9]

Cycling '74. 2005--2015. Max/MSP/Jitter. {online}. (2005--2015). http://cycling74.com/products/max/.

[10]

Peter Elsea. 2007--2013. Max/MSP/Jitter Tutorials. {online}, University of California, Santa Cruz. (2007--2013). ftp://arts.ucsc.edu/pub/ems/MaxTutors/Jit.tutorials/.

[11]

Ben Fry and Casey Reas. 2001--2015. Processing - a programming language, development environment, and online community. {online}. (2001--2015). http://www.processing.org/.

[12]

Google LLC. 2017--2018. Google Brain Team: Machine Learning Algorithms. {online}. (2017--2018). https://magenta.tensorflow.org/.

[13]

Peter Grogono. 2002. Getting Started with OpenGL. {online}. (2002). Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada.

[14]

Intel Corporation, Willow Garage, and Itseez. 2000--2018. Itseez: Image Processing Algorithms. {online}. (2000--2018). https://opencv.org/.

[15]

Joris and The Resolume Team. 2014. Resolume Arena Blog: Spout - Sharing Video between Applications on Windows. {online}. (May 2014). http://resolume.com/blog/11110/spout-sharing-video-between-applications-on-windows.

[16]

Gene Kogan. 2014a. Kinect Projector Toolkit for image mapping and calibration. {online, GitHub}. (July 2014). https://github.com/genekogan/KinectProjectorToolkit.

[17]

Gene Kogan. 2014b. Kinect Projector Toolkit for image mapping and calibration. {online}. (July 2014). https://github.com/genekogan/KinectProjectorToolkit.

[18]

Joseph J. LaViola, Jr. 2015. Context Aware 3D Gesture Recognition for Games and Virtual Reality. In ACM SIGGRAPH 2015 Courses (SIGGRAPH'15). ACM, New York, NY, USA, 10:1--10:61.

Digital Library

[19]

Hao Li, Anshuman Das, Tristan Swedish, Hyunsung Park, and Ramesh Raskar. 2015. Modeling and Capturing the Human Body: For Rendering, Health and Visualization. In ACM SIGGRAPH 2015 Courses (SIGGRAPH'15). ACM, New York, NY, USA, 16:1--16:160.

Digital Library

[20]

V. J. Manzo. 2011. Max/MSP/Jitter for Music: A Practical Guide to Developing Interactive Music Systems for Education and More. Oxford University Press.

[21]

Microsoft. 2012a. Human Interface Guidelines: Kinect for Windows v. 1.5. {online}. (2012). http://go.microsoft.com/fwlink/?LinkId=247735.

[22]

Microsoft. 2012b. The Kinect for Windows SDK v. 1.5. {online}. (21 May 2012). Online at http://www.microsoft.com/en-us/kinectforwindows/develop/developer-downloads.aspx and http://msdn.microsoft.com/en-us/library/hh855347.

[23]

Serguei A. Mokhov, Miao Song, Satish Chilkaka, Zinia Das, Jie Zhang, Jonathan Llewellyn, and Sudhir P. Mudur. 2016. Agile Forward-Reverse Requirements Elicitation as a Creative Design Process: A Case Study of llimitable Space System v2. Journal of Integrated Design and Process Science 20, 3 (Sept. 2016), 3--37.

Digital Library

[24]

Serguei A. Mokhov, Kin-Fung Yiu, Brian Ye, Jie Zhang, Haotao Lai, and Miao Song. 2017. Real-time Motion Capture for Performing Arts and Stage. {online}, TEDxConcordia. (Sept. 2017). https://www.youtube.com/watch?v=YgwnEmHFwI8.

[25]

R. Molich and Jakob Nielsen. 1990. Improving a human-computer dialogue. Commun. ACM 33, 3 (March 1990), 338--348.

Digital Library

[26]

OpenKinect Contributors. 2011--2018. OpenKinect: Open Source Drivers for Kinect v1. {online}. (2011--2018). http://openkinect.org.

[27]

Jean-Marc Pelletier. 2012. jit.freenect.grab - a Max/MSP/Jitter external for Microsoft Kinect. {online}. (7 March 2012). RC5, http://jmpelletier.com/freenect/.

[28]

Bill Polson. 2015. Pipeline Design Patterns. In ACM SIGGRAPH 2015 Courses (SIGGRAPH'15). ACM, New York, NY, USA, 21:1--21:59.

Digital Library

[29]

Konstantinos Psimoulis, Paul Palmieri, Inna Taushanova-Atanasova, Yasmine Chiter, Amjrali Shirkhodaei, Navid Golabian, Mohammad-Ali Eghtesadi, Behrooz Hedayati, Piratheeban Annamalai, and Andrew Laramee. 2018. OpenISS Web Services API Implementation for OpenISS-as-a-Service. {online}, SOEN487 Team 10 and Team 11, Serguei Mokhov. (April 2018). https://github.com/OpenISS/OpenISS/tree/master/src/api/java.

[30]

Miller Puckette and PD Community. 2007--2014. Pure Data. {online}. (2007--2014). http://puredata.org.

[31]

Theresa-Marie Rhyne. 2015. Applying Color Theory to Digital Media and Visualization. In ACM SIGGRAPH 2015 Courses (SIGGRAPH'15). ACM, New York, NY, USA, 5:1--5:112.

Digital Library

[32]

Christian Richardt, James Tompkin, Jiamin Bai, and Christian Theobalt. 2015. User-centric Computational Videography. In ACM SIGGRAPH 2015 Courses (SIGGRAPH'15). ACM, New York, NY, USA, 25:1--25:6.

Digital Library

[33]

Yvonne Rogers, Helen Sharp, and Jenny Preece. 2011. Interaction Design: Beyond Human - Computer Interaction (3rd ed.). Wiley Publishing. Online resources: id-book.com.

Digital Library

[34]

Andreas Schlegel. 2011. oscP5 - A implementation of the OSC protocol for Processing. {online}. (2011). http://www.sojamo.de/libraries/oscP5/.

[35]

Miao Song. 2012. Computer-Assisted Interactive Documentary and Performance Arts in Illimitable Space. Ph.D. Dissertation. Special Individualized Program/Computer Science and Software Engineering, Concordia University, Montreal, Canada. Online at http://spectrum.library.concordia.ca/975072 and http://arxiv.org/abs/1212.6250.

[36]

Miao Song et al. 2014a. Real-Time Motion-Based Shadow and Green Screen Visualization, and Video Feedback for the Like Shadows Theatre Performance with the ISS. {theatre production, video, news}. (2--12 April 2014). http://www.concordia.ca/encs/cunews/main/stories/2014/06/04/digital-art-thatillustratesthelandofthelivingandthedead.html and http://www.concordia.ca/content/dam/encs/csse/news/docs/like-shadows-cse-academy.pdf.

[37]

Miao Song and Serguei A. Mokhov. 2014. Dynamic Motion-Based Background Visualization for the Ascension Dance with the ISS. {dance show, video}. (18--19 Jan. 2014). http://vimeo.com/85049604.

[38]

Miao Song, Serguei A. Mokhov, et al. 2015b. Illimitable Space System at CG in Asia International Resources. Talk and Demo. (10 Aug. 2015). http://s2015.siggraph.org/attendees/acm-siggraph-theater-events.

[39]

Miao Song, Serguei A. Mokhov, Julie Chaffarod, et al. 2015a. Dynamic Motion-Based Visualization for the District 3 Demo Day with the ISSv2 and Processing. {demo, video}. (4 June 2015). https://vimeo.com/130122925 and https://vimeo.com/129692753.

[40]

Miao Song, Serguei A. Mokhov, and Peter Grogono. 2014b. A Brief Technical Note on Haptic Jellyfish with Falcon and OpenGL. In Proceedings of the CHI'14 Extended Abstracts: ACM SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1525--1530. Includes video and poster.

Digital Library

[41]

Miao Song, Serguei A. Mokhov, Peter Grogono, and Sudhir P. Mudur. 2014a. Illimitable Space System as a Multimodal Interactive Artists' Toolbox for Real-time Performance. In Proceedings of the SIGGRAPH ASIA 2014 Workshop on Designing Tools for Crafting Interactive Artifacts (SIGGRAPH ASIA'14). ACM, New York, NY, USA, 2:1--2:4.

Digital Library

[42]

Miao Song, Serguei A. Mokhov, Peter Grogono, and Sudhir P. Mudur. 2014b. On a Non-Web-Based Multimodal Interactive Documentary Production. In Proceedings of the 2014 International Conference on Virtual Systems Multimedia (VSMM'2014), Harold Thwaites, Sarah Kenderdine, and Jeffrey Shaw (Eds.). IEEE, 329--336.

[43]

Miao Song, Serguei A. Mokhov, Alison R. Loader, and Maureen J. Simmonds. 2009. A Stereoscopic OpenGL-based Interactive Plug-in Framework for Maya and Beyond. In Proceedings of VRCAI'09. ACM, New York, NY, USA, 363--368.

Digital Library

[44]

Miao Song, Serguei A. Mokhov, Sudhir P. Mudur, and Peter Grogono. 2015. Rapid Interactive Real-time Application Prototyping for Media Arts and Stage Performance. In ACM SIGGRAPH Asia 2015 Courses (SIGGRAPH Asia'15). ACM, New York, NY, USA, 14:1--14:11.

Digital Library

[45]

Miao Song, Serguei A. Mokhov, Sudhir P. Mudur, and Peter Grogono. 2016. Hands-on: Rapid Interactive Application Prototyping for Media Arts and Stage Production. In ACM SIGGRAPH Asia 2016 Courses (SIGGRAPH Asia'16). ACM, New York, NY, USA, 19:1--19:29.

Digital Library

[46]

Miao Song, Serguei A. Mokhov, Jilson Thomas, et al. 2015b. Dynamic Motion-Based Background Visualization for the Gray Zone Dance with the ISSv2. {dance show, video}. (14 Feb. 2015). https://vimeo.com/121177927.

[47]

Miao Song, Serguei A. Mokhov, Jilson Thomas, and Sudhir P. Mudur. 2015a. Applications of the Illimitable Space System in the Context of Media Technology and On-Stage Performance: a Collaborative Interdisciplinary Experience. In Proceedings of GEM'15. IEEE. To appear.

[48]

Debbie Stone, Caroline Jarrett, Mark Woodroffe, and Shailey Minocha. 2005. User Interface Design and Evaluation (1st ed.). Wiley Publishing.

Digital Library

[49]

Marian F. Ursu, Vilmos Zsombori, John Wyver, Lucie Conrad, Ian Kegel, and Doug Williams. 2009. Interactive Documentaries: A Golden Age. Comput. Entertain. 7, Article 41 (Sept. 2009), 29 pages. Issue 3.

Digital Library

[50]

William W. Wadge and Edward A. Ashcroft. 1985. Lucid, the Dataflow Programming Language. Academic Press, London.

Digital Library

[51]

Todd Winkler. 2001. Compositing Interactive Music: Techniques and Ideas Using Max. MIT Press.

Digital Library

[52]

Jie Zhang, Sebouh Bardakjian, Milin Li, Miao Song, Serguei A. Mokhov, Sudhir P. Mudur, and Jean-Claude Bustros. 2015. Towards Historical Exploration of Sites With an Augmented Reality Interactive Documentary Prototype App. In Proceedings of Appy Hour, SIGGRAPH'2015. ACM.

Cited By

Batista NMonteiro I(2023)How HCI concepts are used in articles featuring interactive digital arts: a literature review.Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems10.1145/3638067.3638116(1-12)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3638067.3638116
Mokhov SSong MMudur SGrogono P(2020)Dataflow VFX Programming and Processing for Artists and OpenISSACM SIGGRAPH 2020 Labs10.1145/3388763.3407760(1-32)Online publication date: 17-Aug-2020
https://dl.acm.org/doi/10.1145/3388763.3407760
Mokhov SLi DLai HSingh JShen YLlewellyn JSong MMudur S(2019)ISSv2 and OpenISS distributed system for real-time interaction for performing artsACM SIGGRAPH 2019 Posters10.1145/3306214.3338539(1-2)Online publication date: 28-Jul-2019
https://dl.acm.org/doi/10.1145/3306214.3338539

Index Terms

Hands-on: rapid interactive application prototyping for media arts and stage performance and beyond

Recommendations

Rapid interactive real-time application prototyping for media arts and stage performance
SA '15: SIGGRAPH Asia 2015 Courses

We explore a rapid prototyping of interactive graphical applications using Jitter/Max and Processing with OpenGL, shaders, and featuring connectivity with various devices such as, Kinect, Wii, iDevice-based controls, and others. Such rapid prototyping ...
Hands-on: rapid interactive application prototyping for media arts and stage production
SA '16: SIGGRAPH ASIA 2016 Courses

We expand on the last year's successful first edition of the course to make it more of a hands-on nature. We explore a rapid prototyping of interactive graphical applications using Jitter/Max and Processing with OpenGL, shaders, and featuring ...
Dataflow programming and processing for artists and beyond
SA '19: SIGGRAPH Asia 2019 Courses

We complement the last three editions of the course at SIGGRAPH Asia (2015, 2016, 2018) and SIGGRAPH (2017) to make it more of a hands-on nature and include OpenISS. We explore a rapid prototyping of interactive graphical applications for stage and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SA '18: SIGGRAPH Asia 2018 Courses

December 2018

2747 pages

ISBN:9781450360265

DOI:10.1145/3277644

Conference Chair:
Carol O'Sullivan
Trinity College Dublin, Ireland

Copyright © 2018 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2018

Check for updates

Author Tags

Qualifiers

Course

Conference

SA '18

Sponsor:

SIGGRAPH

SA '18: SIGGRAPH Asia 2018

December 4 - 7, 2018

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 178 of 869 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
136
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Batista NMonteiro I(2023)How HCI concepts are used in articles featuring interactive digital arts: a literature review.Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems10.1145/3638067.3638116(1-12)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3638067.3638116
Mokhov SSong MMudur SGrogono P(2020)Dataflow VFX Programming and Processing for Artists and OpenISSACM SIGGRAPH 2020 Labs10.1145/3388763.3407760(1-32)Online publication date: 17-Aug-2020
https://dl.acm.org/doi/10.1145/3388763.3407760
Mokhov SLi DLai HSingh JShen YLlewellyn JSong MMudur S(2019)ISSv2 and OpenISS distributed system for real-time interaction for performing artsACM SIGGRAPH 2019 Posters10.1145/3306214.3338539(1-2)Online publication date: 28-Jul-2019
https://dl.acm.org/doi/10.1145/3306214.3338539

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents