research-article

Audio-Visual Interactive Art: Investigating the effect of gaze-controlled audio on visual attention and short term memory

Authors:

Josefine Hölling,

Sandra PaulettoAuthors Info & Claims

AM '21: Proceedings of the 16th International Audio Mostly Conference

Pages 32 - 39

https://doi.org/10.1145/3478384.3478421

Published: 15 October 2021 Publication History

Abstract

This article presents the development and testing of a system for interactive art. The system utilises eye tracking technology to detect the eye movements of people looking at a painting, and then uses this data to trigger sounds related to the painting. The system was developed in collaboration with a visual artist and a musician with the ultimate aim to integrate it into a new piece of interactive art. Described here in detail is the design and development of a prototype which was tested using a copy of a painting by Hieronymus Bosch: The Garden of Earthly Delights. Through this system we tested whether gaze-controlled audio affects visual attention and short term memory. The final system was integrated into a contemporary piece of interactive art currently exhibited in Ronneby, southern Sweden.

References

[1]

Robert Albrecht, Riitta Väänänen, and Tapio Lokki. 2016. Guided by music: pedestrian and cyclist navigation with route and beacon guidance. Personal and Ubiquitous Computing 20, 1 (2016), 121–145.

Digital Library

[2]

Amit Barde, Matt Ward, William S Helton, Mark Billinghurst, and Gun Lee. 2016. Attention redirection using binaurally spatialised cues delivered over a bone conduction headset. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 60. SAGE Publications Sage CA: Los Angeles, CA, 1534–1538.

[3]

Thomas Baumgartner, Michaela Esslen, and Lutz Jäncke. 2006. From emotion perception to emotion experience: Emotions evoked by pictures and classical music. International journal of psychophysiology 60, 1 (2006), 34–43.

[4]

Jennifer Romano Bergstrom and Andrew Schall. 2014. Eye tracking in user experience design. Elsevier.

Digital Library

[5]

Simon Blessenohl, Cecily Morrison, Antonio Criminisi, and Jamie Shotton. 2015. Improving indoor mobility of the visually impaired with depth-based spatial sound. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 26–34.

Digital Library

[6]

Susana Castillo, Tilke Judd, and Diego Gutierrez. 2011. Using eye-tracking to assess different image retargeting methods. In Proceedings of the ACM SIGGRAPH Symposium on Applied Perception in Graphics and Visualization. 7–14.

Digital Library

[7]

Areti Damala, Pierre Cubaud, Anne Bationo, Pascal Houlier, and Isabelle Marchal. 2008. Bridging the gap between the digital and the physical: design and evaluation of a mobile augmented reality guide for the museum visit. In Proceedings of the 3rd international conference on Digital Interactive Media in Entertainment and Arts. 120–127.

Digital Library

[8]

Rae Earnshaw. 2017. Art, Design and Technology: Collaboration and Implementation. Springer.

[9]

Ernest Edmonds. 2017. The art of interaction: What HCI can learn from interactive art. Synthesis Lectures on Human-Centered Informatics 11, 1(2017), i–73.

[10]

Ernest Edmonds, Greg Turner, and Linda Candy. 2004. Approaches to interactive art systems. In Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia. 113–117.

Digital Library

[11]

Ernest A Edmonds, Alastair Weakley, Linda Candy, Mark Fell, Roger Knott, and Sandra Pauletto. 2005. The studio as laboratory: Combining creative practice and digital technology research. International Journal of Human-Computer Studies 63, 4-5 (2005), 452–481.

Digital Library

[12]

Robert W Frick. 1984. Using both an auditory and a visual short-term store to increase digit span. Memory & Cognition 12, 5 (1984), 507–514.

[13]

Sandra Gebbensleben, Jana Dittmann, and Claus Vielhauer. 2006. Multimodal audio guide for museums and exhibitions. In Multimedia on Mobile Devices II, Vol. 6074. International Society for Optics and Photonics, 60740S.

[14]

Myounghoon Jeon, Rebecca Fiebrink, Ernest A Edmonds, and Damith Herath. 2019. From rituals to magic: Interactive art and HCI of the past, present, and future. International Journal of Human-Computer Studies 131 (2019), 108–119.

Digital Library

[15]

Elizabeth A Kensinger, Rachel J Garoff-Eaton, and Daniel L Schacter. 2007. Effects of emotion on memory specificity: Memory trade-offs elicited by negative visually arousing stimuli. Journal of memory and language 56, 4 (2007), 575–591.

[16]

Linda J Levine and Robin S Edelstein. 2010. Emotion and memory narrowing: A review and goal-relevance approach. Psychology Press.

[17]

Kenneth H Norwich. 1982. Perception as an active process. Mathematics and Computers in Simulation 24, 6 (1982), 535–539.

[18]

Rodrigo Quian Quiroga and Carlos Pedreira. 2011. How do we see art: an eye-tracker study. Frontiers in human neuroscience 5 (2011), 98.

[19]

Cliodhna Quigley, Selim Onat, Sue Harding, Martin Cooke, and Peter König. 2007. Audio-visual integration during overt visual attention. Journal of Eye Movement Research 1, 2 (2007).

[20]

J. (ed.) Reichardt. 1968. Cybernetic serendipity. Special Issue of Studio International(1968).

[21]

Daniel Reisberg and Friderike Heuer. 2004. Memory for emotional events.(2004).

[22]

Martin A Safer, Sven-Åke Christianson, Marguerite W Autry, and Karin Österlund. 1998. Tunnel memory for traumatic events. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition 12, 2 (1998), 99–117.

[23]

Ramanathan Subramanian, Divya Shankar, Nicu Sebe, and David Melcher. 2014. Emotion modulates eye movement patterns and subsequent memory for the gist and details of movie scenes. Journal of vision 14, 3 (2014), 31–31.

[24]

Valerie A Thompson and Allan Paivio. 1994. Memory for pictures and sounds: Independence of auditory and visual codes.Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale 48, 3 (1994), 380.

[25]

Yolanda Vazquez-Alvarez, Matthew P Aylett, Stephen A Brewster, Rocio von Jungenfeld, and Antti Virolainen. 2014. Multilevel auditory displays for mobile eyes-free location-based interaction. In CHI’14 Extended Abstracts on Human Factors in Computing Systems. 1567–1572.

[26]

Chi Thanh Vi, Damien Ablart, Elia Gatti, Carlos Velasco, and Marianna Obrist. 2017. Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition. International Journal of Human-Computer Studies 108 (2017), 1–14.

Digital Library

[27]

Jing Yang and Cheuk Yu Chan. 2019. Audio-augmented museum experiences with gaze tracking. In Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia. 1–5.

Digital Library

[28]

Andreas Zimmermann and Andreas Lorenz. 2008. LISTEN: a user-adaptive audio-augmented museum guide. User Modeling and User-Adapted Interaction 18, 5 (2008), 389–416.

Digital Library

Cited By

Overdevest NPatibanda RSaini AVan Den Hoven EMueller F(2024)GazeAway: Designing for Gaze Aversion ExperiencesExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650771(1-6)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650771

Recommendations

Experience evaluation of interactive art: study of GEO landscapes
IE '08: Proceedings of the 5th Australasian Conference on Interactive Entertainment

The cutting edge in the digital arts is a highly fertile ground for the investigation of engagement and the role of new technologies. The nature of audience interaction with artworks and the implication for the consequential nature of the interactive ...
Audio-visual interfaces in digital art
ACE '04: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology

This paper outlines the evolution, in the history of art and technology, of image-sound interface systems. This can be called the evolution of the "audiovisual discourse" in art and technology. Recent software for audio and visual material manipulation ...
Analysis about Application of Learning System into Interactive Digital Art Implementation
ISPAW '11: Proceedings of the 2011 IEEE Ninth International Symposium on Parallel and Distributed Processing with Applications Workshops

The rapid growth of science and technology influences to Digital Arts. Digital art is no more unfamiliar genre in art field. We can see many kinds of digital arts in our surroundings. High technology makes digital art more brilliant and various. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AM '21: Proceedings of the 16th International Audio Mostly Conference

September 2021

283 pages

ISBN:9781450385695

DOI:10.1145/3478384

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

AM '21

AM '21: Audio Mostly 2021

September 1 - 3, 2021

virtual/Trento, Italy

Acceptance Rates

Overall Acceptance Rate 177 of 275 submissions, 64%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
128
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Overdevest NPatibanda RSaini AVan Den Hoven EMueller F(2024)GazeAway: Designing for Gaze Aversion ExperiencesExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650771(1-6)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650771

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents