Just Accepted
Assessing Usefulness, Ease of Use and Recognition Performance of Semi-Automatic Mulsemedia Authoring
Authors: 
Raphael Abreu, Joel dos Santos, 
Gheorghita Ghinea, Débora C. Muchaluat-SaadeAuthors Info & Claims
Raphael Abreu, Joel dos Santos, Gheorghita Ghinea, Débora C. Muchaluat-SaadeAuthors Info & ClaimsAccepted on 19 August 2024
Abstract
Mulsemedia (Multiple Sensorial Media) authoring poses a considerable challenge as authors navigate the intricate task of identifying moments to activate sensory effects within multimedia content. A novel proposal is to integrate content recognition algorithms that use machine learning (ML) into authoring tools to alleviate the authoring effort. As author subjectivity is very important, it is imperative to allow users to define which sensory effects should be automatically extracted. This paper conducts a twofold evaluation of the proposed semi-automatic authoring. The first is from a user perspective within the STEVE 2.0 mulsemedia authoring tool, employing the Goal-Question-Metric (GQM) methodology and a user feedback questionnaire. Our user evaluation indicates that users perceive the semi-automatic authoring approach as a positive enhancement to the authoring process. The second evaluation targets sensory effect recognition using two different content recognition modules, quantifying their automatic recognition capabilities against manual authoring. Metrics such as precision, recall, and F1 scores provide insights into the strengths and nuances of each module. Differences in label assignments underscore the need for ML module result combination methodologies. These evaluations contribute to a comprehensive understanding of the effectiveness of sensory effect recognition modules in enhancing mulsemedia content authoring.
References
[1]
Raphael Abreu and Joel dos Santos. 2017. Using Abstract Anchors to Aid The Development of Multimedia Applications With Sensory Effects. In Proceedings of the 2017 ACM Symposium on Document Engineering (Valletta, Malta) (DocEng ’17). ACM, New York, NY, USA, 211–218. https://doi.org/10.1145/3103010.3103014
[2]
Raphael Abreu, Joel dos Santos, and Eduardo Bezerra. 2018. A Bimodal Learning Approach to Assist Multi-sensory Effects Synchronization. In International Joint Conference on Neural Networks (Rio de Janeiro, Brazil) (IJCNN ’18). IEEE. https://doi.org/10.1109/IJCNN.2018.8489357
[3]
Raphael Abreu, Douglas Mattos, Joel Santos, George Ghinea, and Débora C. Muchaluat-Saade. 2023. Semi-automatic mulsemedia authoring analysis from the user’s perspective. In Proceedings of the 14th ACM Conference on Multimedia Systems (Vancouver, Canada) (MMSys ’23). ACM, New York, NY, USA, 249–256. https://doi.org/10.1145/3587819.3590979
[4]
Raphael Abreu, Douglas Mattos, Joel dos Santos, Gheorghita Ghinea, and Débora Muchaluat-Saade. 2001. Toward Content-Driven Intelligent Authoring of Mulsemedia Applications. IEEE Multimedia 28, 1 (2001), 7–16. https://doi.org/10.1109/MMUL.2020.3011383
[5]
Victor R Basili. 1992. Software modeling and measurement: the Goal/Question/Metric paradigm. Technical Report.
[6]
Stephen R Briggs and Jonathan M Cheek. 1986. The role of factor analysis in the development and evaluation of personality scales. Journal of personality 54, 1 (1986), 106–148. https://doi.org/10.1111/j.1467-6494.1986.tb00391.x
[7]
John Brooke. 1996. SUS: A Quick and Dirty usability scale. Usability evaluation in industry, Chapter 21 (1996), 189–194. https://www.taylorfrancis.com/chapters/edit/10.1201/9781498710411-35/sus-quick-dirty-usability-scale-john-brooke
[8]
Lee Anna Clark and David Watson. 2016. Constructing validity: Basic issues in objective scale development. 7, 3 (2016). https://doi.org/10.1037/1040-3590.7.3.309
[9]
Alexandra Covaci, Longhao Zou, Irina Tal, Gabriel-Miro Muntean, and Gheorghita Ghinea. 2018. Is multimedia multisensorial?-a review of mulsemedia systems. ACM Computing Surveys (CSUR) 51, 5 (2018), 1––35. https://doi.org/10.1145/3233774
[10]
F. Danieau, J. Fleureau, P. Guillotel, N. Mollet, M. Christie, and A. Lécuyer. 2014. Toward Haptic Cinematography: Enhancing Movie Experiences with Camera-Based Haptic Effects. IEEE MultiMedia 21, 2 (Apr 2014), 11–21. https://doi.org/10.1109/MMUL.2013.64
[11]
Fred D Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly (1989), 319–340.
[12]
Fred D Davis, Richard P Bagozzi, and Paul R Warshaw. 1989. User acceptance of computer technology: A comparison of two theoretical models. Management science 35, 8 (1989), 982–1003. https://doi.org/10.1287/mnsc.35.8.982
[13]
Marcello Novaes de Amorim, Estêvão Bissoli Saleme, Fábio Ribeiro de Assis Neto, Celso A. S. Santos, and Gheorghiţă Ghinea. 2019. Crowdsourcing authoring of sensory effects on videos. Multimedia Tools and Applications 78 (2019), 19201––19227. https://doi.org/10.1007/s11042-019-7312-2
[14]
Douglas Paulo de Mattos and Débora C Muchaluat-Saade. 2018. STEVE: a Hypermedia Authoring Tool based on the Simple Interactive Multimedia Model. In Proceedings of the ACM Symposium on Document Engineering 2018. 1–10.
[15]
Douglas Paulo de Mattos, Débora C Muchaluat-Saade, and Gheorghita Guinea. 2020. An Approach for Authoring Mulsemedia Documents Based on Events. In International Conference on Computing, Networking and Communications, 2020. IEEE. https://doi.org/10.1109/ICNC47757.2020.9049485
[16]
Egon Dejonckheere, Febe Demeyer, Birte Geusens, Maarten Piot, Francis Tuerlinckx, Stijn Verdonck, and Merijn Mestdagh. 2022. Assessing the reliability of single-item momentary affective measurements in experience sampling. Psychological assessment 34, 12 (2022), 1138.
[17]
Gheorghita Ghinea, Christian Timmerer, Weisi Lin, and Stephen R. Gulliver. 2014. Mulsemedia: State of the Art, Perspectives, and Challenges. ACM Transactions on Multimedia Computing, Communications, and Applications 11, 1s (2014), 1–23. https://doi.org/10.1145/2617994
[18]
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. 2022. Transformers in vision: A survey. ACM computing surveys (CSUR) 54, 10s (2022), 1–41.
[19]
Sang Kyun Kim, Seung Jun Yang, Chung Hyun Ahn, and Yong Soo Joo. 2014. Sensorial information extraction and mapping to generate temperature sensory effects. ETRI Journal 36, 2 (2014), 224–231. https://doi.org/10.4218/etrij.14.2113.0065
[20]
Yukhe Lavinia, Holly H. Vo, and Abhishek Verma. 2016. Fusion Based Deep CNN for Improved Large-Scale Image Action Recognition. In 2016 IEEE International Symposium on Multimedia (ISM). 609–614. https://doi.org/10.1109/ISM.2016.0131
[21]
Jaebong Lee, Bohyung Han, and Seungmoon Choi. 2015. Motion effects synthesis for 4D films. IEEE Transactions on Visualization and Computer Graphics 22, 10 (2015), 2300–2314. https://doi.org/10.1109/TVCG.2015.2507591
[22]
Jaebong Lee, Bohyung Han, and Choi Seungmoon. 2016. Interactive motion effects design for a moving object in 4D films. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology. ACM, 219–228. https://doi.org/10.1145/2993369.2993389
[23]
Nikola Marangunić and Andrina Granić. 2015. Technology acceptance model: a literature review from 1986 to 2013. Universal access in the information society 14 (2015), 81–95. https://doi.org/10.1007/s10209-014-0348-1
[24]
Douglas Paulo De Mattos, Débora C Muchaluat-Saade, and Gheorghita Ghinea. 2021. Beyond multimedia authoring: On the need for mulsemedia authoring tools. ACM Computing Surveys (CSUR) 54, 7 (2021), 1–31. https://doi.org/10.1145/3464422
[25]
Jum C Nunnally. 1978. An overview of psychological measurement. Clinical diagnosis of mental disorders: A handbook (1978), 97–146.
[26]
José Pinheiro and Douglas Bates. 2006. Mixed-effects models in S and S-PLUS. Springer science & business media.
[27]
Thomhert S Siadari, Mikyong Han, and Hyunjin Yoon. 2017. 4D Effect Video Classification with Shot-Aware Frame Selection and Deep Neural Networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1148–1155. https://doi.org/10.1109/ICCVW.2017.139
[28]
Keith S Taber. 2018. The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in science education 48 (2018), 1273–1296.
[29]
Markus Waltl, Benjamin Rainer, Christian Timmerer, and Hermann Hellwagner. 2013. An end-to-end tool chain for Sensory Experience based on MPEG-V. Signal Processing: Image Communication 28, 2 (2013), 136–150. https://doi.org/10.1016/j.image.2012.10.009
[30]
Markus Waltl, Christian Timmerer, Benjamin Rainer, and Hermann Hellwagner. 2012. Sensory effect dataset and test setups. In 2012 Fourth International Workshop on Quality of Multimedia Experience - QoMEX. IEEE, 115–120. https://doi.org/10.1109/QoMEX.2012.6263841
[31]
Matthew D. Zeiler and Rob Fergus. 2013. Visualizing and Understanding Convolutional Networks. CoRR abs/1311.2901 (2013). http://arxiv.org/abs/1311.2901
[32]
Yuhao Zhou, Makarand Tapaswi, and Sanja Fidler. 2018. Now You Shake Me: Towards Automatic 4D Cinema. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7425–7434. https://doi.org/10.1109/CVPR.2018.00775
Index Terms
- Assessing Usefulness, Ease of Use and Recognition Performance of Semi-Automatic Mulsemedia Authoring
Recommendations
Semi-automatic mulsemedia authoring analysis from the user's perspective
MMSys '23: Proceedings of the 14th ACM Multimedia Systems ConferenceMulsemedia (Multiple Sensorial Media) authoring is a complex task that requires the author to scan the media content to identify the moments to activate sensory effects. A novel proposal is to integrate content recognition algorithms into authoring tools ...
Semi-automatic synchronization of sensory effects in mulsemedia authoring tools
WebMedia '19: Proceedings of the 25th Brazillian Symposium on Multimedia and the WebSynchronization of sensory effects with multimedia content is a non-trivial and error-prone task that can discourage authoring of mulsemedia applications. Although there are authoring tools that assist in the specification of sensory effect metadata in ...
Beyond Multimedia Authoring: On the Need for Mulsemedia Authoring Tools
The mulsemedia (Multiple Sensorial Media (MulSeMedia)) concept has been explored to provide users with new sensations using other senses beyond sight and hearing. The demand for producing such applications has motivated various studies in the mulsemedia ...
Comments
Information & Contributors
Information
Published In
EISSN:1551-6865
Table of ContentsCopyright © 2024 Copyright held by the owner/author(s).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Online AM: 26 August 2024
Accepted: 19 August 2024
Revised: 13 July 2024
Received: 31 January 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 59Total Downloads
- Downloads (Last 12 months)59
- Downloads (Last 6 weeks)38
Reflects downloads up to 16 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in