Multimodal Technologies and Interaction

48 pages, 5649 KiB

Open AccessArticle

Multimodal Dictionaries for Traditional Craft Education

by Xenophon Zabulis, Nikolaos Partarakis, Valentina Bartalesi, Nicolo Pratelli, Carlo Meghini, Arnaud Dubois, Ines Moreno and Sotiris Manitsaris

Multimodal Technol. Interact. 2024, 8(7), 63; https://doi.org/10.3390/mti8070063 - 18 Jul 2024

Viewed by 597

Abstract

We address the problem of systematizing the authoring of digital dictionaries for craft education from ethnographic studies and recordings. First, we present guidelines for the collection of ethnographic data using digital audio and video and identify terms that are central in the description [...] Read more.

We address the problem of systematizing the authoring of digital dictionaries for craft education from ethnographic studies and recordings. First, we present guidelines for the collection of ethnographic data using digital audio and video and identify terms that are central in the description of crafting actions, products, tools, and materials. Second, we present a classification scheme for craft terms and a way to semantically annotate them, using a multilingual and hierarchical thesaurus, which provides term definitions and a semantic hierarchy of these terms. Third, we link ethnographic resources and open-access data to the identified terms using an online platform for the representation of traditional crafts, associating their definition with illustrations, examples of use, and 3D models. We validate the efficacy of the approach by creating multimedia vocabularies for an online eLearning platform for introductory courses to nine traditional crafts. Full article

► Show Figures

Figure 1

15 pages, 13291 KiB

Open AccessArticle

3D Hand Motion Generation for VR Interactions Using a Haptic Data Glove

by Sang-Woo Seo, Woo-Sug Jung and Yejin Kim

Multimodal Technol. Interact. 2024, 8(7), 62; https://doi.org/10.3390/mti8070062 - 15 Jul 2024

Viewed by 705

Abstract

Recently, VR-based training applications have become popular and promising, as they can simulate real-world situations in a safe, repeatable, and cost-effective way. For immersive simulations, various input devices have been designed and proposed to increase the effectiveness of training. In this study, we [...] Read more.

Recently, VR-based training applications have become popular and promising, as they can simulate real-world situations in a safe, repeatable, and cost-effective way. For immersive simulations, various input devices have been designed and proposed to increase the effectiveness of training. In this study, we developed a novel device that generates 3D hand motion data and provides haptic force feedback for VR interactions. The proposed device can track 3D hand positions using a combination of the global position estimation of ultrasonic sensors and the hand pose estimation of inertial sensors in real time. For haptic feedback, shape–memory alloy (SMA) actuators were designed to provide kinesthetic forces and an efficient power control without an overheat problem. Our device improves upon the shortcomings of existing commercial devices in tracking and haptic capabilities such that it can track global 3D positions and estimate hand poses in a VR space without using an external suit or tracker. For better flexibility in handling and feeling physical objects compared to exoskeleton-based devices, we introduced an SMA-based actuator to control haptic forces. Overall, our device was designed and implemented as a lighter and less bulky glove which provides comparable accuracy and performance in generating 3D hand motion data for a VR training application (i.e., the use of a fire extinguisher), as demonstrated in the experimental results. Full article

(This article belongs to the Special Issue 3D User Interfaces and Virtual Reality)

► Show Figures

Figure 1

15 pages, 1793 KiB

Open AccessArticle

The Optimization of Numerical Algorithm Parameters with a Genetic Algorithm to Animate Letters of the Sign Alphabet

by Sergio Hernandez-Mendez, Carlos Hernández-Mejía, Delia Torres-Muñoz and Carolina Maldonado-Mendez

Multimodal Technol. Interact. 2024, 8(7), 61; https://doi.org/10.3390/mti8070061 - 10 Jul 2024

Viewed by 589

Abstract

At present, the development of animation-based works for human–computer interaction applications has increased. To generate animations, actions are pre-recorded and animation flows are configured. In this research, from two images of letters of the sign language alphabet, intermediate frames were generated using a [...] Read more.

At present, the development of animation-based works for human–computer interaction applications has increased. To generate animations, actions are pre-recorded and animation flows are configured. In this research, from two images of letters of the sign language alphabet, intermediate frames were generated using a numerical traced algorithm based on homotopy. The parameters of a homotopy curve were optimized with a genetic algorithm to generate intermediate frames. In the experiments performed, sequences where a person executes pairs of letters in sign language were recorded and animations of the same pairs of letters were generated with the proposed method. Subsequently, the similarity of the real sequences to the animations was measured using Dynamic Time Wrapping. The results obtained show that the images obtained are consistent with their execution by a person. Animation files between sign pairs were created from sign images, with each file weighing an average of 18.3 KB. By having sequences between pairs of letters it is possible to animate words and sentences. The animations generated by this homotopy-based animation method optimized with a genetic algorithm can be used in various deaf interaction applications to provide assistance. From several pairs of letters a file base was generated using the animations between pairs of letters; with these files you can create animations of words and sentences. Full article

► Show Figures

Figure 1

11 pages, 2943 KiB

Open AccessArticle

A Virtual Reality Direct-Manipulation Tool for Posing and Animation of Digital Human Bodies: An Evaluation of Creativity Support

by Youssef Benbelkheir, Alvaro Lerga and Oscar Ardaiz

Multimodal Technol. Interact. 2024, 8(7), 60; https://doi.org/10.3390/mti8070060 - 10 Jul 2024

Viewed by 620

Abstract

Creating body poses and animations is a critical task for digital content creators, movement artists, and sports professionals. Traditional desktop-based tools for generating 3D poses and animations often lack intuitiveness and are challenging to master. Virtual reality (VR) offers a solution through more [...] Read more.

Creating body poses and animations is a critical task for digital content creators, movement artists, and sports professionals. Traditional desktop-based tools for generating 3D poses and animations often lack intuitiveness and are challenging to master. Virtual reality (VR) offers a solution through more intuitive direct-manipulation capabilities. We designed and implemented a VR tool that enables direct manipulation of virtual body parts with inverse kinematics. This tool allows users to pose and animate virtual bodies with one- or two-handed manipulations, while also moving, including bending, jumping, or walking. Our user study demonstrated that participants could produce creative poses and animations using this tool, which we evaluated for creativity support across six factors. Additionally, we discuss further opportunities to enhance creativity support. Full article

(This article belongs to the Special Issue 3D User Interfaces and Virtual Reality—2nd Edition)

► Show Figures

Figure 1

33 pages, 3660 KiB

Open AccessArticle

Mapping or no Mapping: The Influence of Controller Interaction Design in an Immersive Virtual Reality Tutorial in Two Different Age Groups

by Andreas Urech, Pascal Valentin Meier, Stephan Gut, Pascal Duchene and Oliver Christ

Multimodal Technol. Interact. 2024, 8(7), 59; https://doi.org/10.3390/mti8070059 - 9 Jul 2024

Viewed by 561

Abstract

Navigation and interaction in the virtual world will occur via controllers. Previous literature suggests that not all buttons on these controllers are suitable for all functions and that unclear or uncomfortable operations can lead to frustration. This study aimed to determine if an [...] Read more.

Navigation and interaction in the virtual world will occur via controllers. Previous literature suggests that not all buttons on these controllers are suitable for all functions and that unclear or uncomfortable operations can lead to frustration. This study aimed to determine if an operation with freely selectable buttons differs from one with mapped buttons, in which human-factor differences can be identified, and if there are differences between primary and advanced training. A field experiment with professionals employed by Swiss Federal Railways and apprentices and future apprentices (N = 60) was conducted in a VR tutorial that was previously developed using design cues from existing literature. Controller operation was varied in the groups. The results show significant differences and interaction effects, which indicate that a different operation is more appropriate for apprentices than for professionals in further education. Full article

► Show Figures

Figure 1

20 pages, 2884 KiB

Open AccessSystematic Review

Cultural Heritage as a Didactic Resource through Extended Reality: A Systematic Review of the Literature

by Andrea Dordio, Eva Lancho, María José Merchán and Pilar Merchán

Multimodal Technol. Interact. 2024, 8(7), 58; https://doi.org/10.3390/mti8070058 - 5 Jul 2024

Viewed by 848

Abstract

Extended reality, which encompasses virtual, augmented, and mixed reality, emerges as an important didactic resource, offering great potential to bring heritage closer to the educational environment. Heritage education is a challenge for today’s society, contributing to cultural and social development. Therefore, the aim [...] Read more.

Extended reality, which encompasses virtual, augmented, and mixed reality, emerges as an important didactic resource, offering great potential to bring heritage closer to the educational environment. Heritage education is a challenge for today’s society, contributing to cultural and social development. Therefore, the aim of this systematic review is to analyze documents related to heritage in education through extended reality. The search was conducted using Scopus and Web of Sciences databases, allowing for an international search. We excluded any document that was not a scientific article published in a journal. A total of 52 documents were analyzed, all of which were open-access articles. Preliminary results indicate that virtual reality and augmented reality are utilized in education to work with architectural, historical, archaeological, and musical heritage. Extended reality has been applied in all the stages of the education system, and in formal and non-formal settings, thereby demonstrating its versatility. These experiences increase motivation and academic performance in comparison to less innovative methodologies, which is in line with other studies on this topic. In conclusion, extended reality has the potential to make a significant impact on education, particularly when combined with heritage. The creation of multidisciplinary groups will enhance research in the near future. Full article

► Show Figures

Figure 1

15 pages, 1837 KiB

Open AccessArticle

How Blind Individuals Recall Mathematical Expressions in Auditory, Tactile, and Auditory–Tactile Modalities

by Paraskevi Riga and Georgios Kouroupetroglou

Multimodal Technol. Interact. 2024, 8(7), 57; https://doi.org/10.3390/mti8070057 - 2 Jul 2024

Viewed by 618

Abstract

In contrast to sighted students who acquire mathematical expressions (MEs) from their visual sources, blind students must keep MEs in their memory using the Tactile or Auditory Modality. In this work, we rigorously investigate the ability to temporarily retain MEs by blind individuals [...] Read more.

In contrast to sighted students who acquire mathematical expressions (MEs) from their visual sources, blind students must keep MEs in their memory using the Tactile or Auditory Modality. In this work, we rigorously investigate the ability to temporarily retain MEs by blind individuals when they use different input modalities: Auditory, Tactile, and Auditory–Tactile. In the experiments with 16 blind participants, we meticulously measured the users’ capacity for memory retention utilizing ME recall. Based on a robust methodology, our results indicate that the distribution of the recall errors regarding their types (Deletions, Substitutions, Insertions) and math element categories (Structural, Numerical, Identifiers, Operators) are the same across the tested modalities. Deletions are the favored recall error, while operator elements are the hardest to forget. Our findings show a threshold to the cognitive overload of the short-term memory in terms of type and number of elements in an ME, where the recall rapidly decreases. The increase in the number of errors is affected by the increase in complexity; however, it is significantly higher in the Auditory modality than in the other two. Therefore, segmenting a math expression into smaller parts will benefit the ability of the blind reader to retain it in memory while studying. Full article

► Show Figures

Figure 1

27 pages, 354 KiB

Open AccessReview

Harnessing AI and NLP Tools for Innovating Brand Name Generation and Evaluation: A Comprehensive Review

by Marco Lemos, Pedro J. S. Cardoso and João M. F. Rodrigues

Multimodal Technol. Interact. 2024, 8(7), 56; https://doi.org/10.3390/mti8070056 - 1 Jul 2024

Viewed by 749

Abstract

The traditional approach of single-word brand names faces constraints due to trademarks, prompting a shift towards fusing two or more words to craft unique and memorable brands, exemplified by brands such as SalesForce^© or SnapChat^©. Furthermore, brands such as Kodak [...] Read more.

The traditional approach of single-word brand names faces constraints due to trademarks, prompting a shift towards fusing two or more words to craft unique and memorable brands, exemplified by brands such as SalesForce^© or SnapChat^©. Furthermore, brands such as Kodak^©, Xerox^©, Google^©, Häagen-Dazs^©, and Twitter^© have become everyday names although they are not real words, underscoring the importance of brandability in the naming process. However, manual evaluation of the vast number of possible combinations poses challenges. Artificial intelligence (AI), particularly natural language processing (NLP), is emerging as a promising solution to address this complexity. Existing online brand name generators often lack the sophistication to comprehensively analyze meaning, sentiment, and semantics, creating an opportunity for AI-driven models to fill this void. In this context, the present document reviews AI, NLP, and text-to-speech tools that might be useful in innovating the brand name generation and evaluation process. A systematic search on Google Scholar, IEEE Xplore, and ScienceDirect was conducted to identify works that could assist in generating and evaluating brand names. This review explores techniques and datasets used to train AI models as well as strategies for leveraging objective data to validate the brandability of generated names. Emotional and semantic aspects of brand names, which are often overlooked in traditional approaches, are discussed as well. A list with more than 75 pivotal datasets is presented. As a result, this review provides an understanding of the potential applications of AI, NLP, and affective computing in brand name generation and evaluation, offering valuable insights for entrepreneurs and researchers alike. Full article

14 pages, 680 KiB

Open AccessReview

Data Governance in Multimodal Behavioral Research

by Zhehan Jiang, Zhengzhou Zhu and Shucheng Pan

Multimodal Technol. Interact. 2024, 8(7), 55; https://doi.org/10.3390/mti8070055 - 25 Jun 2024

Viewed by 806

Abstract

In the digital era, multimodal behavioral research has emerged as a pivotal discipline, integrating diverse data sources to comprehensively understand human behavior. This paper defines and distinguishes data governance from mere data management within this context, highlighting its centrality in assuring data quality, [...] Read more.

In the digital era, multimodal behavioral research has emerged as a pivotal discipline, integrating diverse data sources to comprehensively understand human behavior. This paper defines and distinguishes data governance from mere data management within this context, highlighting its centrality in assuring data quality, ethical handling, and participant protection. Through a meticulous review of the literature and empirical experience, we identify key implementation strategies and elucidate the benefits and risks of data governance frameworks in multimodal research. A demonstrative case study illustrates the practical applications and challenges, revealing enhanced data reliability and research integrity as tangible outcomes. Our findings underscore the critical need for robust data governance, pointing to future advancements in the field, including the development of adaptive governance frameworks, innovative big data analytics solutions, and user-friendly tools. These enhancements are poised to amplify the utility of multimodal data, propelling behavioral science forward. Full article

► Show Figures

Figure 1

26 pages, 5278 KiB

Open AccessArticle

Emotion-Aware In-Car Feedback: A Comparative Study

by Kevin Fred Mwaita, Rahul Bhaumik, Aftab Ahmed, Adwait Sharma, Antonella De Angeli and Michael Haller

Multimodal Technol. Interact. 2024, 8(7), 54; https://doi.org/10.3390/mti8070054 - 25 Jun 2024

Viewed by 513

Abstract

We investigate personalised feedback mechanisms to help drivers regulate their emotions, aiming to improve road safety. We systematically evaluate driver-preferred feedback modalities and their impact on emotional states. Using unobtrusive vision-based emotion detection and self-labeling, we captured the emotional states and feedback preferences [...] Read more.

We investigate personalised feedback mechanisms to help drivers regulate their emotions, aiming to improve road safety. We systematically evaluate driver-preferred feedback modalities and their impact on emotional states. Using unobtrusive vision-based emotion detection and self-labeling, we captured the emotional states and feedback preferences of 21 participants in a simulated driving environment. Results show that in-car feedback systems effectively influence drivers’ emotional states, with participants reporting positive experiences and varying preferences based on their emotions. We also developed a machine learning classification system using facial marker data to demonstrate the feasibility of our approach for classifying emotional states. Our contributions include design guidelines for tailored feedback systems, a systematic analysis of user reactions across three feedback channels with variations, an emotion classification system, and a dataset with labeled face landmark annotations for future research. Full article

► Show Figures

Figure 1

Journal Menu

Journal Browser

Multimodal Technol. Interact., Volume 8, Issue 7 (July 2024) – 10 articles

Further Information

Guidelines

MDPI Initiatives

Follow MDPI