Studying Natural User Interfaces for Smart Video Annotation towards Ubiquitous Environments

Multimedia resources are important elements to use in order to augment our communication capacity and achieve better audience attention. Video technologies may present a great potential to support and positively impact learning and education [21, 33]. For traditional open education, MOOCs (Massive Open Online Courses), video-based learning content is one of the most critical materials for distance or online learners. Additionally, videos are an essential teaching resource in the context of a flipped classroom, within a blended learning model, where we have online combined with physical classes.

The use of annotations is an important learning and reviewing strategy. Traditionally, after studying a text, the student writes down her/his thoughts and highlights important content summary for invoking their memory or offering hints for the future. This annotations-based approach helps the student by improving the understanding during the learning process.

Video annotation tools are essential tools for stimulating user creativity in order to add and share information. The interfaces for video annotation are often based on a traditional WIMP interaction [24, 40], which is not a problem when working alone on a video. However, presenting video in a classroom context requires the lecturer to be standing next to the device desk all the time to control the video playback or save comments and reflection about the current scene. Still, usually, the lecturer needs to have some mobility in the classroom between the computer, the projection, and the students to keep good communication. Additionally, during the talk, natural behaviour from the lecturer is essential to get the audience's interest. Unnecessary movements are required to manage the video playback, creating an unnatural pause that can decrease the audience's focus.

Researchers have wanted to talk to computers almost since the first one was invented [17, 22]. In the last few years, smart assistants with speech interfaces became very popular; Apple's Siri [45], Amazon's Alexa [2], and Microsoft's Cortana [14] enable users to accomplish complex operations efficiently, including sending text messages, obtaining navigation information, booking a taxi, among other similar tasks, toward pervasive and ubiquitous environments. Speech interfaces can render interactions much friendlier since they enable users to state their objectives without learning the new software interface. However, video annotation is a complex task to be accomplished only through speech. Previous work has shown that visual tasks are best accomplished by combining speech and manual input (e.g., touch, mouse, keyboard) [12, 18, 48].

Previously, we conducted several test sessions where users could be observed working with a previous version of our video annotation tool. We identified a mix consisting of actions, symbols, sketches, and text used during those sessions. Later, we concluded that such communication presents the potential to benefit from interaction based on both speech and manual input. We intend to improve user interaction in video annotation tasks while using our proposed interface. Our research uses a set of design ideas for video annotation collected during previous observations of annotation practices.

We implemented MotionNotes, a web video annotation tool that enables users to express their desired actions using voice recognition and natural language. This tool was developed in the context of dance annotation analysis, yet different application environments are taking advantage of our video annotation features. A user can ask the player to pause the video in a specific frame, saying “Pause the video” and then say, “I want to add a text annotation”. MotionNotes adds a textbox on top of the video and collects the user's speech. Finally, the speech is converted to text and displayed on the screen inside the textbox.

The natural language module was developed to be applied to the video player and annotation domain; this module can map user sentences onto software operations. The system records the user voice with an HTML 5 audio API that already exists in all modern web browsers. Then, the audio is segmented in multiple audio files and then processed and mapped to predefined intents. MotionNotes processes simple instructions in the browser while more complex commands are sent and handled on the server-side. This strategy was followed since we observed while developing the MotionNotes speech module that a two-level interpretation process provides superior results. First, a more domain-specific method and, then, if success is not achieved, it will try another sub-module with a more extensive vocabulary.

We performed tests with 27 people to understand the real benefit of using a speech module in conjunction with the manual interface in video annotation tasks. Afterward, the tool was also tested in a classroom context, achieving positive feedback. This use case appeared after we informally interviewed several teachers, whose experience told them that our solution could work very well in certain classes. Therefore, after conducting the needs assessment, we intended to investigate if this could be the case, intending to learn how it could work properly in a classroom context towards a ubiquitous computing environment.

This paper presents the following main contributions:

• The Web-based application that combines speech with manual interaction for video playback control and digital annotations.
  
• A user study based on the manual and speech interface, reflecting the challenges and benefits behind this approach.
  
• The knowledge gathered with the use case of the classroom context, where the video annotation software was used with the speech module.
  

The paper is structured as follows. We start by analyzing the related work in Section 2, followed by the MotionNotes description, its speech integration architecture, technology, and features in Sections 3, 4, and 5. Afterward, section 6 presents the user study and the results achieved. Finally, we make our conclusions and present future work in sections 7 and 8.

The use of digital content in lectures increases the audience's interest and concentration by using multimedia elements and design customization applied to the respective topic [47, 50]. Presentation software usually requires the presenter to be next to the terminal while running the application and interacting with it to control the presentation flow. This problem was mitigated by introducing wireless remote controllers, giving the presenter better flexibility regarding his position in the presentation room. Researchers have explored other solutions to address this problem, making it possible to find literature regarding approaches to control presentations with human gestures [36] and voice commands [11].

Regarding practices more focused on video content to create debate and brainstorming, it is not easy to find previous studies flexible enough to give us position freedom around a room while controlling the playback. For instance, pre-recorded videos as a teaching technique are becoming very popular across the education area [13]. Videos can optimize face-to-face time, enabling more collaborative activities that provide greater opportunities for students to interact and consequentially foster creativity. Additionally, video has become more interactive, with software enabling timestamped annotation features, where users can make comments, marks, add ideas for the future, and share them with colleagues [26, 41]. Annotations are often used in various tasks daily. A classic scenario is the one in which students add notes in the classroom, which motivated some research on how video annotation could impact learning activities [19, 35].

In a context of a presentation room where the presenter needs to be moving and interacting with the audience, the interaction with the video player can benefit from multimodal interaction. People interact with their surroundings using multimodal channels. The Human-computer Interaction is performed using these capabilities, providing users, as much as possible, the most natural and productive experience to complete tasks [51]. Speech and natural language interfaces have been studied in multiple application fields, including automated web chatbots [37], smartphones [4], home media systems [27], image editing [25], car interfaces [29], industrial applications [3], among others. However, interaction through natural language interfaces for video players and annotation practices are still rare.

Our system supports multiple command types, including video player operations, multiple annotation types, and speech to text. Previous studies on interfaces with speech recognition and natural language understanding demonstrated good outcomes, especially when the voice was combined with other modalities [34].

Video annotation is a valuable resource in different application areas, which motivated the development of several tools: Elan [54] is one of the best-known annotation tools, with applications in many areas such as language documentation, sign language, and gesture research. Choreographer's Notebook [44] was explicitly designed to be used on Choreography workflow, enabling digital-ink and text annotations. BalOnSe [38] is a web implementation that allows users to annotate classical ballet videos with a hierarchical domain-specific vocabulary and provides an archival system for dance videos. Cabral et al. [5, 6] presented Creation-Tool, a project optimized for tablets and pen-based interaction to create different types of annotations, while Ribeiro et al. [39] have expanded this methodology into 3D visualizations. VideoTraces [49] allows users to capture the video and annotate it by talking or gesturing, and it also enables users to evaluate recorded dance performances and share those evaluations with peers for discussion. Commercial video annotation applications such as Wipster [53], Camtasia [8], Frame.io [15], and Vimeo Pro Review [52] have simplified annotating and sharing videos for users. However, none of them supports speech commands for interacting with the applications.

Researchers have studied how to integrate speech and natural language interfaces into video players. Singh et al. [43] created a Voice-Controlled Media Player capable of recognizing seven different speech commands. Chang et al. [10] developed research that led to voice-based navigation for How-To Videos and presented a set of design recommendations for this kind of interaction. Remap is a multimodal search interface that helps users find video assistance via speech, improving users’ focus on their primary task [16]. Interaction in the Internet of Things field can also be enhanced by using multimodal and speech interfaces, as the Minuet system demonstrated [23]. Still, none of these systems supports annotation practices through voice modules.

Video annotation can be very time-consuming, needing to add different levels of detail to transmit ideas properly. Professional dancers, art directors, teachers, football coaches, and choreographers annotate their work to adjust and improve performance. For the suggested modifications, these instructions must be clear and specific for other people such as actors, dancers, students, and athletes to understand easily.

In order to evaluate the tool's potential and understand the language used in the video annotation procedure, a testing session was performed during two lab days, using a previous version of MotionNotes.

3.1 Participants and Design

The first day was conducted with users who had already experience with other annotation tools, and the second day with amateurs, first-time annotation users. These two sessions have been used to collect important information to help build an initial vocabulary structure to support speech recognition. It was also crucial to observing how casual media player users with little experience in annotation reacted to our prototype.

We invited 19 participants for these lab days (ten with little to no experience). The most representative age interval was between 18 and 24 years old with 66.7%, followed by 25-34 with 18.5% and 35-44 with 14.8%. The gender representation was nearly even, with 53% female and 47% male. Regarding the education level, 37% had a master's degree, 32% had a bachelor's degree, 16% had studied until high school, and the remaining 15% held a PhD degree.

The first task requested them to record and interact with a short video demo (see Figure 1). After several minutes, we collected all the observations on how they thought annotations were needed. Next, we asked the participants to annotate videos using the prototype by following a to-do list of exercises. Those exercises were created so that the users would have contact with each type of annotation and software feature. By the end of the test, each participant was able to accomplish all the requested annotation tasks.

MotionNotes preliminary lab experiment environment.

The design guidelines were preserved from the previous versions due to having been very well accepted (see Figure 2). The menus are on the top, on the left multiple input modalities, with their properties on the right. MotionNotes uses a canvas layer overlaid on top of the displayed video frame to enable users to select and customize the position of any annotation. For example, a user can sketch on top of the video, and MotionNotes saves the ink strokes in the particular region with the specific timestamp. Editing and customizing all annotation types is also possible [32, 42]. The speech interface was integrated into this new version of MotionNotes in a careful way to prevent damaging the look and feel of the previous version.

Annotated video frame with text annotation on top-left and drawn annotation in the bottom center

We developed support for multiple commands to execute the most common tasks, e.g., pause, play or stop the video. Moreover, there was a concern with different word sequences for the same action to give users significant freedom when invoking these commands. The objective was to identify commands from specific sentences and also from more subjective and open-ended ones. One example is asking for the desired item: “MotionNotes, I need a new text annotation in this frame, please”.

Therefore, we identified three critical visual features that would be added to the tool:

• Command engaged feedback. Users must have a visual signal indicating if a given speech command was successfully received and interpreted or not.
  
• A Speech Command List. A considerable number of commands are available and can be activated by multiple sentences. Therefore, a mechanism to help users find and learn this topic was required.
  
• Annotation position. Users should be free to annotate in any part of the video frame; the system must enable the selection of multiple regions. For instance, the spatial areas such as the top right and the center can be indicated using speech. Touch, mouse, and digital pen are always available for a more specific position selection.
  

Speech APIs based on machine learning techniques were integrated into our solution in order to extract meaning from user voice interactions. MotionNotes used the extracted meaning to process and execute operations. A considerable number of external speech APIs were analyzed by our team, resulting in categorizing two types of tools: 1) local speech recognition and 2) cloud-based speech recognition. For MotionNotes, we selected an API from each category, providing quicker performance for simple instructions and superior vocabulary support for more complex ones (see Figure 3).

• Local speech processing - When users turn on the speech interface, MotionNotes instantly activates the system microphone, and all words spoken are processed by the local browser-based machine learning algorithm. The data previously collected was used to train a local neural network. For the simplest instructions, this technique provided satisfactory results with minimal latency. MotionNotes uses tenserflow.js [20, 46] and the ML5 sound classification library [31] for the first layer of speech processing.
  
• Cloud speech processing - Sometimes, the local speech algorithm, which is the first layer, returns results with low confidence levels consisting of classifications with a confidence lower than 50%. MotionNotes immediately sends the recorded audio data to a remote speech API service. This approach is less efficient than the local implementation but provides MotionNotes access to a more powerful speech processing system, increasing the success rates. We used the WIT.AI HTTP API [1, 30].
  

System architecture

The study was designed and divided into two parts, each one to understand a different question. The user study one is about how MotionNotes with a speech module compares to MotionNotes without it, both in user satisfaction and performance. The second user study wants to understand how MotionNotes behaves in a classroom video presentation scenario, collecting feedback from the audience towards the design of mobile and ubiquitous environments.

The motivation behind the chosen design was the necessity to confirm, in the first place, if the system is stable enough when working with voice. Therefore, we organized a first user study where the participants performed activities with the manual interface followed by similar activities with the speech module. During these activities, we collected data manually and automatically about the participant's performance. It is essential to highlight that our intention is not to prove that the speech interface is superior to the manual since our goal is not to replace the manual methods. Instead, our focus is to augment the user's chances in terms of natural interaction to enable annotation in situations where the user needs physical mobility.

The inspiration for the second case study had roots in several factors. First, we selected situations where the manual annotation is challenging for the user. These circumstances where people need mobility between the computer, the projection and the audience to foster good communication include talks, work meetings, lectures, classroom teaching, or creative brainstorming. After evaluating each one of these scenarios, we decided to advance with the classroom case. This decision was supported by informal interviews with several teachers, whose experience told them that our solution could work very well in certain classes. The social circumstances that we lived in the last months also contributed since we had to transform our teaching methods and create materials for online classes. After the first lockdown, we had valuable material, including videos, and we would like to use it to improve our traditional teaching. Therefore, the classroom environment, the videos created to assist online classes, and the MotionNotes testing phase converged and resulted in an excellent opportunity that worked out very well.

The user feedback from MotionNotes testing was generally positive; some interesting ideas were also obtained for further development and research work.

Although our speech module in this current version only supports English, supporting other languages could be a valuable feature to implement in future development. While running tests, we observed that users were sensitive to speech module response times, especially when the voice was sent to our server to be processed. The tool took slightly more time to engage the requested command (∼1 sec.); We are aware this issue is dependent on the user's Internet connection speed. Nonetheless, our server's performance and connection-level developments should continue in order to mitigate this issue. Acceptable outcomes were observed regarding speech module accuracy, but we think there is room for improvement in a future version. Since the WIT.AI platform always adds more vocabulary to suggest new utterances after each test session, our team will validate this new vocabulary to create new templates and connections between the new utterances, entities and the respective intents.

The feedback obtained regarding the collaborative experiment in a classroom was also positive. Nonetheless, a few aspects could improve the overall quality, such as 1) Videos created to be presented in the classroom need a higher zoom than others created to be watched on a regular device. 2) The sound of a regular computer is not enough to present in larger rooms, and then a more powerful sound system is needed. 3) The background noise will always reduce the accuracy of the voice commands so that noise reduction algorithms could benefit this speech module in a future version. 4) In this experiment, only the teacher had a microphone to interact with the system, which is appropriate in a classroom context. A future experiment could integrate a more sophisticated input sound system to enable each participant to add annotations with his voice. All these observations and requirements are important principles for designing a more natural user interaction towards smarter environments according to mobile and ubiquitous computing.

We chose the browser as our platform for MotionNotes due to compatibility, given the known availability on practically every workstation and smartphone. Although we did not restrict which electronic device could be used in our test program, all users selected laptops for the testing session. Exploring this prototype with tablets or smartphones would be interesting to determine if similar results could still be achieved.

Finally, as future research, we plan to widen the analysis in several directions. First, extend the video annotation usage to other courses, talks and collaborative creativity moments. Second, we plan to include the effect of assessment strategies in the study to see the influence on audience engagement. Third, we also want to research how gesture input can be integrated to better support video annotation users, providing them with another option to explore and combine with manual and speech interactions.

The web video annotation tool presented in this paper enables users to execute commands with both speech and manual interaction (e.g., touch, mouse, keyboard). This innovation resulted from several discussions and suggestions obtained during tests and meetings working with a previous version of this tool.

Technical developments enabled implementing the required interaction while taking advantage of novel artificial intelligence techniques on speech processing. The MotionNotes speech module architecture successfully processes voice input and facilitates multimodal interaction.

Video annotation tools to support education are becoming common in multiple learning experiences. Using MotionNotes to present course topics with video positively influences class students’ participation and concentration. The tool enables the speaker to be more effective since it supports control from a distance using a wireless sound input device. Moreover, this study's results highlight the importance and need for technology-supported teaching strategies to achieve better educational creativity and productivity.