The Role of Audio Feedback and Gamification Elements for Remote Boom Operation

Abstract

Remote operations have been greatly enhanced by advancements in technology, enabling remote control of machinery in hazardous environments. However, it is still a challenge to design remote control interfaces and provide feedback in a way that would enhance situational awareness without negatively affecting cognitive load. This study investigates how different audio feedback designs can support remote boom operation and, additionally, explores the potential impact of gamification elements on operator performance and motivation. Due to COVID-19 restrictions, this study was conducted remotely with 16 participants using a simulated environment featuring a virtual excavator. Participants performed digging tasks using two audio feedback designs: frequency-modulated beeping and realistic spatialized steam sounds. The findings indicate that both audio designs are beneficial for remote boom operations: the beeping sound was perceived as more comfortable and efficient in determining the proximity of a hidden object and helped in avoiding collisions, whereas spatial sounds enhanced the sense of presence. Therefore, we suggest combining both audio designs for optimal performance and emphasize the importance of customizable feedback in remote operations. This study also revealed that gamification elements could both positively and negatively affect performance and motivation, highlighting the need for careful design tailored to specific task requirements.

1. Introduction

Developments in networking, sensor technology, and artificial intelligence have enabled the rapid development of remote operations in various domains. The benefits of remote operation, also referred to as teleoperations, are evident, as it allows the operation of vehicles in environments that can be hazardous or inaccessible to humans [1,2]. Remote operations remove geographical obstacles and allow operators to work from any location with sufficient networking capabilities. In addition, remote operations can also improve aspects like work safety, labor organization, and ergonomics [3].

The task of remote operators is often more complex due to the absence of contextual information about the environment of operation. The operators usually have to monitor several camera feeds and displays simultaneously, while often lacking the actual feedback from the environment, realistic feel of operating, and other important features, such as noise from the engine [1]. In the context of teleoperations, operators have limited situational awareness and higher cognitive load [1,4,5,6], which increase the chance of misinterpretations and may cause errors that are costly and hard to recover from.

Therefore, obtaining context-sensitive and well-timed feedback during remote operation is crucial for its success, where the human-centered design of remote operation interfaces is a critical factor to consider. In a natural environment, feedback systems are often designed so that they do not overload the operator’s cognitive functions and awareness, but still grab the operator’s attention in scenarios where they must take immediate action. This information is often related to safety, and it is, therefore, crucial to study the most effective ways of conveying this type of information to the operator in a remote location. Whereas multimodal feedback can partly address the challenges of remote operations and better convey missing information [5], it can also cause overstimulation, tiredness, and frustration. Previous studies have demonstrated that audio feedback is a suitable method for alerting the operator and conveying critical safety information [1,6,7]. However, the body of research on the topic is still limited, and there is no shared understanding of how different sound designs may help to maintain situational awareness and the feeling of presence for remote operators [8,9].

To address this challenge and contribute to the topic of excavator remote operations, we designed a study to explore the use of multimodal feedback to support remote boom operations. In particular, we narrowed the evaluations to audio feedback and investigated how different audio designs can influence the work (e.g., performance and behavior) of boom operators. Based on the positive effects of gamification in other domains [10,11,12,13], we also included an additional agenda for our study: we explored how to provide feedback and relevant information in a gamified manner and how gamification elements may affect boom operators’ performance and motivation.

We developed a simulation environment, which consisted of a virtual excavator, with the control layout based on the ISO standards, and a pit full of virtual cement liquid. To simulate a simple digging task, inside the pit, we placed a pipe, which should be avoided while digging and which bursts when touched. Two different audio designs that communicate the proximity of the pipe were tested in the experiment: (1) frequency-modulated beeping, which represents a more traditional approach to audio feedback, and (2) realistic spatialized steam sounds. The simulation included gamification elements, such as a point system, leaderboard, timer, counters, and game-like sounds, to represent information about the task progress.

Our overall goal was to investigate whether, and how, audio feedback together with gamification elements can be used to support effective, yet careful, excavator boom operation. The exact research questions were as follows:

1

How can remote boom operators’ performance be supported through different audio feedback designs?

2

How may gamification elements influence the remote operators’ performance and motivation?

In total, 16 participants took part in the remote and system-moderated experiment and experienced both sound design conditions in a counter-balanced order. Our findings indicate that both sound designs are beneficial for the remote boom operators to locate a hidden object. The frequency-modulated beeping sound was found to be more comfortable and efficient in determining the pipe location, while the spatial sound was found to increase the feel of presence. Further, the insights on gamification elements demonstrated that they might have both negative and positive effects on the remote task and should be designed based on the needs of the task.

The main strength of our work is the detailed exploration of auditory feedback and how the design of it may influence the remote work in question. The research present in the field typically demonstrates the effect of single audio feedback in comparison to its absence or explores the effects of multimodality, comparing combinations of different forms of feedback and how they might affect efficiency, accuracy, and other performance metrics. Our work dives deeper and demonstrates that the design of the feedback is as important as its presence, which emphasizes the need to verify and test various designs before the actual implementation of multimodal remote interfaces.

This article is structured as follows: Section 2 provides the overview of the existing body of research in three subsections, discussing remote operations and related challenges (Section 2.1), followed by the description of how multimodal feedback and, in particular, the auditory modality can address these (Section 2.2), and finishing with the description of gamification in the context of remote operations (Section 2.3). Section 3 provides a detailed overview of the experiment set-up, including the simulator description, system architecture, gamification elements we have designed for it (Section 3.1), and study design specifics, such as the procedure, experimental task, participants, and data collection and analysis methods (Section 3.2). Section 4 presents the results of this study, including the overall user experience and performance (Section 4.1), the results of the sound design comparison (Section 4.2), and gamification-related results (Section 4.3). Section 5 is the Discussion section, which answers the stated research questions and outlines the limitations and future work directions. Section 6 concludes the work and provides a brief summary of the article.

2. Related Work

As remotely operated systems are becoming more easily available, research on their usability is of great importance. Remote operation (or teleoperation) has been studied in various contexts [2,14,15,16], and in this section, we outline some of this research. This includes the early work and state-of-the-art solutions of remote operation, and its different use cases, opportunities, and challenges. We also discuss the related work performed in gamification and audio design in the context of remote operation.

2.1. Remote Operations in a Nutshell

Remote operation refers to the operation of a machine from a distance. Remote control of a vehicle is not a new phenomenon by any means, as the first experiments with remote driving date back to the year 1925 [14]. Since these experiments, remote operation technologies have made huge technological leaps. This development has been even faster during the 21st century.

There are many domains where remote operations are already used daily, for example in mining [2], agriculture farming [15], unmanned aerial vehicles or UAVs for short [16], and medicine, e.g., in robot-assisted surgery [17]. Another more evident example in our everyday urban setting is the remote driving of electric scooters—these devices are now repositioned by remote operators [18]. The most common environments for remotely operated vehicles are those that are dangerous or hard to reach for humans. These environments can be land sea, air, or space environments.

Generally, remote operation is divided into two categories: remote assistance and remote driving and operating. In remote assistance, the operator only provides route instructions which are then translated by the vehicle’s control system. In remote driving and operating, all actions including the vehicle’s steering, throttle, breaking, and so on are performed by the human operator [19]. Our research focuses on the latter (remotely operating an excavator), which is generally much more demanding in regard to bandwidth, network, and hardware requirements [1].

As laid out by Lee et al. [3], the remote operation system of an excavator consists of three modules: (1) operation and environment, (2) human operator, and (3) an interface between these two, which is used to control the operation and provide information feedback from the environment to the operator. The role of human factors in remote operations is evident, and this is also closely related to the design elements of the system operated. Garcia et al. [20] discussed these human factors in the context of underwater robots, but their categorization can be generalized to other types of remote operations too. This categorization was later supplemented by Kallioniemi et al. [1]. Their categorization for the causes of remote-operation-related human errors is as follows:

1

Lack of knowledge on how to act in the presence of given information. The operators require special training to operate the machinery successfully.

2

The time needed to interpret the received information. Human factors such as tiredness may cause the operator to be unable to interpret the received information or stimuli and act on them on time.

3

Not receiving the information at all. Divided attention or other factors may cause the operator to miss important incoming information.

4

Misinterpretation of the information given. This addition by Kallioniemi et al. [1] refers to the scenarios where the remote operator misinterprets the incoming information and/or stimuli, causing them to act in a suboptimal or erroneous way.

Therefore, the primary safety factor of remote operation is maintaining the operator’s situational awareness of the vehicle and the remote environment where they are operating [1,9]. Limited situational awareness and attention increase the likelihood of human error and potentially may reduce the effectiveness of the mission [4]. Another important factor is the feeling of presence (sometimes referred to as telepresence), which occurs when “at the control station, the operator receives sufficient quantity and quality of sensory feedback to provide a feeling of actual presence” [21]. Both factors can contribute to increased performance and safety of remote operators and are directly influenced by the design of the interface and delivered feedback [10,11]. Lee and Ham [22] interviewed expert excavator operators, who pointed out the main differences between novice and expert operators: novice operators have trouble with leveling (the physical location of the boom) and avoidance of utility strikes (e.g., striking a water pipe), and while novice operators tend only to consider the movement of the excavator, experts also consider the surrounding situation (e.g., soil, nearby workers, and equipment). Based on these interviews, operators of all skill levels mostly rely on visual feedback. Auditory information is used for alarm sounds and often signals about parts that are not visible to the operator, and tactile information is commonly used only for identifying object/soil hardiness. Although multiple studies [1,9,14] investigated the use of visual, haptic, and auditive-based interfaces and combinations of them in the context of remote operations, it is still not clear how to aid operators’ performance with the help of multimodal feedback.

2.2. Audtive Feedback and Audio Design in Remote Operations

The body of research has demonstrated that multimodal and multisensory interfaces are a viable solution for remote operations and remote driving since they help to increase spatial understanding and situational awareness, which in turn contributes to overall performance in complex manipulation tasks [5,23].

One of the most common methods of conveying information to a remote operator is through audio, especially in the bimodal combination of audio and visual feedback, which has been demonstrated and evaluated in numerous research articles [5,6,24]. Audio-based feedback is a powerful method for delivering contextual information beyond visual boundaries [25], and thus, it helps to improve immersive experiences in remote operations. The review article by Chen et al. [4] summarized that audio feedback is useful in supplementing visual modality since it may advance situational awareness and awareness of the physical surroundings, express complex information, reduce the workload, and prevent accidents. The study by de Barros and Linderman [6], which reviewed the use of multisensory feedback in virtual teleoperation environments, claimed that sound feedback helps with being more immersed and focused on the tasks and enhances the understanding of distances between the robot and the virtual environment. The study by Bazilinskyy et al. [7] showed that audio was the most preferred type of feedback when compared with visual or haptic, even though audio was also considered to be the most annoying and urgent type of feedback. This finding was also evident in a study by Kallioniemi et al. [1], which concluded that while it may not be the most pleasant modality, auditive feedback is effective in delivering the most critical and urgent information to the operator. These findings suggest that auditive feedback should be used mainly for conveying critical information that the operator should react to as soon as possible.

Similar findings were reported by Lee et al. [3]—the use of auditory feedback to alarm operators in excavator remote operations is an appropriate way to replace or duplicate visual cues in visually demanding situations. Further, a study by Mavridis et al. [9] demonstrated a smaller variance of subjective difficulty with auditory feedback, although no statistically significant difference was found between the conditions with and without the auditive feedback. Kallioniemi et al. [1] also suggested that the auditory interface can be used to complement visual information in dangerous scenarios. Further, the work by Matthew Dunn [26] supported the previous findings and demonstrated that auditory feedback is valuable in addition to visual representation when it lacks critical details. Further, the experiments with remotely piloted aircraft confirmed that auditory feedback not only improves horizontal accuracy and increases the completion time, but also advances the performance of a remote operator.

Novel research in the field of teleoperations also proposes the use of Extended Reality (XR) to advance situational awareness and operators’ perception during a remote task [27,28,29]. Despite certain issues of implementing XR technologies, e.g., real-time data processing, latency, and synchronization issues, the use of multimodal XR interfaces was predicted to revolutionize teleoperations [29] and enhance the execution of complex remote tasks in terms of safety, efficiency, and accuracy [28]. Auditory feedback, being an essential component of XR interfaces [25], similarly to our case, has been utilized to provide information about the proximity of objects in the environment [27].

Therefore, it is evident that the auditory modality is a suitable information representation channel for advancing the situational awareness of remote operators. However, sound can vary in its technical composition and representation, which potentially influences the way auditory feedback is perceived and understood. Since less attention has been paid to the investigation of sound design effects itself, with this article, we fill in this gap by studying the effects of frequency-modulated beeping sound and realistic spatialized steam sounds in addition to a visual representation of the remote task.

2.3. Gamification Elements in Remote Operations

Gamification, which refers to the “use of game mechanics in a non-gaming context” [30], is a potential technique for aiding remote operators in their tasks. Based on the known positive effects of gamification in educational and professional contexts [20,21], the role of gamification should be investigated in the context of remote operations. The work by Rea [31], for instance, suggests gathering inspiration from video-based games to design and evaluate intuitive interfaces for remote operations. Further, gamification elements, such as leaderboards, timers, and counters, may be used to visualize contextual or task-related information, which in turn might positively affect the operator’s behavior, performance, and motivation [32].

In the field of transportation and driving, gamification has been already applied to increase the awareness of good driving [33], demonstrating that the use of gamified elements encouraged the users to think about their driving behavior [12]. Steinberger et al. [13] evaluated the effects of boredom and gamification on driving behavior and concluded that the addition of gamified elements may increase and sustain attention and arousal throughout the operation of the vehicle. Additionally, gamification is widely applied to advance learning and training processes [32,34]. However, to the authors’ knowledge, the research body investigating the application of gamification to support remote operation and its possible effects on operators’ behavior and performance is lacking. To address this, with our experiment, we have performed a primary exploration of gamification in the context of remote boom operations.

3. Materials and Methods

To study how different audio designs can influence the work of a remote boom operator, we developed an excavator simulator and conducted a within-subject experiment with two audio feedback conditions. This section details the design and functionality of the simulator (Section 3.1) and further describes this study (Section 3.2).

3.1. Excavator Simulator

The simulator consisted of a virtual excavator and a virtual environment with a pit full of artificial cement liquid and a steam pipe in it, as seen in Figure 1. The virtual environment was not designed to replicate common earthmoving tasks exactly, but instead, the artificial cement was used mainly to obscure the view of the pit. This more artificial task had clearer goals (avoiding the pipe in the pit) making it more suitable for audio feedback and easier to gamify. The focus of the experiment was on the basic control of the excavator rather than the details of real-life operations, so the artificial task was deemed adequate for this purpose. The excavator controls were implemented according to the ISO 10,968 standard.

3.1.1. System Architecture

The simulator consisted of a virtual excavator and a virtual environment with a pit full of artificial cement. The rest of the environment was a grid-textured plane free of any distractions. Additional 2D user interface elements were included, such as the hidable transparent overlay about controls seen in Figure 2 and Figure 3.

Technically, the simulator was implemented in the Unity game engine, utilizing the Nvidia Flex plugin to simulate the virtual cement liquid. Resistance forces from the virtual cement were not considered in the physics simulation of the excavator. In addition, the PhysSound physics-based audio system was used to produce sound based on the interactions between the bucket and the cement liquid; this audio was enabled regardless of which audio feedback condition was active. To calculate the necessary mesh intersection information for the physics-based audio, a height map was sampled from the Flex simulation volume, and a triangular mesh was generated based on the height map each time the simulation was updated.

Two forms of audio feedback on the position of the bucket relative to the pipe were implemented. The first design was based on the open-source pxStrax sound engine (https://github.com/pixlpa/Unity-Synth-Experiments/blob/master/Assets/Scripts/SoundEngines/pxStrax.cs (accessed on 27 May 2024)), a dual-oscillator square and triangle wave subtractive synthesizer with distortion, which was used to produce a sharp beeping sound similar to a car parking radar. The second design was a spatialized hissing steam sound based on the Resonance Audio spatial audio SDK, which utilizes a generic head-related transfer function (not customized for each participant). The parameterization of the audio feedback is described in Section 3.2.2. To produce the audio feedback, it was necessary to calculate the shortest distance from the excavator bucket to the nearest point on the pipe surface (its collider mesh). Distances were calculated from five virtual sensors on the excavator bucket; two on the leftmost and rightmost teeth of the bucket, respectively, two on the left and right edge at the bottom of the bucket, and one in the joint where the bucket connects to the arm.

3.1.2. Gamification Elements

To provide relevant information for progression in the task, the simulation included gamification elements, such as a point system, leaderboard, timer, counters, and game-like sounds. The point system gave one point for each stone dug from the pit and one penalty (minus point) for each time the pipe burst due to a hit from the bucket. The points, penalties, and the total accumulated score (sum of points and penalties) were each displayed separately. A voluntary online leaderboard was implemented to display the participants’ final scores if they chose to submit them. The timer was displayed next to the scores (lower right corner in Figure 3).

3.2. Study Design

In the beginning, we aimed to perform a moderated experiment in a laboratory setting. However, due to the sudden COVID-19 restrictions, the experiment was redesigned to be remote. Therefore, the experiment procedure was fully moderated by the system: The simulation guided the users through the tasks, provided instructions, and automatically opened all data collection forms. The consent to participate in this study was also collected by the system. The set-up system, consisting of an LCD display with a resolution of 1920 × 1200, a powerful PC with an Intel i7 CPU, an Nvidia GTX 1080 GPU, and a Microsoft Xbox 360 Wired Controller, was located at one of the university’s office rooms, which allowed the experiment to be performed without the direct involvement of a moderator. All the safety guidelines were followed for the duration of the experiment, e.g., disinfection of devices and the rule of a maximum of 4 participants per day with at least a 2 h break in between.

3.2.1. Procedure of the Experiment

The procedure started with an introduction to the experiment and the collection of consent to participate in this study. Next, the participants watched a demo video of the system, went through a checklist to ensure everything was working as it was supposed to, and started with a thorough training process. The training process included three parts: First, the participants learned how to use the controls. Next, they operated the excavator with the task being to touch highlighted spots with the bucket. Finally, they trained to dig stones out from the pit with and without cement liquid in it. A scene from the training process can be seen in Figure 3.

After the training process, participants performed the actual experiment with a task in two conditions (see Section 3.2.2 for details). After each condition, they filled in a condition-related questionnaire, and the questionnaires were similar for both conditions. At the end of this study, participants filled in a final questionnaire and a background information questionnaire. Once this study was finished, log files were sent to our servers, and the leaderboard was displayed to the participants.

3.2.2. Experimental Task and Conditions

In the experimental task, the participants were advised to dig out as many stones as possible without hitting the steam pipe. The exact task phrasing was as follows: “Your job is to dig out as many stones as you can from the cement in 5 min without hitting the steam pipe”.

The stone was hidden within the virtual cement liquid and was marked with a flag since discovering the location of the stone was not relevant to this study. The purpose of this study was to investigate whether, and how, audio feedback can be used to support effective, yet careful, boom operation. The proximity of the steam pipe was communicated with two types of audio feedback—representing different conditions. Both types of audio feedback were also aimed to produce certain types of boom operation behavior. The audio feedback (i.e., conditions) and the desired boom operation behaviors were as follows:

• Frequency-modulated beeping indicating the pipe, similar to a car parking radar: Just remove the stones from the cement without hitting the pipe, only worrying about the pipe if it is becoming too close. A constant sound frequency of 622.25 Hz (MIDI note 75) was used, and the beeping interval was adjusted from 1 Hz to 8 Hz at 0.7 and 0 m away from the pipe, respectively. Above 0.7 m away, the beeping sound was disabled. No spatialization or panning was used.
• Realistic spatialized steam sounds from the pipe: Removing the stones from the cement by smoothly and carefully operating the boom without hitting the pipe, being aware of the pipe’s position at all times. An audio listener (virtual microphone) was placed in the bucket of the excavator, and virtual audio sources (virtual speakers) were placed in three places along the pipe, at the extremes, and in the center of the pipe, playing a hissing steam sound. The sound sources were slightly directional towards the center of the pit. A reverb probe was placed in the center of the pit. The sound falloff behavior was left to realistic default values in the Resonance Audio SDK. This set-up produces realistic hissing sounds originating from the pipe as if heard from virtual binaural microphones in the bucket.

When a participant successfully digs a stone out, they would hear positive audio feedback and receive +1 point. In case the pipe is touched, the participant hears a bursting sound from the pipe and receives −1 point. The counters, shown in Figure 3 in the bottom right corner, were used to inform the participants about their progress. Once the stone was out, the location of the pipe was changed (in a counter-balanced manner), and the task continued until the timer ran out (5 min in total).

3.2.3. Data Collection Methods

In this study, we collected subjective data from the participants in English with electronic forms (made with LimeSurvey tool). The questionnaires and the collected data content were as follows:

• Condition-related questionnaire—similar for both conditions and filled in after both conditions. The questionnaire inquired about the subjective perception of the sound design and gamification elements with 11 statements answered on a 7-point Likert scale (1 = Totally disagree to 7 = Totally agree) and one open-ended question.
• Final questionnaire. The questionnaire inquired about the subjective overall user experience and experiences about the audio designs and gamification elements with 9 statements answered on a 7-point Likert scale, 2 preference selections, and 4 open-ended questions.
• Background information questionnaire. The questionnaire inquired about participants’ age, gender, and experience with gaming, driving, and heavy machinery.

In addition to the questionnaire data, we collected objective log data from the application, including the number of pipes hit and stones collected, positions and rotations of various parts of the excavator, controller inputs, and condition switch points.

3.2.4. Participants

In total, 16 participants aged from 25 to 59 (M = 35.4, SD = 9.91) performed the experiment and filled in all questionnaires. There were 5 female and 11 male participants. Seven of the participants were researchers, three were students, and others were from varying occupations. Seven participants were from the field of ICT, and others were from varying fields (e.g., education, medicine, and metal industry). English was their mother tongue for only one participant.

All except one participant had a “regular” driving license and had had it on average for 17.7 years (7–41 years). Only 3 participants reported driving a regular car daily and 5 weekly, while the rest drove less frequently. Five of the participants had a truck driving license with an average of 25.4 years of having had it (5–41 years), but they reported driving a vehicle requiring a truck driving license very seldom (less than monthly (1) or never or almost never (4)). All five stated that they never or almost never drive or operate heavy or work machinery (like forklifts or excavators).

As our simulator had game-like features, i.e., the excavator was operated with a game controller and there were gamification elements, we also inquired about participants’ gaming experience and experience with games including driving or operating a vehicle of some kind. How often the participants played on a computer or console games varied quite a lot: two participants played them more than 3 h daily, four a few times a week, six a few times a month, and the rest (4) less frequently or never. Playing games that include driving or operating a vehicle of some kind (e.g., simulators) was rather rare among the participants as two played such games monthly, seven less than monthly, and the rest (7) never or almost never.

4. Results

In this section, we present the results of the performed remote user study, firstly describing the general user experiences and then going into the experiences regarding sound design and gamification for remote operation.

4.1. Overall User Experience and Performance

The digging task seems to have been quite difficult for some participants as the mean number of stones dug out was 2.53 (0–6 stones) in the frequency-modulated beeping sound condition and 2.65 (0–7 stones) in the realistic spatialized steam sound condition. Two participants were not able to dig any stones out of the pit in one of the conditions.

The overall user experience questionnaire results are shown in Figure 4. It demonstrates that despite the majority being engaged in the remote digging operation (Mdn = 5 out of 7), the feeling of actually operating a heavy machine remotely was low (Mdn = 3.5). Furthermore, even though the training process was found to be useful (Mdn = 6), the participants disagreed that they had no difficulties with the controls (Mdn = 2.5). Also, the results demonstrate that it was not clear how to operate the excavator (Mdn = 4). As an outcome, for most of the participants, the task of operating a remote excavator via a game controller was hard, which caused negative experiences and emotions during the experiment.

Participants’ verbal comments are shown in

Table 1. Participants’ comments about controlling the virtual excavator.

Participant ID	Citation
P1 (male, 30)	“The design on its own was interesting and fun. However, the overall experience was one of frustration. This might have to do with me not being familiar with how excavator works but it distracted me from the objectives given to me.”
P2 (female, 38)	P6: “Getting used to the controls was hard as they didn’t feel intuitive at first.”
P3 (female, 30)	P7: “I liked using the game controller, but I think it would take some time to get used to the controls and understand how the excavator should be used to be the most efficient in certain situations."
P4 (male, 34)	P8: “I felt the task was difficult to perform with a game controller. I couldn’t use my eyes to estimate the location because of the foam, and there was no realistic haptic controls or feedback to help with the movements.”
P5 (male, 47)	“Overall, controlling the machine felt quite complex task.”
P6 (female, 30)	“The focus was so much on the complex controls needed to balance in order to perform the task, that the gamification elements were often not noticeable.”
P7 (male 43)	“At the end, I was still not very experienced in control. Moving the “excavator pot” up and down was not intuitive—I’d like to turn it the other way round.”
P8 (male, 32)	“It was quite hard to control the machine with controller and the corresponding buttons. The sound was helping me to avoid but controlling 6 different joints independently was a very tough task.”

. One of the possible reasons for difficulties may be the reversed logic of the controls, which were based on existing ISO standards and would be familiar to expert users with prior experience of operating heavy machinery (e.g., moving the controller up would lower the boom). Still, even a person with extensive experience in operating heavy vehicles found the task to be “quite complex”.

4.2. Sound Design

The log data on the scores (number of pipes hit and stones dug out) were analyzed using the paired samples Wilcoxon signed-rank test. No statistically significant differences were observed between the two audio designs, either in the number of pipes hit or stones dug out. In addition, the mean velocity of controller inputs was calculated in both audio conditions for each participant, and the mean velocities were compared using the paired samples t-test, but no statistically significant differences in controller input velocity were observed between the two audio designs.

The condition-related questionnaires for the two audio designs were analyzed using descriptive statistics and the paired samples Wilcoxon signed-rank test. Statistically significant differences were found concerning four statements. The participants found that it was easier to avoid hitting the pipe with the car parking radar-like sound (Mdn1 = 4.5; Mdn2 = 3), and the Wilcoxon test indicated this difference was statistically significant (W = 57, z = 2.14, p = 0.032). The car parking radar-like beeping sound was also found to be more reliable in determining the pipe location (Mdn1 = 5.5; Mdn2 = 4), showing a statistically significant difference (W = 66, z = 2.14, p = 0.033). Further, the car parking radar-like beeping sound was found more helpful in performing the digging task more accurately (Mdn1 = 5; Mdn2 = 3.5), and the Wilcoxon test indicated this difference was statistically significant (W = 82, z = 2.57, p = 0.01). Finally, the results indicate that the realistic-like spatial sound was found to be more promising for positively influencing the feel of presence for remote operations, since with this sound design, the participants had the feeling that the steam pipe was actually in the pit (Mdn1 = 2.5; Mdn2 = 5); the difference was found to be statistically significant (W = 15, z = 1.97, p = 0.049). The positive and negative differences for the above-mentioned statements can be seen in Figure 5.

When it comes to the subjective preferences for the sound design, the results demonstrated that there was no obvious preference between the two audio designs. However, the majority of the participants found the car parking radar-like beeping sound to be most comfortable (62.5%) and most effective (68.8%). This may be explained by the familiarity of this sound design as well as by the clarity of the meaning in comparison to rather uncommon realistic-like spatial sound, which sometimes was unnoticed by the participants. On the other hand, the beeping sound was found to be distracting by some of the participants. Based on the comment of P5 (who has extensive experience in operating heavy machinery), both sound designs could be efficiently used in combination to provide feedback in different stages of the task. The above-mentioned and other participants’ comments are shown in

Table 2. Participants’ insights on audio designs.

Participant ID	Citation
P9 (male, 38)	“The audio feedback was pretty useful for determining where the pipe was, although I felt the volume and tempo range of the sound could have been wider, or the development more gradual”
P8 (male, 32)	“It was much easier to determine the location of the pipe with radar sounds, as I could scan the ditch as a first action.” “The beeping feedback was good as it was not disturbing me when I was trying to concentrate. But the realistic sound was giving me constant feedback when I really didn’t need. Also, I didn’t know how much close I was from the pipe. The beep sound helped me better as the faster it beeps means I should move in opposite direction.”
P1 (male, 30)	“Especially the beeping noise was frustrating and might’ve actually caused to distract from the objective.”
P5 (male, 47)	“I think I was able to locate the pipe based on the stereo sound quite well. However, it was more difficult for me to figure out the near-far dimension based on the sound.” “I think that the combination of the radar-beep and realistic sound could be efficient sonification for spotting the pipe location. While first hovering above, the natural sound could help in estimating the rough location and detecting the direction of the pipe. Then, while digging the beeps would indicate the proximity.”
P7 (male 43)	about the stereo sound: “This feedback was less intuitive, and the sound was present all the time. It was less clear how far I’m from the pipe”
P10 (female, 32)	“With constant sound it was easier to determine where the pipe was by moving the bucket over the cement, and easier to remain aware of where it lay while operating the excavator.” “… but it was also more difficult to tell when the bucket came close to the pipe—especially when it was about to be too close”
P11 (male, 32)	“The stereo audio helped much more compared to the beeping noise.”
P2 (female, 38)	“Locating the pipe was easier for me with the beeps than the spatial noise.”

.

4.3. Gamification in Remote Operation

The results on gamification demonstrate that for remote operations, the use of gamification elements should be investigated further. The gamification-related statements were also analyzed using descriptive statistics. For the statements from the condition-related questionnaire, we also used the paired samples Wilcoxon signed-rank test. No significance was found. However, the participants’ comments indicate that gamification elements should be carefully selected and tested separately, as they may affect the remote operations in both negative and positive ways—the variety of opinions among the participants in

Table 3. Participants’ insights on gamification elements.

Participant ID	Citation
P9 (male, 38)	“As a novice in the task, I did not find the gamification elements particularly helpful or reassuring. In fact, they added complexity to the already difficult task.”
P7 (male, 50)	“Maybe something more complex could guide operator for more accurate and safer working (like “style points” of going carefully, not hitting outside the area etc).”
P4 (male, 34)	“In general, I think gamification elements could make this kind of repetitive work more fun.”
P1 (male, 30)	“The gamification elements made me nervous and frustrated that might contribute to less accurate and less safe style of work. On the other hand, the time limit and score motivated me to be faster and more efficient.”
P8 (male, 32)	“I do agree gamification elements helped me to work faster, but it did create a panic that the time is running, and I couldn’t finish the task. So at that points it made me to take more risk and care less about hitting the pipe and finish the task. “
P12 (female, 30)	“Gamification elements, especially the timer, promote haste and stress. The pipe-counter does promote being more careful”
P3 (female, 30)	“I’m not a construction worker, but I would be afraid that the gamification elements would make the task less real and might make the operators focus on getting points rather than perform the task with careful accuracy. [...] However, it might make more sense in a training situation where mistakes are not made in real life, just the testing software”
P2 (female, 38)	“At least for me, trying to perform faster decreases the accuracy. Accuracy and safety should maybe be the main points here, but I also understand the need for efficiency. I feel that these simple gamified elements might also backfire if they feel glued on top or are used (or felt like they are used) by employers to get more out of the workforce.”

highlights this.

The participants’ comments, in summary, convey that gamification elements, such as a timer and points, may negatively affect the safety and accuracy of remote operations by making them more stressful and complex. On the other hand, they may affect positively the speed and efficiency of remote operations and make a routine task (of digging stones remotely) more engaging. The qualitative findings are supported by questionnaire statement results, shown in Figure 6.

5. Discussion

While remote operation is becoming more desired in many industries, it is still not clear how to design interfaces in a human-centered manner and how to positively influence the situational awareness of the operator. The auditory modality is seen as an appropriate modality to provide critical information on top of the visual representation of the remote work process [1,3,8,9,26]. The body of research showed numerous examples of how audio feedback positively affects the operator’s situational awareness, while not overloading their cognitive load [1,6]. However, less attention has been paid to the effect of audio design in representing critical information and providing feedback to the operator.

To address these shortcomings, in this study, we investigated the role of two audio designs in the context of remote boom operation: car parking radar-like beeping sound and realistic-like spatial sound. To achieve this study’s goal, we conducted a user study with 16 participants, where they performed digging tasks in an excavator simulator using controllers to manipulate the simulated machine. Due to the pandemic restrictions, the experiment was designed to be remote, whereas the moderation of the experiment was performed automatically via the system, providing all necessary information, guidance, and training to the participants.

Overall, the remote and system-moderated nature of the experiment worked quite well, delivering insightful information about two audio designs. All the participants completed the experiment task, although the performance metrics were somewhat low. One of the reasons for this was the lack of experience and unfamiliarity with operating a boom. Many of the participants found it difficult to operate the simulated boom with a controller, although for most of them, their gaming experience was somewhat high, and using a controller was not a new phenomenon.

Despite these minor challenges, the experiment demonstrated that the auditory modality is indeed a good method for positively influencing situational awareness and providing information about task-specific dangers, such as a hidden pipe. As follows, this article confirms and provides an extension of the previous research, which found positive effects of the auditory modality in addition to visual representation [1,3,4,6,7,8,9,26].

Table 4. Summary of previous research with a focus on auditory modality findings.

References	Context of Work	Key Findings on Auditory Modality
Kallioniemi et al. [1]	Remote operation of heavy machinery	Auditory feedback effectively delivers critical information, complementing visual data in dangerous scenarios, and is perceived as urgent.
Lee et al. [3]	Remote operation of excavator	Auditory feedback is an appropriate modality for alarming and may replace or duplicate visual cues in visually demanding situations.
Chen et al. [4]	Remotely operated robots	Audio feedback supplements visual modality, advancing situational awareness, conveying complex information, and decreasing operator workload.
Triantafyllidis et al. [5]	Multimodal interfaces in remotely operated task	No significant impact of auditory stimulation was found, except for a minor improvement in spatial accuracy of less than 5%.
de Barros and Linderman [6]	Multisensory displays in remote operations	Auditory feedback enhances task immersion, focus, and understanding of distances.
Bazilinskyy et al. [7,8]	Automated driving	Auditory modality is the most preferred type of feedback for take-over requests, despite being perceived as urgent and annoying.
Mavridis et al. [9]	Training for remotely operated robots	Auditory feedback reduces subjective difficulty variance, though no statistically significant difference in performance was observed.
Matthew Dunn [26]	Remotely piloted aircraft	Auditory feedback is valuable when visual representation lacks critical details, improving accuracy and performance.
Burova et al. (current article)	Remote operation of excavator	Auditory feedback is a suitable method for conveying critical information about the environment of remote operations (e.g., hidden objects). Moreover, the design of audio feedback and technical specifics of sound also affect the performance metrics of remote operators.

summarizes the positive effects of the auditory modality from the existing research in similar contexts as key findings.

In summary, with this article, we claim not only that auditory feedback enhances situational awareness, alarming remote operators about the contextual details of the environment of operation, but also that the design of the auditory feedback itself influences operator accuracy and the feeling of presence, based on subjective findings. More details are presented in the next section.

5.1. The Effect of Audio Design in Remote Boom Operation

In this section, we address the first research question: How can the remote boom operators’ performance be supported through different audio feedback designs?

The remote experiment demonstrated that there is no significant difference in performance between the two different audio designs when looking at the participants’ objective performance (total score per feedback condition). Therefore, we cannot claim that sound design has a direct effect on the performance of remote boom operation. However, subjective data demonstrated statistically significant differences in how the two audio designs affected their task execution. The parking radar-like beeping sound was seen as more prominent feedback for aiding the boom operation—it was subjectively found to be more comfortable and more efficient by the participants. Further, this sound design made it easier to determine the location of the pipe and better assisted in avoiding hitting the pipe. Finally, the parking radar-like beeping sound helped the participants to perform more accurately. These findings support the claim that the beeping nature of sound with the variation of frequency to determine the distance is a better alternative for communicating information about a hidden object or possible collision with it.

On the other hand, with the spatial sounds, the illusion of a steam pipe in the cement was stronger. Therefore, it can be concluded that both sound designs are beneficial for different stages of the task. Hence, we propose that, in remote operations, the identification of hidden objects can be split into two steps:

(1)

Identify the approximate location, for example, via realistic-like spatial sound;

(2)

Avoid hitting the object with more concrete feedback sound, such as a car parking radar-like beeping sound.

As follows, in the context of boom operation, we suggest implementing the combination of both audio designs, where spatial sound can be used to help identify the presence of a hidden object, such as a pipe, while parking radar-like sound should be used to understand the proximity of it when critically close and assist in avoiding the collision.

Further, the participants’ comments helped to identify specifics of how to use the auditory modality to influence operators’ performance. For instance, audio feedback should not be constant but rather appear to provide context-sensitive information. The volume, frequency, and other sound specifications can be entangled to add more detailed information to the operator via the auditory modality. However, it was also noted that increasing the frequency of sound is suited for determining proximity to the hidden object, while volume is not.

Finally, the experiment demonstrated that audio feedback was perceived differently by different individuals; for some, the beeping sounds were appropriate feedback, and for others, they were distracting. This insight indicates the need for customization of audio feedback in remote operations. It can be claimed that there is no unique sound design to support the remote operators in their tasks, and similarly to Lathan and Tracey [24], we argue that feedback in remote operations should be customizable to an extent where operators should be able to select their preferred sound design for each phase of the remote task. This feature would help to avoid the negative impacts of the specific sound designs and strengthen the human-centric perspective in the remote interfaces.

5.2. Gamification in the Context of Remote Boom Operation

In this section, we answer the second research question: How may gamification elements influence the remote operator’s performance and motivation?

The results of this study did not provide a strong verification that a set of gamification elements as a whole can influence operators’ performance and motivation during remote boom operations. Nevertheless, the subjective insights collected from the participants indicate that gamification elements can deliver both negative and positive effects to the operators.

Most of the participants were inclined toward the idea that gamification elements could make the work process more fun, motivate operators, and affect their speed. However, that would require a more thoughtful and complex design of the gamification implementation. Fewer participants believed that gamification elements can support the accuracy and safety of work. For instance, a timer element can easily create a feeling of nervousness and frustration, influencing the operators to rush and panic to perform the work faster, which in turn would sacrifice in terms of accuracy. A point system may also shift the focus of the operators to obtaining points but not prioritizing accuracy. Therefore, gamification elements in the context of remote boom operation should be further investigated, and obviously, they should be designed with prior user studies, based on the concrete needs of the industry and industrial tasks in question.

5.3. Limitation and Future Work

A major limitation of our study was the inability to recruit participants with real field experience in boom operations, meaning expert participants, who possess domain knowledge. Despite this, we managed to obtain statistically significant results in comparisons of the two audio designs as well as insights into the use of auditory feedback. Regardless, we still believe that expert participants with prior experience of operating heavy machinery are a must to study the aspects of remote operations. Therefore, in our future research activities, we aim to verify the results gained based on expert involvement.

Further, in our study, we lightly covered the topic of gamification elements and managed to draw conclusions about the possible effects of gamification in the context of remote operations. Future studies should look deeper into the application of gamification elements and, for instance, design an experiment to compare the effects of gamification elements, such as points, timers, leaderboards, and others.

In addition, we believe that another viable direction for future research is to include a haptic modality and test unimodal and multimodal feedback options with the goal of positively affecting situational awareness. We also believe that simplification of the boom controls can be performed to reduce the difficulty and negative impacts of task complexity when testing different multimodal feedback options.

6. Conclusions

In this article, we demonstrated utilizing an excavator simulator to investigate the application of audio feedback and gamification elements in the context of remote boom operation. To accomplish our study goal and investigate two audio designs, we conducted a remote and system-moderated experiment with 16 participants. The participants engaged in a comprehensive training session and completed digging tasks with the simulator in two conditions.

Based on the participants’ involvement and collected data, we found that both audio designs, traditional car parking radar-like beeping sound and realistic spatial sound, could be utilized in different stages of boom operations. Both sounds deliver their benefits to the process of remote boom operation: spatial sound increases the feel of presence and can be used to identify the approximate location of a hidden object, while car parking radar-like beeping better helps to locate and avoid hitting the object. Based on the insights from the experiment, we also point out how to use audio design to deliver additional information via the auditory modality and recommend some level of customization for the sounds. The use of gamification elements was also explored, drawing indications for how different elements may influence remote work. However, this study possesses no strong verification of its effects on motivation and performance; therefore, we suggest investigating these further.

In conclusion, our study verifies the power of auditory feedback in the context of remote operations and demonstrates how different audio designs can support it.