Solar Energy Systems Design Using Immersive Virtual Reality: A Multi-Modal Evaluation Approach

Abstract

As the demand for renewable energy sources continues to increase, solar energy is becoming an increasingly popular option. Therefore, effective training in solar energy systems design and operation is crucial to ensure the successful implementation of solar energy technology. To make this training accessible to a wide range of people from different backgrounds, it is important to develop effective and engaging training methods. Immersive virtual reality (VR) has emerged as a promising tool for enhancing solar energy training and education. In this paper, a unique method is presented to evaluate the effectiveness of an immersive VR experience for solar energy systems design using a multi-modal approach that includes a detailed analysis of user engagement. To gain a detailed analysis of user engagement, the VR experience was segmented into multiple scenes. Moreover, an eye-tracker and wireless wearable sensors were used to accurately measure user engagement and performance in each scene. The results demonstrate that the immersive VR experience was effective in improving users’ understanding of solar energy systems design and their ability to perform complex tasks. Moreover, by using sensors to measure user engagement, specific areas that required improvement were identified and insights for enhancing the design of future VR training experiences for solar energy systems design were provided. This research not only advances VR applications in solar energy education but also offers valuable insights for designing effective and engaging training modules using multi-modal sensory input and real-time user engagement analytics.

1. Introduction

Addressing some of the world’s most pressing environmental challenges demands a shift in the way we design and develop renewable energy systems. Conventional methods rely on experts creating designs on 2D flat screens using outdated CAD models, often at large distances from the actual site or community [1]. This separation between the physical and digital environments often leads to challenges for designers who must adapt real-world site images to the limitations of 2D flat screens. However, research suggests that enabling users to virtually experience the site as if they were physically present can enhance design effectiveness and user understanding [2]. In fact, 3D visualisation tools, with their interactive and representational capabilities can further facilitate higher levels of user engagement, ultimately enhancing learning and understanding [3]. Moreover, teaching people about climate change is important because it helps them understand and solve such problems in their own communities [4,5].

Virtual reality (VR) is one of the most promising 3D visualisation tools in education, training and instructional design, offering users an innovative immersive experience. Several studies have demonstrated the benefits of VR, which can replicate complex scenarios and environments, offering practical learning experience [6]. It has even been called the “learning aid of the 21st century” [7], since it lets students see and interact with abstract ideas in virtual worlds, making them easier to understand. [8]. Risky scenarios can also be simulated in virtual reality, enabling a safe environment for students to learn or practice. For instance, neurosurgical procedures training [6], use a turning lathe [9] and aerial firefighting training scenarios [10]. VR technology enables remote collaboration where users can work together and interact with virtual objects in real-time, regardless of their physical locations [11].

According to Al Farsi et al., despite their overwhelming promise, VR in education faces three main limitations [12]:

• High cost and uncertain effectiveness: VR technology can be expensive and its effectiveness in education is not always guaranteed.
• Reliance on powerful technology: VR experiences require powerful computers, complex software and sophisticated supporting technologies to function properly.
• Instructors’ lack of confidence: Educators may not feel confident using VR due to a lack of training or experience with the technology.

Furthermore, some VR systems are bulky and uncomfortable to wear, which can hinder the immersive experience [13]. Some users may also experience nausea, dizziness, or other symptoms of cyber-sickness when using VR [14,15]. Nevertheless, despite these limitations, research continues to explore the potential of VR in education and training. Numerous VR studies investigate the relationship between user performance and their sense of presence within the virtual environment, particularly in educational and training contexts [16,17]. It is argued that this immersion and sense of presence leads to a deeper understanding of climate change issues [18,19].

The working principle of a VR experience for solar energy systems includes developing immersive VR environments that mimic actual solar energy generation and system design. These VR environments integrate virtual objects of solar energy system components such as solar panels, inverters, batteries and charge controller that closely resemble their real-world counterparts. These features provide users with an immersive and visually authentic experience. The VR environments allow users to navigate the virtual environment and interact with 3D objects, which are the elements of the solar energy system. Many factors should be considered to provide real-time data for the solar energy system, such as solar irradiance, panel orientation and shading effects.

In the literature, several studies have explored using VR for solar energy systems education [20]. For example, VR has been used to create realistic simulations of solar energy systems where users can explore different system configurations [21], see the effect of orientation and shadowing on solar panel performance and display energy generation in real-time [22]. Moreover, Lopez et al. in [23] presented a VR study designed to foster students’ self-directed learning in photovoltaic power plant installation. Similarly, Abichandani et al. in [21] introduced a VR educational system that taught students the fundamentals of photovoltaic (PV) cells, solar modules and various PV array configurations. Furthermore, Alqallaf et al. [14] proposed a VR game-based approach for teaching basic solar energy system design concepts to higher education students. A comprehensive literature review of PV systems education has also been published [24]. However, in all these situations, the evaluation of training has been carried out using subjective self-reported data and questionnaires.

Researchers are increasingly exploring and evaluating this immersive environment to enhance user experience, comprehension of content, decision-making and problem-solving [25]. Maintaining positive psychological states such as motivation and engagement is crucial to prevent boredom and loss of focus after repeated exposure to VR [26]. The latest VR headsets, including HTC Vive and Oculus Rift, deliver high levels of immersion to users [27,28]. This immersion influences the level of presence, which is the sensation of being in the virtual world [29]. Jennett et al. in [30] discussed the three core concepts frequently used to characterise engagement experiences: flow, cognitive absorption and presence.

In recent human-computer interaction studies, measuring user experience (UX) typically relies on self-reported data, questionnaires, and user performance. However, questionnaires as self-assessment methods face two main challenges: the potential for misinterpretation and misunderstanding of the items’ meanings, and the risk of eliciting stereotypical responses [31]. Current research advocates for the integration of physiological measures into immersive virtual reality applications and experiments, as they can significantly complement self-report data when estimating users’ emotions and stress levels [32]. Furthermore, combining both objective and subjective methods leads to more reliable results [31].

Many contemporary theories of emotion view the autonomic nervous system’s (ANS) activity as a significant contributor to emotional responses [33]. Bio-signals, such as electrocardiography, electroencephalography and blood pressure monitoring can provide objective data. Engagement is associated with physiological changes, including increased heart rate, sweating, tensed muscles and rapid breathing [34]. The degree of engagement affects the autonomic nervous system, which in turn influences physiological changes in the body [34]. McNeal et al. in [35] presented a study using galvanic skin response (GSR) to measure students’ engagement levels in an Introductory Environmental Geology Course. Similarly, Lee et al. in [36] used electrodermal activity (EDA) measurements to gauge cognitive engagement in Maker learning activities. Darnell and Krieg used heart rate measurements via wristwatch monitors to assess cognitive engagement among medical school students [37].

Since engagement levels influence the autonomic nervous system and are linked to physiological changes in the body, these responses can be measured through specific biosignals, including those reflecting learner engagement in class. By combining physiological parameters like heart rate, breathing rate, skin conductance and other sensor data, we can gain a comprehensive understanding of users’ emotional state. Therefore, this research aims to demonstrate the effectiveness of combining these evaluation methods to gain a deeper understanding of user behaviour while interacting with our 3D solar energy design tool. By shifting data capture from post-experiment to during the experiment, we aim to gain richer data on physiological and behavioral cues, allowing for a more detailed mapping of emotional states and cognitive load.

2. Eye Tracking in VR

Eye trackers are tools that measure what users are unconsciously focusing on, giving valuable insights for educators [38]. Tiny sensors mounted on a headset track eye movements and where users fix their gaze [39]. Researchers use this data to understand what grabs user attention and how long they focus on certain areas, which helps them design more user-friendly and engaging digital experiences.

Studying eye tracking in VR environments offers new ways to understand user attention and thinking, especially with head-mounted displays (HMDs) that have built-in eye tracking [40,41,42]. Companies such as FOVE and Tobii are developing advanced eye-tracking systems specifically for VR [43,44]. In fact, eye tracking is a well-established method for assessing usability and user experience in various settings [45]. A review by Li et al. confirmed that eye tracking is useful for measuring mental workload, immersion level and user experience in VR [46].

Analysing eye movements is important for evaluating cognitive function since they can reveal a person’s mental state more accurately than other biological signals [47]. Eye tracking measures fixations (where the eye stays still) and saccades (quick eye movements) [48,49]. By studying this data, researchers can gauge user engagement by looking at how many times and for how long users fixate on something [50,51].

Another indicator of user attention is pupil dilation, which is controlled by the autonomic nervous system [52,53,54]. Research has shown connections between pupil dilation, how engaged someone is with a task, and how difficult the task is [55,56,57].

Eye blinking rates are also influenced by how someone is thinking, how engaged they are with a task and how tired they are [58]. Studies have shown a link between blinking rate and task difficulty or engagement [59,60,61].

A theory called the Cognitive Load Theory (CLT) suggests that learning is best when our working memory (the part of our brain that holds onto information for short periods) is not overloaded [62,63,64]. Researchers use eye-tracking data to measure cognitive load and ensure VR experiences are not overwhelming learners [65]. By combining eye tracking with other data, researchers can gain valuable insights into how people learn in VR environments. This information can then be used to improve VR learning experiences and ultimately help people learn more effectively.

3. Approach and Hypotheses

A previous study [66] compared learner engagement levels in a 2D application and a 3D immersive virtual reality application for designing solar energy systems. Learner vital signs were measured using a non-invasive radar sensor and the data were validated with self-reported questionnaires. The study confirmed the hypothesis that a 3D virtual reality application leads to higher engagement levels than a 2D application. Figure 1 illustrates the heart rate of a participant who designed a solar energy system using both 2D and 3D applications. According to the figure, the participant experienced higher levels of heart rate in the 3D experiment, which can be interpreted as increased engagment.

In the present study, our goal was to measure learner engagement while designing a solar energy system in a 3D immersive environment, analysing biofeedback and eye-tracking data. We compared biofeedback reflecting engagement levels within the VR experience across different scenes. Identifying areas of low learner engagement can guide modifications to the application’s design, thereby increasing learner attention and focus. Moreover, our approach to collecting data offers the key advantages of being wireless, wearable and unobtrusive compared to other methods of measuring vital signs and eye-tracking data. By integrating physiological and behavioural data such as heart rate and eye tracking, with an immersive virtual reality environment, we aimed to gain deeper insights into user interaction and perspectives. This integration allowed us to conduct in-depth analyses that reflected users’ cognitive state and engagement with the virtual experience.

To further explore learner engagement and experience within the 3D immersive virtual reality environment, we will focus on specific scenes within the application and analyse eye-tracking and biofeedback data. With this focus, we propose the following three research questions (RQs) and hypotheses (Hs):

• RQ1: How does user engagement differ across various scenes within the 3D immersive virtual reality environment when designing solar energy systems?
  • H1: Users’ engagement levels vary across different scenes in the 3D immersive virtual reality environment, with certain scenes eliciting higher engagement levels than others during the solar energy system design process.
• RQ2: How do eye-tracking data and biofeedback correlate with user experience (UX) and engagement levels in the 3D immersive virtual reality environment?
  • H2: Eye-tracking data and biofeedback effectively reflect users’ UX and engagement levels, with an increased number of fixations and lower blink rates indicating higher engagement in the 3D immersive virtual reality environment.
• RQ3: To what extent do modifications in the 3D immersive virtual reality environment, informed by biofeedback and eye-tracking data, improve user engagement and focus in designing solar energy systems?
  • H3: Modifications to the 3D immersive virtual reality environment, based on biofeedback and eye-tracking data, lead to significant improvements in user engagement and focus in designing solar energy systems.

In accordance with Dale’s cone theory [67], which structures learning experiences, we chose to focus on active and passive learning through hands-on experiences, since these are situated at the base of the cone, as shown from Figure 2. This decision stems from the fact that teaching sustainable energy subjects typically relies on a theoretical approach, which may not provide the most effective learning experience [68].

4. Methodology

As mentioned earlier, traditional evaluation methods such as quizzes, multiple-choice questions and self-reported questionnaires may not always provide the most accurate assessment of VR experiences. Instead, researchers should consider using operational, protocol and behavioural measurements that are combined with neurocognitive methods to evaluate user experience for a more comprehensive evaluation [69,70,71,72]. Operational measurements often assess a learner’s ability to correctly operate equipment or machinery, while protocol measurements evaluate whether the learner adheres to a prescribed process for a specific job task. In contrast, behavioural measurements examine whether the learner exhibits the desired behaviour in a given situation. Given that solar energy systems design involves a set of procedures and best practices (a protocol) that designers must follow, we developed a methodology to evaluate the effectiveness of our VR experience.

To gain a deeper understanding of learner engagement, we segmented the VR experience into three scenes. Our objective was to observe participants as they interacted with the VR experience, identifying the aspects that captured their interest, the elements they grasped quickly, the parts they wanted to explore further and the areas they found challenging. We also aimed to pinpoint unclear rules and mechanics that were not yet fully developed within our VR experience. By analysing participants’ interactions, we sought to determine which mechanics were enjoyable and which ones needed fine-tuning to balance the experience and guide users toward the intended learning objectives at an appropriate pace. Our motivation behind this approach was to gain insights into the design of the VR experience and refine it accordingly. To collect real-time data during this process, we used physiological sensors as an additional evaluation tool.

As previously mentioned, we divided the VR experience into three main scenes, each corresponding to a crucial task in a typical solar energy system design project. These tasks are vital for ensuring the efficiency, functionality and optimal performance of the solar energy system. Here is a summary of the tasks associated with each scene:

Scene 1: Site Selection—Users begin by choosing a location for installing the solar energy system. Accurate site selection is crucial for maximising solar energy production, as it accounts for factors such as available sunlight, local weather conditions and physical constraints.

Scene 2: Power Room—In this scene, users explore the power room, where they can interact with the essential system components such as batteries, inverters and charge controllers. Users can use the VR controllers to grab and install the components on a stand. Familiarising themselves with these components and understanding their roles is essential for designing a functional solar energy system that meets energy production and storage requirements.

Scene 3: Solar Panel Installation—The third scene takes users to the house’s roof, where they can experiment with the arrangement of solar panels on a stand. Users can add, remove and adjust the tilt of the panels, observing the effects of these changes on the solar power output and electricity generation, as displayed by a gauge. This task is critical in the design process, as optimising the solar panel arrangement can significantly impact the system’s overall efficiency and energy production.

Users can move around the VR environment using a teleporting system with the ability to walk steps in the virtual environment. We chose to use this system to minimise motion sickness while moving around the environment.

By incorporating these essential tasks into the VR experience, we wanted to give users hands-on experience and develop a comprehensive understanding of the solar energy systems design process.

In the following sections, we will provide a detailed account of the hardware and software used for developing and evaluating the effectiveness of our VR experience.

4.1. Hardware

Psychophysiological signals were collected using the Shimmer Sensing Kit [73], which featured a sampling rate of 204 Hz for producing an electrocardiogram (ECG). The ECG electrodes were placed on users as shown in Figure 3: Right Arm (RA), Right Leg (RL), Left Arm (LA), Left Leg (LL), and Chest (V1).

The virtual environment was displayed through the HP Reverb G2 Omnicept [74], equipped with an integrated eye-tracking system powered by Tobii. This Head Mounted Display (HMD) offers a field of view of 114 degrees, presenting the scene with a resolution of 2160 × 2160 pixels per eye and a combined resolution of 4320 × 2160 pixels. The headset also features a refresh rate of 90 Hz. Furthermore, the integrated sensors in the headset provide heart rate, cognitive load, and eye-tracking data, enabling the tracking of user engagement and the evaluation of user responses in real time. These data also facilitate a deeper understanding of user performance and inform decision-making regarding the application’s design.

4.2. ECG and Eye-Tracking Data

Heartbeats are decomposed into five main waves: P, Q, R, S and T [75]. The R waves can be used from the electrocardiogram to determine the heart rate in beats per minute (BPM). This wave is a part of the QRS complex, which is the main spike shown in the ECG signals representing Ventricular depolarisation. Figure 4 shows the ECG Waveform and QRS complex, which can be used to calculate the heartbeats from the ECG signals.

The Shimmer software, known as ConsensysPRO, uses an ECG-to-HR algorithm to provide users with access to heart rate data collected from the ECG sensor. ECG signals were collected simultaneously from participants using five disposable electrodes attached to their skin. These data were streamed via Bluetooth to ConsensysPRO and saved as individual CSV files for each participant.

For eye-tracking, cognitive load and heart rate data capture, three custom scripts were developed. These scripts extended the HP Omicept headset’s Software Development Kit (SDK) version 1.14.0 using C# programming language. The scripts subscribed to the relevant information messages provided by the SDK.

These scripts were linked with their corresponding scenes in the VR application and activated upon scene start. Each script recorded eye-tracking data associated with each frame in real-time. This data was stored in separate files for each scene and each participant.

The header information captured from the HP Omicept VR headset can be represented succinctly as the sum of the following components, $\mathcal{D}$:

$$\begin{array}{cc}\mathcal{D}=\hfill & T+{\overrightarrow{G}}_{\mathrm{L}}+{\overrightarrow{P}}_{\mathrm{L}}+D_{\mathrm{L}}+C_{D_{\mathrm{L}}}+O_{\mathrm{L}}+C_{O_{\mathrm{L}}}+{\overrightarrow{G}}_{\mathrm{R}}+{\overrightarrow{P}}_{\mathrm{R}}+D_{\mathrm{R}}+C_{D_{\mathrm{R}}}+O_{\mathrm{R}}+C_{O_{\mathrm{R}}}+\hfill \\ & {\overrightarrow{G}}_{\mathrm{C}}+HR+SDNN+RMSSD+CL+{\sigma }_{CL}+S_{CL}\hfill \end{array}$$

(1)

where each term is defined as follows:

• T: OmniceptTimeMicroSeconds.
• ${\overrightarrow{G}}_{\mathrm{L}},{\overrightarrow{G}}_{\mathrm{R}},{\overrightarrow{G}}_{\mathrm{C}}$: Gaze vectors for the left, right, and combined eyes respectively, with components in $x,y,z$ coordinates.
• ${\overrightarrow{P}}_{\mathrm{L}},{\overrightarrow{P}}_{\mathrm{R}}$: Pupil position vectors for the left and right eyes, respectively, with components in $x,y$ coordinates.
• $D_{\mathrm{L}},D_{\mathrm{R}}$; $C_{D_{\mathrm{L}}},C_{D_{\mathrm{R}}}$: Pupil dilation and its confidence for the left and right eyes.
• $O_{\mathrm{L}},O_{\mathrm{R}}$; $C_{O_{\mathrm{L}}},C_{O_{\mathrm{R}}}$: Eye openness and its confidence for the left and right eyes.
• $HR$: Heart Rate in BPM. $SDNN$ and $RMSSD$: Heart Rate Variability metrics.
• $CL,{\sigma }_{CL},S_{CL}$: Cognitive Load, its standard deviation, and state, respectively.

At this stage, a low-pass filter was applied to the ECG signals to preserve crucial low-frequency components while attenuating high-frequency noise.

Similarly, eye-tracking data were cleaned and pre-processed using Python (3.9). Filtering the eye-tracking data from the VR headset involved using a confidence level of 1, which refers to selecting only the most certain and reliable data for analysis, ensuring the analysed data are free from errors and biases.

As the three scenes in the VR application have varying durations and are task-based, blinking and the number of fixations were normalised using the MinMaxScaler function. This transformation adjusted the values to a range between 0 and 1, facilitating data comparison across different scenes while also eliminating the effects of varying scene durations or individual characteristics.

4.3. Virtual Reality Setup

The experiment was conducted in a VR lab, with participants taking part voluntarily via an ethically approved consent form that was approved by our university. Initially, participants were briefed on the instructions and the purpose of the experiment. Then, they were asked for permission to place the ECG electrodes at the specified positions. Figure 5A shows a participant wearing the ECG electrodes and the VR headset. Omnicept Overlay was used to visualise heart rate and cognitive load data concurrently. Open Broadcaster Software (OBS) version 28.1.2 from OBS studio was used to record and live-stream the VR application, including the overlay screen, as shown in Figure 5B. Prior to starting the VR application, calibration was carried out for each participant to ensure optimal accuracy when performing the eye-tracking measurements, as depicted in Figure 6.

Moreover, the 3D application was developed using the Unity3D game engine [76], version 2020.3.25f1. All the scripts were developed using C# in Visual Studio within the game engine. OpenXR was used in this application to create VR functionality such as grabbing and locomotion. The OpenXR Plugin package was used for implementing all VR-specific features. The 3D objects for the VR application were taken from the Unity assets store. Blender software version 2.93.4 was used for modelling the solar panels’ stand and its handle. As previously mentioned, our application was divided into the three main scenes shown in Figure 7. The VR application development involved overcoming several obstacles which are represented in simulating PV system behaviour in real-time. This step required creating many scripts that ensure all the system components are installed in the power room (the battery, inverter and charge controller), and calculating the amount of electricity based on the number of solar panels and the angle of the stand. To display the electricity generated from the system, we have added a sprite image to show the amount of electricity generated using the fill amount property.

Furthermore, we implemented the VR application on a Lenovo laptop with Windows 11 having a 64-bit operating system and an Intel Core i7 with an NVIDIA GeForce RTX 3070 graphics card.

4.4. Finding the Average Brightness for the VR Scenes

Numerous studies have reported that eye-tracking data, particularly pupil dilation, can be influenced by the luminance of the environment. As a result, we analysed the brightness levels of the three scenes in the VR application. We recorded a 30-s video for each scene and used Python code with the OpenCV video processing library to extract frames from the videos and convert them to grayscale. Subsequently, we calculated the lightness value for each frame by determining the mean pixel value of the grayscale frame, summing up all pixel values, and dividing by the total pixel count. Finally, we computed the average lightness values for the entire video by adding all the lightness values and dividing by the total number of frames.

4.5. Self-Report Questionnaire Design

Participants took part in the project voluntarily via ethically approved consent using anonymous and confidential online self-report questionnaires after experiencing the virtual reality application and collecting the vital data using the headset and the ECG sensor. They were also informed that they were able to withdraw their participation from the project at any time. The questionnaire consists of fourteen questions designed to measure the engagement and immersion level of the participants in general and for each scene. The first question was whether the participants had previous experience with virtual reality. The second question asked participants to rate how easy the application was from 1 to 5, where 1 means ‘Extremely Difficult’ and 5 means ’Extremely Easy’. The third question was about asking the participants if they felt engaged in the virtual environment or not. The fourth question asked participants to select the scene they felt more engaged in, in front of the house, the power room, or the roof. In questions 5, 6 and 7, participants rated their engagement level in each scene from 1 to 5, where 5 is the highest engagement level. Questions 8 to 14 were taken from the unified questionnaire on user experience (UX) in an immersive virtual environment(IVEQ) proposed by [31] related to measuring engagement and immersion sub-scales.

4.6. Participants

A total of 27 students volunteered to take part in our experiment. These participants included 17 males and 10 females, ranging from 25 to 42 years old. All participants were healthy and did not take medication for heart problems or mental diseases.

5. Results

5.1. Self-Reported Questionnaire Results

Participants completed self-report questionnaires that assessed their sense of immersion and engagement in each scene of the application. Out of the 27 users, 12 (44% of the participants) had previous experience with VR. On a scale of 1 to 5, with 1 being extremely difficult and 5 being extremely easy, 23 participants (85% of the population) rated the application as easy to use, selecting scores of 4 and 5. Meanwhile, 4 participants (15%) chose scores of 3 or below.

Regarding engagement in VR, 48% of participants strongly agreed that they felt engaged, 44% agreed, 4% were neutral, and 4% strongly disagreed. When asked which scene they felt most engaged in, 4% chose the front of the house scene, 30% chose the power room scene, and 67% chose the house roof scene. Figure 8 illustrates participants’ responses when rating their engagement level during the three scenes.

When asked if the visual aspects of the virtual environment engaged them, 44% of participants rated the engagement as extremely high (5), 41% chose 4, and 15% chose 3. In terms of feeling compelled or motivated to move around inside the virtual environment and complete the application, 41% of participants rated this aspect as extremely high (5). Furthermore, 59% of participants rated their involvement in the virtual environment as extremely high (5).

Regarding stimulation from the virtual environment, 52% of participants selected an extremely high rating (5), while only one participant (4%) chose a neutral rating (3). As for becoming so involved in the virtual environment that they were unaware of things happening around them, 33% rated this aspect as extremely high (5) and 37% rated it as 4. When asked if they felt physically present in the virtual environment, 30% rated this aspect as 5 and 52% rated it as 4. Finally, 33% rated their involvement in the virtual environment as extremely high (5) when it came to losing track of time, while 37% rated it as 4.

5.2. Vital Signs and Eye Tracking Data

Based on the literature, we analysed the data representing users’ engagement level, such as heart rate, cognitive load, blinking rate, pupil dilation and the number of fixations. Headset data was subsequently analysed to provide real-time insights.

5.2.1. Heart Rate

After analysing the heart rate data from the ECG sensor and the sensor from the HP headset, we found that the accuracy for the HP headset was 86.72% compared with the ECG sensor. Therefore, we decided to rely on the ECG data for analysing the heartbeat signals. Interestingly, the mean heart rate across the three scenes was nearly identical (see Figure 9). This suggests that heart rate data may not be a significant factor in measuring engagement in a VR environment.

5.2.2. Cognitive Load

As anticipated, our results show that the highest cognitive load was observed in Scene 1 due to the amount of text included in this scene. Users in the first scene had to read and comprehend the role of each component in the solar energy system. This may indicate that the cognitive demands of processing textual information, particularly when learning new concepts, are higher compared to the other tasks.

Scene 2 exhibited the lowest cognitive load values, as the task involved grabbing and placing 3D objects, the system components, on a stand. This suggests that the task in Scene 2 was more intuitive and relied primarily on users’ motor skills, thus requiring less cognitive effort.

Scene 3 was intermediate, as it combined interactions with 3D objects and observation of the results generated by the system. This could be interpreted as an indication that users were engaged in both cognitive processing and motor skills, balancing the overall cognitive load.

Figure 10 displays the box plot of cognitive load data across the three scenes. These findings can inform future refinements of the VR experience by optimising the amount and presentation of information in each scene, balancing cognitive load and ensuring that users remain engaged throughout the experience.

5.2.3. Pupil Dilation

We examined the changes in pupil dilation among participants across the three scenes. Our findings show the greatest pupil dilation for participants in Scene 2, the power room. As previously mentioned, pupil size tends to decrease as the brightness of the visual environment increases. Figure 11 illustrates the noticeable difference in average brightness in Scene 2 compared to the other scenes, which led to increased pupil size for participants.

The brightness level for Scenes 1 and 3 is nearly the same, but the pupil size for participants in Scene 3 is larger than in Scene 1. This discrepancy may be attributed to the difference in the tasks’ difficulty and the nature of the environment, as pupil size typically increases with a rise in mental activity and engagement level.

These results suggest that the tasks in Scene 2 and Scene 3 could have been more cognitively engaging or demanding for participants, while Scene 1, despite its textual information, may not have induced the same level of mental effort. Additionally, the variations in brightness between scenes may have influenced pupil dilation, further affecting the interpretation of cognitive load or engagement. Future iterations of the VR experience may benefit from taking these factors into account to optimise the balance between engagement, cognitive load, and visual design.

5.2.4. Blinking Rate

Figure 12 illustrates the normalised blinking rate for participants across the three scenes. We normalised the data for the blinking rate in each scene to account for the differences in duration between them. The highest number of blinks occurred in Scene 2, followed by Scene 3 and Scene 1, respectively.

The differences in blinking rates among the scenes could suggest varying levels of cognitive load, attention, or visual engagement for the participants. A higher blinking rate in Scene 2 might indicate increased cognitive effort, possibly due to the interaction with 3D objects or the lower brightness level in that scene. Meanwhile, the lower blinking rates in Scenes 1 and 3 might be indicative of reduced cognitive load or increased focus on the tasks at hand. It is essential to consider these factors when evaluating user engagement and cognitive load in the VR experience. Further analysis of the relationship between blinking rate and other physiological or behavioural data might offer additional insights into the effectiveness of each scene in promoting learning and engagement.

5.2.5. Number of Fixations

Figure 13 presents the number of fixations for participants during the three scenes. Our findings reveal a significant difference in the number of fixations across the scenes, with Scene 2 having the highest number of fixations.

The higher number of fixations in Scene 2 could be indicative of increased visual attention or cognitive effort, as participants may have been more focused on manipulating and placing the 3D objects in the power room. This could also suggest that Scene 2 was more engaging or required more intricate interactions, drawing the participants’ gaze more frequently to various elements within the scene.

Conversely, the lower number of fixations in Scenes 1 and 3 might imply that participants found these scenes less visually demanding or cognitively challenging. However, it is essential to consider the context of the tasks and the nature of the interactions within each scene when interpreting these findings. A more detailed analysis of the spatial distribution and duration of fixations, alongside other physiological or behavioural data, could offer a deeper understanding of the participants’ engagement, learning, and cognitive load during each scene in the VR experience.

Finally,

Table 1. Summary of the Collected Data Analysis.

	Scene#	Mean	SD
Heart Rate	1	88.15	16.28
	2	88.06	16.07
	3	88.04	13.94
Cognitive Load	1	0.582	0.132
	2	0.513	0.163
	3	0.567	0.135
Pupil Dilation	1	3.260	0.717
	2	4.314	0.929
	3	3.636	0.857
Blinking Rate	1	0.20	0.18
	2	0.36	0.28
	3	0.34	0.20
Number of Fixations	1	0.24	0.20
	2	0.40	0.25
	3	0.38	0.20

provides a summary of the mean and standard deviation values for all parameters across the three scenes in the VR application.

6. Discussion

Human–computer interaction innovations depend heavily on understanding users’ mental states. This study investigated user experience during a 3D immersive virtual environment using an HP Omnicept headset and the ECG sensor on a solar energy systems design task. We investigated whether eye tracking, heart rate and cognitive load data would be associated with increasing engagement and cognitive level in each scene of the VR application. We also formulated three RQs and hypotheses at the start of the experiment around these areas and we will now discuss our findings related to these hypotheses.

We hypothesised that diverse emotional responses within the three scenes would have produced various attention patterns. Results from our study showed that the user experience and engagement level could be estimated by analysing eye-tracking data, as the eyes can reveal much more about a person’s emotions than most people realise and are a slightly more enigmatic indicator of their emotions.

Despite the survey showing greater engagement in a specific scene, there is no difference in the heart rate data. The participants in the three scenes had almost equal heart rate levels, and there was no significant impact from the ECG signals related to the engagement level. The VR game was likely immersive and engaging for all participants during the three scenes. Our finding with the ECG sensor contradicts what was shown by the study of Murphy and Higgins in [77]. They used ECG and EEG sensors to assess user engagement in an immersive virtual reality environment. Their findings showed that the heart rate is a good indicator for measuring the users’ engagement level in virtual reality.

It was hypothesised that using an approach that increased cognitive load when teaching a topic to students would have an adverse effect on their performance [78]. A task becomes more challenging and places a heavier intrinsic load on working memory when it has a greater number of interconnected informational components. The cognitive load in scene 1 is higher than in scenes 2 and 3. This was to be expected because participants needed to read and understand the role of the solar energy system components, as there is a lot of text in this scene.

Psychologists have long been curious about how changes in pupil size and mental activity relate to one another [79]. Also, mental activities are closely connected with problem difficulty, which affects pupil response [56,80]. We predicted that pupil dilation would be affected by the users’ engagement and performance in the VR application in different scenes. Pupil diameter is a complicated parameter in eye-tracking because it is affected by the brightness of the visual stimulation [81] and cognitive load [82]. Our results showed increased pupil dilation in scene 2, which had a lower average brightness level than in scenes 1 and 3. This was aligned with the literature that the pupil size and brightness of the visual environment are found to be inversely proportional [83]. This finding suggests the importance of considering the effect of visual brightness on pupil size while reliably measuring the user experience in virtual reality applications. Moreover, when comparing scenes 1 and 3, there is no significant difference in the average brightness level between them, but the pupil dilation in scene 3 is larger than in scene 1. This indicated the relationship between pupil dilation and task difficulty [84], increasing the level of interest and arousal [85] and users’ attention [86].

As we mentioned earlier, Scenes 1 and 3 have different physical efforts. In scene 1, users read and understand the role of the solar energy system components by clicking on buttons and demonstrating the information. In scene 3, users interact more with the 3D objects as they grab and place the solar panels and try different situations of the system. This physical activity has an impact on pupil dilation. Our results revealed that the size of the pupil increased in response to physical exertion like one of a previous study indicated that pupil size increased during physical effort [87].

Higher blink rates are frequently observed in insight problem-solving situations and creativity performance [88,89]. However, a previous study observed the relationship between the blinking rate and visual attention [59]. The result of this study indicated that the blinking rate increases when visual attention is engaging and vice versa. Our findings indicated the relationship between the blinking rate and task difficulty. The blinking rate in Scene 1 was lower than in Scenes 2 and 3, as the task in scene 1 was very easy to perform. This aligned with the finding from the study by Tanaka and Yamaoka [61], which shows that the blinking rate with the difficult task was higher than for the easy task. In addition, the blinking rate is affected by the nature of the task. Users experienced different types of tasks during the three scenes, which produced different levels of blinking rates. Also, endogenous blinks are reduced when a task demands more concentration [90], which was clearly obvious with the blinking rate in Scene 1. The amount of text in Scene 1 reduced the blinking rate of the participants as they had to be focused on comprehending the presented information. This finding matches the literature that indicated that the blinking rate reduced during reading [60].

The relationship between blink rate and cognitive load is often found to be inverse. The fundamental hypothesis from earlier studies is that when cognitive load is at its lowest, we blink more frequently because we believe we can blink without missing anything. Moreover, blinking inhibition may be an adaptive strategy that shields delicate cognitive processes from disruption when a mental load is raised [91].

The number of fixations in eye-tracking data can provide insights into several aspects of visual processing, attention, and cognitive engagement. The number of fixations might indicate which parts of a visual environment appeal to the user. A scene with a higher fixation number may show that the user finds that area more visually attractive or interesting. The number of fixations may also be a good indicator of the difficulty of a task or stimuli. As we expect a user moves his eyes to absorb information and make sense of the visual input. Scene 2 had the highest number of fixations which suggests the high level of users’ interest in this scene. Increasing the number of fixations in Scene 3 may suggest that the user spent more time processing and integrating information from various scene areas. We assumed that participants would look more at the solar energy power, the output, of the system they already built simultaneously with trying a new design. This process attracted users’ attention and motivated them to try different design scenarios.

A high number of fixations, which occur when a user revisits the same place or object repeatedly, may indicate a high level of interest. That might, however, also be a sign of understanding issues [92]. It is crucial to know that fixation number may not always give a full understanding of visual processing. Other eye-tracking elements, such as fixation duration, saccades or sensory data, should be considered to interpret the data precisely. It is important to note that eye-tracking research is a complicated field, and analysing data should be conducted in conjunction with other relevant measures.

Our findings indicated the importance of capturing several physiological data for monitoring the user experience. Data from eye tracking can be highly helpful in testing the usability and user experience of any VR game. The capacity of eye tracking to detect variations in involvement during particular tasks enables the researcher to link particular contexts to particular outcomes and demonstrate that engagement was the mediating factor [84]. Researchers can also learn more about the motivations behind users’ responses and behaviours as they engage with photorealistic items, environments, and pretty much any stimuli by submerging research participants in a virtual reality world.

Numerous studies in the literature on solar energy system design have evaluated the effectiveness, usability, and user experience of their VR applications via common assessment techniques. These techniques include gathering quantitative and qualitative data from users about their experience via surveys [23,93,94], pre and post test [95,96] and interviews [97]. Despite the advantages of using these methods in assessing VR experiences and proving their validity in measuring UX [31], they also have certain limitations. As mentioned earlier, these methods rely on self-reported data and subjective interpretations of users’ behaviours and attitudes. This can lead to biases and inaccurate feedback, as many users prefer to provide answers that are socially acceptable or have difficulty expressing how they feel.

Our study provides valuable contributions to the learning technology field and highlights the educational benefits of using virtual reality in learning. Analysing eye-tracking and Electrocardiography data offers insights into the enhancement of instructional design and understanding what keeps engaged to develop educational content more effectively. Also, through monitoring engagement and bio-feedback alongside post-experience assessments, researchers will be able to assess the VR effectiveness in conveying knowledge and skills. Researchers can detect the most effective elements of the VR application that motivate and engage learners. This information can be utilised to develop better strategies to enhance involvement in educational materials. In addition, our findings can inspire further research that combines engagement, multi-modal physiological data and learning outcomes for assessing VR applications that help comprehend the mechanisms of immersive learning.

7. Summary

We summarise our findings and illustrate how our results are aligned with our original research hypotheses (H).

H1: We hypothesised that diverse emotional responses within the three scenes would produce various attention patterns. Results from our study showed that user experience and engagement could be estimated by analysing eye-tracking data, as the eyes can reveal much more about a person’s emotions than most people realise and are a slightly more enigmatic indicator of their emotions. The number of fixations might indicate which parts of a visual environment appeal to the user. Scene 2 had the highest number of fixations, which suggests a high level of user interest in this scene. The high number of fixations in Scene 3 may suggest that users spent more time processing and integrating information from various scene areas.

H2: We predicted that pupil dilation would be affected by user engagement and performance in the VR application in different scenes. Our results showed increased pupil dilation in Scene 2, which had a lower average brightness level than in Scenes 1 and 3. Our findings align with existing literature, which suggests an inverse relationship between pupil size and the brightness of the visual environment. This finding highlights the importance of considering the impact of visual brightness on pupil size when reliably measuring user experience in virtual reality applications. Moreover, when comparing Scenes 1 and 3, there is no significant difference in average brightness between them, but the pupil dilation in Scene 3 is larger than in Scene 1. This indicated the relationship between pupil dilation and task difficulty, potentially leading to heightened user engagement and attention.

H3: We hypothesized that using an approach that increased cognitive load during instruction would negatively impact student performance. Our results confirmed this hypothesis, showing that cognitive load was highest in Scene 1. This was unsurprising, as Scene 1 presented a complex topic with extensive text requiring participants to read and understand various solar energy system components. Furthermore, our findings aligned with previous research by Tanaka and Yamaoka. Blink rate in Scene 1, which involved a simpler task, was lower compared to Scenes 2 and 3. This supports the notion that blink rate increases with task difficulty.

In summary, our VR application fosters higher engagement and intuitive interaction with 3D objects. By incorporating physiological sensors and eye-tracking, it provides valuable insights into user engagement and cognitive load, allowing future optimization of the learning experience. The VR setup’s ability to simulate real-world conditions and provide immediate feedback on design choices accelerates learning and improves accuracy compared to traditional 2D design methods. Furthermore, the application’s user-friendly interface and positive user reception suggest that our VR approach can significantly enhance learning outcomes in solar energy system design education. However, while integrating physiological measures offers a more comprehensive understanding of user engagement, it comes with the added complexity of data analysis and interpretation, requiring specialized tools and expertise. Moreover, although the impact of scene brightness on pupil dilation was explored, further work is clearly required for careful visual design to optimise user engagement and learning outcomes without causing visual strain or misinterpreting cognitive load.

8. Conclusions

This paper highlights how combining multimodal data channels, which include a variety of objective and subjective metrics, can offer insights into a more comprehensive understanding of learner engagement and evaluate user experience. We compared the captured data in different scenes of a VR solar energy systems design task to investigate learner engagement levels. Our findings showed that analysing eye-tracking and vital signs data allows a comprehensive analysis of user engagement and cognitive load measurements for designers and developers to develop applications that meet user expectations. For example, the majority of our participants found the VR application easy to use and felt highly engaged. Most users rated their engagement as extremely high, indicating the application’s effectiveness in capturing user interest and attention. Moreover, cognitive load varied across different scenes, with the highest cognitive load observed in the text-heavy first scene and the lowest in the more interactive second scene. This demonstrates the importance of balancing informational content and interactive elements to maintain user engagement. Moreover, pupil dilation varied significantly across scenes, with the greatest dilation in Scene 2. This finding suggests a correlation between pupil dilation and the level of cognitive engagement, offering a valuable metric for assessing user attention in VR environments. The blinking rate and number of fixations varied among scenes, providing insights into the varying cognitive demands and user focus in different parts of the VR experience.

The impact of these findings is multifaceted, as they contribute to understanding user engagement and experience in immersive virtual environments. By highlighting the importance of multimodal data channels and revealing the limitations of heart rate data in representing user engagement, our paper offers valuable insights for developers, designers and educators. Future research should explore the optimisation of content and interaction balance in VR applications, aiming to maximise user engagement and learning outcomes. Further investigation into the relationship between physiological responses and user experience can provide deeper insights into effective VR design strategies.