4.1. Factors Influencing Operation of Control Elements in VR
In this study, we identified, for the first time, the factors that influence the positioning accuracy, time on task, presence, user experience, and mental workload when operating joysticks and rotary controls in VR. Mathematical models were used to calculate how these factors must be designed to enable users to operate the control elements efficiently in VR and to experience a high presence and user experience and a low mental workload. Finally, the VR-adapted control elements were validated by user tests.
Figure 4 and
Figure 6 show the influence of the
angular resolution and the
positioning speed of the control elements on the evaluation parameter time on task and positioning accuracy. The higher the angular resolution or the positioning speed, the less time the test participants required to complete the tasks. However, the test participants with sensitive control elements also made more errors. The 8%/s calculated by the optimization calculation is therefore a compromise between time on task and positioning accuracy.
The diameter of the rotary control had a significant influence on the efficiency dimension of the UEQ and on mental workload. Test participants rated the mental workload higher and the efficiency dimension lower when the rotary control had a large diameter of 110 mm. However, the diameter of the rotary control had no influence on the positioning time or positioning accuracy. Thus, the test participants only had the feeling that the operation of the big rotary control was not as efficient. Objects are often displayed larger in VR for better readability or representation. The results of this study show that, at least for control elements, an enlarged display may not be useful despite good operability, because test participants perceived the operation of the control element as less efficient.
The factor input device had a significant effect on the positioning accuracy. The test participants with the Valve Index controller primarily had problems releasing the control element at the end of the positioning process. The test participants had to remove their fingers from the controller simultaneously to release the control element in VR. If this was unsuccessful and the test participants continued to move their hands, the control element continued to rotate. With the HTC Vive controller, only the trigger button must be released. Problems with the release did not occur with this controller.
Test participants who tested rotary controls with haptic feedback were able to complete the tasks significantly faster than test participants who tested rotary controls without haptic feedback. We expected that haptic feedback would also influence the positioning accuracy. However, the test participants apparently acted more cautiously and thus slower with the rotary controls without haptic feedback in order to make as few errors as possible. The test participants were not informed that the time for the positioning tasks or the number of errors were measured.
The shape of the rotary control had no significant influence on the time on task and positioning accuracy, but it influenced the mental workload. The test participants with the knurled rotary control experienced the lowest mental workload. The highest mental workload was reported with the cylindrical rotary control. Because of the knurling, the rotation of the rotary control was better recognizable than with the other two variants. We assumed that this additional visual feedback reduced the mental workload of the test participants.
Test participants were able to operate the joystick significantly better while
sitting than while standing. Fewer errors were made in the sitting position, and the test participants needed less time for the tasks. In real-life standing workplaces where control elements are operated, forearm supports exist or the operators can rest their arms on a table [
73]. The test participants in this study stood freely in the room. Movements in the upper body consequently had a direct effect on the operation of the control element. In reality, small movements in the upper body or arm have no or only a minimal effect because the fixed mounting and the restoring force of the joystick prevent accidental or unintentional operation. In VR, this force is nonexistent. Further studies should investigate how joysticks can be adapted to operation while standing. We suggest that the sensitivity of the joystick should be further reduced. The results show that the position of the test participant should be considered in usability tests of new prototypes and can influence the usability of the prototype in VR.
The factors of
acoustic feedback and
visualization of the hand influence presence and user experience. Overall, the test participants rated the user experience of the joystick better when it had acoustic feedback and when the hand enclosing the joystick was visualized. Conversely, rotary control was rated better when no hand was visualized. Other studies have described clear benefits of visualizing hands in VR [
74,
75]. In the study by Lougiakis et al. [
75], the effects of virtual hand representation on interaction and embodiment in VR environments were investigated. The test participants in this study had to move an object through a type of obstacle course. Although more errors were made with the hand representation than with the representation of a controller, it was preferred by most users. We assumed that, in our study, the hand covered the rotary control too much and that the participants were better able to detect the rotation of the rotary control without a visualized hand. Another reason for the lower user experience could be the virtual hand posture. In our study, the virtual hand gripped the cylindrical rotary control completely with all fingers (cf.
Figure 3). Another type of grip for rotary controls would be a two-finger grip with thumb and index finger or thumb and middle finger. Whether this type of grip is more suitable for VR and feels natural to the user must be investigated in further studies.
Table 8 shows that the
size of the joystick in VR had a significant effect on the time on task. The test participants who evaluated the joystick with a total height of 32 cm took significantly longer to complete the tasks. We hypothesized that a large joystick suggests a high actuating force and large range of motion, causing the test participants to pull up on the VR joystick more quickly. Participants who evaluated the smallest VR joystick took less time to complete the task than participants who evaluated the large joystick but took more time than participants who evaluated the medium joystick. We suspected that the small joystick, in contrast, suggested a small range of motion and fine motor work, which did not correspond to the actual work task.
The interaction between the factors positioning speed and angular resolution had a significant influence on the perceived presence of the test participants. The angular resolution was varied because we suspected that a linear angular resolution, i.e., the proportional increase in the positioning speed depending on the deflection of the joystick, could lead to a poor positioning accuracy due to the lack of force feedback in the VR. However, the angular resolution had no influence on the positioning accuracy. With the three-stage angular resolution, the angular range of the joystick was divided into three equal sections in which the same positioning speed prevails. If the test participant moved the joystick within the defined section, the positioning speed did not change. It only changed when the joystick was moved from one section to the next section. The three-stage angular resolution in combination with a high positioning speed led to a strong increase in the change in the positioning speed between the sections. We suspected that this behavior felt very unnatural to the test participants, causing presence to be rated very low for this combination. Overall, it could be concluded that a linear angular resolution is suitable in VR, despite the lack of force feedback, because both the three-stage and the five-stage subdivisions did not lead to a reduction in position errors. A linear angular resolution is more in accordance with the user’s expectations and, therefore, also makes sense in VR.
In addition to the control-element-specific factors, we also varied the
VR environment. The test participants performed the trials either in a detailed crane environment or in an empty virtual room. In the literature, the advantage of VR is stated to be the ability to create realistic virtual environments that simultaneously provide a high degree of control over experimental conditions and good ecological validity [
76]. In our study, the test participants rated the user experience of the control elements higher when the control elements were tested in the detailed crane environment (refer to
Figure 2). This could either have been due to the realistic virtual environment or the detailed crane environment increased the novelty effect, in which test participants rate products better due to the novel technology [
77]. A comparative study between VR and reality should further investigate whether the same control elements are similarly evaluated in a real crane environment. If not, this could be an indication of a bias caused by the novelty effect.
Overall, the factor positioning speed had a significant influence on most of the evaluation parameters. Objective parameters such as time on task or positioning accuracy as well as subjective parameters such as presence and user experience depend on the positioning speed of the joystick. Therefore, when evaluating prototypes in VR and simulating operating processes, it should be especially considered that the positioning speed is adapted to the changed modalities in VR. The recommendations for the design of a joystick in VR developed in this work offer a first orientation.
4.2. Optimized Control Elements for Operation in VR
A comparison between the calculated optimal rotary control in VR and the recommendations of Anguelov [
40] and Schmidtke and Jastrzebska-Fraczek [
30] shows that the calculated optimal diameter of 33–45 mm corresponds to the recommendations for real rotary controls (refer to
Table 1). The optimization calculation computed a sensitivity of 24–36 detents per revolution. This is close to the recommended range of Anguelov [
40] with 25–35 detents per revolution. We assume that haptic feedback in the form of vibration pulses per detent can compensate for the missing detent torque.
The handle length of a real joystick as recommended Schmidtke and Jastrzebska-Fraczek [
30] is 75–125 mm. This recommendation only refers to the grip handle and not to the overall length of the joystick. In our study, we varied the overall length including the grip handle. The joystick optimized for VR had an overall length of 240 mm. In the optimized vertical shape, the length of the VR handle was 95 mm and thus corresponds to the recommendations of Schmidtke and Jastrzebska-Fraczek [
30].
The recommended diameter for real joysticks is between 20 and 40 mm, and the actuating angle should be less than 45°. The calculated optimal handle diameter in VR was 43 mm, which is slightly higher than the recommendations. The calculated optimal angle range for joysticks in VR was 10–17°, which is within the recommendations for real joysticks. Overall, the design recommendations for real rotary controls can be transferred to VR if haptic and acoustic feedback types in the form of vibration pulses and clicking sounds are implemented. The recommendations for the design of real joysticks are also transferable to VR; however, the feedback should only be acoustic.
4.3. Limitations
In this study, we used the DOE method and examined the control elements with an fractional factorial I-optimal response surface test design. The test design used had the resolution IV, i.e., the effects of the main factors (A, B, C, etc.) were neither mixed with each other nor with effects of the two-factor interactions (AB, AC, etc.). However, effects of the main factors were mixed with three-factor interactions (ABC, ACD, etc.), as well as effects of the two-factor interactions among themselves. This type of design is used to identify factors that have a significant effect on the evaluation parameters. The mathematical models created and optimization calculations performed in this study provide initial guidance. In a follow-up study, detailed investigations with the significant factors will be conducted with a full factorial design. The important factors will be evaluated on more levels and with a significantly increased number of experiments in order to further increase the accuracy of the mathematical models.
The main field of application of the DOE is chemical and process engineering [
59]. However, the DOE is also being used more and more in the field of human factors [
78]. Compared with chemical and process engineering, signal-to-noise ratios (SNRs) for human test participants must be expected to be higher owing to human variabilities such as age, affinity for technology, and prior VR experience. The SNR presented in
Table 6 and
Table 9 indicates that the tests had high scatter in some cases. In our estimation, these are mainly due to the different prior experiences of the test participants with the VR systems. In addition, the standard deviation of questionnaires in the field of usability and user experience is high [
79].
This also made it difficult to perform confirmatory runs.
Table 7 and
Table 10 show the results of the confirmation runs, in which four additional test participants evaluated the optimized control elements. The theoretical parameter values calculated using the mathematical models for these control elements deviated for both the joystick and rotary control for the dimensions of the User Experience Questionnaire. However, the calculated 95% prediction intervals were partially outside the scale of the questionnaire. The upper limits of the 95% prediction interval were above a value of three for all UEQ dimensions, and thus above the scale. In addition, it could be assumed that some test participants tended towards the middle [
80,
81]. Consequently, extreme values of the questionnaire were selected less frequently. However, during the optimization calculations, the design of the control elements was calculated to precisely achieve these extreme values. Therefore, the mathematical model for the mental workload for operating the rotary control could not be confirmed. According to the optimization calculation, the optimized rotary control should achieve a score of zero. However, this score is unlikely in practice. The optimized rotary control achieved a score of 2.63 in the confirmation runs. This corresponded to a low mental workload.
Overall, the experiments showed that although the response behavior of the test participants could not be precisely modeled mathematically, the models were sufficient to calculate suitable control elements that achieved good results on the questionnaires. Comparatively high values were achieved for the UEQ and SUSQ. The positioning times (time on task) and positioning accuracy were even better than those predicted by the mathematical models. Generally, these human performance data, such as positioning accuracy or the time on task, can be better represented mathematically than factors for perception, such as mental workload.