Exploring the Impact of Interconnected External Interfaces in Autonomous Vehicles on Pedestrian Safety and Experience

Pedestrians have historically depended on implicit cues from vehicles, such as distance, speed, and deceleration, to determine when it is safe to cross the street [19, 47, 59]. This is still the case with AVs, as demonstrated by several field studies employing the ghost driver method [55, 62]. However, in situations where vehicle movement is ambiguous, like in slow-moving traffic, pedestrians tend to seek explicit cues from drivers. As we transition to a world with a greater presence of AVs, the need for these cues does not disappear; rather, it evolves. The emergence of eHMIs now facilitates effective communication between AVs and pedestrians in their shared environments.

In recent years, there has been a marked increase in the variety and sophistication of eHMI concepts [17, 63]. Initially, eHMIs echoed traditional vehicular signals: brake lights indicating deceleration [54, 57], and marker lamps indicating the activation of an automated driving system [41]. Yet, technology and design advancements brought forth innovative and intuitive eHMI designs. Some of these simulated human behaviours, using lights to mimic eye contact with pedestrians [6, 29], or animations showing a vehicle’s directional intent—akin to a human indicating a direction before walking. Some eHMIs even employed advanced technologies like augmented reality to overlay information on real-world scenes [33, 58, 67, 70] or used dynamic road projections [71] to provide crucial signals or warnings directly.

However, an observation made during the initial stages of eHMI development is that many concepts were evaluated in isolation, often emphasising singular vehicle-pedestrian interactions [9, 68]. While this approach provides a detailed and nuanced understanding of AV-pedestrian interactions, it fails to encompass the broader context of real-world scenarios. A recent scoping review conducted by Tran et al. [69] highlighted seven key scalability issues of eHMIs. In multi-pedestrian environments, the most pressing issue is the Clarity of Recipients, underscoring the necessity for eHMIs to communicate clearly to individual pedestrians in a vehicle’s vicinity [20, 40, 73]. In multi-vehicle environments, which are a focus of this paper, the critical challenges that may arise include:

Information Overload: In urban environments, pedestrians encounter various information sources, from vehicle cues to environmental prompts like billboards and smartphone distractions [38, 46]. With the expected rise in eHMIs, pedestrians may face more diverse interfaces and messages from different AVs [17, 61]. The impact of this increased information and visual clutter on pedestrian safety and traffic efficiency is a growing concern in AV-pedestrian research. Some investigated approaches include determining if AVs need to explicitly communicate at all times [16] or aggregating multiple AV communications using infrastructure (e.g., curbstones [37]) or pedestrian devices (e.g., AR glasses [70] and smartphones [38]).

Safety: The potential liability of instructing pedestrians to cross, as well as the risk of overtrust, where pedestrians might over-rely on eHMIs and potentially neglect other safety cues, has been highlighted [1, 17, 36, 45]. A study by Mahadevan et al. [50] further underscores this point, revealing that some participants, after seeing positive eHMI signals, failed to notice other critical risks like distracted drivers in adjacent lanes. This potential for misjudgment is especially alarming in multi-lane crossings due to the direct negative influence of eHMIs on pedestrian safety.

To address the aforementioned challenges, the domain of eHMIs has begun to overlap with the emerging V2X (Vehicle-to-Everything) communication technologies. In this broader communication network, vehicles can interact with virtually every entity in their vicinity, including pedestrians. Recent studies have presented intriguing V2X integrated solutions. The ‘Smart Curbstones’ concept involves urban infrastructural sensors that communicate safety information to pedestrians through curb-embedded LEDs [37]. Another proposition involves decentralised eHMIs, shifting their functionalities to personal devices [38]. Notably, Tran et al. [70] proposed a system wherein pedestrians, equipped with AR glasses, can communicate and negotiate safe crossing opportunities with oncoming AVs.

The interconnected eHMI system is another V2X-based solution. By networking eHMIs, they can derive and communicate messages based on data from multiple AVs rather than just one. This interconnectedness promises a more coherent and consistent message delivery to pedestrians, potentially eliminating the challenges of information overload and message conflicts seen with traditional eHMI designs. While these systems hold great potential, so far, the only study on them has been by Colley et al. [8]. Their research focused on auditory interconnected eHMIs, designed to assist visually impaired individuals in navigating traffic. To delve deeper into interconnected eHMIs, our study focus on visual-based design concepts, because the majority of the pedestrian population relies heavily on visual cues when navigating traffic [17]. Recognising this predominance of visual reliance is essential for creating a comprehensive eHMI solution.

A key challenge facing interconnected eHMI research is the prevalent lack of awareness about the connected vehicle technology. A study by Colley et al. [8] suggested that gaps in participant knowledge concerning vehicle connectivity could adversely affect their trust. This finding underscores the importance of prior knowledge in shaping pedestrian perceptions of interconnected eHMIs. To address this issue, our study firstly aims to design interconnected eHMIs capable of visualising their connectivity. Secondly, we investigate how prior knowledge of the concept, acquired through pre-instruction, influences user perceptions. We hypothesise that users with foundational understanding of interconnected eHMIs will demonstrate greater trust and more efficient interaction with these systems. This hypothesis aligns with previous research on internal HMIs, which suggests that user instruction can significantly enhance drivers’ understanding, interaction performance, acceptance, and trust [21, 27]. Furthermore, this investigation contributes to the ongoing discourse of eHMIs regarding the necessity for public education [18, 22]. While longitudinal studies related to the learnability and familiarity of eHMIs exist [24, 34], the role of user instruction as a means to bridge initial knowledge gaps remains largely unexplored.

Given the above challenges and gaps identified in the literature, our study addresses two main research questions (RQ):

RQ1: To what extent will interconnected eHMIs affect pedestrian cognitive load, safety, and trust during multi-lane crossings? - This question is motivated by the need to compare the performance of interconnected eHMIs with unconnected eHMIs and baseline scenarios where no eHMI is present.

RQ2: How does prior knowledge about the concept of interconnected eHMIs alter pedestrian cognitive load, safety, and trust during multi-lane crossings? - Given that interconnected eHMIs are a relatively new concept, we are driven to determine if the impact of interconnected eHMIs is contingent on the user’s understanding of the technology.

Informed by existing literature, three distinct visual representations were selected for the unconnected eHMI, each corresponding to a specific state of the AV (refer to Figure 1). When the vehicle is cruising autonomously, a solid light band on its bumper lights up [18, 31, 49, 65]. As the vehicle prepares to yield to pedestrians (i.e., when applying the brakes), this light band transitions to an inward-sweeping pattern. This choice of visualisation was informed by the need for clear differentiation between the actions of giving way and continuing on its path [16]. Upon coming to a complete stop, the vehicle projects a zebra crossing, signalling pedestrians about crossing opportunities. This crossing pattern has garnered significant attention in academic research [20, 37, 70] and is also a feature of the Mercedes Benz F 015 concept [53].

The recommended colour for eHMI is typically a blue-green hue, such as cyan or turquoise, chosen for its perceived neutrality, which is not associated with a specific meaning in traffic communication [3, 18, 23]. As such, the light band is coloured cyan. While Dey et al. [20] and Holländer et al. [37] employed cyan for their zebra crossings, the symbolic representation of a zebra crossing already provides a clear invitation for pedestrians. In this particular context, the distinction between cyan and green is less pronounced. Therefore, we opted for a green crossing pattern, aligning more closely with universal traffic interpretations.

The interconnected eHMI retains many visual cues from its unconnected counterpart, as seen in Figure 1. However, there are several distinct features.

The Red Crosswalk: When the AV comes to a full stop, it projects a red zebra crossing. The red colour is widely recognised for indicating caution and alertness [43, 74]. In Australia, red-painted areas on the roads are used to trigger alertness and capture the attention of drivers when they are approaching, for example, a school zone, speed bumps [2], or township entry [14]. As a result, a combination of red colour and the familiar crosswalk pattern is expected to signal to pedestrians that they can proceed but with increased vigilance. When safety is assured, for instance, when vehicles in adjacent lanes also stop, the crossing transitions from red to green. Thus, the red crosswalk serves a dual function: it helps to mitigate the potential over-reliance of pedestrians on eHMIs for safety decisions and communicates both the intentions of the AV and the broader traffic situation.

Single AV Communication: In designs where eHMIs communicate independently, each vehicle projects its own crossing pattern when stopping, even if they stop sequentially in the same lane. A study by Holländer et al. [37] exemplified such an approach. Our proposed design designates one AV to take charge of the communication and projects the zebra crossing, adopting the ‘omniscient narrator’ concept of the auditory interconnected eHMIs [8]. This centralised approach is intended to minimise visual clutter that could arise from multiple crosswalk projections and aims to provide pedestrians with a single, clear source of information on which to focus their attention. However, understanding that pedestrians might expect to see individual AV signals [8, 70], we retain the light band on each AV. Moreover, we synchronise the inward sweeping animation across all AVs, reinforcing the perception of a connected vehicle fleet.

The Wi-Fi Signal: Given the unfamiliarity and skepticism towards connected vehicles noted by Colley et al. [8], our design integrates an animated Wi-Fi symbol. This universally recognised sign of connectivity aims to bridge knowledge gaps, fostering immediate understanding of interconnected eHMI operations.

We employed a mixed-methods approach using a 2x3 mixed design to explore the influence of knowledge about interconnected eHMIs and differences among three interface conditions, whilst controlling for individual differences.

For the between-subjects factor, participants were randomly allocated to two distinct groups. The knowledge group was provided with information about interconnected eHMIs, whereas the no knowledge group was not given this information. As for the within-subjects factor, all participants underwent three interface conditions: a baseline without eHMI, an unconnected eHMI, and an interconnected eHMI. Each condition was presented in blocks in a counterbalanced order to avoid learning effects.

Within each block, participants experienced three different scenarios. The order of these scenarios was randomised among the conditions, ensuring that the sequence of scenarios did not influence participant responses. In total, each participant completed nine crossing trials. In each scenario, an AV on the nearest lane was programmed to yield to the pedestrian 13 seconds after the start of the VR scenario. However, the behaviour of the vehicles on the furthest lane varied to mitigate potential habituation and ensure that participants could experience the design concepts in various traffic situations:

A VR simulation was selected due to its inherent safety advantages, removing potential risks of direct interactions between pedestrians and AVs in a physical setting [68]. In situations with multiple vehicles on multi-lane roads, this safety aspect is noteworthy, as the possibility for collisions might increase (e.g., one collision recorded in a related study [37]). Moreover, VR simulations provide a cost-effective alternative to real-world experiments. The platform’s tools enable efficient data collection, offering insights into pedestrian crossing behaviours.

Hardware: The Oculus Quest 2, a standalone and untethered VR headset, was employed in this study. Its design allows participants to physically cross a simulated street without being constrained by cables and cords. The headset has a resolution of 1832 x 1920 pixels per eye and an approximate field of view of around 90-95 degrees, depending on factors like the fit of the headset and the user’s interpupillary distance. Participants used a Touch controller to navigate from one scenario to the next. Experiments took place in a space with dimensions of 8 m x 5 m (see Figure 2).

Virtual Environment: The virtual setting was developed in the Unity 3D game engine¹, using readily available assets from the Unity Asset Store. It depicted an unmarked midblock location on a two-way two-lane urban road, with traffic originating from both directions. Each lane measured 3.5 m in width [68]. To provide context for the pedestrian crossing task, a bus stop was situated on the opposite roadside. To enhance the realism of the environment, Mixamo² 3D characters were incorporated to mimic typical human sidewalk activities, such as conversing and exercising. These characters are positioned at a distance to not distract or influence participants’ behaviours [35] (see Figure 3). An urban auditory backdrop, featuring bird chirps and traffic noises, was also integrated.

In the simulation, black vehicles symbolised AVs, devoid of a driver, while white vehicles represented manually operated vehicles with a visible driver. The colour distinction was solely to facilitate easier identification, especially in the Baseline condition where AVs are not equipped with an eHMI. Vehicles were spawned from a location outside the participant’s line of sight, travelling with a maximum speed of 50 km/h. When 50 m away, they decelerated at a rate of 3.5 m/s², coming to a full stop at 3 m from a pedestrian, adhering to recommended stopping distances [28]. Emergency braking was not implemented to ensure that any collision events could be directly attributed to the design concepts rather than automated interventions. In each lane, mixed traffic is generated with a sequence of vehicles: AV, Manual, AV, AV, Manual, AV, and then the sequence repeats. The gap between these vehicles is 2.5 - 6.5 seconds.

Design Elements: To closely mirror real-world implementations of the LED light band, we referenced related field studies [49] and developed a custom 3D model of the light band. The light band was affixed to the lower front of the vehicle and activated when braking at a distance of 50 m. For the zebra crossing, we used an emissive material to give it a glowing effect. Additionally, a point light was placed on the AV bumper to enhance the brightness of the projection and indicate the light’s origin. In Unity, we employed this approach to achieve the appearance of a laser projection. The Wi-Fi signal displays waves radiating outward. Refer to Figure 4 for the VR implementation of these design elements across interface concepts.

Three days prior to the study, all participants received information about the eHMI light band via email. However, participants from the knowledge group received additional information about interconnected eHMIs. The material was refined based on insights from a small-scale pilot study involving four people and can be found in Appendix A. On the day of the study, participants signed a consent form and were briefed about the study. Researchers then confirmed that participants had a good understanding of the previously provided material by asking them to explain the concept. This was followed by a familiarisation session in which participants were introduced to both the VR equipment and the virtual environment.

Within this virtual setting, participants learned two distinct types of stationary vehicles. Following this, they had the opportunity to practise crossing the street within the VR environment at least twice, without the presence of vehicles. To contextualise the task, they were instructed to cross the street to reach the bus stop located on the opposite side.

Before initiating the experiment, participants were reminded to behave as they would under real traffic conditions, meaning they were expected to cross the street only when they deemed it safe to do so. If they found no suitable crossing opportunities, they were advised to inform the researchers to advance to the next scenario. It was clarified that deciding not to cross the street would not impact the outcome of the study in any manner. These instructions were provided to ensure that participants didn’t feel pressured to cross the street merely to fulfil the task.

Participants subsequently underwent three experimental blocks, each comprising three scenarios. Following each block, participants were asked to briefly recall the events and respond to paper questionnaires. Upon completion of all blocks, participants were requested to rank the conditions based on their overall preference and engage in semi-structured interviews. The study lasted between 60 to 90 minutes, depending on the participant’s familiarity with VR. An overview of the procedure can be seen in Figure 5.

We recruited participants from the university’s mailing list, social media networks, and through word of mouth. Our sample consisted of 32 individuals, including 20 males and 12 females. Among them were both working professionals and university students, all of whom were fluent in English. To be eligible, participants needed to have normal vision or use glasses to achieve corrected-to-normal vision. Participants with glasses were permitted to wear them with the headset, and all were given ample time to adjust the headset for comfort. Colour blindness was an exclusion criterion. Those with mobility impairments that might hinder movement within a VR environment were also excluded from participation. Each participant was compensated with a $20 gift card. The study received approval from the university’s Human Ethics Committee (ID 2020/779), with which the authors are affiliated. For a detailed breakdown of the demographics and VR experience of the participants, refer to Table 1.

We employed both subjective (questionnaires) and objective (simulation-logged data) methods for data collection.

Safety: The Head-Mounted Display (HMD) recorded positional data (x, y, and z coordinates) and logged user events at 60 Hz. This data facilitated the collection of metrics related to safety and crossing behaviour. Key metrics included the number of collisions, crossing initiation time, and crossing duration per lane. These metrics have been utilised in related studies to holistically assess pedestrian safety [13, 37]. In the simulation, a collision with the vehicle triggers a fast beep sound, alerting participants to the collision and immediately pausing the scenario, thus minimising potential psychological effects.

Perceived Safety: We assessed perceived safety using a 3-item questionnaire with a 5-point Likert scale (based on Holländer et al. [37]).

Workload: We used NASA Task Load Index [32] to assess workload in six dimensions: mental, physical, temporal demand, performance, effort, and frustration. Each of these dimensions is rated on a scale from 0 (very low) to 100 (very high), used to compute an overall workload score.

Trust: Trust in AVs was evaluated using the Trust In Automation questionnaire [44], focusing on three subscales: Reliability/Competence, Understandability/Predictability and Trust in Automation.

Motion Sickness and Presence: We used the Misery Scale [5] to monitor simulator sickness, suspending the study if ratings exceeded three. The Igroup Presence Questionnaire³ [64] assessed the degree of participants’ immersion in the virtual environment, serving as an indicator of simulation validity.

Semi-structured Interviews: At the end of the study, we asked participants to rank eHMI concepts from 1 (most preferred) to 4 (least preferred) and elaborate on their preferences. We aimed to understand how different eHMI concepts affected their experiences and elements shaping their crossing decisions. Additionally, participants gave feedback on the virtual environment and indicated if their crossing behaviour in the VR setting mirrored their actions in the real world.

The Group Effect assesses the differences in responses between the two groups: the knowledge and the no knowledge group. From our results, there were no significant differences between these groups in terms of their workload and perceived safety. Regarding trust subscales, Trust In Automation and Reliability/Competence also did not show significant differences between the two groups. However, the Understandability/Predictability subscale presented a notable distinction. The knowledge group registered lower scores compared to the no knowledge group, as indicated by a value of Mdiff = -0.24, SE = 0.10, p = .024. This suggests that the knowledge group perceived the system as less predictable or understandable than the no knowledge group did.

The Interaction Effect examines how the difference between these two groups changes based on specific interface conditions. The data suggests that the knowledge group appears to perceive the Interconnected eHMI more favourably on measures such as perceived safety, workload, and trust subscales when compared to the Unconnected and Baseline conditions (see Figure 7) Nonetheless, the interaction effects were not statistically significant (see Table 3). This suggests that the impact of the different interface conditions on these measures remains uniform regardless of whether participants had prior knowledge of interconnected eHMIs.

We observed variability in workload, perceived safety, and trust subscales across different conditions, as presented in Table 4. The data reveals two primary insights:

The Misery Scale showed that participants experienced either no problems or only vague symptoms of motion sickness. The average Presence score for our 32 participants was 0.43 (SD = 0.50), with scores ranging from a low of -0.71 to a high of 1.36, suggesting a moderate sense of presence.

Given the prior information on interconnected eHMIs, we would expect the knowledge group to have a better grasp of this design concept. Surprisingly, regardless of the conditions, those in the knowledge group perceived a significantly lower level of Understandability/Predictability compared to the no knowledge group. The no knowledge group might be making assumptions or relying on their intuition to gauge Understandability/Predictability, leading to an inflated sense of confidence. On the other hand, the knowledge group, having received explicit information, might become more aware of the nuances and complexities, leading to a reduced Understandability/Predictability ratings.

Quantitative analysis suggests both groups had a similar responses when exposed to the interface conditions. However, the knowledge group had a more favourable view of the Interconnected eHMI across all measures. Even though the interaction effects were not statistically significant, these trends could offer valuable insights for eHMI education (e.g. via public campaigns). In particular, the absence of an interaction effect suggests that the information alone does not uniquely alter the way users respond to different eHMI designs. This observation might be explained by the theory-practice gap. While reading materials provide a theoretical perspective, experiencing the concept in a scenario emphasises practice. For complex design concepts, education should ideally balance both to ensure a holistic understanding and effective user interaction with the system.

This balance is evident in a study by Faas et al. [24] where participants were introduced to eHMI concepts and directly experienced them during the first session. In the following sessions (7-9 days later), participants’ recall of these concepts was tested, and if necessary, corrected before they experienced the concepts again. The authors discovered that the positive impacts of eHMIs not only persist over time in areas such as acceptance and user experience but may also intensify in others like trust and reliance. This underscores the importance of firsthand interaction and longitudinal assessment in understanding the true impact of eHMIs on users.

Balancing Trust and Overtrust: Trust in automation is complex. While users need to trust the technologies they use, overtrust can lead to complacency and risky behaviours. Striking a delicate balance between clear guidance and user caution is crucial. For instance, Faas et al. [22] suggested teaching pedestrians to detect AV malfunctions via eHMI. Our red crosswalk solution aimed to instill vigilance, as seen in the study’s observed crossing behaviour. However, feedback was mixed: half perceived this combination as a reinforced cautionary signal (interpreting red as caution), while the other half found it conflicting (interpreting red as danger/stop). In light of these divergent perceptions, future research might explore more nuanced design strategies. For instance, a less alarming colour, such as a darker shade of red typically seen in red coloured asphalt for road markings [52], orange or yellow, could be considered to communicate warnings more subtly [74]. Enlarging the eHMI’s crosswalk projection to span both lanes may clarify the extent of the covered space. Additionally, integrating symbols or text-based cues like ‘Cross with caution’ could further enhance clarity and accessibility, echoing the suggestions of some study participants. These suggestions align with the existing practice of using highly visible pavement markings to caution pedestrians⁴.

Communicating Traffic Intentions: Traditional eHMIs primarily convey the intentions of individual vehicles, akin to a pedestrian seeking cues from a single driver. Interconnected eHMIs, however, offer insights into multiple vehicles. Our qualitative findings on collisions suggest that interconnected eHMIs can reduce signal misinterpretations, leading to safer interactions in traffic. This is particularly beneficial in complex traffic scenarios like multi-lane crossings or roundabouts, where the intentions of several vehicles must be interpreted cohesively. The term interconnected eHMIs serves as an overarching concept encompassing various representations. While our study explored a centralised communication approach, future research might explore alternative representations. For example, while each AVs activates its eHMI individually, their yielding behaviour, as well as the timing and visual aspects of the communication signals (e.g., light band and projection), could be synchronised. This coordination would ensure temporal alignment and visual consistency in the communication across all AVs, enhancing safety and clarity in complex traffic situations.

Testing eHMIs in Mixed Traffic Environments: In settings where vehicles, with and without eHMIs, coexist, eHMI designs should not complicate the pedestrian’s decision-making process [50]. Instead, they should complement the kinematic vehicle cues pedestrians have historically depended on. Given the potential challenges pedestrians face in mixed traffic situations, there is a pressing need for eHMI designers and researchers to undertake rigorous testing within these contexts, ensuring designs effectively enhance, rather than impede, pedestrian safety.

Our design choices, such as the implementation of the red crosswalk and the Wi-Fi visual, while reasonable and informed, proved insufficient in facilitating a clear understanding of the interconnected eHMI operations among participants. As a result, the study findings pertain to this specific representation of interconnected eHMIs rather than the broader concept of interconnected eHMIs. This issue of generalisability is indeed a common challenge in eHMI design studies, where findings are often specific to the tested design. Despite these limitations, our study provides valuable benchmarks and insights for the future development of more effective eHMI systems, particularly in understanding user needs and preferences.

Our study assumed that participants’ behaviour would be similar across designed scenarios. However, individual variations, influenced by past experiences, cognitive abilities, and risk tolerance, were significant. This variability was evident when some participants quickly crossed the street before the red crosswalk changed from red to green, potentially leading to an incomplete understanding of the interconnected eHMIs and their influence on measured effectiveness. Moreover, to simplify the study design and focus solely on interactions between AVs and pedestrians, all scenarios involved an AV stopping in the first lane. This decision resulted in a missed opportunity to compare pedestrian behaviour towards AVs and manually-driven vehicles in mixed traffic. Future work should aim to evaluate the interconnected eHMIs holistically under more varied situations.

Our reliance on a VR simulation introduced inherent limitations in terms of transferability to real-world scenarios, especially regarding field of view and the accuracy of estimating distance and speed. Additionally, the repetitive nature of having participants cross the street multiple times might not accurately reflect real-world pedestrian spontaneity.