Accessing Passersby Proxemic Signals through a Head-Worn Camera: Opportunities and Limitations for the Blind

Why computer vision and wearable cameras.

Assistive technologies for the blind that employ wearable cameras and leverage advances in computer vision, such as pedestrian detection [5, 36, 54, 96], could provide access to spatial behavior of passersby to help blind users increase their autonomy in practicing social distancing or initiating social interaction; the latter motivated our work since it was done right before COVID-19. ‘Speak up’ and ‘Embrace technology’ are two of the tips that the LightHouse Guild provided to people who are blind or have low vision for safely practicing social distancing during COVID-19 [38], mentioning technologies such as Aira [11] and BeMyEyes [27]. However, these technologies rely on sighted people for visual assistance. Thus, beyond challenges around cost and crowd availability, they can pose privacy risks [6, 13, 94]. Prior work, considering privacy concerns for parties that may get recorded, has shown that bystanders tend to be amicable toward assistive uses of wearable cameras [7, 80], especially when data are not sent to servers or stored somewhere [54]. However, little work explores how blind people capture their scenes with wearable cameras and how their data could work with computer vision models. As a result, there are few design guidelines for assistive wearable cameras for blind people.

To better understand the opportunities and challenges of employing such technologies for accessing passersby proxemic signals, we collect and analyze video frames, log data, and open-ended responses from an in-person study with blind (N=10) and sighted (N=40) participants³. As shown in Fig. 1, blind participants ask passersby for information while wearing smart glasses with our testbed prototype for real-time pedestrian detection. We explore what visual information about the passerby is captured with the head-worn camera; how well pedestrian detection algorithms can extract proxemic signals in camera streams from blind people; how well the real-time estimates can support blind users at initiating interactions; and what limitations and opportunities head-worn cameras can have in terms of accessing proxemic signals of passersby.

Our exploratory study shows that there are still many limitations to overcome for such assistive technology to be effective. When the pedestrians faces are included, the results are promising. Yet, idiosyncratic movements of blind participants and the limited field of view in current wearable cameras call for new approaches in estimating passersby proximity. Moreover, analyzing images from smart glasses worn by blind participants, we observe variations in their scanning behaviors (i.e., head movement) leading to capturing different body parts of passersby, and sometimes excluding essential visual information (i.e., face), especially when they are near passersby. Blind participants’ qualitative feedback indicates that they found it easy to access proxemics of passersby via smart glasses as this form factor did not require camera aiming manipulation. They generally appreciated the potential and importance of such an wearable camera system, but commented on how experience degraded with errors. They mentioned trying to adapt to these errors by aggregating multiple estimates, instead of relying on a single one. At the same time, they shared their concern that there was no guaranteed way for them to check errors in some visual estimates. These findings help us discuss the implications, opportunities, and challenges of an assistive wearable camera for blind people.

2.1. Accessing the Spatial Behavior of Others With a Camera

Prior work on spatial awareness for blind people with wearable cameras and computer vision has mainly focused on tasks related to navigation [29, 55, 68, 70, 99] and object or obstacle detection [4, 55, 56, 64, 70]. We find only a few attempts exploring the spatial behavior of others whom a blind individual may pass by [48, 61] or interact with [51, 91, 96]. For example, McDaniel et al. [61] proposed a haptic belt to notify blind users of the location of a nearby person, whose face is detected from the wearable camera. While closely related to our study, blind participants were not included in the evaluation of this system. Thus, there are no data on how well this technology would work for the intended users. Kayukawa et al. [48], on the other hand, evaluate their pedestrian detection system with blind participants. However, the form factor of their system, a suitcase equipped with a camera and other sensors, is different. Thus, it is difficult to extrapolate their findings for other wearable cameras such as smart glasses that may be more sensitive to blind people’s head and body movements. Designing wearable cameras for helping blind people in social interactions is a longstanding topic [32, 51, 66, 76]. Focusing on this topic, our work is also inspired by Stearns and Thieme [96]’s preliminary insights on the effects of camera positioning, field-of-view, and distortion for detecting people in a dynamic scene. However, they focused on interactions during a meeting (similar to [91]) and analyzed video frames captured by a single blindfolded user, which is not a proxy for blind people [3, 79, 88, 100]. In contrast to our work, the context of a meeting does not allow for extensive movements and distance changes — typical tasks when proxemics of passersby are needed. In our paper, we explore how blind people’s movements and distances can lead to visual characteristics of their camera frames. In this paper, we explore how movements and distance are reflected in cameras’ field of view and related to the performance of the computer vision system.

2.2. Estimating Proximity to Others Beyond Accessibility

Distance between people is key to understanding spatial behavior. Thus, epidemiologists and behavioral scientists have long been interested in automating the estimation of people’s proximity [14, 60]. Often their approaches employ mobile sensors [60] or Wi-Fi signals [75]. For example, Madan et al. [60] recruited participants in a residence hall and tracked foot traffic based on their phones’ signals to understand how their behavior change relates to reported symptoms. While these approaches work well for the intended objective, they are limited in the context of assistive technology for real-time estimates of approaching passersby in both indoor and outdoor spaces.

Since the recent pandemic outbreak [74], spatial proximity estimation (i.e., looking at social distancing) has gained much attention. The computer vision community started looking into this using surveillance cameras [33, 86] or camera-enabled autonomous robots [2]. Using cameras capturing people from the perspective of a third person, they monitor the crowd and detect people who violate social distancing guidelines. However, for blind users, an egocentric perspective is needed. Other approaches include using ultrasonic range sensors that can measure a physical distance between people wearing the sensors (e.g. [41]). This line of work is promising in that the sensors consume low power, but the sensors can help detect only presence and approximate distance. They cannot provide access to other visual proxemics that blind people may care about [54]. More so, they assume that sensors are present on every person. Thus, in our work, we prioritize RGB cameras that can be worn only by the user to estimate presence, distance, and other visual proxemics.

Specifically, we borrow from pedestrian detection approaches in computer vision, but some do not really work for our context. For example, they estimate distances by looking at visual differences of the pedestrian between two images from a stereo-vision camera [43, 69]. They first track the pedestrian and then change the camera angle to position the person in the center of frames [43]. Alternatively, they use two stationary cameras [69]. Neither case is appropriate for our objective. However, approaches that estimate the distance based on a person’s face or eyes captured from an egocentric user perspective could work [30, 83, 104, 110]. For our testbed, we use the face detection model of Zhang et al. [110] to estimate a passerby’s presence, distance, relative position, and head pose.

2.3. Choosing Handheld versus Head-worn Cameras in Assistive Technologies for the Blind

Smartphones and thus handheld cameras are the status quo for assistive technologies for the blind [9, 16, 17, 45, 47, 53, 101, 102, 116], though, there are still many challenges related to camera aiming [9, 17, 45, 47, 52, 53, 102, 116], which can be trickier for a pedestrian detection scenario. Nonetheless, what might start as research ideas quickly translate to real-world applications [1, 11, 21, 23, 24, 26, 27, 57, 84, 92, 98]. However, the use of cameras along with computationally intensive tasks can rapidly heat up the phone and drain its battery [62, 111]. Although this issue is not unique to this form factor, it can harm phone availability that blind users may rely on; many have reported experiencing anxiety when hearing battery warnings in their phones, especially at their work or during commuting [106].

With the promise of hands-free and thus more natural interactions, head-worn cameras are falling on and off the map of assistive technologies for people who are blind or have low vision [37, 42, 58, 78, 95, 97, 113, 115]. This can be partially explained by hardware constraints, such as battery life and weight, limiting their commercial availability [93] which can also relate to factors such as cost, social acceptability, privacy, and form factor [54, 80]. Nonetheless, we see a few attempts in commercial assistive products [1, 11, 26, 71], and people with visual impairments remain amicable towards this technology [117], especially when it resembles a pair of glasses [67] similar to the form factor used in our testbed. However, to our knowledge, there is no prior work looking at head-worn camera interaction data from blind people, especially in the context of accessing proxemics to others. Although wearable cameras do not require wearers to aim a camera, including visual information of interest in the frame can still be challenging for blind people. Prior work with sighted [105] and blind-folded [96] participants indicate that the camera position can lead to different frame characteristics, though such data are not proxy for blind people’s interactions nor related to the pedestrian detection scenario. Thus, collecting and analyzing data from blind people, our work takes a step towards understanding their interactions with a head-worn camera in the context of pedestrian detection.

4.1. Participants

We recruited ten blind participants; nine were totally blind, and one was legally blind. On average, blind participants were 63.6 years old (SD = 7.6). Five self-identified as female and five as male. Three participants (P4, P8, P10) mentioned having light perception, and four participants reported having experience with wearable cameras such as Aira [11] with frequencies shown in Table 1. Responses to technology attitude questions⁵ indicate that they are positive about technology in general (mean = 2.02, SD = 0.42) and wearable technology such as smartglasses (mean = 1.89, SD = 0.90).

We also recruited 40 sighted participants to serve as passersby in the study — four sighted participants, one at a time, passed by a blind participant. On average, sighted participants were 25.3 years old (SD = 4.8). Twenty-four self-identified as male and fifteen as female, and one chose not to disclose their gender. Participants’ responses to technology attitude questions⁵ indicate that they were slightly positive about wearable technology (mean = 1.08, SD = 0.95).

4.2. Procedure

We shared the consent form with blind participants via email to provide ample time to read it, ask questions, and consent prior to the study. Upon arrival, the experimenter asked the participants to complete short questionnaires on demographics, prior experience with wearable cameras, and attitude towards technology. The experimenter read the questions; participants answered them verbally. Blind participants first heard about the system and then practiced walking with it in the study environment and detecting the experimenter as he approached in the opposite direction. During the main data collection session, blind participants were told to walk in the corridor and ask a person, if detected, about the office number nearby. On the other hand, sighted participants, one at a time, were told to simply walk in a corridor with no information about the presence of blind participants in the study. As illustrated in Fig. 3, a blind participant walked in the corridor eight times — four times with four sighted participants, respectively, and four times with four sighted members of our research lab, respectively. Our study was designed not to extend beyond 2 hours.

All sighted participants were recruited on site and thus strangers to blind participants; blind participants did not meet any sighted participants before the study. A stationary camera was used to record the dyadic interactions in addition to the head-worn camera and the server’s logs. Both blind and sighted participants consented to this data collection prior to the study — sighted participants were told that there will be camera recording in the study, which may include their face, but these images will not be publicly available without being anonymized. The study ended with a semi-structured interview for blind participants with questions⁵ created for eliciting their experience and suggestions towards the form factor, proxemics feedback delivery, error interaction, and factors impacting future uses of such head-worn cameras.

4.3. Data Collection, Annotation, and Analysis

We collected 183-minute-long session recordings from a stationary camera, more than 1,700 camera frames from the smart glasses testbed sent to the server, estimation logs on proxemic signals of sighted passersby that were communicated back to blind participants, and 259-minute-long audio recordings of the post-study interview.

4.4. Limitations

We simplified our user study to have only one passerby in a corridor since our prototype system was able to detect only one person at a time. In the future, designers should consider handling a case where more than one passerby appears. When more than one passerby is included in a camera frame, a system needs to detect a person of the most interest to blind users and then delivering the person’s proxemics to the users. Such a method should help the users perceive the output, efficiently and quickly. For example, detecting a person within a specific distance range could be a naive but straightforward way.

5.1. Visual Information Accessed by a Head-worn Camera

In a real-world setting, where both blind users and other nearby people are constantly in motion, it is important to understand what is being captured by assistive wearable cameras — smart glasses in our case. What is visible dictates what kind of visual information regarding passersby’s proxemics can be extracted. In this section, results are based on our analysis of 3,175 camera frames from the head-worn camera and those ground-truth annotations.

5.2. Errors in Estimating Passersby Presence, Distance, Position, and Head Pose

We evaluate how well models that rely on face detection can estimate passersby’ proxemic signals (i.e., presence, position, distance, and head pose) in camera frames from blind participants. Specifically, using visual estimates and those ground-truth annotations, we report results in terms of precision, recall, and F1-score (Table 3). These results capture blind participants’ experience with the testbed, and thus provide context for interpreting their responses related to machine learning errors.

Table 3 presents that, when the presence of a passerby was detected, there was actually a passerby (precision=1.0)⁶. However, the model’s ability to find all the camera frames where a passerby was present is low (recall=0.33) — even when their head was visible in the camera frame (recall=0.34). Our visual inspection on these false negatives informed that, for the majority (80%), this was due to the person being too far (> 16 feet); thus, their face was too small for the model to detect, raising questions such as How far away should a system detect people?

Since the estimation of position, distance, head pose depended on the presence detection, we reported metrics on both (i) frames where passersby were present and (ii) frames where they were first detected. Once passersby were detected, the estimation of their relative position worked relatively well (precision=0.90; recall=0.84). However, the head pose estimation (i.e., whether the passerby is looking at the user) was not as accurate (precision=0.77; recall=0.77). The distance estimation was also challenging (precision=0.62; recall=0.66). This might be partially due to the limited training data, both in size and in diversity. A passerby’s head size in an image, a proxy for estimating the distance, can depend on the blind user’s height and camera angle.

5.3. Participant Feedback on Accessing Proxemics

The majority of blind participants (seven) appreciated having access to proxemics about nearby people through the smart glasses. P4 highlighted, “I liked having the [proxemic] information. That means a lot when I know that there’s a person, or how close they are to you or where they are[.] It’s a good thing to be able to know who’s there and that there’s somebody there. That’s important.” Blind participants also shared certain situations where they can benefit from such assistive cameras (e.g., when nearby people do not provide enough non-visual cues). P2 highlighted, “People are sometimes quite, and [I] don’t know if they are approaching.” Some participants remarked that having access to proxemics related to a near passerby can help them make their decision proactively. For example, P7 mentioned, “With the distance estimations, it gives [me] a couple of cues within range, where [I] can make a decision ‘Do I want to do something or not?’ ”

Participants also envisioned how these visual estimations could enhance their social engagement. For example, P1 said, “It was novel [and] interesting to have it tell me things that I normally wouldn’t even be paying attention to[.] I think it would be a great social interaction thing,” and P4, “It means a lot to be able to know who it is [and] who’s there — that’s a big deal to me for social interaction.” Some participants see its potential in helping them predict the outcome of their attempted social interaction by saying, “If you know somebody is in front of you and looking [at you] or you know [where] to look (left or right), you can probably get their attention more easily” (P5), and “[GlAccess] certainly made it easy to know, when there was a potential [social interaction] — you didn’t talk to the air[.] I would say that I like the ability to possibly predict the outcome of an attempted interaction” (P7). They also envisioned using proxemics detection and person recognition to initiate their communication. P6 said, “Walk down the hall and be able to greet a person by name when they hadn’t spoken to me; that would be cool,” and P7, “Because [I] know that there is at least a person and how far away they are, [I’m] not simply chattering with air, which just occurred to me a couple times [in my daily life].”

On the other hand, our analysis also reveals potential reasons why blind participants might not pay much attention to visual estimates. Since such visual information was mostly inaccessible in their daily lives, they did not rely on the visual information and also was unsure how to utilize them. More specifically, P7 and P9 remarked, “[Getting visual information] never occurred to me. [Not sure if] I would want or need anything more than just listening to the voices around me and hearing the walking around me because I’ve never had that experience,” and “It’s visual feedback that I’m not used to, I never functioned with, so I don’t know, I’m not used to processing or analyzing things in a visual way. I love details, but I just never often think about them much related to the people,” respectively. To address this issue, participants suggested providing training that helps blind users interpret such visual estimates as P7 said, “I would need maybe to be trained on what to do [with the visual feedback].”

5.4. Participant Feedback on the Form Factor

Six participants commented on how easy it was to have the smart glasses on and access information, saying “Just wearing the glasses and knowing that would give you feedback” (P2); “Just have to wear it and listen” (P4); and “Once [I] put on and just walk” (P8). Some supported their statements by proving a rationale such as the head being a natural way to aim towards areas of interest and the fact that they didn’t have to do any manipulations. For example, P4 and P9 commented, “If it’s just straight, then [I’m] gonna see what’s straight in front of [me]. That’s a good benefit[.] Seeing what directly in front of [me] is a bonus” and “I could turn my head to investigate or find out what’s to my right or my left — that part was kind of nice,” respectively. P7 and P10 commented on the manipulation, saying “You don’t have a lot of controlling and manipulating to do” and “And [it] doesn’t take any manipulation — it just feeds new information. And that was easy. It took no interaction manually changed,” respectively.

However, anticipating that some control would be eventually needed, P8 tried to envision how one could access the control panel for the glasses and suggested them being paired with the phone or support voice commands by stating “Altering [it] would be an interface with your phone so that I could [control] it using the voice over on my phone, or using Siri to do it. The voice assistant [would] let you control.” Only one participant (P1) suggested deploying similar functionalities into one’s phone “And just use your rear facing camera and have it as a wearable, a pouch or something like that. Then you just flip that program on [the phone] when you want to use it.”

Other suggestions focused on the sound. At the time, Vuzix Blade did not have a built-in speaker, and their shape conflicted with bone conduction headphones. Four participants quickly pointed out this limitation with statements such as “Bone conduction [would] seem to work [with GlAccess] really good” (P4) and “[Need to keep] your ear canals open there for being able to hear traffic or whatever” (P8). P4 suggested extending the field of view for capturing more visual information, stating “I wonder if the field of vision for the camera could go a little bit wider because then it would catch somebody coming around the side more[.]”

5.5. Participant Feedback on Interactions with the System

Four participants shared concerns on outside noises blocking out the system’s audio feedback, or vice versa. P4 mentioned, “The only time it could be harder [to listen to the system feedback] is in a situation with lots of noise or something. If it’s very noisy, then it’s going to be hard to hear,” and “There’s a limit to how much you can process because you’re walking. When we’re walking[,] we’re doing more than just listening to the [the feedback.] We listen to [others]. Listening is big [to us].” Although GlAccess reported only feedback changed between two contiguous frames, this is possibly why they wanted to control the verbosity of the system feedback. P7 said, “It was reacting to you every couple seconds or literally every second. Until I’m looking for something specific, it [is] a little overwhelming to have all of it coming in at the same time.”

Moreover, blind participants emphasized that they would need to access such proxemic signals in real time, especially when they are walking. They mentioned being misguided by stale estimations caused by network latency at times — P9 commented on this latency issue, “I know about the latency in it. Because when [a person] and I were talking, I knew [the person] walked away, [but GlAccess] still reported that [the person was] in front of me.” To minimize the effects of the latency, some reported that they made quick adaptation, such as “walking slow[ly]” (P1) or “stop[ping] and paus[ing]” (P10), during the study. Nonetheless, the latency issue would make the use of such assistive wearable cameras impractical in the real world, as P8 put, “If I’m walking at a normal pace [and] it’s telling me somebody is near, I’m probably going to be 10 to 15 feet past [due to the latency].”

Participants also suggested providing options for customizing the feedback delivery. For example, P7 wanted the system to detect moments for pausing or muting the feedback delivery, saying “Once a conversation is going on, it’s not reporting everyone else around; [otherwise, it’s] distracting [me].” Also, four blind participants commented on verbosity control, which we explored a bit in our study by implementing two ways of communicating the distance estimation. P2 mentioned, “I like that you had options [for verbosity], and [I] could choose that.” At the same time, we observe that blind people may have different definitions of spatial perception — i.e., the definition of far and near. P4 remarked on this, “ ‘Far’–’near’ thing wasn’t so good. I liked hearing the the number of feet for the distance or my concept of far is different than your concept of far and near[.]”

5.6. Participant Feedback on Estimates and Errors

Seven blind participants mentioned being confident of interpreting person recognition output. It seems that they gained their trust in the person recognition feature since they were able to confirm the recognition result with a person’s voice. P2 said, “I learned [a lab member]’s voice, and when it said ‘[lab member]’, it was her; or at least I thought it was.” Similar to any other machine learning models, however, our prototype system also suffered from errors such as false negatives (Section 5.2). Participants seemed to be familiar with technical devices making errors and understood that it is part of technology improvement, as P1 commented, “If it didn’t work, then I figured that’s just part of the the system growing, so that’s okay. It’s just part of learning how to use it.” To prevent from making their decision based on errors, six participants mentioned aggregating multiple estimates from the system, instead of relying on a singular estimate, after realizing the possibility of errors in visual estimates. P9 highlighted, “If [the] information was not what I felt either incorrect or not certain, I would wait and give the system a chance to focus better or to process it differently.”

Even with errors in visual estimates, blind participants said that it can help them know that “there was somebody [at least]” (P6). On the other hand, seven participants pointed out that there would be no guaranteed way for them to verify the visual estimates — i.e., detecting errors in visual estimates is inaccessible. For example, P3 did not know if there was even an error because “[P3] can’t really recall that where it wasn’t right.” Participants who even experienced something odd mentioned that it was difficult for them check whether that was an error, or not. P4 commented on this issue by saying, “One time [GlAccess] goofed, I think, but I don’t know if that [was an error.] I don’t know if there were more people in the hallway.” Due to the inaccessibility of feedback verification, P10 mentioned being dependent on the system feedback, “Unless someone was there saying that listening to same thing I was listening to and tell me if it was right or wrong, I had to depend on it.” We suspect that these experiences led them to expecting future assistive systems to provide more reliable visual estimates, as P3 commented, “[Future glasses need to] reassure that it’s accurate in telling you like distance [how far] people are.”

5.7. Participant Feedback on Teachable Passerby Recognition

Although blind participants did not experience personalizing their smart glasses in the study, they envisioned that such assistive technologies would be able to recognize their family and friends once they provide their photos to the device. For instance, P1 expected this teachable component by saying, “[Person recognition] means [that] I have to create a database of pictures. And I wouldn’t mind doing that if my friends wanted to.” Recognizing our lab members during the study seemed to help them see the benefits of recognizing their known people via smart glasses. P6 was excited about using this recognition feature to greet her family and friends by their name even “when they hadn’t spoken to [P6].” Also, P10 mentioned training smart glasses to find, ”out of a group, a person that [P10 is] looking for.” P1 commented on sharing photos with other blind people to recognize each other through such smart glasses, saying “When I was at the American Council of the Blind convention, there were rooms full of blind people, [and] none of [us] knew who all was in the room. So wouldn’t it be cool if we all had each other’s pictures [to teach smart glasses about us]?”

6.1. Implications regarding Accessing Proxemic Signals using Head-Mounted Cameras

Section 5.1 reveals that some camera frames from blind participants did not capture passersby’s face but other body parts (i.e., torso, arms, or legs), especially when they were near the passersby. Although some visual cues such as head pose and facial expressions [116] cannot be extracted from the other body parts, detecting those may help increase accessibility of someone’s presence [109, 112]. Also, we see that proximity sensors can enhance the detection of a person’s presence and distance as those sensors can provide depth information. In particular, such detection can help blind users to practice social distancing, which has posed unique challenges to people with visual impairments [31]. The sensor data however needs to be processed along with person detection, which often requires RGB data from cameras. Future work that focuses on more robust presence estimation for blind people may consider incorporating proximity sensors with cameras.

Regarding delivery of proxemic signals, it is important to note that blind people may have different spatial perception [22] and could thus interpret such proxemics feedback differently, especially if the feedback was provided with adjectives related to spatial perception, such as ‘far’ or ‘near’ (Section 5.5). Other types of audio feedback (e.g., sonification) or sensory stimulus (e.g., haptic) may overcome the limitations that the speech feedback has. On the other hand, blind participants’ experiences with our testbed may have been affected by its performance and thus could differ if an assistive system had more accurate performance or employed different types of feedback. Future work should consider investigating such a potential correlation.

6.2. Implications regarding the Form Factor

Several factors, such as blind people’s tendency of veering [39] and behavior of scanning the environment when using smart glasses (Section 5.1), can change camera input on wearable cameras and thus may affect the performance of detecting proxemic signals. From our study, we observed that the head movement of blind people often led to excluding a passerby from camera frames of a head-worn camera. Although we did not observe blind participants’ veering tendency in our study, the veering, which can vary by blind people [39], could change their camera aiming and consequently affect what is being captured. Also, as discussed in earlier work [22], blind people’s onset age affects the development of their spatial perception. Observing that our blind participants had different onset ages, we suspect that some differences in their spatial perception may explain their different scanning behaviors with smart glasses. Future work should investigate these topics further to enrich our community’s knowledge on these matters.

Moreover, using a wide-angle camera may help assistive systems to capture more visual information (e.g., a passerby coming from the side of a user) and tolerate some variations in blind people’s camera aiming with wearable cameras. The smart glasses in our user study, having only a front view with the limited horizontal range (64-degree angle), are unable to capture and detect people coming from the side or back of a blind user. As described in Table 4, some prior work has employed a wide-angle camera [96] or two cameras side by side [49] to get more visual data, especially for person detection. However, accessibility researchers and designers need to investigate which information requires more attention and how it should be delivered to blind users. Otherwise, blind users may be overwhelmed and often distracted by too much feedback (Section 5.5).

In addition, aiming with a wearable camera can vary by where the camera is positioned (e.g., a chest-mounted camera) as earlier work on wearable cameras for sighted people indicates that the position of a camera can lead to different visual characteristics of images [105]. In the context of person detection, prior work investigated assistive cameras for blind people in two different positions (i.e., either on a suitcase [48, 49] or on a user’s head [54, 96]). However, little work explores how camera positions can change the camera input while also affecting blind users’ mental mapping and computer vision models’ performances. We believe that it is important to study how blind people interact with cameras placed in different positions to explore design choices for assistive wearable cameras.

6.3. Implications regarding the Interactions with the System

Data-driven systems (e.g., machine learning models) inherently possess the possibility of generating errors due to several factors, such as data shifts [82] and model fragility [34]. Since the system that study participants experienced merely served as a testbed, it was naturally error prone (Section 5.2), which is largely true for any AI system, and consequently affected blind participants’ experiences. (Section 5.6). In particular, blind participants pointed out inaccessibility of error detection in visual estimates. To help blind users interpret feedback from such data-driven assistive systems effectively, it is imperative to tackle this issue (i.e., How to make such errors accessible to blind users?). One simple solution may be to provide the confidence score of an estimation, but further research is needed to understand how blind people interpret this additional information (i.e., confidence score) and to design effective methods of delivering such information.

Furthermore, different feedback modalities are worth exploring. In this work, we provided visual estimates in speech, but blind participants sometimes found it too verbose and imagined that it would be difficult for them to pay attention to speech feedback in certain situations, e.g. when they are in noisy areas or talking to someone. To prevent their auditory sensory system from being overloaded, other sensory stimuli such as haptic can be employed to provide feedback [15, 90]. It is however important to understand what information each feedback modality can deliver to blind users and how effectively they would perceive the information.

6.4. Implications regarding Teachable Interfaces

Blind participants envisioned personalizing their smart glasses to recognize their family members and friends (Section 5.7). To provide this experience for users, it would be worth exploring teachable interfaces, where users provide their own data to teach their systems [46]. Our community has studied data-driven approaches in several applications (e.g., face recognition [116], object recognition [9, 47, 53], and sound recognition [44]) to investigate the potential and implications of using machine learning in assistive technologies. However, little work explores how blind users interact with teachable interfaces, especially for person recognition. Our future work will focus on this to learn opportunities and challenges of employing teachable interfaces in assistive technologies.

Moreover, in Section 5.5, blind participants suggested enabling future smart glasses to detect specific moments (e.g., sitting in a table or talking with someone) to control feedback delivery. We believe that egocentric activity recognition [59, 65, 107], a widely-studied problem in the computer vision community, can help realize this experience. However, further investigation is required for adopting such models in assistive technologies, especially for blind users. For instance, one recent work proposed an egocentric activity recognition model assuming that input data inherently contains the user’s gaze motion [65]. However, that model may not work on data collected by blind people since the assumption, (i.e., eye gaze), only applies to a certain population (i.e., sighted people).

6.5. Implications regarding Hardware, Privacy Concerns, and Social Acceptance

There are still challenges related to hardware, privacy, and public acceptance, which keep wearable cameras from being employed for assistive technologies. For example, Aira ended their support for Horizon glasses in March 2020, due to hardware limitations [10]. Also, prior work investigated blind users’ privacy concerns over capturing their environments without being given access to what is being captured [8, 12], and social acceptance of assistive wearable cameras as the always-on cameras can capture other people without notification [7, 54, 80].

Beyond the high cost of real-time support from sighted people (e.g., Aira), as of now, there is no effective solution that can estimate the distance between blind users and other people in real time. Computer vision has the potential to provide this proxemic signal in real time on a users’ device. No recordings [54] and privacy-preserving face recognition [25, 85] may help address some privacy issues, but more concerns may arise if assistive wearable cameras need to store visual data (even on the device itself) to recognize the users’ acquaintances. Future work should consider investigating associated privacy concerns and societal issues, as these factors can affect design choices for assistive wearable cameras.