A Scoping Review of Assistance and Therapy with Head-Mounted Displays for People Who Are Visually Impaired

A great number of studies have been conducted in the broad field of ophthalmology, and tremendous resources have been invested in the prevention and treatment of a variety of visual conditions. As our global population grows, and human lifespans increase, the need for mechanisms for visual assistance and therapy continues to grow. As reported by the World Health Organization (WHO) in 2019, a growing range of effective strategies are available to address the needs associated with eye conditions and vision impairments, including treatment (“curing as well as addressing symptoms and progression”) and rehabilitation (“maximizing the use of residual vision and providing practical adaptations to address the social, psychological, emotional, and economic consequences of vision impairment”) strategies [72].

While there is evidence that the pervasiveness of personal electronics has increased the prevalence of ocular symptoms [38], it also seems that the increasing availability and prevalence of HMDs is increasing the opportunities for related treatment and rehabilitation. In 1999, Harper et al. discussed the “emerging” head-mounted Low Vision Aid (LVA) devices based on video technology, characterizing them as having a promising future, and articulating fundamental advantages including hands-free operation and ready adaptation to a wide range of tasks, and disadvantages (at that time) including bulky size, high cost, and complexity of use [32]. The Low Vision Enhancement System (LVES) by Visionics Corporation was one of the leading players in the head-mounted LVA market in the 1990s. However, at that time Head-Mounted Displays (HMDs) did not seem to be practical for the general public. In 2003, Peterson et al. conducted a study with 70 visually-impaired subjects, comparing three types of electronic vision enhancement systems with a conventional optical magnifier. At the time, the HMD-based systems performed the worst [57]. In 2004, Culham et al. compared the clinical performance of four head-mounted LVAs with the performance of conventional optical LVAs, and suggested that practitioners should only consider head-mounted LVAs in certain circumstances [19]. Not surprisingly, it seems that the early head-mounted LVAs ended up quietly disappearing.

A seminal use of HMDs for vision therapy was for Unilateral Spatial Neglect (USN), which is a neurological condition characterized by a failure to explore and allocate attention in a particular region [65]. In 2009, Tsirlin et al. reviewed the use of Virtual Reality (VR) in assessing, treating, and studying USN, admitting that there were little data documenting the effectiveness of VR-related treatments [68]. They articulated advantages and disadvantages of HMDs similar to those by Harper et al. (id.), concluding that the technology was attractive, but that several characteristics of current VR technologies—including ergonomics, complexity, and cost, can pose a challenge to the development of new applications for USN.

In the 2010s, HMDs have significantly improved in several respects, including cost and availability. As such, there has been a resurgence in the market related to the use of HMDs in visual assistance and therapy. For example, founded in 2006 in Canada, eSight released its first generation product, eSight 1, in the same year, while the second generation came to the market in 2015 [2]. Founded in 2014 in the US, Iris Vision employs phone-based VR devices to help certain visually impaired individuals see better [4]. Founded in 2014 in the UK, Give Vision’s first generation product SightPlus is another example of a phone-based VR aid. They carried out a clinical trial in 2019 on 60 low-vision participants testing their visual acuity, contrast sensitivity, and reading performance with and without SightPlus [18]. Nearly half of the participants indicated willingness to use this product. Their second-generation product is based on a pair of Optical See-Through (OST) glasses for AR [3]. Also founded in 2014, the US company Vivid Vision has offered products for the treatment of eye conditions including amblyopia (“lazy eye”), strabismus (“crossed eyes”), and convergence insufficiency [5]. Their system, which was already in use by clinicians in 2018, provides the HMD version of dichoptic training [9]. They conducted a study on 17 adults with amblyopia, a vision disorder regarded as incurable after the age of eight, and received positive preliminary results [83].

Here, we provide a scoping review of the progress researchers have made using HMDs for visual assistance and therapy to date. While we identified eight review articles during our screening, they turned out to be focused on very specific vision conditions or were not conducted for scoping purpose. Three of the identified survey articles focused purely on the VR-based assessment and treatment of USN, all of which emphasized the importance of comparing the performances of VR methods with the conventional methods, the need to customize the ergonomics of VR devices and user study settings for people with USN as they are likely to have other medical issues such as mobility difficulties, and the necessity of lowering the cost to make the apparatus practical for clinical uses [54, 56, 68]. Tsirlin et al. described different implementations of the conventional and VR methods, while no systematic literature search was conducted [68]. Pedroli et al. and Ogourtsova et al. looked systematically into 13 and 22 studies, respectively, with detailed information of participants [54, 56]. Two out of the eight survey articles looked into the use of VR as a vision therapy for stereo vision dysfunctions. Fortenbacher et al. went through the history and development of this research direction, and envisioned a bright future with improved HMDs and software designs [26]. They highlighted that the long-term professionally instructed therapy was more effective than therapies in other settings. Coco-Martin et al. reviewed clinical studies associated with the use of VR for the treatment of a specific visual condition, amblyopia, and suggested that repetitive training, multisensory simulation, user engagement, and customizable designs are key for studies in neurorehabilitation [16]. The remaining three survey articles are all about HMDs for low vision rehabilitation. Besides the aforementioned envisioning of Harper et al. in 1999 [32], Ehrlich et al. investigated different types of HMD technologies that could be useful for vision enhancement and rehabilitation, and various optical and human factors considerations for each display type [24]; Deemer et al. focused on the software side of AR approaches to vision enhancement with HMDs, including contrast enhancement, image remapping, and motion compensation [20]. Though no systematic literature searches were conducted, the two articles provided useful insights into what the display technology and software design for low vision rehabilitation have evolved into, and further asserted the promising future envisioned two decades ago.

Rather than focus on treatment or rehabilitation of specific vision conditions, we provide here an up-to-date scoping review of previous work using HMDs to assist or treat people who are visually impaired. We focus on the following research questions:

Following the scoping study framework [8, 51], we describe how the literature pieces were searched and screened in this section (see Figure 1 for overall screening process).

From an initial set of 24 related articles, we extracted the frequently-seen AR/VR/HMD-related and vision-related keywords as shown in Table 1. We conducted the literature search using those keywords on September 22, 2020, in six digital libraries, namely, Institute of Electrical and Electronics Engineers (IEEE), Association for Computing Machinery (ACM), Springer Science+Business Media (Springer), ScienceDirect (SD), Web of Science (WoS) and PubMed. We intended to search in the fields of abstract and author keywords to not go down into every detail but also not miss important information; however, due to different restrictions of the search engines, only IEEE, ACM and WoS were able to meet this expectation. In the case of Springer, we applied full body in search range and refined the topic to computer science; in SD, we searched the title, abstract, and author keywords; and in PubMed, we searched title and abstract. No time constraints were set during the search. The search result (1,248 articles) is composed of 104 articles from IEEE, 61 articles from ACM, 613 articles from Springer, 37 articles from SD, 264 articles from WoS, and 169 articles from PubMed. We also included three additional articles that did not appear in the search result but were previously read by the first author considering their relevance [9, 18, 32]. In total, 1,251 articles were forwarded to the screening stage.

To filter out articles that are not qualified or relevant to the topic of this literature review, the following exclusion criteria were applied during the screening stage: no HMDs were employed; not written in English; focused only on hardware design; early-stage poster/demo presentation, e.g., the length was less than four-/eight-page in double/single-column; focused mostly on non-visual affordances, e.g., audio or haptic feedback; HMDs were not used for visual assistance or therapy purpose, e.g., HMD as visual condition simulator.

Given the above criteria, our team, including experts in VR/AR, vision science, and scoping and systematic literature reviews, conducted four rounds of screening: the first author merged duplicates and checked the title and abstract of the articles for relevance in the first and second round of screening; the first six authors conducted the third round of relevance checking on full texts, and the fourth round of explicit tagging. The full text of a article was checked by at least one author. Ambiguous articles were discussed in group meetings or assigned to multiple authors. The tagging categories were agreed upon by all seven authors and are introduced in the following section. After screening, 917 articles were excluded. The remaining 69 articles (Table 2) were used for our detailed full-text analysis, and eight review articles were excluded from the high-level meta-analyses. To balance the volume of articles and our workforce, we did not conduct a reference harvesting.

For the high-level analysis of the 61 articles, we collected potential terms for classification with respect to the aspects of research and technologies for visual impairments during the review and determined the following categories after iterative refinements. Note that articles can potentially be tagged with multiple categories since a single article can generally cover multiple aspects.

Objectives. While reviewing the collected articles, we pinpointed the articles using HMD technology for visual assistance and therapy, and classified them into two categories:

Approaches. Based on how the HMDs were used for visual assistance or therapy regarding the visualization approaches, the articles were classified into the following three categories:

HMD Types. The types of the HMDs used in the articles were collected and classified into the following three categories:

Visual Conditions. Various visual conditions were addressed in the articles. To identify technological trends in the use of HMDs for visual assistance and therapy associated with the visual conditions, we determined four categories to classify the articles as shown below. Note that one condition may lead to multiple limitations and thus fall into multiple categories.

User Study. To evaluate the effectiveness of HMD-based visual assistive and therapeutic technology, user studies that investigate visual performance and human factors are critical and essential—especially involving qualitative research of the people who are visually impaired. We classified the articles into two categories according to the inclusion of a user study in the article: yes and no. The number of participants was also recorded. In the case of multiple studies in one article, the sum was taken.

This section presents the results of our high-level quantitative analysis, which provides insights into our first research question about the use of HMDs for vision care and human vision (RQ1 in Section 1). During the analysis, we first evaluated the chronological changes in the volume of the articles that we collected through the literature search using keywords related to AR/VR/HMD and visual impairments and then classified all the 61 screened articles for more detailed analyses based on the categories defined in Section 3.

The chronological distribution of the screened articles with and without user studies is presented in Figure 2. While this increasing trend generally aligns with the recent increase in the volume of literature in the VR/AR research community [22, 39] and the popularity of powerful, yet affordable low-cost HMD devices [1], it also indicates the growing use of VR/AR HMDs for visual impairments.

We analyzed the chronological trends under the categories of objectives, approaches, and HMD types over the years, presenting the results in Figure 3. Regarding the objectives categories (shown in Figure 3(a)), two-thirds (41) of the 61 articles were classified as assistive technology while the remaining (20) were classified as therapeutic technology. This phenomenon could be explained by the fact that a sizeable portion of the targeted visual illnesses is not treatable to date through external equipment. This is also supported by the proportions among the objectives and visual conditions categories shown in Figure 4(a) (63 data points are presented due to multi-tagging [44, 52]). All the works focusing on impaired central vision and color vision, which are mostly caused by incurable or hard-to-cure illnesses, are not therapeutic. The same applies to the articles falling under the visual field category except for one specific type of illness—hemineglect, which is known to be treatable. In contrast, all the articles working on deficiencies in stereopsis aimed at treating their targeted user group, which can be explained by the fact that the ability of HMDs to display two different images to the left and right eyes of the wearer is naturally compatible with dichoptic training, a promising novel therapy for stereoblindness.

Two articles were tagged with multiple approaches [60, 75], resulting in 63 data points in Figure 3(b) which shows the annual distribution of the articles classified in the augmented, modified, and virtual approaches. The modified approach is the most frequently implemented method (32 articles), followed by virtual approach (19) and augmented approach (12). We also present the proportions among the categories of objectives and approaches in Figure 4(c) to see if there are any correlations between them. We found the majority of the articles using modified approach were for assistive technology—31 out of the 32 articles (see Figure 4(c)). The single article falling under therapeutic technology is about therapy for hemineglect, the treatable type, in which the idea of real-time stimuli overlaying the real-world is proposed but not implemented [12]. In contrast to the modified approach, the virtual approach is most often used for therapeutic technology. As shown in Figure 4(c), only one out of the 19 articles is categorized as assistive technology. And even in this article, the virtual approach was not designed to be used in the ultimate assistive tool, but used by the researchers to simulate scotomas in VR before experimenting with visual assistance with the proposed strategy [77]. Many articles using VR as illness simulators or virtual testers were excluded based on our exclusion criteria (see Section 2), while this article was kept for the reason that it described a means to assist the people who are visually impaired. This article also contributed to the only data point in Figure 4(b) using IVR for central vision. Compared with modified and virtual, the augmented approach is a relatively new and less developed method but its appearance has been continuously increased since 2016 (Figure 3(b)). The augmented approach is mostly applied to assist the people who are visually impaired—11 out of 12 articles were classified as assistive technology as shown in Figure 4(c). The one exception implemented dichoptic training using the augmented approach, instead of the common virtual approach in an attempt to leverage some advantages of AR, such as less cybersickness; however, their research was not comprehensively examined by a formal user study [53]. This data point is also reflected in Figure 4(b) as the only article used OST devices for Stereopsis.

Regarding the categories of HMD types: IVR, OST, and VST, many on-brand devices were used in the articles, such as Oculus Rift (8 appearances), Microsoft HoloLens (7), HTC Vive (5), Google Cardboard (6), Google Glass (3), and Gear VR (2). DIYs are frequently used as well, especially before the year 2015—note that the consumer versions of Oculus Rift and HTC Vive appeared, and HoloLens Development Edition was shipped in 2016. Figure 3(c) shows the number of articles employing different HMD types throughout the years with 63 data points as there are articles including multiple HMDs. In total, there are 18 papers under the category of OST, 26 under VST, and 19 under IVR. By observing the proportions between the categories of approaches and HMD types in Figure 4(d), we can clearly see most articles with the augmented approach used OST HMDs, while the modified approach used both OST and VST HMDs. For the modified approach that requires any vision enhancement techniques, VST HMDs would be convenient and tractable to modify the captured imagery from the mounted cameras and merge with the real-world scene.

As user evaluation is a key factor in accessibility research and clinical trials play an important role in medical-related research, we recorded the existence of user studies in an attempt to gain some insights into this aspect. Figure 2 shows the number of articles with and without a user study among the screened 61 independent research articles over the years, and we noticed a general trend of increasing involvement of user studies in the recent decade. An overview of all 61 articles based on the number of participants in the user studies is shown in Figure 5(a). Though over half of the articles have no user study or a user study with less than ten participants, we observed that the proportions varied between articles focusing on different objectives—assistive and therapeutic, in Figure 5(b) and(c). There are 50% of therapeutic articles that did not conduct a user study, whereas the statistics of assistive articles is less than 30%. This could be due to stricter user study rules for medical-related research. For example, research for assistive uses can be more likely to get approved or get approved quicker by human research ethics committees or review boards than research for therapeutic uses as clinical user studies are subjected to stricter inspections. Nevertheless, as far as time is concerned, all identified articles with more than 30 participants were published in the recent three years, and the two articles with the highest number of user study participants (100) among the analyzed articles were both published in 2020 with one for assistive and one for therapeutic uses [29, 47]. As rigorous evaluations are increasingly demanded by the research community, we expect that the trend towards more user studies will continue and accelerate in the future.

In this section, we present in-depth qualitative reviews of the collected articles based on the four visual conditions categorized in Section 3: central vision, visual field, stereopsis, and color vision. We cover most of the articles that we identified but focus more on seminal highly-cited articles and research presenting novel approaches, considering the impact of the work. These reviews are intended to provide an understanding of the current state of HMD-based visual assistance and therapy and to provide insights that may support the identification of research gaps and potential future research directions, which are discussed in Section 6.

In this section we present research gaps and opportunities to promote visual impairment assistance and therapy technology to a larger population (RQ2) and discuss how technological advances can influence the adoption of HMD-based applications (RQ3).

HMD Technology Trends. The resurgence of the use of HMDs in visual impairment assistance and therapy is partially a result of recent developments and ease of access to low-cost consumer-grade commercial off-the-shelf HMDs. We believe this stimulus-response relationship will keep growing in the near future, as more specialized versions of HMDs emerge. For instance, child-friendly HMDs with adjustable IPDs and suitable sizes and weights can help solve the device limitations for children mentioned by Martin et al. [47].

Advances in display technologies should also bring progress into this research field. For instance, higher resolutions will improve the user experience of all systems we discussed above, larger FoVs can further help people with visual field loss, and robustness in harsh lighting conditions can expand the user scenarios to places like outdoor environments. Virtual Retinal Displays, which project a raster image directly onto the retina of the eye, have been around for a while and keep promising some unique advantages, such as the potential to shrink the size of HMDs and be more eye-friendly [43]. However, no recent work using this type of display has been found, indicating an opportunity for further exploration. Similarly, Light Field Displays have shown tremendous potential to correct optical aberrations in the human eye, such as near-sightedness or far-sightedness, which may at some point in the future provide a versatile solution to many vision problems, once the hardware challenges and computational complexity are overcome [34]. We noticed a trend that more and more researchers have been using on-brand HMDs since year 2016 as mentioned in Section 4. These HMDs have greatly lowered the threshold to conduct research in this area. However, we would also like to point out that customized HMDs have shown their own irreplaceable advantages—flexibility and innovation.

Looking at the stereopsis category where the majority of the therapeutic solutions were presented to users in VR, we identified very few examples where simulator sickness or its associated symptoms, such as fatigue, dizziness, and nausea, were discussed either as a factor influencing the system design or as a measure [35, 47, 61]. As simulator sickness is an ongoing issue experienced by many users of VR applications and can be dependent on a wide range of variables [23], methods should be included for measuring simulator sickness. Collecting such data may shed light on the effectiveness of the suggested VR solutions, the potential interference of simulator sickness symptoms on users’ performance, and provide guidance for future design and development. Another possible strategy to lower the likelihood of sickness is to adopt OST HMDs [71], which have recently received more attention, for instance, the OST AR version of dichoptic training by Nowak et al. [53].

More Clinical Evaluations. User study is a key feature of research related to clinical use. It could not only help evaluate the usability of the proposed methods but also help researchers identify specific needs of the users. One of the major problems that the aforementioned off-the-shelf products face is the high rate of device abandonment due to mismatches of the device to the end-user. Customized service could be useful, but sufficient user studies in early stages could help solve the issue earlier and with a lower cost. Birckhead et al. proposed a three-phase framework of VR clinical trials that in the first stage, participants are involved in designing through the format of interviews; in the second stage, participants focus on evaluating whether the method is feasible, suitable, and accountable; in the third stage, Randomized Controlled Trials are conducted to compare the outcomes of the VR methods and conventional methods [11].

As discussed in Sections 3 and 4, we are seeing an increasing percentage of work conducting user studies and the scale has been getting larger, but still, only nine articles among the screened 61 conducted studies contained user studies with more than 30 subjects; hereby, we would like to emphasize that larger sample sizes are important to assess the efficacy of the proposed methods, as noted by Lee and Kim in their work on residual amblyopia treatment [42]. Though we speculate that the growing HMD market could help increase sample sizes both with at-home participants and facilitation of parallel recruitment of patients [1], the situation can be unpredictable for the next few years considering the ongoing COVID-19 pandemic. In 2021, Radiah et al. summarized a framework for conducting remote VR studies from an online survey with 227 valid submissions and two case studies [59]. This framework could be a good reference but might not be enough for studies involving participants with visual impairments. Whether participants have the capability to complete the studies remotely needs to be considered, for example, people with USN are likely to have mobility issues and thus might require special study settings [54, 56, 68]. Also, the study effectiveness could be lowered as Fortenbacher et al. stated in their survey article on using VR as a therapy for stereo vision dysfunctions that therapy sessions conducted in an office environment with professional supervisions were significantly more effective than sessions conducted in other settings [26].

Besides the scale, we would also like to stress the importance of user study duration. Overall, we observed a few instances where multi-session or long-term study designs were considered under both therapeutic and assistive technology categories [31, 42, 73, 83]. In one of the large multi-session studies conducted by Halička et al. for amblyopia treatment, they noted that some participants might require longer sessions compared to the eight one-hour sessions utilized in their work and longer treatment duration could better indicate the effectiveness of their findings in terms of stability [31]. In the work of Deemer et al., participants used HMDs at home to understand and experience the visual assistive technology for about a week, which helped to collect more ecologically valid results [21].

More Field Studies and Use Cases. The majority of the user studies that were included in our analysis take place in laboratory settings as opposed to in the field, which may be explained by the need to provide the same experience for all participants consistently [41]. While this provides a good starting point for understanding how HMDs can be used as assistive or therapeutic technology, it does not provide us with the whole picture capturing a variety of vision-support use cases. As we move forward in this domain, more work needs to take place that involves field studies focusing on active daily life tasks in the environments that users would be typically using the devices, rather than passive observation and identification tasks. For instance, it has been emphasized that CVD introduces many obstacles to the daily life of individuals with this condition; however, we observed only one example that utilized tasks that reflect to some degree in daily life activities [66]. Although standard measurements (e.g., Ishihara test [37] for color deficiencies) are highly valuable and informative, adoption of field studies can shed more light on the needs of different populations with vision impairment and the efficacy of the presented prototypes.

Even beyond the visually impaired population, developing new use cases for vision-support HMDs may present the potential for non-disabled users to extend their vision capability through these assistive HMD technologies—especially considering environments or contexts where some of these users may experience visual challenges, for example, first-responders scenarios [6].

Novel Sensing and Processing Techniques. Our detailed review in Section 5 revealed that the modified approach was most often used for visual assistance, enhancing the user’s view through different visual enhancement methods. While the research community and industry in computer vision and machine learning are experiencing unprecedented technological achievements based on the enormous multimedia data and advanced deep learning techniques, the HMD-based assistive technology can adopt such advanced technologies to provide more context-relevant semantic information to the people who are visually impaired. Most recent articles in our review captured such a trend; for example, vision-based obstacle detection and identification using Azure Service [33] and facial expression recognition for better social interactions [40].

The visual assistance context involves not only the surrounding environment but also the users, such as understanding their behavior and intent. Overall, we observed a few examples where eye-tracking data was collected to better understand participants’ behaviors [25, 44]. For instance, Luo et al. tracked users’ gaze paths to measure their performance with and without an assistive system in a search task [44]. In several examples, researchers discussed the future use cases of eye trackers as a means to provide objective measurements, such as measuring deviation angles [47], eye movement velocity [9, 35], and accuracy [9]. Others discussed the future use of eye trackers in the context of adaptive systems [80, 81, 82], user interaction, and automatic calibration mechanisms [41, 53]. In recent years eye trackers are becoming more robust and more commonly available in many commercial HMDs, e.g., FOVE, VIVE Pro Eye, and HoloLens 2. This trend increases opportunities for realizing many of the aforementioned use cases in the context of visual impairment assistance and therapy and adopting some of the already established trends in the realm of eye-based interactions [46].

However, novel or more complex sensing and processing approach also tend to increase the demands on the underlying computing capabilities of the mobile setups, which can slow visual assistive technologies down and make them unusable from a practical point of view. HMD-based visual assistance systems need to provide very-low-latency and highly accurate feedback to users or they could impair their users’ safety. To achieve such real-time operation in this field, researchers have started employing advanced Graphics Processing Unit (GPU) and Field-Programmable Gate Array techniques [48, 69]. We see more demand in the future for such specialized high-performance processing solutions.

In this article, we presented a scoping review of previous work using HMDs as visual assistive or therapeutic tools. We provided a high-level analysis and detailed reviews of 61 related articles selected from a pool of 1,251 articles. The analysis revealed the increasing use of HMDs for visual impairments while indicating the prevalent visualization approaches (augmented, modified, virtual) with respect to the technical objectives and the users’ vision conditions. Our review captured the current state of assistive and therapeutic technologies with different types of HMDs associated with different visual conditions. We shared our insights on the research gaps, emerging trends, and future directions for the research and development of visual assistive and therapeutic technology with HMDs.

We would also like to acknowledge the limitations of this work: Our utilized search terms included commonly-seen vision conditions but could not explicitly cover all conditions that could potentially benefit from HMD assistance or therapy. We also did not include articles focusing on non-visual assistance and therapy solutions, such as audio or haptic VR/AR head-mounted devices, which led to a gap in covering blind users who could not benefit from the enhancement of the visual sense. Future work may continue this literature research by expanding the database of related articles to all uses of HMDs for the people who are visually impaired.