Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM

Within the domain of Human-Computer Interaction, researchers have extensively delved into accessibility challenges prevalent in various categories of small-scale mobile applications [56, 63, 83, 84, 87, 107, 134], spanning domains like health [39, 61, 93, 110, 129], smart cities [95, 127, 132], and government engagement [13, 66]. While these works scrutinize distinct facets of accessibility, a recurring theme has been the conspicuous absence of descriptions for image-based components [68, 98, 112, 122]. The gravity of this issue has been explicitly recognized across studies. Notably, Park et al. [107] conducted an evaluation that assigned both severity and frequency ratings to different accessibility errors, with missing labels emerging as the most severe among various issues. Kane et al. [62] conducted an investigation into mobile device adoption and accessibility for users with visual and motor disabilities. Ross et al. [112] conducted a comprehensive analysis of image-based button labeling within a relatively large corpus of Android apps, pinpointing prevalent labeling deficiencies. Based on the above analysis of label missing issues, Chen et al. [30] proposed LabelDroid, and employed deep-learning techniques to train a model on a dataset of existing icons with labels to automatically generate labels for visually similar, unlabeled icons. To further improve the performance of label generation, Mehralian et al. [91] considered more GUI information and proposed a context-aware label generation approach, COALA, that incorporated several sources of information from the icon in generating accurate labels. The above studies have conducted in-depth research on the accessibility issues of image components. In addition, statistical data shows that the accessibility issues associated with text input are also serious, but this issue has received relatively little attention in existing research. So our study not only presents the most expansive scrutiny of text input components but also introduces an LLM-based solution to generate hint-text.

Several works also focused on detecting and rectifying accessibility gaps, particularly for users with visual impairments [11, 12, 15, 20, 35, 74, 85, 86, 113, 121]. Eler et al. [43] proposed an automated test generation model to dynamically evaluate mobile apps. Salehnamadi et al.  [113] designed a high-fidelity form of accessibility testing for Android apps, Latte, that automatically reused tests written to evaluate an app’s functional correctness to assess its accessibility as well. Zhang et al. [143] leveraged crowd-sourcing to annotate GUI elements devoid of original content descriptions. Although these researches help enhance mobile accessibility, they still can’t fix the issue of missing hint-text. This also further motivates us to design an automated method to generate hint-text that satisfies contextual semantics.

In order to help visually impaired individuals understand the meaning of GUI pages and components, researchers have attempted to use computational vision technology for GUI modeling and semantic understanding of GUI pages [16, 19, 28, 41, 44, 46, 51, 60, 70, 131, 136, 142]. Schoop et al. [115] designed a novel system that models the perceived tappability of mobile UI elements with a vision-based deep neural network and helps provide design insights with dataset-level and instance-level explanations of model predictions. He et al. [54] designed a new pre-trained UI representation model, ActionBert, aimed at utilizing visual, linguistic, and domain-specific features from user interaction traces to pre-train the universal feature representation of UI and its components. Although these studies can help users understand the GUI information of pages, they have not identified and understood the relevant information of text input components. Considering that visually impaired users cannot obtain GUI information on the page through their eyes, it is difficult to determine input requirements. To address the above challenges, this paper repairs the hint-text missing issue in input components and helps visually impaired users understand the input components.

The portability of mobile devices has led to an increasing number of visually impaired people using smartphones for daily life [57, 64, 111, 114]. Researchers have explored innovations in alternative sensory modalities like speech systems [17, 104], auditory [25, 49], and multimodal interaction [26], etc. It has opened a new era for accessible usage of the smartphone for visually impaired people [40, 99]. To assist visually impaired people in filling in the input content, researchers proposed some input solutions rely on indicating Braille mappings to enter characters [14, 18, 24, 47, 48, 67, 88, 90, 102, 123]. While input methods to assist mobile text entry have been extensively studied in the recent literature [79, 81], text entry research has focused much less on the needs of persons with vision problems. For visually impaired people, how to use the text input functionality of an app is a challenging task. They not only need to understand the intent of the input component, but also fill in the correct input.

Recently, the great success of pre-trained Large Language Models [27, 32, 37, 116, 140] in a variety of NLP tasks. Considering the powerful performance of LLM, researchers have successfully leveraged it to solve various tasks in the field of human-machine interaction and software engineering [100, 108, 135, 139, 141]. Supported by code naturalness [55], researchers applied the LLMs to code writing in different programming languages [45, 133]. A similar work QTypist [81] leveraged the LLM to generate the text inputs for triggering the next GUI page in order to improve the testing coverage of mobile testing. Different from its sole focus on text input generation to boost existing GUI testing tools, HintDroid is designed for generating the hint-text of text input component, which can help visually impaired users fill in the correct input.

LLMs were also successfully applied in research related to the HCI community  [38, 58, 59, 65, 73, 80, 128]. Stylette [65] allowed users to modify web designs with language commands and used LLMs to infer the corresponding CSS properties. Lee et al. [72] presented CoAuthor, a dataset designed to reveal GPT-3’s capabilities in assisting creative and argumentative writing. Othman et al. [103] proposed an automated accessibility issue repair method based on LLM, which utilizes LLM to repair website accessibility issues for the first time. Wang et al. [128] investigated the feasibility of enabling versatile conversational interactions with mobile UIs using a single LLM and designed prompting techniques to adapt an LLM to mobile UIs. These researches on LLM also inspires us to use LLM knowledge to understand UI and generate hint text. Unlike them, this paper addresses the challenge of missing hint-text that cannot be fixed by current accessibility repair tools. To our knowledge, it is the first time to propose using LLM’s automation to repair hint-text missing issues, helping users further understand input requirements.

In order to assess the support of popular mobile apps for vision impairment users, we randomly crawl 4,950 apps from 33 categories (150 apps per category) from Google Play [7], all of which were updated between December 2022 and June 2023, with installations ranging from 1K to 100M. We use the Application Explorer [29] (ensuring that each app can run normally on the browser) to automatically explore different screens in the app through various operations such as clicking, editing, and scrolling. During the exploration process, we get screenshots of the app GUI and their view hierarchy file (runtime front-end code file), which identifies the type of each element (such as EditText, TextView), coordinates in the screenshot, content description, and other metadata. After removing all duplicate screenshots, we ultimately collected 73,286 GUI screenshots and their corresponding View hierarchy files from 4,950 apps. In the collected data, 45,803 (63%) screenshots from 4,501 apps (91%) included text input components, which formed the dataset we analyzed in this study.

Statistical results show that out of 4,501 apps with text input components, 3,398 (76%) of them are without hint-text. Among all 45,803 screens, 30,226 (66%) had at least one text input without explicit hint-text content. This means that more than half of the text input components do not provide hint-text to users. These statistical data confirm the severity of the issue of missing hint-text, which may seriously hinder visually impaired users from using the mobile app. Then, we further analyze the issue of missing hint-text in the text input component of different categories of mobile apps. The statistical results show that the issue of hint-text missing widely exists in different types of mobile apps, and some categories of hint-text missing problems are severe. As shown in Figure 4 (a), more than 80% of the hint-text of 18 categories of apps are missing. The missing rate of hint-text in Tools, Shopping, Book and Reference and Travel and local categories, which commonly used in the daily lives of visually impaired individuals exceeds 90%, greatly affecting their daily use.

According to Google Play’s download statistics standards [7], We further analyze the hint-text missing rate for apps with different download numbers (popularity). We find that regardless of the number of downloads, the apps have a serious hint-text missing rate, of 71%-89%. Meanwhile, we also find that popular apps with high downloads also have serious issues of missing hint-text, i.e., apps with 100K-100M downloads have similar missing rates, with an average missing rate of 72%. Such hint-text missing issues with popular apps may have a greater negative impact, as these apps have a larger audience. These findings confirm the severity of the issue of missing hint-text, therefore an automated method is needed to generate missing hint-text for text input components.

The first module of our approach is to understand, analyze, organize, and extract the entities from the GUI page with the text input component. Although LLM is equipped with knowledge learned from large-scale training corpora and excels in various tasks, its performance may be significantly affected by the quality of its input, that is, whether the input can accurately describe what to ask. Therefore, we design an approach to extract and organize the GUI information.

It is usually difficult for LLM to perform well on domain-specific tasks as our hint-text generation, and a common practice would be employing the in-context learning [42, 94, 119] schema to boost the performance. It provides the LLM with examples to demonstrate what the instruction is to enable the LLM to better understand the task. Following the in-context learning schema, along with the GUI prompt for the text input as described in Section 5.1.2, we additionally provide the LLM with examples of the hint-text. To achieve this, we first build a basic example dataset of hint-text from the popular mobile apps in our motivational study. Research shows that the quality and relevance of examples can significantly affect the performance of LLM [42, 94, 119]. Therefore, based on the dataset we built, a retrieval-based example selection method (in Section 5.2.2) is designed to select the most appropriate example according to the text input and its hint-text, which further enables the LLM to learn with pertinence.

Since the LLM cannot always generate the correct hint-text as we want, we ask it to generate not only the hint-text, but also the input content that aligns with the hint-text, and use the input content to help determine whether the hint-text is appropriate or not. The design idea of HintDroid is to role-play LLM as a low vision user, providing it with GUI context and hint-text to see if it can generate input content correctly. So, we optimize the hint-text by determining whether the generated input content can trigger the next page, rather than simply triggering the next page which is what QTypist [81] did for software testing. In detail, we enter the generated content into the text input component and check whether it triggers the next page (transition). If it doesn’t trigger the next page (transition), we use the failed information to serve as the feedback and add it to the prompt to re-query the LLM.

We implement HintDroid based on the Turbo-3.5 which is released on the OpenAI website². It obtains the view hierarchy file of the current GUI page through UIAutomator [124] to extract GUI information of the text input components. HintDroid can be integrated by the automated GUI exploration tool, which automatically extracts the GUI information and generates the hint-text. When we obtain the generated hint-text from HintDroid, we further design an automated script to add its missing hint-text to the app. Specifically, as shown in Figure 7, given any app, the automated script consists of the following five steps. (1) We use Application Explorer [29] to automatically run the app and get the view hierarchy files for each page. (2) We detect GUI pages (view hierarchy files) with missing hint-text based on text input components. (3) HintDroid automatically generates the corresponding hint-text. (4) Automatically decompile the APK file of the app and retrieve the code of text input. We design a script to automatically execute to decode the APK. (5) Automatically repackage APK files. We design a script to automatically execute to encode and package the modified code into APK, and complete the repair. Please note that if it is an open-source app, we can directly modify its source code and repackage it. So HintDroid is not a dynamic interactive process, it is a one-time offline job. We also calculate that the average time for generating hint-text on each GUI page is 1.86 seconds.

To ensure the representativeness of test data, We begin with the 3,398 apps from Google Play described in Section 4, which have text input components without hint-texts (Please note that the data in this Section is not used for model training or in-context learning). To further confirm the universality and usefulness of our model, we first filter them according to the following rules. (1) At least 3 pages requiring text input components, (2) generated hint-texts can be integrated into the app, and (3) the number of activities in the app is more than 10. According to the above rules, we obtained 371 apps, and further selected the app with the highest download number of each app category as our experimental data. As shown in Figure 11, we describe the data selection process according to the PRISMA Flow Diagram[105]. We end up with 33 apps (1 app per category) with 237 text input components, and use them for the final evaluation, with details in Table 4.

We recruit 36 visually impaired users to participate in the experiment, of whom 20 are male and 16 are female. Their ages ranged from 20 to 55 years (median = 36 years). 22 participants have no residual vision, 8 have only light/dark perception and 6 have very little central vision. The participants have visual impairment for a period between 7 and 41 years. All participants use screen readers (Talkback [8]) as their primary assistive technology to use mobile apps. All of the participants have been using mobile devices for 5 years or more. Every participant receives a $100 as a reward after the experiment. At the beginning of the experiment, we ask participants to use app functions as much as possible. We also conduct a follow-up survey among the participants regarding their experiment experience.

The study involves two groups of 36 participants: the experimental group from P1 to P18 who use the mobile apps with the hint-text generated by our HintDroid, and the control group from P19 to P36 who use the app without hint-text. Each pair of participants ⟨ Px, P(x+18) ⟩ has a comparable app using experience to ensure that the experimental group has similar expertise and capability to the control group in total [36, 97, 109, 120]. Specifically, given that all participants are affiliated with the same rehabilitation and education institution, we seek the collaboration of the institution’s director to assist in the matching process. This process ensures a one-to-one ratio between the experimental and control groups, with pairings based on comparable personal competencies. The director’s extensive familiarity with the participants, stemming from over two years of close association, provided invaluable insight into their capabilities, ensuring an equitable and balanced distribution between the two study groups.

To avoid potential inconsistency, we pre-install the 36 apps in the Samsung Galaxy Note 10 with Android 9.0 OS. For each app in the experiment group, we first run Application Explorer [29] to observe the GUI page file. Then run our HintDroid to complete the missing hint-text for each app. Finally, we repackage the APK file according to the automated script in Section 5.4. Please note that to ensure the correctness of the experiment, we check that each app can still run correctly after repackaging.

We start the screen readers on the devices. and ask them to explore each app separately. The participants in the two groups need to use the 33 given mobile apps. They are required to fully explore the app and cover as many functionalities as possible. Each participant has up to 15 minutes to use a mobile app which is far more than the typical app session (71.56 seconds) [23]. Each participant can choose whether to end the exploration early based on their perceived exploration situation. Each of them conducts the experiment individually without any discussion with each other. During their exploring, all their screen interactions are recorded, based on which we derive their exploring performance.

Following previous studies [34, 76, 81], we use the following metrics to evaluate the effectiveness of HintDroid.

We present the HintDroid’s input accuracy, average activity, state coverage and the average filling time across the two groups, as shown in Table 4.

According to the visually impaired users’ feedback, all of them confirm the usefulness of our HintDroid in assisting their app exploration. They all appreciate that the hint-text generated by our HintDroid can help the understanding of the input requirement to successfully fill in the correct input, increasing the activity and state coverage. For example, “The hint-text generated by HintDroid is very helpful for us to fill in the input.”(P1), “The hints were super helpful in guiding me through the input fields. Thanks for making it clear!”(P3), “I really like the straightforward expression of these hints.”(P8), “The hints gave me exactly what I needed to know.”(P14), “They made it easy for me to fill in the blanks.”(P15). Participants express they like the hint-text, such as “Great, it’s useful. I like it!”(P2), “Yo, these hints were bang on!”(P5), “Plain and simple!”(P9), “These hints were crystal clear!”(P11), “I liked how the hints were friendly.”(P12). Participants express that our HintDroid can save their exploration time such as “Nice job! The hint-text of HintDroid saves our time.”(P17), “These hints made the whole input process so much faster.”(P13).

The participants also mention the drawback and potential improvement of our HintDroid. They hope that we can also provide some examples of correct input or provide input formats (our method also can generate the input content based on hint-text). For example, “If the hints could guide me better for specific formats like phone numbers or emails, that’d be awesome!”(P4), “Providing links or references for more info in the hints would be really helpful.”(P7), “Providing links or references for more info in the hints would be helpful.”(P10), “If these prompts can show me how much I need to input, it might be more useful”(P18). Participants also hope that HintDroid can be adjusted in real-time based on their input situation in the future. For example, “Can it be made into interactive hints? When I encounter problems, your tool can provide more details.”(P12), “If we make a mistake, it’d be awesome if the hint could help us figure out what went wrong.”(P16).

HintDroid is designed to generate the hint-text of text input component, which can help visually impaired users successfully fill in the correct input. In addition to Android, there are also many other platforms such as iOS, Web and Desktop. To conquer the market, developers tend to develop either one cross-platform app or separate native apps for each platform considering the performance benefit of native apps. Although our HintDroid is designed specifically for Android, since other platforms have similar types of information, it can also be extended to other platforms.

We conduct a small-scale experiment for another two popular platforms, and experiment on 20 iOS apps with 34 text input and 20 Web apps with 57 text input, with details on our website. Results show that HintDroid achieves the average exact match, BLEU@1, of 0.73, 0.88 for iOS apps and 0.71, 0.85 for Web apps. This further demonstrates the generality and usefulness of HintDroid, and we will conduct more thorough experiments in the future.

In addition to generating the hint-text in helping the visually impaired users successfully fill in the correct input, our HintDroid can also be applied to help end-users in their daily app usage. Given the increasing complexity of mobile apps, filling in the correct input of mobile apps is a challenging task, especially for aged users. For example, there may be too many text input components in one GUI page for aged users to try the correct input. They may stuck on one page with repetitive attempts but not working. Even normal users may stay on the input page due to unclear input requirements(without hint-text), especially for new apps.

Based on HintDroid, according to the GUI information of the text input component and knowledge from popular app datasets, the HintDroid can automatically generate hint-text and complete the “hint” fields of the text input component in the app. Therefore, HintDroid can also be integrated into UI automation tools [53, 69, 130] to provide developers with more diverse hint-text. In addition, HintDroid generates corresponding input content based on the generated hint-text, which can be used in software testing to help testers generate diverse test cases.

Our experiment shows that hint-text can help visually impaired individuals understand input needs. However, developers lack a unified style and approach when designing hint-texts, which may lead to ambiguity for visually impaired individuals. HintDroid utilizes generative models to generate hint-text, which could potentially optimize this process in the future. For example, a hint-text generation model can be customized based on the historical usage records of visually impaired individuals, or a generative model can be customized based on the usage scenarios of different types of applications, providing personalized hint-text.

Although the average metric of the hint-text generated by HintDroid exceeds 70%, there are still some inaccuracies in the generation of hint-texts. As analyzed in Section 6.2.1, different developers have different design styles when designing text-input components, some of them have little or no contextual information. All these could influence the correct generation of hint-text. We will keep improving HintDroid for generating hint-text more accurately through the information from the previous GUI page.

For the correctness and rationality of the hint-text generated by HintDroid, we only consider whether the input content generated based on the hint-text can trigger page transitions. As described in Section 5.3, not triggering page transitions is just the worst-case scenario. In addition, factors such as whether the information conveyed by hint-text is reasonable, complete, and ambiguous need to be considered. We will also incorporate these evaluation indicators in our future work.

In addition, HintDroid is currently an offline one-time automated hint-text missing issue repair approach that uses repackaging to complete hint-text injection. For some closed-source apps that use code encryption, code obfuscation, and other techniques that prevent decompilation and repackaging, we will also send the hint-text generated by HintDroid to the developers via email. Considering the low cost of the approach, i.e., the average time to generate hint-text on each GUI page is 1.86 seconds. Meanwhile, adopting repackaging technology may potentially bring some security risks, so we prefer to implement our HintDroid through real-time interaction. As suggested by the participants in 7.6, we will design a real-time interaction approach in the future and integrate it into the screen reader. It is possible to dynamically adjust the description of hint-text based on user input.