-
Surveyor: Facilitating Discovery Within Video Games for Blind and Low Vision Players
Authors:
Vishnu Nair,
Hanxiu 'Hazel' Zhu,
Peize Song,
Jizhong Wang,
Brian A. Smith
Abstract:
Video games are increasingly accessible to blind and low vision (BLV) players, yet many aspects remain inaccessible. One aspect is the joy players feel when they explore environments and make new discoveries, which is integral to many games. Sighted players experience discovery by surveying environments and identifying unexplored areas. Current accessibility tools, however, guide BLV players direc…
▽ More
Video games are increasingly accessible to blind and low vision (BLV) players, yet many aspects remain inaccessible. One aspect is the joy players feel when they explore environments and make new discoveries, which is integral to many games. Sighted players experience discovery by surveying environments and identifying unexplored areas. Current accessibility tools, however, guide BLV players directly to items and places, robbing them of that experience. Thus, a crucial challenge is to develop navigation assistance tools that also foster exploration and discovery. To address this challenge, we propose the concept of exploration assistance in games and design Surveyor, an in-game exploration assistance tool that enhances discovery by tracking where BLV players look and highlighting unexplored areas. We designed Surveyor using insights from a formative study and compared Surveyor's effectiveness to approaches found in existing accessible games. Our findings reveal implications for facilitating richer play experiences for BLV users within games.
△ Less
Submitted 15 March, 2024;
originally announced March 2024.
-
Help Supporters: Exploring the Design Space of Assistive Technologies to Support Face-to-Face Help Between Blind and Sighted Strangers
Authors:
Yuanyang Teng,
Connor Courtien,
David Angel Rios,
Yves M. Tseng,
Jacqueline Gibson,
Maryam Aziz,
Avery Reyna,
Rajan Vaish,
Brian A. Smith
Abstract:
Blind and low-vision (BLV) people face many challenges when venturing into public environments, often wishing it were easier to get help from people nearby. Ironically, while many sighted individuals are willing to help, such interactions are infrequent. Asking for help is socially awkward for BLV people, and sighted people lack experience in helping BLV people. Through a mixed-ability research-th…
▽ More
Blind and low-vision (BLV) people face many challenges when venturing into public environments, often wishing it were easier to get help from people nearby. Ironically, while many sighted individuals are willing to help, such interactions are infrequent. Asking for help is socially awkward for BLV people, and sighted people lack experience in helping BLV people. Through a mixed-ability research-through-design process, we explore four diverse approaches toward how assistive technology can serve as help supporters that collaborate with both BLV and sighted parties throughout the help process. These approaches span two phases: the connection phase (finding someone to help) and the collaboration phase (facilitating help after finding someone). Our findings from a 20-participant mixed-ability study reveal how help supporters can best facilitate connection, which types of information they should present during both phases, and more. We discuss design implications for future approaches to support face-to-face help.
△ Less
Submitted 12 March, 2024;
originally announced March 2024.
-
StreetNav: Leveraging Street Cameras to Support Precise Outdoor Navigation for Blind Pedestrians
Authors:
Gaurav Jain,
Basel Hindi,
Zihao Zhang,
Koushik Srinivasula,
Mingyu Xie,
Mahshid Ghasemi,
Daniel Weiner,
Sophie Ana Paris,
Xin Yi Therese Xu,
Michael Malcolm,
Mehmet Turkcan,
Javad Ghaderi,
Zoran Kostic,
Gil Zussman,
Brian A. Smith
Abstract:
Blind and low-vision (BLV) people rely on GPS-based systems for outdoor navigation. GPS's inaccuracy, however, causes them to veer off track, run into unexpected obstacles, and struggle to reach precise destinations. While prior work has made precise navigation possible indoors via additional hardware installations, enabling precise navigation outdoors remains a challenge. Ironically, many outdoor…
▽ More
Blind and low-vision (BLV) people rely on GPS-based systems for outdoor navigation. GPS's inaccuracy, however, causes them to veer off track, run into unexpected obstacles, and struggle to reach precise destinations. While prior work has made precise navigation possible indoors via additional hardware installations, enabling precise navigation outdoors remains a challenge. Ironically, many outdoor environments of interest such as downtown districts are already instrumented with hardware such as street cameras. In this work, we explore the idea of repurposing street cameras for outdoor navigation, and investigate the effectiveness of such an approach. Our resulting system, StreetNav, processes the cameras' video feeds using computer vision and gives BLV pedestrians real-time navigation assistance. Our user evaluations in the COSMOS testbed with eight BLV pedestrians show that StreetNav guides them more precisely than GPS, but its performance is sensitive to lighting conditions and environmental occlusions. We discuss future implications for deploying such systems at scale.
△ Less
Submitted 30 September, 2023;
originally announced October 2023.
-
Social Wormholes: Exploring Preferences and Opportunities for Distributed and Physically-Grounded Social Connections
Authors:
Joanne Leong,
Yuanyang Teng,
Xingyu "Bruce" Liu,
Hanseul Jun,
Sven Kratz,
Yu Jiang Tham,
Andrés Monroy-Hernández,
Brian A. Smith,
Rajan Vaish
Abstract:
Ubiquitous computing encapsulates the idea for technology to be interwoven into the fabric of everyday life. As computing blends into everyday physical artifacts, powerful opportunities open up for social connection. Prior connected media objects span a broad spectrum of design combinations. Such diversity suggests that people have varying needs and preferences for staying connected to one another…
▽ More
Ubiquitous computing encapsulates the idea for technology to be interwoven into the fabric of everyday life. As computing blends into everyday physical artifacts, powerful opportunities open up for social connection. Prior connected media objects span a broad spectrum of design combinations. Such diversity suggests that people have varying needs and preferences for staying connected to one another. However, since these designs have largely been studied in isolation, we do not have a holistic understanding around how people would configure and behave within a ubiquitous social ecosystem of physically-grounded artifacts. In this paper, we create a technology probe called Social Wormholes, that lets people configure their own home ecosystem of connected artifacts. Through a field study with 24 participants, we report on patterns of behaviors that emerged naturally in the context of their daily lives and shine a light on how ubiquitous computing could be leveraged for social computing.
△ Less
Submitted 16 May, 2023;
originally announced May 2023.
-
Supporting Piggybacked Co-Located Leisure Activities via Augmented Reality
Authors:
Samantha Reig,
Erica Principe Cruz,
Melissa M. Powers,
Jennifer He,
Timothy Chong,
Yu Jiang Tham,
Sven Kratz,
Ava Robinson,
Brian A. Smith,
Rajan Vaish,
Andrés Monroy-Hernández
Abstract:
Technology, especially the smartphone, is villainized for taking meaning and time away from in-person interactions and secluding people into "digital bubbles". We believe this is not an intrinsic property of digital gadgets, but evidence of a lack of imagination in technology design. Leveraging augmented reality (AR) toward this end allows us to create experiences for multiple people, their pets,…
▽ More
Technology, especially the smartphone, is villainized for taking meaning and time away from in-person interactions and secluding people into "digital bubbles". We believe this is not an intrinsic property of digital gadgets, but evidence of a lack of imagination in technology design. Leveraging augmented reality (AR) toward this end allows us to create experiences for multiple people, their pets, and their environments. In this work, we explore the design of AR technology that "piggybacks" on everyday leisure to foster co-located interactions among close ties (with other people and pets. We designed, developed, and deployed three such AR applications, and evaluated them through a 41-participant and 19-pet user study. We gained key insights about the ability of AR to spur and enrich interaction in new channels, the importance of customization, and the challenges of designing for the physical aspects of AR devices (e.g., holding smartphones). These insights guide design implications for the novel research space of co-located AR.
△ Less
Submitted 18 March, 2023;
originally announced March 2023.
-
ImageAssist: Tools for Enhancing Touchscreen-Based Image Exploration Systems for Blind and Low Vision Users
Authors:
Vishnu Nair,
Hanxiu 'Hazel' Zhu,
Brian A. Smith
Abstract:
Blind and low vision (BLV) users often rely on alt text to understand what a digital image is showing. However, recent research has investigated how touch-based image exploration on touchscreens can supplement alt text. Touchscreen-based image exploration systems allow BLV users to deeply understand images while granting a strong sense of agency. Yet, prior work has found that these systems requir…
▽ More
Blind and low vision (BLV) users often rely on alt text to understand what a digital image is showing. However, recent research has investigated how touch-based image exploration on touchscreens can supplement alt text. Touchscreen-based image exploration systems allow BLV users to deeply understand images while granting a strong sense of agency. Yet, prior work has found that these systems require a lot of effort to use, and little work has been done to explore these systems' bottlenecks on a deeper level and propose solutions to these issues. To address this, we present ImageAssist, a set of three tools that assist BLV users through the process of exploring images by touch -- scaffolding the exploration process. We perform a series of studies with BLV users to design and evaluate ImageAssist, and our findings reveal several implications for image exploration tools for BLV users.
△ Less
Submitted 17 February, 2023;
originally announced February 2023.
-
Towards Inclusive Avatars: Disability Representation in Avatar Platforms
Authors:
Kelly Mack,
Rai Ching Ling Hsu,
Andrés Monroy-Hernández,
Brian A. Smith,
Fannie Liu
Abstract:
Digital avatars are an important part of identity representation, but there is little work on understanding how to represent disability. We interviewed 18 people with disabilities and related identities about their experiences and preferences in representing their identities with avatars. Participants generally preferred to represent their disability identity if the context felt safe and platforms…
▽ More
Digital avatars are an important part of identity representation, but there is little work on understanding how to represent disability. We interviewed 18 people with disabilities and related identities about their experiences and preferences in representing their identities with avatars. Participants generally preferred to represent their disability identity if the context felt safe and platforms supported their expression, as it was important for feeling authentically represented. They also utilized avatars in strategic ways: as a means to signal and disclose current abilities, access needs, and to raise awareness. Some participants even found avatars to be a more accessible way to communicate than alternatives. We discuss how avatars can support disability identity representation because of their easily customizable format that is not strictly tied to reality. We conclude with design recommendations for creating platforms that better support people in representing their disability and other minoritized identities.
△ Less
Submitted 3 February, 2023;
originally announced February 2023.
-
"I Want to Figure Things Out": Supporting Exploration in Navigation for People with Visual Impairments
Authors:
Gaurav Jain,
Yuanyang Teng,
Dong Heon Cho,
Yunhao Xing,
Maryam Aziz,
Brian A. Smith
Abstract:
Navigation assistance systems (NASs) aim to help visually impaired people (VIPs) navigate unfamiliar environments. Most of today's NASs support VIPs via turn-by-turn navigation, but a growing body of work highlights the importance of exploration as well. It is unclear, however, how NASs should be designed to help VIPs explore unfamiliar environments. In this paper, we perform a qualitative study t…
▽ More
Navigation assistance systems (NASs) aim to help visually impaired people (VIPs) navigate unfamiliar environments. Most of today's NASs support VIPs via turn-by-turn navigation, but a growing body of work highlights the importance of exploration as well. It is unclear, however, how NASs should be designed to help VIPs explore unfamiliar environments. In this paper, we perform a qualitative study to understand VIPs' information needs and challenges with respect to exploring unfamiliar environments, with the aim of informing the design of NASs that support exploration. Our findings reveal the types of spatial information that VIPs need as well as factors that affect VIPs' information preferences. We also discover specific challenges that VIPs face that future NASs can address such as orientation and mobility education and collaborating effectively with others. We present design implications for NASs that support exploration, and we identify specific research opportunities and discuss open socio-technical challenges for making such NASs possible. We conclude by reflecting on our study procedure to inform future approaches in research on ethical considerations that may be adopted while interacting with the broader VIP community.
△ Less
Submitted 29 November, 2022;
originally announced November 2022.
-
Exploring Immersive Interpersonal Communication via AR
Authors:
Kyungjun Lee,
Hong Li,
Muhammad Rizky Wellyanto,
Yu Jiang Tham,
Andrés Monroy-Hernández,
Fannie Liu,
Brian A. Smith,
Rajan Vaish
Abstract:
A central challenge of social computing research is to enable people to communicate expressively with each other remotely. Augmented reality has great promise for expressive communication since it enables communication beyond texts and photos and towards immersive experiences rendered in recipients' physical environments. Little research, however, has explored AR's potential for everyday interpers…
▽ More
A central challenge of social computing research is to enable people to communicate expressively with each other remotely. Augmented reality has great promise for expressive communication since it enables communication beyond texts and photos and towards immersive experiences rendered in recipients' physical environments. Little research, however, has explored AR's potential for everyday interpersonal communication. In this work, we prototype an AR messaging system, ARwand, to understand people's behaviors and perceptions around communicating with friends via AR messaging. We present our findings under four themes observed from a user study with 24 participants, including the types of immersive messages people choose to send to each other, which factors contribute to a sense of immersiveness, and what concerns arise over this new form of messaging. We discuss important implications of our findings on the design of future immersive communication systems.
△ Less
Submitted 30 November, 2022; v1 submitted 28 November, 2022;
originally announced November 2022.
-
Evaluating Temporal Patterns in Applied Infant Affect Recognition
Authors:
Allen Chang,
Lauren Klein,
Marcelo R. Rosales,
Weiyang Deng,
Beth A. Smith,
Maja J. Matarić
Abstract:
Agents must monitor their partners' affective states continuously in order to understand and engage in social interactions. However, methods for evaluating affect recognition do not account for changes in classification performance that may occur during occlusions or transitions between affective states. This paper addresses temporal patterns in affect classification performance in the context of…
▽ More
Agents must monitor their partners' affective states continuously in order to understand and engage in social interactions. However, methods for evaluating affect recognition do not account for changes in classification performance that may occur during occlusions or transitions between affective states. This paper addresses temporal patterns in affect classification performance in the context of an infant-robot interaction, where infants' affective states contribute to their ability to participate in a therapeutic leg movement activity. To support robustness to facial occlusions in video recordings, we trained infant affect recognition classifiers using both facial and body features. Next, we conducted an in-depth analysis of our best-performing models to evaluate how performance changed over time as the models encountered missing data and changing infant affect. During time windows when features were extracted with high confidence, a unimodal model trained on facial features achieved the same optimal performance as multimodal models trained on both facial and body features. However, multimodal models outperformed unimodal models when evaluated on the entire dataset. Additionally, model performance was weakest when predicting an affective state transition and improved after multiple predictions of the same affective state. These findings emphasize the benefits of incorporating body features in continuous affect recognition for infants. Our work highlights the importance of evaluating variability in model performance both over time and in the presence of missing data when applying affect recognition to social interactions.
△ Less
Submitted 7 September, 2022;
originally announced September 2022.
-
Uncovering Visually Impaired Gamers' Preferences for Spatial Awareness Tools Within Video Games
Authors:
Vishnu Nair,
Shao-en Ma,
Ricardo E. Gonzalez Penuela,
Yicheng He,
Karen Lin,
Mason Hayes,
Hannah Huddleston,
Matthew Donnelly,
Brian A. Smith
Abstract:
Sighted players gain spatial awareness within video games through sight and spatial awareness tools (SATs) such as minimaps. Visually impaired players (VIPs), however, must often rely heavily on SATs to gain spatial awareness, especially in complex environments where using rich ambient sound design alone may be insufficient. Researchers have developed many SATs for facilitating spatial awareness w…
▽ More
Sighted players gain spatial awareness within video games through sight and spatial awareness tools (SATs) such as minimaps. Visually impaired players (VIPs), however, must often rely heavily on SATs to gain spatial awareness, especially in complex environments where using rich ambient sound design alone may be insufficient. Researchers have developed many SATs for facilitating spatial awareness within VIPs. Yet this abundance disguises a gap in our understanding about how exactly these approaches assist VIPs in gaining spatial awareness and what their relative merits and limitations are. To address this, we investigate four leading approaches to facilitating spatial awareness for VIPs within a 3D video game context. Our findings uncover new insights into SATs for VIPs within video games, including that VIPs value position and orientation information the most from an SAT; that none of the approaches we investigated convey position and orientation effectively; and that VIPs highly value the ability to customize SATs.
△ Less
Submitted 30 August, 2022;
originally announced August 2022.
-
ARcall: Real-Time AR Communication using Smartphones and Smartglasses
Authors:
Hemant Bhaskar Surale,
Yu Jiang Tham,
Brian A. Smith,
Rajan Vaish
Abstract:
Augmented Reality (AR) smartglasses are increasingly regarded as the next generation personal computing platform. However, there is a lack of understanding about how to design communication systems using them. We present ARcall, a novel Augmented Reality-based real-time communication system that enables an immersive, delightful, and privacy-preserving experience between a smartphone user and a sma…
▽ More
Augmented Reality (AR) smartglasses are increasingly regarded as the next generation personal computing platform. However, there is a lack of understanding about how to design communication systems using them. We present ARcall, a novel Augmented Reality-based real-time communication system that enables an immersive, delightful, and privacy-preserving experience between a smartphone user and a smartglasses wearer. ARcall allows a remote friend (Friend) to send and project AR content to a smartglasses wearer (Wearer). The ARcall system was designed with the practical limits of existing AR glasses in mind, including shorter battery life and a reduced field of view. We conduct a qualitative evaluation of the three main components of ARcall: Drop-In, ARaction, and Micro-Chat. Our results provide novel insights for building future AR-based communication methods, including, the importance of context priming, user control over AR content placement, and the feeling of co-presence while conversing.
△ Less
Submitted 8 March, 2022;
originally announced March 2022.
-
Understanding the Role of Context in Creating Enjoyable Co-Located Interactions
Authors:
Szu-Yu,
Liu,
Brian A. Smith,
Rajan Vaish,
Andrés Monroy-Hernández
Abstract:
In recent years, public discourse has blamed digital technologies for making people feel "alone together," distracting us from engaging with one another, even when we are interacting in-person. We argue that in order to design technologies that foster and augment co-located interactions, we need to first understand the context in which enjoyable co-located socialization takes place. We address thi…
▽ More
In recent years, public discourse has blamed digital technologies for making people feel "alone together," distracting us from engaging with one another, even when we are interacting in-person. We argue that in order to design technologies that foster and augment co-located interactions, we need to first understand the context in which enjoyable co-located socialization takes place. We address this gap by surveying and interviewing over 1,000 U.S.-based participants to understand what, where, with whom, how, and why people enjoy spending time in-person. Our findings suggest that people enjoy engaging in everyday activities with individuals with whom they have strong social ties because it helps enable nonverbal cues, facilitate spontaneity, support authenticity, encourage undivided attention, and leverage the physicality of their bodies and the environment. We conclude by providing a set of recommendations for designers interested in creating co-located technologies that encourage social engagement and relationship building.
△ Less
Submitted 3 February, 2022;
originally announced February 2022.
-
Friendscope: Exploring In-the-Moment Experience Sharing on Camera Glasses via a Shared Camera
Authors:
Molly Jane Nicholas,
Brian A. Smith,
Rajan Vaish
Abstract:
We introduce Friendscope, an instant, in-the-moment experience sharing system for lightweight commercial camera glasses. Friendscope explores a new concept called a shared camera. This concept allows a wearer to share control of their camera with a remote friend, making it possible for both people to capture photos/videos from the camera in the moment. Through a user study with 48 participants, we…
▽ More
We introduce Friendscope, an instant, in-the-moment experience sharing system for lightweight commercial camera glasses. Friendscope explores a new concept called a shared camera. This concept allows a wearer to share control of their camera with a remote friend, making it possible for both people to capture photos/videos from the camera in the moment. Through a user study with 48 participants, we found that users felt connected to each other, describing the shared camera as a more intimate form of livestreaming. Moreover, even privacy-sensitive users were able to retain their sense of privacy and control with the shared camera. Friendscope's different shared camera configurations give wearers ultimate control over who they share the camera with and what photos/videos they share. We conclude with design implications for future experience sharing systems.
△ Less
Submitted 15 December, 2021;
originally announced December 2021.
-
NavStick: Making Video Games Blind-Accessible via the Ability to Look Around
Authors:
Vishnu Nair,
Jay L. Karp,
Samuel Silverman,
Mohar Kalra,
Hollis Lehv,
Faizan Jamil,
Brian A. Smith
Abstract:
Video games remain largely inaccessible to visually impaired people (VIPs). Today's blind-accessible games are highly simplified renditions of what sighted players enjoy, and they do not give VIPs the same freedom to look around and explore game worlds on their own terms. In this work, we introduce NavStick, an audio-based tool for looking around within virtual environments, with the aim of making…
▽ More
Video games remain largely inaccessible to visually impaired people (VIPs). Today's blind-accessible games are highly simplified renditions of what sighted players enjoy, and they do not give VIPs the same freedom to look around and explore game worlds on their own terms. In this work, we introduce NavStick, an audio-based tool for looking around within virtual environments, with the aim of making 3D adventure video games more blind-accessible. NavStick repurposes a game controller's thumbstick to allow VIPs to survey what is around them via line-of-sight. In a user study, we compare NavStick with traditional menu-based surveying for different navigation tasks and find that VIPs were able to form more accurate mental maps of their environment with NavStick than with menu-based surveying. In an additional exploratory study, we investigate NavStick in the context of a representative 3D adventure game. Our findings reveal several implications for blind-accessible games, and we close by discussing these.
△ Less
Submitted 2 September, 2021;
originally announced September 2021.
-
Predicting Infant Motor Development Status using Day Long Movement Data from Wearable Sensors
Authors:
David Goodfellow,
Ruoyu Zhi,
Rebecca Funke,
Jose Carlos Pulido,
Maja Mataric,
Beth A. Smith
Abstract:
Infants with a variety of complications at or before birth are classified as being at risk for developmental delays (AR). As they grow older, they are followed by healthcare providers in an effort to discern whether they are on a typical or impaired developmental trajectory. Often, it is difficult to make an accurate determination early in infancy as infants with typical development (TD) display h…
▽ More
Infants with a variety of complications at or before birth are classified as being at risk for developmental delays (AR). As they grow older, they are followed by healthcare providers in an effort to discern whether they are on a typical or impaired developmental trajectory. Often, it is difficult to make an accurate determination early in infancy as infants with typical development (TD) display high variability in their developmental trajectories both in content and timing. Studies have shown that spontaneous movements have the potential to differentiate typical and atypical trajectories early in life using sensors and kinematic analysis systems. In this study, machine learning classification algorithms are used to take inertial movement from wearable sensors placed on an infant for a day and predict if the infant is AR or TD, thus further establishing the connection between early spontaneous movement and developmental trajectory.
△ Less
Submitted 14 October, 2018; v1 submitted 7 July, 2018;
originally announced July 2018.
