CopyCat is an America Sign Language (ASL) game, which uses gesture recognition technology to help... more CopyCat is an America Sign Language (ASL) game, which uses gesture recognition technology to help young Deaf children practice ASL skills. Our database of signing samples was collected from user studies of Deaf children playing a Wizard of Oz version of the game at the ...
... School of Interactive Computing, College of Computing, Georgia Institute of Technology {zahoo... more ... School of Interactive Computing, College of Computing, Georgia Institute of Technology {zahoor brashear harley.hamilton thad}@cc.gatech.edu ... During experimentation we found that rather than just using the reverse pass, verification per-formance could be improved by ...
The CopyCat game is an interactive educational adventure game to help deaf children improve their... more The CopyCat game is an interactive educational adventure game to help deaf children improve their language and memory abilities. As part of the CopyCat project, several computer-assisted language learning games have been designed, one of the games “Alien” is shown in Figure 1. Each game entails some sort of quest by the hero to collect items in order to solve a problem. In each quest, the child interacts with the hero (Iris the white cat) via American Sign Language (ASL) to warn her of a villain or identify where a hidden object is located. The child may view the tutor repeatedly if they so choose (see Figure 1). After the child talks to the hero, the child's signing is classified as correct or incorrect. If the child's signing is incorrect, a question mark appears above the hero's head to simulate misunderstanding by the hero, and the child must try again to communicate accurately. If the child's sign is correct, the hero, with the wave of a paw, “poofs” the villain, turning it into an innocuous item, and the hero continues on the quest.
Abstract We investigate the potential of the Kinect depth-mapping camera for sign language recogn... more Abstract We investigate the potential of the Kinect depth-mapping camera for sign language recognition and verification for educational games for deaf children. We compare a prototype Kinect-based system to our current CopyCat system which uses colored gloves ...
The Gesture and Activity Recognition Toolit (GART) is a user interface toolkit designed to enable... more The Gesture and Activity Recognition Toolit (GART) is a user interface toolkit designed to enable the development of gesture-based applications. GART provides an abstraction to machine learning algorithms suitable for modeling and recognizing different types of gestures. The toolkit also provides support for the data collection and the training process. In this paper, we present GART and its machine learning abstractions. Furthermore, we detail the components of the toolkit and present two example gesture recognition applications.
We perform real-time American Sign Language (ASL) phrase verification for an educational game, Co... more We perform real-time American Sign Language (ASL) phrase verification for an educational game, CopyCat, which is designed to improve deaf children's signing skills. Taking advantage of context information in the game we verify a phrase, using Hidden Markov Models (HMMs), by applying a rejection threshold on the probability of the observed sequence for each sign in the phrase. We tested this approach using 1204 signed phrase samples from 11 deaf children playing the game during the phase two deployment of CopyCat. The CopyCat data set is particularly challenging because sign samples are collected during live game play and contain many variations in signing and disfluencies. We achieved a phrase verification accuracy of 83% compared to 90% real-time performance by a sign linguist. We report on the techniques required to reach this level of performance.
CopyCat is an America Sign Language (ASL) game, which uses gesture recognition technology to help... more CopyCat is an America Sign Language (ASL) game, which uses gesture recognition technology to help young Deaf children practice ASL skills. Our database of signing samples was collected from user studies of Deaf children playing a Wizard of Oz version of the game at the ...
... School of Interactive Computing, College of Computing, Georgia Institute of Technology {zahoo... more ... School of Interactive Computing, College of Computing, Georgia Institute of Technology {zahoor brashear harley.hamilton thad}@cc.gatech.edu ... During experimentation we found that rather than just using the reverse pass, verification per-formance could be improved by ...
The CopyCat game is an interactive educational adventure game to help deaf children improve their... more The CopyCat game is an interactive educational adventure game to help deaf children improve their language and memory abilities. As part of the CopyCat project, several computer-assisted language learning games have been designed, one of the games “Alien” is shown in Figure 1. Each game entails some sort of quest by the hero to collect items in order to solve a problem. In each quest, the child interacts with the hero (Iris the white cat) via American Sign Language (ASL) to warn her of a villain or identify where a hidden object is located. The child may view the tutor repeatedly if they so choose (see Figure 1). After the child talks to the hero, the child's signing is classified as correct or incorrect. If the child's signing is incorrect, a question mark appears above the hero's head to simulate misunderstanding by the hero, and the child must try again to communicate accurately. If the child's sign is correct, the hero, with the wave of a paw, “poofs” the villain, turning it into an innocuous item, and the hero continues on the quest.
Abstract We investigate the potential of the Kinect depth-mapping camera for sign language recogn... more Abstract We investigate the potential of the Kinect depth-mapping camera for sign language recognition and verification for educational games for deaf children. We compare a prototype Kinect-based system to our current CopyCat system which uses colored gloves ...
The Gesture and Activity Recognition Toolit (GART) is a user interface toolkit designed to enable... more The Gesture and Activity Recognition Toolit (GART) is a user interface toolkit designed to enable the development of gesture-based applications. GART provides an abstraction to machine learning algorithms suitable for modeling and recognizing different types of gestures. The toolkit also provides support for the data collection and the training process. In this paper, we present GART and its machine learning abstractions. Furthermore, we detail the components of the toolkit and present two example gesture recognition applications.
We perform real-time American Sign Language (ASL) phrase verification for an educational game, Co... more We perform real-time American Sign Language (ASL) phrase verification for an educational game, CopyCat, which is designed to improve deaf children's signing skills. Taking advantage of context information in the game we verify a phrase, using Hidden Markov Models (HMMs), by applying a rejection threshold on the probability of the observed sequence for each sign in the phrase. We tested this approach using 1204 signed phrase samples from 11 deaf children playing the game during the phase two deployment of CopyCat. The CopyCat data set is particularly challenging because sign samples are collected during live game play and contain many variations in signing and disfluencies. We achieved a phrase verification accuracy of 83% compared to 90% real-time performance by a sign linguist. We report on the techniques required to reach this level of performance.
Uploads
Papers by Helene Brashear