- Authors:
- Tony Ezzat,
- Tomaso Poggio
We present MikeTalk, a text-to-audiovisual speech synthesizer which converts input text into an audiovisual speech stream. MikeTalk is built using visemes, which are a small set of images spanning a large range of mouth shapes. The visemes are acquired from a recorded visual corpus of a human subject which is specifically designed to elicit one instantiation of each viseme. Using optical flow methods, correspondence from every viseme to every other viseme is computed automatically. By morphing along this correspondence, a smooth transition between viseme images may be generated. A complete visual utterance is constructed by concatenating viseme transitions. Finally, phoneme and timing information extracted from a text-to-speech synthesizer is exploited to determine which viseme transitions to use, and the rate at which the morphing process should occur. In this manner, we are able to synchronize the visual speech stream with the audio speech stream, and hence give the impression of a photorealistic talking face.
Cited By
Radovan M, Pretorius L and Kotzé A Towards a Northern Sotho talking head Proceedings of the 5th international conference on Computer graphics, virtual reality, visualisation and interaction in Africa, (125-133)- Edge J and Maddock S Image-based Talking Heads using Radial Basis Functions Proceedings of the Theory and Practice of Computer Graphics 2003
- Giese M and Poggio T (2019). Morphable Models for the Analysis and Synthesis of Complex Motion Patterns, International Journal of Computer Vision, 38:1, (59-73), Online publication date: 30-Jun-2000.
Recommendations
Visual Speech Synthesis by Morphing Visemes
special issue on learning and vision at the center for biological and computational learning, Massachusetts Institute of TechnologyWe present MikeTalk, a text-to-audiovisual speech synthesizer which converts input text into an audiovisual speech stream. MikeTalk is built using visemes, which are a small set of images spanning a large range of mouth shapes. The visemes are acquired ...
Speech-Input Speech-Output Communication for Dysarthric Speakers Using HMM-Based Speech Recognition and Adaptive Synthesis System
Dysarthria is a motor speech disorder that causes inability to control and coordinate one or more articulators. This makes it difficult for a dysarthric speaker to utter certain speech sound units, thereby producing poorly articulated, slurred, and ...