Abstract
The paper describes a field evaluation of the automated ‘reverse directory assistance’ service presently in use in Italy in which information about names and addresses is provided by a TTS system. A simulation of the service using a natural voice was also run to get comparative data. Both services were accessed from an office room and a call-box on the street. Different evaluation metrics, such as intelligibility, task completion, task correctness, transaction success, and user's reactions were used. The aim of the work was to evaluate TTS synthesis in real world use and to make a comparison between laboratory data and data on system performance in a real application. Such a comparison suggested that in laboratory tests more attention should be dedicated to simulate more closely the conditions that can be predicted in real world use, by including important aspects that are generally not taken into consideration in laboratory tests and that are likely to have a large influence on TTS system performance such as environmental noise, prosody, and task complexity. The results also underline the importance of field evaluations to get an overall view of the usability of a service in real applications and with users who are as similar as possible to actual users.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Allen, J. (1992). Overview of text-to-speech Systems. In S. Furui and M.M. Sondhi (Eds.),Advances in Speech Signal Processing. New York, USA: Dekker.
Balestri, M., Foti, E., Nebbia, L., Oreglia, M., Salza, P.L., and Sandri, S. (1992). Comaprison of natural and synthetic speech intelligibility for a reverse telephone directory service. InProc. ICSLP' 92, Banff, Canada, pp. 559–562.
Basson, S., Yashchin, D., Silverman, K., and Kalyanswamy, A. (1991). Assessing the acceptability of automated customer name and address: A rigorous comparison of text-to-speech synthesizers. InProc. American Voice I/O Society, Atlanta, USA, pp. 200–204.
Basson, S., Yashchin, D., Kalyanswamy A., and Silverman, K. (1993). Comparing synthesizers for names and address provision: Field trial results. InProc. Eurospeech-93, Berlin, Germany, pp. 2165–2168.
Bladon, A. (1990). Evaluating the prosody of text-to-speech synthesizers. InProc. Speech Tech'90, New York, USA, pp. 215–220.
Carlson, R., Granstrom, B., and Lindstrom, A. (1989). Predicting name pronounciation for a reverse directory service. InProc. Eurospeech-89, Parigi, France, pp. 113–116.
Delogu, C., Paoloni, A., and Pocci, P. (1991). New directions in the evaluation of voice input/output systems.IEEE Journal on Selected Areas in Communications 9(4):566–573.
Delogu, C., Paoloni, A., Ridolfi, P., and Vagges, K. (1993). Intelligibility of speech produced by text-to-speech systems over the orthophonic and the telephonic channel. InProc. Eurospeech-93, Berlin, Germany, pp. 1893–1896.
Delogu, C., Paoloni, A., Ridolfi, P., and Vagges, K. (1995a). Intelligibility of speech produced by text-to-speech systems in good and telephonic conditions.ACTA ACUSTICA, 3(1):89–96.
Delogu, C., Paoloni, A., and Ridolfi, P. (1995)b. Confusions among Italian consonants in good and in telephonic conditions: Differences between text-to-speech systems and natural speech with noise. InProc. Eurospeech-95, Madrid, Spain, pp. 1109–1112.
Delogu, C., Sementina, C., and Conte, S. (1995)c. Cognitive factors affecting the perception of synthetic speech: An applicative perspective. Speech Processing Group, Fondazione Ugo Bordoni, Rome, Italy, Memo. 5C05395.
Grice, M., Vagges, K., and Hirst, D. (1991). Assessment of intonation in text-to-speech synthesis system—A pilot test in English and Italian. InProc. of Eurospeech-92, Genoa, Italy, pp. 879–882.
ITU-T (1993). Draft recommendation P.8S—Subjective performance assessment on the quality of speech output devices. Study group 12—contribution 6.
Jekosch, U. (1994). Speech intelligibility testing: On the interpretation of results.Journal of the American Voice Input/Output Society, 15:63–80.
Kalyanswamy, A. and Silverman, K.E.A. (1991). “Say what?” —Problems in preprocessing names and addresses for text-to-speech conversion. InProc. American Voice I/O Society, San Jose, CA pp. 205–209.
Levinson, E.S., Oliver, J.P., and Tschirgi, J.S. (1993). Speech synthesis in telecommunications.IEEE Communications Magazine, 46–53.
Nusbaum, H.C., Dedina, M.J., and Pisoni, D.B. (1984). Perceptual confusions of consonants in natural and synthetic CV syllables. Research on speech perception progress, Speech Research Laboratory, Psychology Department, Indiana University, Bloomington, IN. Memo. 10.
Rabiner, L.R. (1994). Applications of voice processing to telecommunications. IbdInProc. IEEE, 83(2):199–228.
Ralston, J.V., Pisoni, D.B., and Mullenix J.W. (1995). Perception and comprehension of synthetic speech. In A. Syrdal, R. Bennet, and S. Greenspan (Eds.),Applied Speech Technology. Boca Raton, Florida: CRC Press.
Rosson, M.B. and Cecala, A.J. (1986). Designing a quality voice: An analysis of listener's reactions to synthetic voices. In M. Mantei and P. Orbeton (Eds.),Human Factors in Computing Systems. CHI'86, Conference Proceedings.
Salza, P.L., Foti, E., and Oreglia, M. (1994). Intelligibility of natural speech and TTS synthesis: An experiment on reading of acronyms. InProc. American Voice I/O Society, San Jose, USA, pp. 97–106.
Schmandt, C. (1995). Voiced mail: Speech synthesis of electronic mail. In A. Syrdal, R. Bennet, and S. Greenspan (Eds.),Applied Speech Technology. Boca Raton, Florida: CRC Press.
Silverman, K., Basson, S., and Levas, S. (1990). Evaluating synthesizers performance: Is intelligibility enough? InProc. ICSLP'90, Kobe, Japan, pp. 981–984.
Spiegel, M.F. (1993). Coping with telephone directories that were never intended for synthesis applications. InProc. American Voice I/O Society, San Jose, CA, pp. 75–81.
Spiegel, M.F., Altom, M.S., Macchi, M., and Wallace, K.L. (1990). Comprehensive assessment of the telephone intelligibility of synthetic and natural speech.Speech Communication, 9:279–291.
Spiegel, M.F. and Winslow, E. (1995). Advances in the implementation of effective reverse directory (ACNA) services. InProc. American Voice I/O Society, San Jose, CA, pp. 145–152.
Van Coile, B., Leys, S., and Mortier, L. (1992). On the development of a name prouunciation system. InProc. ICSLP'92, Banff, Canada, pp. 487–490.
van Donselaar, W. (1995). The influence of speech intelligibility on the use of accentuation and given/new information in speech processing. InProc. of the XIIIth International Congress of Phonetic Sciences, Stockholm, Sweden, pp. 184–187.
Wright, J. T., Malsheen, B. J., and Peet, M. 1986. Comparison of segmental intelligibility and pronunciation accuracy for two commercial text-to-speech systems. InProc. American Voice I/O Systems Conference, pp. 235–261.
Yuschik, M., Schwab, E., and Griffith, L. (1993). Ameritech customer name and addresses service design, field trial and implemetation. InProc. American Voice I/O Society, San Jose, CA, pp. 67–73.
Yuschuk, M., Schwab, E., and Griffith, L. (1994). ACNA—The Ameritech customer name and address service.Journal of the American Voice Input/Output Society, 15:21–33.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Delogu, C., Paoloni, A., Ridolfi, P. et al. A field evaluation of the Italian “Automated reverse directory assistance” service. Int J Speech Technol 1, 161–169 (1997). https://doi.org/10.1007/BF02277197
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/BF02277197