extended-abstract

DesertWoZ: A Wizard of Oz Environment to Support the Design of Collaborative Conversational Agents

Authors:

Hamish Stening,

Patrick Nalepka,

Erik D. Reichle,

Christopher J. Best,

Deborah Richards,

Michael J. RichardsonAuthors Info & Claims

CSCW'22 Companion: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing

Pages 188 - 192

https://doi.org/10.1145/3500868.3559711

Published: 08 November 2022 Publication History

Abstract

The use of conversational agents to support team-based training in virtual environments is a nascent area of research within the Computer Supported Collaborative Work (CSCW) and Conversational User Interface (CUI) communities. To support this research, we developed a Wizard of Oz (WoZ) system enabling experimenters to play the role of the conversational agent who must work with a team of three human participants to herd together a group of autonomous robots (target agents) within a virtual task environment. The experimenter can select from one of several tailored responses. The aim of this environment is to determine whether the identified responses facilitate effective team performance and to collect training data to build a conversational agent capable of engaging in team based virtual learning environments.

References

[1]

Julia Cambre, Jessica Colnago, Jim Maddock, Janice Tsai, and Jofish Kaye. 2020. Choice of Voices: A Large-Scale Evaluation of Text-to-Speech Voice Quality for Long-Form Content. Conference on Human Factors in Computing Systems - Proceedings 20, (April 2020).

Digital Library

[2]

Heloisa Candello, Mauro Pichiliani, and Claudio Pinhanez. Making Conversational Robots Learn from Children in a Collaborative Museum Exhibition; Making Conversational Robots Learn from Children in a Collaborative Museum Exhibition. Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing.

Digital Library

[3]

Anders Drachen and Jonas Heide Smith. 2008. Player talkthe functions of communication in multplayer role-playing games. Computers in Entertainment (CIE) 6, 4 (December 2008).

Digital Library

[4]

Jamie C. Gorman, Nancy J. Cooke, and Polemnia G. Amazeen. 2010. Training adaptive teams. Hum Factors 52, 2 (April 2010), 295–307.

[5]

J. F. Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS) 2, 1 (January 1984), 26–41.

Digital Library

[6]

Seunghwan Kim, Jinkyun Park, Sangyong Han, and Hanjeom Kim. 2010. Development of extended speech act coding scheme to observe communication characteristics of human operators of nuclear power plans under abnormal conditions. J Loss Prev Process Ind 23, (July 2010), 539–548.

[7]

June Lee. 2019. Celebrating 100,000 Alexa Skills -100,000 Thank Yous to You.

[8]

Martin Porcheron, Joel E. Fischer, and Michel Valstar. 2020. NottReal: A Tool for Voice-based Wizard of Oz studies. ACM International Conference Proceeding Series (July 2020).

Digital Library

[9]

Deborah Richards and Patrina Caldwell. 2018. Improving Health Outcomes Sooner Rather Than Later via an Interactive Website and Virtual Specialist. IEEE J Biomed Health Inform 22, 5 (September 2018), 1699–1706.

[10]

Joseph Seering, Michal Luria, Geoff Kaufman, and Jessica Hammer. 2019. Beyond Dyadic Interactions: Considering Chatbots as Community Members. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), Association for Computing Machinery, New York, NY, USA, 1–13.

Digital Library

[11]

James Simpson and Cassandra Crone. 2021. Can Alexa be a Team Player? Proceedings of the 3rd Conference on Conversational User Interfaces.

Digital Library

[12]

James Simpson, Franziska Gaiser, Miroslav MacÍk, and Timna Breßgott. 2020. Daisy: A Friendly Conversational Agent for Older Adults. ACM International Conference Proceeding Series (July 2020).

Digital Library

[13]

James Simpson, Patrick Nalepka, Cassandra L. Crone, Rachel W. Kallen, Mark Dras, Erik D. Reichle, Simon G. Hosking, Christopher J. Best, Deborah Richards, and Michael J. Richardson. 2021. Tip of the Finger or Tip of the Tongue? The Effects of Verbal Communication on Online Multi-Player Team Performance. Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW (October 2021), 175–178.

Digital Library

[14]

Anoop K. Sinha, Scott R. Klemmer, Jack Chen, James A. Landay, and Cindy Chen. 2001. SUEDE: Iterative, informal prototyping for speech interfaces. Conference on Human Factors in Computing Systems - Proceedings (2001), 203–204.

Digital Library

[15]

Matthew J. Smith, Emily J. Ginger, Katherine Wright, Michael A. Wright, Julie Lounds Taylor, Laura Boteler Humm, Dale E. Olsen, Morris D. Bell, and Michael F. Fleming. 2014. Virtual reality job interview training in adults with autism spectrum disorder. J Autism Dev Disord 44, 10 (October 2014), 2450–2463.

[16]

Travis J. Wiltshire, Jonathan E. Butner, and Stephen M. Fiore. 2018. Problem-Solving Phase Transitions During Team Collaboration. Cogn Sci 42, 1 (January 2018), 129–167.

[17]

Zoom. 2020. 90-Day Security Plan Progress Report: April 22. Retrieved December 20, 2020 from https://blog.zoom.us/90-day-security-plan-progress-report-april-22

[18]

Amazon-Powered Smart Speakers See Big 2021 Shipment Decline. Retrieved April 30, 2022 from https://au.pcmag.com/smart-home/92621/amazon-powered-smart-speakers-see-big-2021-shipment-decline

Cited By

Nalepka PPatil GKallen RRichardson M(2025)Human-inspired strategies for controlling swarm systemsPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences10.1098/rsta.2024.0147383:2289Online publication date: 30-Jan-2025
https://doi.org/10.1098/rsta.2024.0147
Rayan JKanetkar DGong YYang YPalani SXia HDow S(2024)Exploring the Potential for Generative AI-based Conversational Cues for Real-Time Collaborative IdeationProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656184(117-131)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656184

Index Terms

DesertWoZ: A Wizard of Oz Environment to Support the Design of Collaborative Conversational Agents
1. Applied computing
  1. Education
    1. Collaborative learning
2. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing

Recommendations

A Wizard or a Fool? Initial Assessment of a Wizard of Oz Agent Supporting Collaborative Virtual Environments
HAI '22: Proceedings of the 10th International Conference on Human-Agent Interaction

A popular technique for evaluating envisioned AI agents without committing significant engineering resources to implement such an agent is the Wizard of Oz (WoZ) methodology. Thus far, many of the WoZ studies concerned with the design of conversational ...
Wizard of Oz experiments and companion dialogues
BCS '10: Proceedings of the 24th BCS Interaction Specialist Group Conference

Novel speech systems such as the conversational agents being developed by the Companions Project (www.companions-project.org) can be simulated using the Wizard of Oz methodology. In this approach technologies that are not yet ready for testing by people ...
Wizard of Oz experiments for companions
BCS-HCI '09: Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology

Wizard of Oz experiments allow designers and developers to see the reactions of people as they interact with to-be-developed technologies. At the Centre for Interaction Design at Edinburgh Napier University we are developing a Wizard of Oz system to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW'22 Companion: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing

November 2022

318 pages

ISBN:9781450391900

DOI:10.1145/3500868

Editors:
Gary Hsieh
University of Washington, Seattle, USA
,
Anthony Tang
University of Toronto, Toronto, Canada
,
Morgan G. Ames
University of California Berkeley, Berkeley, USA
,
Sharon Ding
University of Glasgow, Glasgow, UK
,
Susan Fussell
Cornell University, Ithaca, USA
,
Vera Liao
Microsoft Research, Montreal, Canada
,
Andrés Monroy-Hernández
Princeton University, Princeton, USA
,
Sean Munson
University of Washington, Seattle, USA
,
Irina Shklovski
University of Copenhagen, Copenhagen, Denmark
,
John Tang
Microsoft Research, Sunnyvale, USA

Copyright © 2022 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2022

Check for updates

Author Tags

Qualifiers

Extended-abstract
Research
Refereed limited

Funding Sources

Australian Department of Defence

Conference

CSCW '22

Sponsor:

SIGCHI

CSCW '22: Computer Supported Cooperative Work and Social Computing

November 8 - 22, 2022

Virtual Event, Taiwan

Acceptance Rates

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
308
Total Downloads

Downloads (Last 12 months)90
Downloads (Last 6 weeks)8

Reflects downloads up to 01 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nalepka PPatil GKallen RRichardson M(2025)Human-inspired strategies for controlling swarm systemsPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences10.1098/rsta.2024.0147383:2289Online publication date: 30-Jan-2025
https://doi.org/10.1098/rsta.2024.0147
Rayan JKanetkar DGong YYang YPalani SXia HDow S(2024)Exploring the Potential for Generative AI-based Conversational Cues for Real-Time Collaborative IdeationProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656184(117-131)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656184

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten