Article

Free access

Situated evaluation for cooperative systems

Authors:

Michael Twidale,

Richard BentleyAuthors Info & Claims

CSCW '94: Proceedings of the 1994 ACM conference on Computer supported cooperative work

Pages 441 - 452

https://doi.org/10.1145/192844.193066

Published: 22 October 1994 Publication History

Abstract

This paper discusses an evaluation of the MEAD prototype, a multi-user interface generator tool particularly for use in the context of Air Traffic Control (ATC). The procedures we adopted took the form of opportunistic and informal evaluation sessions with small user groups, including Air Traffic Controllers (ATCOs). We argue that informal procedures are a powerful and cost effective method for dealing with specific evaluation issues in the context of CSCW but that wider issues are more problematic. Most notably, identifying the “validity” or otherwise of CSCW systems requires that the context of use be taken seriously, necessitating a fundamental re-appraisal of the concept of evaluation.

References

[1]

Bannon, L. Use, design and evaluation: Steps towards an integration. In Proceedings of the International Workshop on the Design of CSCW and Groupware Systems. To appear in D. Shapiro, et al. (Eds.), The Design of Computer Supported Cooperative Work and Groupware Systems. Elsevier Science, Amsterdam, forthcoming.]]

[2]

Barnard, P. Applied cognitive psychology: Research for human-computer interaction. In A. Downton (FA.), Engineering the Human-Computer Interface (pp. 28- 61). McGraw-Hill, Maidenhead, 1991.]]

[3]

Bentley, R. Supporting multi-user interface development for cooperative systems. Unpublished doctoral dissertation, available as a technical report. Computing Department, Lancaster University, January 1994.]]

[4]

Bentley, R., Hughes, J. A., Randall, D., Rodden, T., Sawyer, P., Shapiro, D., and Sommerville, I. Ethnographically-informed system design for air traffic control. In Proceedings of CSCW'92 (Nov., Toronto, Ontario) ACM Press, 1992, pp. 123-129.]]

Digital Library

[5]

Bentley, R., Rodden, T., Sawyer, P. and Sommerville, I. An architecture for tailoring cooperative multi-user displays. In Proceedings of CSCW'92 (Nov., Toronto, Ontario) ACM Press, 1992, pp. 187-194.]]

Digital Library

[6]

Bodker, S., and Gronbaek, K. Cooperative prototyping: Users and designers in mutual activity. In Int. J. of Man-Machine Studies 34 (1991), 452- 478.]]

Digital Library

[7]

Cool, C., Fish, R. S., Kraut, R. E., and Lowery, C. M. Iterative design of video communication systems. In Proceedings of CSCW'92 (Nov., Toronto, Ontario), ACM Press, 1992, pp. 25-32.]]

Digital Library

[8]

Davenport, T. Process Innovation. Ernst and Young, London, 1993.]]

[9]

Downton, A. Evaluation techniques for humancomputer systems design. In A. Downton (Ed.), Engineering the Human-Computer Interface (pp. 325- 355). McGraw-Hill, Maidenhead, 1991.]]

[10]

Feigenbaum, A. V. Total Quality Control (3rd ed.). McGraw-Hill, New York, 1991.]]

[11]

Draper, S., and Wately, D. Practical methods for measuring the performance of human-computer interfaces. Distributed notes accompanying talk at the JCI HCI Summer School, Queen Mary College, Aug. 1991.]]

[12]

Grudin, J. Why CSCW applications fail: Problems in the design and evaluation of organisational interfaces. In Proceedings of CSCW'88 (Sept., Portland, Ore.), 1988, pp. 85-93.]]

Digital Library

[13]

Hammersley, M., and Atkinson, P. Ethnography Principles in Practice. Routledge, London, 1983.]]

[14]

Harper, R., Hughes, J. A., and Shapiro, D. Harmonious working and CSCW: Computer technology and air traffic control. In J. Bowers and S. Benford (Eds.), Studies in Computer Supported Cooperative Work: Theory, Practice and Design (pp. 225-234), North Holland, Amsterdam, 1991.]]

Digital Library

[15]

Heath, C., and Luff, P. Collaborative activity and technological design: Task coordination in London underground control rooms. In Proceedings of ECSCW'91 (Amsterdam), 1991, pp. 65-80.]]

Digital Library

[16]

Hughes, J. A., Shapiro, D., and Randall, D. CSCW: Discipline or paradigm? In Proceedings of ECSCW'91 (Amsterdam), 1991, pp. 309-323.]]

Digital Library

[17]

Hughes, J. A., Randall, D., and Shapiro, D. Faltering from ethnography to design. In Proceedings of CSCW'92 (Nov., Toronto, Ontario), ACM Press, 1992, pp. 115-122.]]

Digital Library

[18]

Karsenty, S., Landay, J., and Weikart, C. Inferring graphical constraints with Rockit. In Proceedings of HCI'92 (York), 1992, pp. 137-153.]]

Digital Library

[19]

Landauer, T. Relations between cognitive psychology and computer systems design. In J. M. Carroll (Ed.), Interfacing Thought, M1T Press, Cambridge, 1987.]]

Digital Library

[20]

Nielsen, J. Usability engineering at a discount. In G. Smith and M. Salvendy (Eds.), Designing and Using Human Computer Interfaces and Knowledge Based Systems (pp. 394-401). Amsterdam, North-Holland, 1989.]]

Digital Library

[21]

Randall, D. Perspectives on evaluation: Putting your money where your mouth is. Paper presented at the UK CSCW SIG (Sept., DTI, London), 1993. (Available from the author.)]]

[22]

Randall, D., Hughes, J. A., and Shapiro, D. Systems development--the fourth dimension: Perspectives on the social organisation of work. In P. Quintas (Ed.), The Social Dimensions of Software Engineering. Ellis Horwood, London and New York, 1993.]]

[23]

RSRE, Prototype Electronic Flight Strips {Video}. Malvern, 1991.]]

[24]

Self, J. Journal of Artificial intelligence and Education 4, 2-3, 1993. (Special issue on evaluation.)]]

[25]

Shapiro, D., Hughes, J. A., Randall, D., and Harper, R. Visual re-representation of database information: The flight strip in air traffic control. In M. J. Tauber, et. al. (Eds.), Cognitive Aspects of Visual Languages and Visual Interfaces (pp. 349-376). Elsevier Science, Amsterdam, 1994.]]

[26]

Suchman, L. Plans and Situated Actions. Cambridge University Press, Cambridge, 1987.]]

Digital Library

[27]

Tatar, D. G., Foster, G., and Bobrow, D. G. Design for conversation: Lessons from Cognoter. Int. J. of Man Machine Studies 34, 2, 1991, 185-210.]]

Digital Library

[28]

Twidale, M. B. Redressing the balance: The advantages of informal evaluation techniques for intelligent learning environments. In J. of A1 and Ed. 4, 2-3, 1993, 155-178.]]

[29]

Twidale, M. B., Rodden, T., and Sommerville, I. The Designers' Notepad: Supporting and understanding cooperative design. In Proceedings of ECSCW'93 (Milan, Italy), 1993, pp. 93-108.]]

Digital Library

[30]

Twidale, M. B., Rodden, T., and Sommerville, I. Developing a tool to support collaborative dialogues and graphical representation of ideas. In F. Verdejo (Ed.), Collaborative Dialogue Technologies in Distance Learning, in press.]]

[31]

Vincent, D. The Information Based Corporation. Dow Jones Irwin, 1990.]]

Cited By

Kaufhold MRiebe TBayer MReuter C(2024)‘We Do Not Have the Capacity to Monitor All Media’: A Design Case Study on Cyber Situational Awareness in Computer Emergency Response TeamsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642368(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642368
Kuntke FKaufhold MLinsner SReuter C(2023)GeoBox: design and evaluation of a tool for resilient and decentralised data management in agricultureBehaviour & Information Technology10.1080/0144929X.2023.218574743:4(764-786)Online publication date: 10-Mar-2023
https://doi.org/10.1080/0144929X.2023.2185747
Mosconi Gde Carvalho ASyed HRandall DKarasti HPipek V(2023)Fostering Research Data Management in Collaborative Research Contexts: Lessons learnt from an ‘Embedded’ Evaluation of ‘Data Story’Computer Supported Cooperative Work (CSCW)10.1007/s10606-023-09467-632:4(911-949)Online publication date: 15-May-2023
https://doi.org/10.1007/s10606-023-09467-6
Show More Cited By

Index Terms

Situated evaluation for cooperative systems
1. Human-centered computing
  1. Collaborative and social computing
  2. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Design implications for teamwork in ATC
Abstract
In air traffic control (ATC), teamwork is a key component among air traffic control operators (ATCOs) to safely direct the aircraft through the sky and on the ground. To be able to design for future ATC systems, we must understand how ATCOs work ...
Supporting air traffic control collaboration with a TableTop system
CSCW '11: Proceedings of the ACM 2011 conference on Computer supported cooperative work

Collaboration is key to safety and efficiency in Air Traffic Control. Legacy paper-based systems enable seamless and non-verbal collaboration, but trends in new software and hardware for ATC tend to separate controllers more and more, which hinders ...
Investigating Communication Infrastructure of Next Generation Air Traffic Management
ICCPS '12: Proceedings of the 2012 IEEE/ACM Third International Conference on Cyber-Physical Systems

Growing demand for use of the National Airspace System (NAS) has resulted in research and development programs to modernize the air traffic control system. The primary focus of the US FAA's Next Generation Air Transportation System (NextGen) plan is to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '94: Proceedings of the 1994 ACM conference on Computer supported cooperative work

October 1994

464 pages

ISBN:0897916891

DOI:10.1145/192844

Chairmen:
John B. Smith
Univ. of North Carolina
,
F. Don Smith
IBM Corporation
,
Thomas W. Malone
Massachusetts Institute of Technology

Copyright © 1994 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 1994

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CSCW94

Sponsor:

CSCW94: Computer-Supported Cooperative Work 94

October 22 - 26, 1994

North Carolina, Chapel Hill, USA

Acceptance Rates

CSCW '94 Paper Acceptance Rate 42 of 200 submissions, 21%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

70
Total Citations
View Citations
1,063
Total Downloads

Downloads (Last 12 months)90
Downloads (Last 6 weeks)34

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kaufhold MRiebe TBayer MReuter C(2024)‘We Do Not Have the Capacity to Monitor All Media’: A Design Case Study on Cyber Situational Awareness in Computer Emergency Response TeamsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642368(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642368
Kuntke FKaufhold MLinsner SReuter C(2023)GeoBox: design and evaluation of a tool for resilient and decentralised data management in agricultureBehaviour & Information Technology10.1080/0144929X.2023.218574743:4(764-786)Online publication date: 10-Mar-2023
https://doi.org/10.1080/0144929X.2023.2185747
Mosconi Gde Carvalho ASyed HRandall DKarasti HPipek V(2023)Fostering Research Data Management in Collaborative Research Contexts: Lessons learnt from an ‘Embedded’ Evaluation of ‘Data Story’Computer Supported Cooperative Work (CSCW)10.1007/s10606-023-09467-632:4(911-949)Online publication date: 15-May-2023
https://doi.org/10.1007/s10606-023-09467-6
Kaufhold MReuter CLudwig T(2023)Big Data and Multi-platform Social Media Services in Disaster ManagementInternational Handbook of Disaster Research10.1007/978-981-19-8388-7_172(573-593)Online publication date: 1-Oct-2023
https://doi.org/10.1007/978-981-19-8388-7_172
Berumen GFischer JBaumers M(2022)Data as a Resource for Designing Digitally Enhanced Consumer Packaged GoodsMultimodal Technologies and Interaction10.3390/mti61101016:11(101)Online publication date: 17-Nov-2022
https://doi.org/10.3390/mti6110101
Gupta MEden G(2022)The Human-Air Interface: Responding To Poor Air Quality Through Lived Experience and Digital InformationProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533563(1085-1098)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533563
Fries MLudwig T(2022)‘Why are the Sales Forecasts so low?’ Socio-Technical Challenges of Using Machine Learning for Forecasting Sales in a BakeryComputer Supported Cooperative Work (CSCW)10.1007/s10606-022-09458-zOnline publication date: 13-Dec-2022
https://doi.org/10.1007/s10606-022-09458-z
Hoffmann SLudwig TJasche FWulf VRandall D(2022)RetrofittAR: Supporting Hardware-Centered Expertise Sharing in Manufacturing Settings through Augmented RealityComputer Supported Cooperative Work (CSCW)10.1007/s10606-022-09430-x32:1(93-139)Online publication date: 30-Jun-2022
https://doi.org/10.1007/s10606-022-09430-x
Kaufhold MReuter CLudwig T(2022)Big Data and Multi-platform Social Media Services in Disaster ManagementInternational Handbook of Disaster Research10.1007/978-981-16-8800-3_172-1(1-21)Online publication date: 17-Nov-2022
https://doi.org/10.1007/978-981-16-8800-3_172-1
Berumen GFischer JBaumers M(2022)Interactions of fast‐moving consumer goods in cooking: Insights from a quantitative ethnographic studyPackaging Technology and Science10.1002/pts.271036:4(265-279)Online publication date: 26-Dec-2022
https://doi.org/10.1002/pts.2710
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents