poster

Test-Retest Reliability of the Open Card Sorting Method

Authors:

Styliani Pampoukidou,

Christos KatsanosAuthors Info & Claims

CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 330, Pages 1 - 7

https://doi.org/10.1145/3411763.3451750

Published: 08 May 2021 Publication History

Abstract

Open card sorting is a popular method for designing information architectures. However, there is a lack of empirical evidence on the validity and reliability of the method. This paper explores the test-retest reliability of open card sorting. To this end, a within-subjects study was conducted. The same participants performed open card sorts twice with a time interval of 15-20 days. Content from three website domains was used: an eshop, a travel and tourism website, and a university website. Results showed that participants provided highly similar groupings and labels between the two card sorts per domain. A high agreement of the produced navigation structures was also found. These findings provide support for the test-retest reliability of open card sorting.

References

[1]

George Adamides, Georgios Christou, Christos Katsanos, Michalis Xenos, and Thanasis Hadzilacos. 2015. Usability guidelines for the design of robot teleoperation: A taxonomy. IEEE Transactions on Human-Machine Systems 45 (2), 256–262.

[2]

William Albert and Thomas S. Tullis. 2013. Measuring the user experience: Collecting, analyzing, and presenting usability metrics (2nd ed.). Morgan Kaufmann.

[3]

Miranda G. Capra. 2005. Factor analysis of card sort data: an alternative to hierarchical cluster analysis. In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting, HFES, Santa Monica, CA, 691–695.

[4]

Brendan Cassidy, Dipti Saurabh Antani, and Janet C C. Read. 2013. Using an open card sort with children to categorize games in a mobile phone application store. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13), ACM, New York, NY, USA, 2287–2290.

[5]

Barbara S. Chaparro, Veronica D. Hinkle, and Shannon K. Riley. 2008. The usability of computerized card sorting: a comparison of three applications by researchers and end users. J. Usability Stud. 4 (1), 31–48.

Digital Library

[6]

Jacob Cohen. 1992. A power primer. Psychological Bulletin 112 (1), 155–159.

[7]

Katherine Deibel, Richard Anderson, and Ruth Anderson. 2005. Using edit distance to analyze card sorts. Expert Systems 22 (3), 129–138.

[8]

Jianming Dong, Shirley Martin, and Paul Waldo. 2001. A user input and analysis tool for information architecture. In CHI ’01 extended abstracts on Human factors in computing systems, ACM, Seattle, Washington, 23–24.

[9]

Brian Dorn and Mark Guzdial. 2010. Learning on the job: characterizing the programming knowledge and learning strategies of web designers. In Proceedings of the 28th international conference on Human factors in computing systems - CHI ’10, Atlanta, Georgia, USA, 703–712.

Digital Library

[10]

Michelle E. Harper, Florian Jentsch, Lori Rhodenizer Van Duyne, Kimberly Smith-Jentsch, and Alicia D. Sanchez. 2002. Computerized card sort training tool: Is it comparable to manual card sorting? In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, SAGE Publications Inc, 2049–2053.

[11]

William Hudson. 2013. Card sorting. In The Encyclopedia of Human-Computer Interaction, 2nd Ed., Mads Soegaard and Rikke Friis Dam (eds.). Interaction Design Foundation. Retrieved from https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed

[12]

Rebecca Jeong and Sonia Chiasson. 2020. “Lime”, “open lock”, and “blocked”: children's perception of colors, symbols, and words in cybersecurity warnings. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), ACM, New York, NY, USA, 14 pages.

[13]

Karin Kappel, Martin Tomitsch, Thomas Költringer, and Thomas Grechenig. 2006. Developing user interface guidelines for DVD menus. In CHI ’06 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’06), ACM, New York, NY, USA, 177–182.

[14]

Christos Katsanos, Nikolaos Tselios, and Nikolaos Avouris. 2008. AutoCardSorter: designing the information architecture of a web site using latent semantic analysis. In Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, CHI 2008, ACM, Florence, Italy, 875–878.

Digital Library

[15]

Christos Katsanos, Nikolaos Tselios, and Nikolaos Avouris. 2008. Automated semantic elaboration of web site information architecture. Interact Comput. 20 (6), 535–544.

Digital Library

[16]

Christos Katsanos, Nikolaos Tselios, Nikolaos Avouris, Stavros Demetriadis, Ioannis Stamelos, and Lefteris Angelis. 2019. Cross-study reliability of the open card sorting method. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19), ACM, New York, NY, USA, LBW2718:1-LBW2718:6.

Digital Library

[17]

Christos Katsanos, Nikolaos Tselios, Jorge Goncalves, Tomi Juntunen, and Vassilis Kostakos. 2014. Multipurpose public displays: Can automated grouping of applications and services enhance user experience? International Journal of Human-Computer Interaction 30 (3), 237–249.

[18]

Christina Kelley, Bongshin Lee, and Lauren Wilcox. 2017. Self-tracking for mental wellness: understanding expert perspectives and student experiences. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), ACM, New York, NY, USA, 629–641.

Digital Library

[19]

Alexandros Liapis, Christos Katsanos, Michalis Xenos, and Theofanis Orphanoudakis. 2019. Effect of personality traits on UX evaluation metrics: a study on usability issues, valence-arousal and skin conductance. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19), ACM, New York, NY, USA, LBW2721:1-LBW2721:6.

Digital Library

[20]

Nathan Mantel. 1967. The detection of disease clustering and a generalized regression approach. Cancer Res. 27 (2), 209–220.

[21]

Georgios Melissourgos and Christos Katsanos. 2020. CardSorter: Towards an open source tool for online card sorts. In Proceedings of the 24th Panhellenic Conference on Informatics, ACM, New York, NY, USA, 77-81.

Digital Library

[22]

Ather Nawaz. 2012. A Comparison of card-sorting analysis methods. In Proceedings of the 10th Asia Pacific Conference on Computer-Human Interaction, APCHI ’12, ACM Press, 583–592.

[23]

Jacob Nielsen. 2004. Card Sorting: How many users to test. Alertbox. Retrieved from http://www.useit.com/alertbox/20040719.html

[24]

Sione Paea and Ross Baird. 2018. Information Architecture (IA): Using multidimensional scaling (MDS) and K-Means clustering algorithm for analysis of card sorting Data. J. Usability Stud. 13 (3), 138–157.

Digital Library

[25]

Celeste Paul. 2008. A modified Delphi approach to a new card sorting methodology. J. Usability Stud. 4 (1), 7–30.

Digital Library

[26]

Celeste Paul. 2014. Analyzing card-sorting data using graph visualization. J. Usability Stud. 9 (3), 87–104.

Digital Library

[27]

Helen Petrie, Christopher Power, Paul Cairns, and Cagla Seneler. 2011. Using card sorts for understanding website information architectures: technological, methodological and cultural Issues. In Human-Computer Interaction – INTERACT 2011 (Lecture Notes in Computer Science), Springer Berlin Heidelberg, 309–322.

[28]

Carol Righi, Janice James, Michael Beasley, Donald L. Day, Jean E. Fox, Jennifer Gieber, Chris Howe, and Laconya Ruby. 2013. Card sort analysis best practices. J. Usability Stud. 8 (3), 69–89.

Digital Library

[29]

Louis Rosenfeld, Peter Morville, and Jorge Arango. 2015. Information architecture: For the Web and beyond (4 edition ed.). O'Reilly Media, Sebastopol, CA.

[30]

Hasti Seifi, Michael Oppermann, Julia Bullard, Karon E. MacLean, and Katherine J. Kuchenbecker. 2020. Capturing experts’ mental models to organize a collection of haptic devices: affordances outweigh attributes. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), ACM, New York, NY, USA, 13 pages.

[31]

Donna Spencer. 2009. Card sorting: designing usable categories. Rosenfeld Media, Brooklyn, N.Y.

[32]

Donna Spencer. 2010. A practical guide to information architecture. Five Simple Steps.

[33]

Thomas S. Tullis. 1985. Designing a menu-based interface to an operating system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’85), ACM, New York, NY, USA, 79–84.

Digital Library

[34]

Thomas Tullis and Larry Wood. 2004. How can you do a card-sorting study with LOTS of cards? In Usability Professionals Association (UPA) 2004 Conference, Minneapolis, MN.

[35]

Thomas Tullis and Larry Wood. 2004. How many users are enough for a card-sorting study? In Usability Professionals Association (UPA) 2004 Conference, Minneapolis, MN.

[36]

Ian H. Witten and Eibe Frank. 1999. Data mining: Practical machine learning tools and techniques with java implementations (1st ed.). Morgan Kaufmann.

[37]

Jed Wood and Larry Wood. 2008. Card sorting: current practices and beyond. J. Usability Stud. 4 (1), 1–6.

Digital Library

[38]

Panayiotis Zaphiris, Mariya Ghiawadwala, and Shabana Mughal. 2005. Age-centered research-based web design guidelines. In CHI ’05 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’05), ACM, New York, NY, USA, 1897–1900.

[39]

Panayiotis Zaphiris and Sri Kurniawan. 2001. Senior-centered design of health information architecture. In CHI ’01 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’01), ACM, New York, NY, USA, 465.

Cited By

Putjorn TPutjorn P(2023)Augmented Imagination: Exploring Generative AI from the Perspectives of Young Learners2023 15th International Conference on Information Technology and Electrical Engineering (ICITEE)10.1109/ICITEE59582.2023.10317680(353-358)Online publication date: 26-Oct-2023
https://doi.org/10.1109/ICITEE59582.2023.10317680
Pechlevanoudis CZilidis GKatsanos C(2023)How Many Participants Do You Need for an Open Card Sort? A Case Study of E-commerce WebsitesHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42293-5_7(80-89)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-42293-5_7
Katsanos CChristoforidis VDemertzi C(2023)Task-Based Open Card Sorting: Towards a New Method to Produce Usable Information ArchitecturesHuman Interface and the Management of Information10.1007/978-3-031-35132-7_5(68-80)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-35132-7_5
Show More Cited By

Recommendations

CardSorter: Towards an Open Source Tool for Online Card Sorts
PCI '20: Proceedings of the 24th Pan-Hellenic Conference on Informatics

Structural design of shared information environments is rather challenging. Card sorting constitutes a valuable method towards this end. However, there is a lack of free software tools for conducting card sorts, especially from a distance, and analysing ...
Validity of the Open Card Sorting Method for Producing Website Information Structures
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

When it comes to creating website information structures, open card sorting is the main approach used. However, scientific research for the method's validity is lacking. This paper explores the validity of open card sorting for website structural ...
Effect of Sense of Direction on Open Card Sorts for Websites
CHI Greece 2021: CHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI Chapter

Card sorting is one widely used method to design website information architectures based on an understanding of their users’ mental models. This paper studies the effect of participants’ sense of direction on card sorting results. To this end, a two-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

2965 pages

ISBN:9781450380959

DOI:10.1145/3411763

Editors:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
260
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)4

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Putjorn TPutjorn P(2023)Augmented Imagination: Exploring Generative AI from the Perspectives of Young Learners2023 15th International Conference on Information Technology and Electrical Engineering (ICITEE)10.1109/ICITEE59582.2023.10317680(353-358)Online publication date: 26-Oct-2023
https://doi.org/10.1109/ICITEE59582.2023.10317680
Pechlevanoudis CZilidis GKatsanos C(2023)How Many Participants Do You Need for an Open Card Sort? A Case Study of E-commerce WebsitesHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42293-5_7(80-89)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-42293-5_7
Katsanos CChristoforidis VDemertzi C(2023)Task-Based Open Card Sorting: Towards a New Method to Produce Usable Information ArchitecturesHuman Interface and the Management of Information10.1007/978-3-031-35132-7_5(68-80)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-35132-7_5
Ntouvaleti MKatsanos C(2022)Validity of the Open Card Sorting Method for Producing Website Information StructuresExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519734(1-7)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519734
Goundar MKumar BAli A(2022)Development of Usability Guidelines: A Systematic Literature ReviewInternational Journal of Human–Computer Interaction10.1080/10447318.2022.214100940:5(1298-1316)Online publication date: 9-Nov-2022
https://doi.org/10.1080/10447318.2022.2141009
Katsanos CZafeiriou GLiapis A(2022)Effect of Self-efficacy on Open Card Sorts for WebsitesHuman Interface and the Management of Information: Visual and Information Design10.1007/978-3-031-06424-1_7(75-87)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-06424-1_7
Greifeneder EBressel P(2022)Hybrid Digital Card Sorting: New Research Technique or Mere Variant?Information for a Better World: Shaping the Global Future10.1007/978-3-030-96960-8_4(50-67)Online publication date: 28-Feb-2022
https://dl.acm.org/doi/10.1007/978-3-030-96960-8_4
Zafeiriou GKatsanos CLiapis A(2021)Effect of Sense of Direction on Open Card Sorts for WebsitesCHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI Chapter10.1145/3489410.3489416(1-8)Online publication date: 25-Nov-2021
https://dl.acm.org/doi/10.1145/3489410.3489416

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents