research-article

Testability and validity of WCAG 2.0: the expertise effect

Authors:

Giorgio Brajnik,

Yeliz Yesilada,

Simon HarperAuthors Info & Claims

ASSETS '10: Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility

Pages 43 - 50

https://doi.org/10.1145/1878803.1878813

Published: 25 October 2010 Publication History

Abstract

Web Content Accessibility Guidelines 2.0 (WCAG 2.0) require that success criteria be tested by human inspection. Further, testability of WCAG 2.0 criteria is achieved if 80% of knowledgeable inspectors agree that the criteria has been met or not. In this paper we investigate the very core WCAG 2.0, being their ability to determine web content accessibility conformance. We conducted an empirical study to ascertain the testability of WCAG 2.0 success criteria when experts and non-experts evaluated four relatively complex web pages; and the differences between the two. Further, we discuss the validity of the evaluations generated by these inspectors and look at the differences in validity due to expertise.

In summary, our study, comprising 22 experts and 27 non-experts, shows that approximately 50% of success criteria fail to meet the 80% agreement threshold; experts produce 20% false positives and miss 32% of the true problems. We also compared the performance of experts against that of non-experts and found that agreement for the non-experts dropped by 6%, false positives reach 42% and false negatives 49%. This suggests that in many cases WCAG 2.0 conformance cannot be tested by human inspection to a level where it is believed that at least 80% of knowledgeable human evaluators would agree on the conclusion. Why experts fail to meet the 80% threshold and what can be done to help achieve this level are the subjects of further investigation.

References

[1]

S. Abou-Zahra. Web accessibility evaluation. In S. Harper and Y. Yesilada, editors, Web Accessibility: A Foundation for Research, Human-Computer Interaction Series, chapter 7, pages 79--106. Springer, London, first edition, Sept. 2008.

[2]

F. Alonso, J. L. Fuertes, A.L. González, and L. Martínez. On the testability of wcag 2.0 for beginners. In Web for All - W4A 2010, Raleigh, USA, April 2010. ACM.

Digital Library

[3]

G. Brajnik. Comparing accessibility evaluation tools: a method for tool effectiveness. Int. Journal on Universal Access in the Information Society, 3(3-4):252--263, Oct. 2004.

Digital Library

[4]

G. Brajnik. Beyond conformance: the role of accessibility evaluation methods. In S. Hartmann, X. Zhou, and M. Kirchberg, editors, WISE 2008: 9th Int. Conference on Web Information Systems Engineering - 2nd International Workshop on Web Usability and Accessibility IWWUA08, LNCS 5176, pages 63--80, Auckland, New Zealand, Sept. 2008. Springer-Verlag. Keynote speech.

Digital Library

[5]

B. Caldwell, M. Cooper, L. G. Reid, and G. Vanderheiden. Web Content Accessibility Guidelines (WCAG) 2.0. W3C, 2008. http://www.w3.org/TR/WCAG20/.

[6]

M. Catani and D. Biers. Usability evaluation and prototype fidelity: users and usability professionals. In Proc. of the Human Factors and Ergonomics Society 42nd Annual Meeting, 1998.

[7]

K. P. Coyne and J. Nielsen. How to conduct usability evaluations for accessibility: methodology guidelines for testing websites and intranets with users who use assistive technology. http://www.nngroup.com/reports/accessibility/testing, Nielsen Norman Group, Oct. 2001.

[8]

DRC. The web: Access and inclusion for disabled people. Technical Report, Disability Rights Commission (DRC), UK, 2004.

[9]

A. D. N. Edwards. Assistive technologies. In S. Harper and Y. Yesilada, editors, Web Accessibility: A Foundation for Research, Human-Computer Interaction Series, chapter 10, pages 142--162. Springer, London, first edition, Sept. 2008.

[10]

S. L. Henry and M. Grossnickle. Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA, 2004. On-line book: www.UIAccess.com/AccessUCD.

[11]

M. Hertzum and N. E. Jacobsen. The evaluator effect during first-time use of the cognitive walkthrough technique. In Proc. of HCI International on Human-Computer Interaction: Ergonomics and User Interfaces-Volume I, pages 1063--1067, 1999.

Digital Library

[12]

M. Hertzum and N. E. Jacobsen. The evaluator effect: a chilling fact about usability evaluation methods. Int. Journal of Human-Computer Interaction, 1(4):421--443, 2001.

[13]

M. Hertzum, N. E. Jacobsen, and R. Molich. Usability inspections by groups of specialists: Perceived agreement in spite of disparate observations. In CHI 2002 Extended Abstracts, pages 662--663. ACM, ACM Press, 2002.

Digital Library

[14]

K. Hornbæk and E. Frøkjær. A study of the evaluator effect in usability testing. Human-Computer Interaction, 23(3):251--277, 2008.

[15]

N. E. Jacobsen, M. Hertzum, and B. John. The evaluator effect in usability tests. In CHI '98, pages 255--256. ACM, 1998.

Digital Library

[16]

C. Jay, D. Lunn, and E. Michailidou. End user evaluations. In S. Harper and Y. Yesilada, editors, Web Accessibility: A Foundation for Research, Human-Computer Interaction Series, chapter 8, pages 107--126. Springer, London, first edition, September 2008.

[17]

T. Lang. Comparing website accessibility evaluation methods and learnings from usability evaluation methods. http://www.peakusability.com.au/about-us/pdf/website_accessibility.pdf, Visited May 2008, 2003.

[18]

C. Ling and G. Salvendy. Effect of evaluators' cognitive style on heuristic evaluation: Field dependent and field independent evaluators. Int. Journal of Human-Computer Studies, 67(4):382--393, 2009.

Digital Library

[19]

J. Nielsen. Finding usability problems through heuristic evaluation. In Proc. of CHI 1992, pages 373--380, Monterey, CA, USA, May 1992. ACM.

Digital Library

[20]

J. Nielsen. Usability Engineering. Morgan Kaufmann, San Francisco, 1993.

Digital Library

[21]

H. Petrie and O. Kheir. The relationship between accessibility and usability of websites. In Proc. CHI 2007, pages 397--406, San Jose, CA, USA, 2007. ACM.

Digital Library

[22]

J. Rubin and D. Chisnell. Handbook of Usability Testing. Wiley, second edition, 2008.

Digital Library

[23]

J. Slatin and S. Rush. Maximum Accessibility: Making Your Web Site More Usable for Everyone. Addison-Wesley, 2003.

Digital Library

[24]

J. Thatcher, M. Burks, C. Heilmann, S. Henry, A. Kirkpatrick, P. Lauke, B. Lawson, B. Regan, R. Rutter, M. Urban, and C. Waddell. Web Accessibility: Web Standards and Regulatory Compliance. FriendsofED, 2006.

Digital Library

[25]

W3C/WAI. Requirements for WCAG 2.0 Checklists and Techniques. http://www.w3.org/TR/2003/WD-wcag2-tech-req-20030207, 2003.

[26]

Y. Yesilada, G. Brajnik, and S. Harper. How Much Does Expertise Matter? A Barrier Walkthrough Study with Experts and Non-Experts. In Proc. of 11th Int. ACM SIGACCESS Conference on Computers and Accessibility - ASSETS 2009, pages 203--210, Pittsburgh, PA, Oct. 2009.

Digital Library

Cited By

Bahram SChakraborty ARavindran SSt. Amant R(2113)Intelligent Interaction in Accessible ApplicationsA Multimodal End-2-End Approach to Accessible Computing10.1007/978-1-4471-5082-4_5(93-117)Online publication date: 26-Mar-2113
https://doi.org/10.1007/978-1-4471-5082-4_5
Laitano MOrtiz-Núñez RStable-Rodríguez Y(2024)User participation in digital accessibility evaluations: Reviewing methods and implicationsEducation for Information10.3233/EFI-240014(1-16)Online publication date: 12-Sep-2024
https://doi.org/10.3233/EFI-240014
Freire AParreira Junior PRigatto Sda Silva TSalton BNervis LRibeiro Ee Silva GAntonio Jde Oliveira Rosa JSouza JSilveira PRocha Rde Andrade S(2024)Accessibility Inspections Using the Web Content Accessibility Guidelines by Novice Evaluators: an Experience ReportProceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems10.1145/3702038.3702040(1-10)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3702038.3702040
Show More Cited By

Index Terms

Testability and validity of WCAG 2.0: the expertise effect
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. Interactive systems and tools
      1. User interface management systems
2. Social and professional topics
  1. Professional topics
    1. Computing profession
      1. Assistive technologies
  2. User characteristics
    1. People with disabilities

Recommendations

Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0

The Web Content Accessibility Guidelines (WCAG) 2.0 separate testing into both “Machine” and “Human” audits; and further classify “Human Testability” into “Reliably Human Testable” and “Not Reliably Testable”; it is human testability that is the focus ...
On the testability of WCAG 2.0 for beginners
W4A '10: Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A)

Web accessibility for people with disabilities is a highly visible area of research in the field of ICT accessibility, including many policy activities across many countries. The commonly accepted guidelines for web accessibility (WCAG 1.0) were ...
How much does expertise matter?: a barrier walkthrough study with experts and non-experts
Assets '09: Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility

Manual accessibility evaluation plays an important role in validating the accessibility of Web pages. This role has become increasingly critical with the advent of the Web Content Accessibility Guidelines (WCAG) 2.0 and their reliance on user evaluation ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASSETS '10: Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility

October 2010

346 pages

ISBN:9781605588810

DOI:10.1145/1878803

General Chair:
Armando Barreto
Florida International University, USA
,
Program Chair:
Vicki L. Hanson
University of Dundee, UK

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 October 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASSETS '10

Sponsor:

SIGACCESS

ASSETS '10: The 12th International ACM SIGACCESS Conference on Computers and Accessibility

October 25 - 27, 2010

Florida, Orlando, USA

Acceptance Rates

Overall Acceptance Rate 436 of 1,556 submissions, 28%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

42
Total Citations
View Citations
832
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)5

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bahram SChakraborty ARavindran SSt. Amant R(2113)Intelligent Interaction in Accessible ApplicationsA Multimodal End-2-End Approach to Accessible Computing10.1007/978-1-4471-5082-4_5(93-117)Online publication date: 26-Mar-2113
https://doi.org/10.1007/978-1-4471-5082-4_5
Laitano MOrtiz-Núñez RStable-Rodríguez Y(2024)User participation in digital accessibility evaluations: Reviewing methods and implicationsEducation for Information10.3233/EFI-240014(1-16)Online publication date: 12-Sep-2024
https://doi.org/10.3233/EFI-240014
Freire AParreira Junior PRigatto Sda Silva TSalton BNervis LRibeiro Ee Silva GAntonio Jde Oliveira Rosa JSouza JSilveira PRocha Rde Andrade S(2024)Accessibility Inspections Using the Web Content Accessibility Guidelines by Novice Evaluators: an Experience ReportProceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems10.1145/3702038.3702040(1-10)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3702038.3702040
Mowar PPeng YSteinfeld ABigham J(2024)Tab to Autocomplete: The Effects of AI Coding Assistants on Web AccessibilityProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688513(1-6)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3688513
Bansal NSiddharth Jain BPatel S(2024)Automating Error Identification and Evaluating Web Accessibility for Differently Abled UsersProceedings of Data Analytics and Management10.1007/978-981-99-6547-2_22(275-292)Online publication date: 3-Jan-2024
https://doi.org/10.1007/978-981-99-6547-2_22
Mateus DSouza MFreire A(2023)Accessibility of Mobile Apps for Visually Impaired Users: Problems Encountered by User Evaluation, Inspections and Automated ToolsProceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems10.1145/3638067.3638101(1-11)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3638067.3638101
Mateus DFerreira Sde Almeida Souza MFreire A(2023)Accessibility Inspections of Mobile Applications by Professionals with Different Expertise Levels: An Empirical Study Comparing with User EvaluationsHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42280-5_9(135-154)Online publication date: 25-Aug-2023
https://doi.org/10.1007/978-3-031-42280-5_9
Colley MKränzle TRukzio E(2022)Accessibility-Related Publication Distribution in HCI Based on a Meta-AnalysisExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519701(1-28)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519701
Reyes Arias JKurtzhall KPham DMkaouer MElglaly Y(2022)Accessibility Feedback in Mobile Application Reviews: A Dataset of Reviews and Accessibility GuidelinesExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519625(1-7)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519625
Akram MAli GSulaiman Aul Hassan M(2022)Accessibility evaluation of Arabic University websites for compliance with success criteria of WCAG 1.0 and WCAG 2.0Universal Access in the Information Society10.1007/s10209-022-00921-822:4(1199-1214)Online publication date: 3-Oct-2022
https://doi.org/10.1007/s10209-022-00921-8
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten