1 Introduction

The development of technologies, including medical technologies, medical care and a general increase in prosperity contribute to a greater detection of diseases and the effectiveness of their treatment. Modern medical technologies give us the possibility to save lives of patients after serious injuries and extend lives of people suffering from degenerative diseases, what influences the increase in the number of persons with special needs [1,2,3,4,5]. WHO reports that over a billion people in the world are affected by disability, which constitutes about 15% of the society [6]. In order to improve the quality of life of the persons with impairments it is crucial to introduce the concept of accessibility to information and communications technologies (ICTs) [7]. The goal of accessibility in ICT is to ensure that all electronic information, resources and services are accessible to people using alternative ways of human-computer interaction [8]. In the context of digital accessibility, one can also come across the relatively “young” term “users with special needs”. This concept means that a person who, due to internal or external characteristics, or due to the circumstances in which one is, must take additional measures to overcome barriers so that one can use digital resources on an equal basis with others. This approach eliminates all divisions into groups in terms of legal requirements, suggesting the need to develop facilities for all persons in need (Design for All) [9].

Massive open online course (MOOC) is a course available to a large number of participants that has online access via the Web. The name MOOC defines three features of the course. Massiveness suggests the appropriate number of participants, often referred to as the level of 2000. Openness is defined as accessibility to a huge number of participants around the world, regardless of their location, age, income, ideology or level of education, without any prerequisites or access fees. The third feature, ’online’, is the availability of courses from anywhere in the world via an Internet connection to ensure synchronous and asynchronous interaction between participants [10]. The method of learning described above, apart from many advantages, has a number of drawbacks. The most frequently raised comments concern the lack of feedback, low level of interaction with the teacher or a large percentage of people who did not complete the courses, but also the quality of the materials created. There is also a lot of talk about possible barriers–linguistic, economic or technical [11,12,13]. These issues are even more visible in case of users with disability. The aim of this study is to verify compliance with the criteria for the accessibility of massive online open courses (MOOCs) on the Coursera platform for users with disabilities, with particular emphasis on people with visual and motor dysfunction. The review was carried out against the guidelines developed by the W3C consortium, in particular the web content accessibility guidelines (WCAG) 2.1.

2 Related works

Massive open online course (MOOC) services offer people all over the world access to education regardless of their age, gender, culture, location and physical fitness. Using new technologies followed by new functionalities makes MOOCs more attractive and accessible to the students [14]. According to the research performed by the Open University in UK [15], one out of seven users of the MOOC courses is impaired. This means that online learning makes education more reachable to people who have limited mobility because of their disability [16]. The application of WCAG principles also brings wide benefits to the elderly [17]. It was found that participation of people over the age of 66 in some courses is over 15%, and even around 40% of participants are over the age of 56. Taking into consideration the variety of MOOC users, it is very important to ensure high level of the accessibility of MOOC platforms as it has a significant impact on the attitude and motivation of the students [18, 19]. Research by Sanchez-Gordon et al. from 2013 [20] included analysis of five selected courses that were tested for compliance with WCAG 2.0. In their research from 2016 they analyzed compliance of the EdX platform with accessibility guidelines and provided the authors of the course content with a set of recommendations about improving the quality of the courses [21]. Subsequent studies were described in an article by Al-Mouh et al. [22] and concerned the analysis of the availability of selected courses by two blind people and a specialist in the field of accessibility using the blindfold. As in the previous case, the tests were based on WCAG 2.0. In the same year, the results of their research were published by M. Bohnsack and S. Puhl. They assessed the accessibility of five platforms offering MOOCs (Coursera, Udacity, edX, OpenCourseWorld, Iversity), in which the blind users participated [23]. The results of similar studies were published in 2017 by Królak et al. [24]. The research included seven users with vision impairments, all of whom used screen readers, and two additional people using Braille-based technologies. Each participant completed the 4-week course and after the completion submitted one’s comments. Analysis of three MOOC platforms, edX, Coursera and Khan Academy was presented in [31]. This research also concentrated on learners with visual disability and was composed of user studies with three volunteers and heuristic evaluation performed by five experts and based on WCAG 2.1 recommendation. The main problems identified were frequent lack of alternative text for multimedia content, hard to access drop-down menus and no bypass buttons allowing to avoid listening to repetitive information, located on the page, read by screen readers. Coursera, Canvas and edX platforms were also evaluated in [34] using Universal Design for Learning (UDL) framework. The results showed that the UDL principles related to providing multiple means of engagement and multiple means of representation were partly achieved in Canvas and edX and largely achieved in Coursera. The principle concerning providing multiple means of action and expression was only partially achieved in all the three platforms. The main problem related to UDL guidelines in examined MOOCs was the lack of variable methods for response and navigation. All of the described studies relate to the courses offered, among others, by the Coursera platform, indicating significant accessibility issues. In recent studies, the accessibility of MOOCs has also been examined from the perspective of providers and stakeholders [32, 33]. The results show that MOOC providers are mostly aware of the users with special needs; however, their priority is legislation over the accessibility preferences of the disabled learners. Identified problems that were pointed out include the use of third-party software and technologies that often do not comply with accessibility requirements and lack of information collected by the providers about the MOOC users and their accessibility needs.

In the literature, we can find several approaches to testing the accessibility of MOOC platforms. A classification into five groups was presented in [25]: (1) user-based empirical studies, (2) heuristic evaluations, (3) online surveys, (4) accessibility in terms of legal framework, and (5) MOOC in terms of teaching accessibility. Another method is automatic accessibility audit using online tools, such as AChecker of WAVE, which is often reported in the literature [25,26,27]. Research studies using user-based approach usually include small-size testing groups representing only one type of disability [22, 23], while the heuristic approach tends to combine automatic tools and expert evaluation [20, 21, 23, 29, 30]. The heuristic approach should include various aspects of accessibility evaluations, as described in [28], including: (1) technical accessibility evaluation, (2) user experience (UX) evaluation, (3) quality evaluation, and (4) learning design evaluation. In the presented study the heuristic evaluation is proposed, being a combination of used-based evaluation and automatic audit. Technical accessibility is verified in addition to user experience tests with specific scenario performed by persons with different disability types.

3 Methods

Coursera is a global learning platform that was set up in 2012. It offers over 7000 free and paid courses that are delivered not only in English but also in many other languages. Due to its popularity Coursera has been assessed for its accessiblity by many researchers since 2013 [20, 22,23,24, 30, 31, 34]. It gives a possibility to verify how the quality of the courses, their accessibility and conformance with WCAG principles change over the years. For these reasons Coursera was chosen as a test platform in the presented study. The research problem was defined as follows:

RP: Are MOOCs on the Coursera platform accessible to users with disabilities?

In order to answer the above research question, the following research hypothesis was formulated:

RH: Massive online open courses on the coursera platform meet the criteria of the web content accessibility guidelines (WCAG) 2.1.

3.1 Methods and Research Tools

In the presented study, a qualitative approach was used to identify problems and barriers faced by people with visual and / or movement impairment in MOOCs. The study participants could choose any course available on the coursera platform. Such courses typically last several weeks, divided into weekly modules with learning materials, so the users have submitted their observations and comments in weekly reports after completing the selected course. Thus, the methods used to collect the data include case study elements and the analysis of the weekly reports. Users carried out the survey in November and December 2020, and their applications were obtained thanks to the cooperation with the office for persons with disabilities of the Lodz University of Technology. All users who volunteered for the study were informed about the goals of the research, requirements and its anonymous nature. Nine out of ten participants were not familiar with WCAG recommendation and one person with light vision impairment was an expert in website accessibility. Participants were asked to complete an initial questionnaire to collect demographic data and information about the assistive technologies they use or previous experience with online courses. Then, users wrote down their observations in a specially prepared report in the form of a text document, containing predefined open-ended questions. These questions allowed the testers to participate in the study without the knowledge of WCAG, in an accessible form, paying attention to the most important aspects of this standard. The questionnaire was divided into six sections: (1) general information about the participant’s disability and used technologies, (2) non-text content quality with questions related to Perceivable principle, (3) course website operation with questions related to WCAG Operable principle, (4) course content with questions related to Understandable principle, (5) technical problems with questions related to Robust principle, (6) additional comments and remarks. The answers were evaluated by two accessibility experts. The success criteria selected for accessibility analysis of Coursera platform are presented in Table 1.

Table 1 Success criteria required to test the website
Full size table

Taking into account that barriers faced by people with disabilities may occur both in the courses themselves and in the user interface, it was decided to limit the scope of the study to only one MOOC provider. Taking into account aspects such as popularity, long presence on the market, extensive cooperation with academic units and the number of available research materials, the Coursera platform was selected. Following the methodologies used for assessing the accessibility of MOOCs [35] the study also used a simplified automatic accessibility audit to detect additional non-compliance with the WCAG 2.1. guidelines what complemented the information received from the study participants and present the results from a broader perspective. For this purpose, three generally available tools were used: (1) markup validation service by W3C which allows to detect possible errors in the code, (2) PowerMapper Validator for checking the compliance with WCAG 2.1 guidelines, and (3) Color Contrast Analyzer browser extension that examines the color contrast. The last two tools were developed at NC State University. Markup Validation Service is a free solution that allows to check the compliance of the HTML code with HTML or XHTML standards and the correctness of CSS declarations. Its main goal is to verify browser compatibility and site usability. Markup validation tools return the general information about the technical quality of web pages, however manual verification by experts is recommended to ensure correct rendering in all browsers. PowerMapper consists of a large number of automated tests examining websites for compliance with WCAG. It includes over 340 tests for WCAG 2.0 and over 350 tests for WCAG 2.1 that cover A, AA and AAA guidelines. The Color Contrast Analyzer (CCA) tool was developed by the IT Accessibility Office at NC State University and is available as an extension to Google Chrome browser. It allows to analyze the contrast on a selected page by imposing characteristic borders on the elements, but the results must be interpreted by a user. The level of WCAG 2.1 conformance can be chosen by the user as AA or AAA, also the size of the text to be analyzed can be specified. In case of all automatic tools used it is recommended to verify the results by an expert and not rely on the generated reports only. Some of the tools return feedback on the same WCAG guidelines, like contrast requirements verification by CCA and PowerMapper; therefore, the results were compared with each other and with the remarks of the study participants.

3.2 Description of research group

Ten people took part in the study, forming a research group. The average age of the participants was 29, with the youngest participant being 21 and the oldest 39 (Table 2). Among the respondents, four people had a formal decision related to visual impairment, including one blind person. Five people had a certificate of motor disability and one person had a multiple disability diagnosed for sight and movement. Figure 1 presents the bar chart showing distribution and types of disability of the study participants.

Table 2 Statistics of the research group
Full size table
Fig. 1
figure 1

Research group disability statistics

Full size image

As shown in Fig. 2, each person from the research group used a computer on a daily basis, most often to browse the Internet, for professional purposes, and to use the Microsoft Office suite. Participants often used the computer for receiving and sending e-mails, as well as to operate other specialized programs. They least often used the computer for programming, instant messaging or entertainment. All members of the research group used the Lodz University of Technology’s e-learning platform, but only two users had previous experience with other MOOC providers, namely Udemy and Quizlet.

Fig. 2
figure 2

Way of computer use by members of research group

Full size image

Half of the study participants did not use assistive technologies. The blind user used JAWS, NVDA and Window-Eyes screen readers interchangeably. In addition, one person used the NVDA screen reader, two persons reported using operating system magnifier and high magnifications, and one used the additional magnifying programs combined with the operating system magnifier. These data are presented in Fig. 3.

Fig. 3
figure 3

Use of assistive technologies by a research group

Full size image

The study participants completed a total of seven different Coursera courses. Three persons took the “Learning How to Learn: Powerful mental tools to help you master tough subjects” course, and two persons took the “English for Career Development” course. All tested courses were conducted in English. Detailed information is presented in Table 3. Selected courses were targeted to improving soft skills of learners. No scientific courses were taken in order to avoid various methodologies for evaluation of the accessibility.

Table 3 List of courses completed by participants
Full size table

Seven users took the courses on a laptop computer and three of them used a desktop computer. The most popular operating system was Windows 10 (eight users), the other two persons used Windows 7. The most frequently used browsers were Google Chrome and Mozilla Firefox, used by four people each. The other two participants used the Opera browser. The most complicated configurations are as follows:

  • Mozilla Firefox 52 with JAWS 18 screen reader. Google Chrome 87 browser and NVDA reader were used alternatively;

  • Mozilla Firefox 83+ with NVDA screen reader. Google Chrome 87 was used interchangeably.

4 Results

The results of the research conducted are presented in two parts: in the first subsection the results obtained with the use of automated tools are described, and the next subsection presents the comments of the participants of the qualitative study, in accordance with the principles of the web content accessibility guidelines (WCAG) 2.1.

4.1 Automatic audit

The automatic audit was performed using Markup Validation Service, PowerMapper Validator and Color Contrast Analyzer. The Coursera platform test with Markup Validation Service showed fifteen errors, seven of which are CSS syntax errors, and another four are invalid declaration in cascading style sheets. The remaining errors indicate metadata and character encoding problems. These problems violate criterion 4.1.1 of the WCAG. The results are presented in Fig. 4.

Fig. 4
figure 4

Errors detected by the W3C validator

Full size image

The PowerMapper tool presents the results in the form of six groups of problems and tests up to ten subpages in the free version. This paper focuses only on accessibility considerations reported. The most important accessibility issues detected by PoweMapper are presented in Fig. 5 and listed as follows:

  • On the subpage devoted to cooperation with business, an element responding only to a mouse click was detected, while omitting the keyboard and links with the description such as “Click here”. Also found a picture with a double alternative description;

  • On the main page there are images with the description “Placeholder”;

  • ARIA hidden items to which the focus can be moved were detected;

  • In the course catalog, an element with the tabindex attribute, buttons and links were detected, which appear as a descendant of the element with the link role;

  • Missing aria-label or aria-labeled by attribute detected on homepage;

  • A button with an image with no alternative description was detected on the subpages devoted to business and universities;

  • Carousel arrows on the home page have multiple duplicate IDs;

  • On the subpage devoted to universities, an element with the menu role was found, which does not contain any items and graphics in SVG format with the image role assigned, but without the title attribute;

  • Incorrect assignments of IDs to ARIA attributes;

  • General headline level problems.

Fig. 5
figure 5

Results generated by PowerMapper tool

Full size image

The color contrast analyzer (CCA) tool was used with the setting of 4.5:1. The Coursera platform ensures the correct contrast, one can only have minor comments about the text in the search box and in the fields in the “Tell us about yourself” section.

4.2 Qualitative analysis conducted in a research group

The results of the qualitative analysis based on the questionnaires filled in by the users after each week of the course are presented in four subsections related to WCAG principle groups, namely perceivable, operable, understandable and robust.

4.2.1 Perceivable

The requirements of the “Perceivable” principle concern the presentation of information and interface elements in such a way that they are fully perceptible to the senses of users. The most frequently reported problem was the lack of descriptions of alternative graphics included in the courses. As an example, the first quiz from the course “Learning How to Learn: Powerful mental tools to help you master tough subjects”, is given in Fig. 6. The tips or even ready-made answers to questions were presented as pictures with the lack of alternative description.

Fig. 6
figure 6

Example of an image with meaningful content with no alternative description

Full size image

Users also noted that the icons of the authors creating a given week of the course also do not have an alternative description and do not have any other caption. There is also a note that in the account settings, in the section of linking the profile to other platforms, there is a duplicated element in the form of the described icon and a text caption next to it. All the problems mentioned are related to criterion 1.1.1 of WCAG. The tested platform meets criterion 1.1.2, providing videos with extended subtitles and transcription. During the study, comments were also made regarding the inconsistent use of regions and landmarks, including ARIA tags (WCAG 1.3.1, 4.1.2), and the transmission of information based solely on sensory properties (WCAG 1.3.3). Another example is the “Learning How to Learn [...]” course. The first quiz asked if it is true that you can turn on subtitles for a movie with the “small icon” in the “lower left” or “upper right” corner of the video player. In the same course, users have encountered an issue with using the underscore as a missing word to complete, which the screen reader is unable to identify. Further reported problems related to contrast, that is, criteria 1.4.3 and 1.4.11 of WCAG. The gradient background on the home page of some courses caused some of the texts to have too little contrast. As an example, the above-mentioned course, in which pink is turning to gray, made the white text no longer clearly visible. However, users reported the most issues related to the focus across the entire Coursera platform. In the case of Firefox, there is only the default, very faint focus, which makes the website much more difficult to use. Google Chrome offers better support, but also in this case many elements do not meet the requirements. Poor focus visibility is related to WCAG criterion 2.4.7. The last criterion for which comments were submitted is 1.4.13, i.e., the content under the cursor or focus. They mainly concerned the inability to dismiss the content that appears without moving the mouse cursor, eg. in the case of the “Explore” or “For Enterprise” menu. There is also no possibility of rejecting or hovering over the appearing tooltips, for example during registration and logging in, and an attempt to dismiss them with the Escape key ends with closing the entire window instead of the hint. Objections were also made to the icon responsible for the search. Clicking on it would immediately bring the user to the results page, regardless of whether one entered anything into the search field, attempted to reject this action or tried to open it in a new tab.

4.2.2 Operable

The “Operable” principle implies that users must be able to operate and navigate the page regardless of the input device used, at their own pace. Many comments were made to control the website using only the keyboard (WCAG 2.1.1). The most serious of these concerned the video player used in the courses. The video can be started from the keyboard level, but there is no possibility of scrolling it or changing the volume. It is also problematic to set subtitles or change the settings, there is no possibility of quitting the displayed windows (WCAG 1.4.13). It also happened that the focus was shifted in front of the player, without warning, in addition to the element that the reader is not reading. In full-screen playback mode, there is no limitation of focus only to visible elements and one can freely move the keyboard over the entire, invisible at the moment, page. In the case of the JAWS 18 screen reader, playback was even more problematic. The reported issues also concerned the timeline in the course review, however this solution was updated at the time of compiling the results of the study and replaced with a text presenting completion percentage. Objections were also raised regarding the handling of modal windows and multiple choice checkboxes and radio buttons. They were partially updated during the course of the study, but still posed difficulties. It was also found that these difficulties are strongly related to the browser and screen reader used. Google Chrome with NVDA worked best with virtually no problems. NVDA with Firefox had minor problems, including the inability to set weekly goals in the course. However, the JAWS 18 reader caused the greatest difficulties with keyboard navigation in both browsers. Using modal windows and controls was very difficult and sometimes impossible. Subsequent comments submitted concerned the checking of their answers in the completed quiz. By blocking the focus on the questions, it was impossible to check the status of the answers, i.e., whether they are correct, incorrect or whether the answers were not marked at all (WCAG 2.1.2). Users of screen readers also raised objections to the mechanism of skipping content blocks on the website (WCAG 2.4.1). The surveyed website has one skip link, only in specific locations, for example in the course overview. Moreover, it is placed after the search engine. Such a procedure may also violate the consistency of navigation and identification, that is criteria 3.2.3 and 3.2.4, respectively. It is a good practice to put skip links at the beginning of each subpage where a given block of content is repeated. Study participants also suggested adding additional links to facilitate navigation. Problems with accessibility reported by users also concerned links in some courses and their content, and the lack of a mechanism informing about opening links in new tabs (WCAG 2.4.4). Many comments were made on the wrong header hierarchy (WCAG 2.4.6), which in turn caused problems in navigation (Fig. 7). An example is a course overview where there are two level two headings before the level one heading, and level 3 headings are omitted from the rest of the page.

Fig. 7
figure 7

Hierarchy of headings in the course overview

Full size image

The last WCAG criterion in the ’Operable’ principle to which comments were submitted is 2.5.3 in combination with 2.4.6, that is related to labels. No labels were found in the search engine, and the carousel control arrows on the home page had a label set as the default icon name, namely “chevron right”. Users had enough time to complete the tasks, no flashing elements were found on the platform, and the focus order was satisfactory (WCAG 2.2.1, 2.3.1, and 2.4.3).

4.2.3 Understandable

The ’Understandable’ principle requires that the interface and content of the website is understandable to participants, and that the platform provides support in the case of an error. The only major error reported by the participants of the study was the lack of content language (WCAG 3.1.2) for courses with foreign names and for transcriptions under videos. Users reported that the courses had a logical and understandable structure, easy-to-understand instructions and clear error messages (WCAG 3.2.3, 3.2.4, 3.3.1 and 3.3.4). One of the participants with severe visual disability mentioned the importance of proper spelling of words and names in the course as: “errors in spelling make the blind user, who uses keywords to search on the website, may not find such information.”

4.2.4 Robust

According to the ’Robust’ principle, content should be created in such a way that it can be correctly interpreted by as many user programs as possible, including assistive technologies. Survey participants made some general comments about the platform and courses, including the slow performance of the JAWS 18 reader page:

  • “The course best collaborated with Firefox browser. On the Google Chrome the course was much worse”—person with medium visual impairment;

  • “Playing videos greatly slows down the screen reader operation”—person with severe visual impairment;

  • “When filling the profile information I found some form elements with no labels”—person with visual and motor impairment.

Another problem is that privacy and cookie notices appear late and without a warning or message. This can cause many problems that are difficult to predict. In the study, the critical situation was the loss of control over the modal window after closing the above-mentioned message.

5 Discussion

Study participants’ reports revealed many significant inconsistencies with web content accessibility guidelines (WCAG) 2.1, both at the A and AA levels. The most important of them are:

  • lack of descriptions for alternative graphic elements;

  • requiring sensory perception (recognizing shape, location of elements, etc.) to understand and manipulate content;

  • problems while operating the website and courses using the keyboard;

  • insufficient possibilities of skipping repeated blocks of content;

  • problems with understanding the goals of some links;

  • missing labels;

  • very poor focus visibility.

The main problems identified in the study conducted by Sanchez-Gordon et al. [20] were defined as gaps in descriptions of alternative images, poorly visible focus on the page and ambiguous links, which was confirmed by the participants of this study. Bohnsack and Puhl [23] used Internet Explorer 8, which was recognized by the Coursera platform as out of date with an update recommendation. For technical reasons, this issue was not possible to verify in the presented study. The next results of the Coursera accessibility analysis were published in 2017 by Królak et al. [24]. The discussion in this paper included the most important results obtained and presented in the article written by Al-Mouh et al. [22], which were mostly confirmed in the presented study. The participants of the research conducted in [24] noted the lack of labels for buttons, deficiencies in alternative texts for graphic elements, incorrect hierarchy of header levels, problems with the use of the page using the keyboard, including controls in tests, and slow operation of the web browser in combination with the screen reader application. All these issues were confirmed by the participants of the presented research study. Some of the errors mentioned are now on a smaller scale than in previous studies. The problem with automatic multimedia playback was completely fixed, resigning from this solution, page titles and link intelligibility were also improved. Obtained results give partial answer to the research question stated, show that MOOCs on the Coursera platform are to some extent accessible and allow learners with disabilities complete the course. The accessibility problems reported in previous studies are constantly being removed or reduced. The research hypothesis of the presented research, stating that Massive online open courses on the Coursera platform meet the criteria of the Web Content Accessibility Guidelines 2.1. was partially confirmed.

6 Conclusions

The purpose of this study was to identify problems and non-compliance with the web content accessibility guidelines (WCAG) 2.1. An informal analysis of data obtained on the basis of the reports sent by the participants of the study was carried out. Significant non-compliances with the guidelines have been demonstrated, mainly from the perceivable and operable principle groups. The inconsistencies listed in the Discussion are significant and should be considered as a priority by all MOOC providers and developers of the content of e-Learning services. Comparing results of the studies on MOOCs accessibility since 2014, it can be observed that the platforms have improved their functionalities and are more user friendly for learners with special needs.

Future works should include an extended form of research, i.e., accessibility verification by automatic tools, full expert research and usability tests with the participation of users with disabilities using specific scenarios. To achieve reliable heuristic evaluation it is recommended to enroll accessibility experts representing each disability group. The next stage of research should take into account a more diverse research group, including people using Braille displays, additional assistive technologies, including, for example, people operating devices using the eyeball movement or users with color vision impairments. It is also worth testing a wider range of devices such as tablets or smartphones, operating systems such as Linux, Android, iOS or macOS, and applications and services such as VoiceOver or TalkBack. The inclusion of more MOOCs and the extension of the research to other platforms providing online courses can also provide meaningful information.