The Transparency of Automatic Web Accessibility Evaluation Tools: Design Criteria, State of the Art, and User Perception

Ninety people answered that they use accessibility tools for their work, while the remaining 48 did not use them. Regarding the tool they use most often for their work they answered: MAUVE++ is used by 32 users (16.49%); WAVE by 27 users (13.92%); Siteimprove by 21 people (10.82%); W3C Markup Validation Service by 14 persons (7.22%), Lighthouse by 11 people (5.67%); axe by 11 users (5.67%); Achecker by 10 people (5.15%); Vamola by 10 users (5.15%), Accessibility Insights by 6 users (3.09%); IBM Equal Access Accessibility Checker by 6 users (3.09%); ARC Toolkit by 3 users (1.55%), Cynthia Says by 3 participants (1.55%), tota11y by 3 users (1.55%), and WebAIM by 3 users (1.55%). Then, FAE is used by 2 users, aCe is used by 2 users, pdf checker is used by 2 users, TPGi Colour Contrast Analyzer (CCA) is used by 2 users, 2 users declared to use a mix of browsers’ extensions and add-ons. Other tools (such as Jigsaw, Imergo) were mentioned by just one user. We also collected information about the frequency with which the users exploit validation tools for their job (we asked to indicate a maximum of three tools in the survey). 29 users (which correspond to the 32.22% of the 90 abovementioned users), declared to use at least one tool once a month. 26 users (28.89%) declared to use one or more tools once a day, 18 users (20%) once a year, while the remaining 17 (18.89%) once a week.

The answers provided by the users are grouped according to the criteria that we identified previously. Please note that it sometimes happened that, in their responses, users mentioned more than one aspect/criterion.

What standards, success criteria, and techniques are supported. The standard(s) with the tool is compliant with have been mentioned by 16 users as a way to characterize the transparency of tools. Seventeen persons mentioned that when a tool explicitly highlights the criteria used for the evaluation and how many of them are covered, this highly contributes to an increase in its transparency. One user explicitly mentioned that one factor that affects the transparency of a tool is “when the tool highlights the methods and the parameters that characterize the evaluation criteria”. Seven persons highlighted that one factor that affects the transparency of a tool is whether the tool highlights the specific techniques/tests that it applied (or not) when checking the various success criteria.

How accessibility issues are categorized. Six users mentioned this aspect, in particular the importance of having the results categorized according to different types of content (e.g., according to the implementation language, such as HTML or CSS).

How the validation results are provided by the tool. Forty-one users mentioned aspects associated with this point. In particular, the information that the tool provides to users about errors/violations of accessibility was judged an aspect that strongly characterizes the transparency of a tool. In particular, many users declared that it is important that tools provide correct and clear explanations/visualizations of such errors (possibly both in the page and in the code), and in a way that is comprehensible also by non-technical users. Furthermore, they should offer good coverage of the errors using relevant references to the page/code to better identify/localize them, and also using relevant references to the corresponding concerned criteria. Finally, they should provide clear explanations about why an issue was pointed out and what its consequences are in terms of accessibility. Other users mentioned that transparency is highly impacted by how clear the results provided by the analysis are, and also how comprehensible is the way in which the analysis is carried out, also referring to broader information on the resulting analysis, i.e., not just focusing on the errors, but indicating how the results have been obtained, the pages used for the evaluation, the clarity and comprehensibility of the results produced by the analysis and the way in which they have been obtained, the completeness of the analysis, its reliability, and verifiability/replicability.

Whether the tool provides practical indications about how to solve the identified problems. Fifteen users mentioned as a key aspect when the tools provide users with concrete suggestions for possible solutions to the accessibility violations identified, more precisely how and where to intervene in order to solve the identified accessibility violations/issues.

Whether the tool is able to provide information about its limitations. Another aspect that users (N = 5) judged important for the transparency of the tool is that it should clearly highlight the situations that it is not able to address automatically, and therefore for which situations there is a specific need for manual checking. In particular, one user said that one aspect characterizing transparency is when the tool clearly states “what needs a manual validation and what would imply, in concrete terms, performing this manual validation”, thus highlighting that not only it is important to remark the need for manual checking in general (as tools are never exhaustive), but also to provide guidance to the users about how to perform such manual check in concrete terms. One user highlighted that the tool is transparent when it highlights any part of the web site that the tool was not able to test. Another aspect that users (N = 4) rated highly in terms of transparency was related to the situations when the evaluations are ambiguous, or when false positive and false negatives could occur. In such cases, one user highlighted that it would be better that the tool explained the choices made in order to arrive at the provided results. Four users highlighted that further information about the methodology and objectives of the tool should be provided to users, to increase transparency. Moreover, the importance of declaring the limitations of the assessment provided was mentioned by four users as a way to improve transparency. Among further aspects mentioned by participants, two of them highlighted that the inconsistencies occurring between the evaluations provided by different tools can affect transparency. One user mentioned that a tool is transparent when it is actually possible to perform some modifications to the validation results (e.g., when it is possible to declare that a “fail” is actually a “pass”).

Additional aspects mentioned. Twenty-four users mentioned some more general characteristic/quality that tools should have in order to be transparent. Among the qualities mentioned there was ease of use, clarity, and reliability; also, the fact that the tool is free or open source, that it is independent from specific stakeholders/vendors, and that it can be used by any user in an “open” manner, and also the accountability/credibility of the organization that develops the tool. Nine users mentioned specific features of the tool that can affect transparency. Among them, five users highlighted the users’ need to get further details and documentation about the tool, also in terms of its strengths and features. One user highlighted that the tools generally do not provide many details, thereby accessibility experts tend not to trust them. Two users mentioned that the transparency of a tool could be increased by the personalization/configuration possibilities offered by the tool.

Have you ever experienced NOT to be able to understand the results of an accessibility evaluation performed by an automated tool? 89 answered yes, 49 users answered no.

If YES, do you remember what kind of difficulties you encountered? The answers to this question are grouped according to four themes: results, errors, solutions, and lack of clarity.

Seventeen users mentioned difficulties connected with the results provided by tools. The aspects that they mentioned regarding the results can be grouped according to five sub-themes (with the most frequent ones appearing first): the mismatches between what was reported and what the user observed, the interpretation of results, divergences between evaluations provided by other tools, unclear/inefficient presentation of results, and lack of completeness in providing such results. In particular, most users complained about mismatches: some users reported experiencing a mismatch between what they observed and what was reported by the tool, e.g., this happened –one user said- when there was a criterion which was reported as not satisfied, whereas actually there was no error in the page. On the other hand, another user said that, although the page was not accessible, the tool said that it was. Another user highlighted that this mismatch could affect the trust that users have in tools, as sometimes users can have the feeling that the indications are not correct and therefore the tool seems buggy. Indeed, a pair of users reported having actually experienced a bug in the validation tool: “I do recall that the technical support team were able to explain the issues and that some issues were due to bugs in the tool.” Another point mentioned by several users regarded the interpretation of results: people complained about the difficulty of understanding the results provided by the tool. One user said that sometimes the tools are limited to provide a technical summary, which makes it difficult to understand their results, especially by non-technical people. Another user said that “it is sometimes difficult to understand what success criteria it was mapping to, why it was only picking some techniques over others, or why certain lines of code were tagged as wrong.” Another sub-theme referred to the divergences among different evaluations. In particular, two users highlighted that the provided results are not always in line with the results provided by other tools. A pair of users highlighted that sometimes tools present the results in an uneasy-to-read and non-efficient manner. In particular, one user highlighted that sometimes tools provide “a long list of results which is not useful if you are willing to prioritize due to lack of resources”. A final sub-theme regarded the lack of completeness in providing the results: one user said that she would have preferred to see in detail also the criteria that successfully passed the evaluation, and not only those which failed.

Sixteen users mentioned difficulties connected with errors: in many cases the reported difficulty is in understanding the reason why a point is reported as a violation of accessibility as the indications (error messages) are sometimes ambiguous, generic (i.e., do not specifically indicate where the error is), do not even actually relate to a real accessibility violation, are not always correctly associated with the concerned element, and overall are unclear and not exhaustive. One user complained about an unclear distinction between errors and warnings. Fourteen users provided further comments about reported difficulties concerning the solutions provided by the tools. Most of them highlighted the difficulty of having specific, clear and correct indications about how to solve the issues, reporting that currently there is a scarcity of them. A pair of users highlighted that sometimes the proposed suggestions about how to solve a specific accessibility issue are not correct or do not work. One user suggested providing “more contextualized solutions, perhaps with small examples, to understand whether the tool has understood the context of the analysed content or not”. Eleven users complained, more in general, about lack of clarity of the information provided by the tool: sometimes tools provide messages that are generic and ambiguous, or they use a too technical language, which is not suitable for unskilled people.

Are there any other features you think an automated accessibility evaluation tool should have in order to be transparent? 42 users (30.43% of the total users) answered YES, the remaining 96 answered no.

If YES, which ones? The answers from users were grouped into six themes.

Seventeen users mentioned some features and/or characteristics that the tool should have. Aspects that were highlighted by users regarded: the possibility to evaluate also PDFs, to be open source, to include further localization possibilities (i.e., to be able to select the language to more easily understand the errors, or to be able to select a specific country, as accessibility norms can change according to them), to have the possibility to export the reports. One user suggested having a blog reporting the updates done over time on the tool, which could help −the user noted− for understanding why some evaluation results changed over time. Another user suggested making available some practical tutorials about how to write HTML and CSS. Another user highlighted that it would be important to know who is developing and releasing the tool, as well as its mission and the goals, to better evaluate its reliability and degree of confidence; the same user highlighted that another added value could be the availability of an effective support service. One user highlighted that it could be useful to know who supports/promotes the tool. One user suggested indicating how often the tool is updated. Another user highlighted that some tools (e.g., aCe) have a manual checking model that should be followed by other tools, as it helps in doing manual checks. In particular, to this regard, another user said that tools should suggest which manual tests can be done in order to verify semi-automated success criteria. A user suggested publishing the list of the tests that tools do, highlighting that some tools already do this. One user suggested proving the results delivered by the tool against a standardized set of examples, like the ones provided by ACT rules. Another user highlighted the need of solving the inconsistencies that can be found among different tools. One user declared that it would be useful to understand whether a site/app meets the WAD requirements. Another user said that tools should state outright the known statistics of how many accessibility problems can be determined through automated scans. Thirteen users mentioned the need of having further info on the results/analysis. Among the most relevant comments provided, one highlighted the need of having further information to precisely replicate the tests; another person suggested to clearly indicate what was not tested, and what needs to be tested manually. Another user highlighted the clarity of the results as a key aspect. A pair of users highlighted that the tool should give a precise indication of the conformance level (i.e., A, AA, AAA) that it considers. In addition, one of them highlighted that it would be useful to indicate what kind of problems would be faced by people with which disability(ies) if an error is not corrected or a criterion is not met. Another user would appreciate further information about the ARIA rules the tool evaluated. A pair of users highlighted the need to have more references to WCAG, i.e., which WCAG checkpoints are covered, how they are covered, and more code snippets with a hint on what to fix and what is missing. A user highlighted that, if the tool provides a general overall measure, it should be explicit in how it is calculated and what its limitations are. Two users emphasized the importance of having some visual indications directly in the concerned Web page (i.e., to show the layout of the Web page, to highlight key parts in it e.g., the tables). In addition, another user highlighted the importance of having further information about how the check has been done by the tool, in order to facilitate the user to verify whether there is a bug in the tool. Five users highlighted the need of providing concrete and operative indications to solve the errors identified, also by showing one or more examples of the solution, especially for the most common errors. Among them, one user highlighted the need of providing hands-on examples especially when specific assistive technologies are involved (i.e., screen readers), as in such situations it is not obvious that all the users of the tool know all their implications and how to solve possible problems connected with their use. Four users also pointed out the need to provide better support to non-technical users. This would imply, for instance, providing users with easily understandable results, possibly accompanied by visual graphs, as well as easy-to-understand explanations of the motivations why an accessibility violation was found by the tool. One user suggested having an icon that should allow users to keep track of the current state of the evaluation easily. Three users mentioned the need to emphasize that manual check is always needed. Users highlighted that tools should clearly state that the automatic checks they provide should in any case be complemented by manual validation. One, in particular, declared: “They should state outright the known statistics of how many accessibility problems can be determined through automated scans and should also make clear that it is not possible for an automated tool to identify or remediate the vast majority of websites to 100% WCAG conformance.”. Finally, three users pointed out the need of highlighting and addressing the occurrence of false positives, false negatives, and possible errors in the analysis. They said that sometimes tools highlight issues that are not real ones or, on the contrary, could fail in identifying actual accessibility violations, thereby it would be better to solve this issue. One user, in particular, suggested that the tools should highlight the possibility of false positives in the most critical WCAG criteria. Another one suggested that the tools should report the confidence level associated with a specific error when it cannot be sure that it is an actual error.

The answers addressed various themes: (i) the aspects that the tool is not able to automatically evaluate, (ii) the preferences and parameters that users can specify for the analysis, (iii) the situations in which the results can be wrong (e.g., false positive or false negatives) or ambiguous, (iv) the lack of clear indications highlighting the need of manual check (with possible guidance on this manual check), and also (v) further aspects.

Thirty-six users mentioned aspects that the tool is not able to automatically evaluate as a limitation. Many of them generically pointed out that tools should indicate the situations that they cannot automatically assess, either e.g., because they do not cover the corresponding criterion or because the success criterion is just partially checked. Other users were more specific in identifying such cases: when a tool is not able to access URLs that are protected by login; when tools are not able to perform their assessment when specific technologies are considered or when dynamic pages are considered; when issues are in the content of the page, rather than in its structure; when checking colour contrast on pseudo-elements; when they have to analyse mobile apps, when they have to analyse different types of documents/formats (i.e., svg, pdf), and other specific situations (e.g., shadow DOMs, content inside frames). Also, one user mentioned the inability of tools to perfectly emulate a braille reader; another user mentioned the inability for tools to evaluate how properly images and alternative texts are used in a website. One of the users mentioned that tools are not able to cover all WCAG success criteria, and it would be useful to know the rules (testing algorithms) used and which published ACT rules are covered. Another theme regarded limitations concerning the preferences and parameters that users can specify for the analysis (8 users mentioned this point). Some users mentioned as a limitation the number of pages to consider for the evaluation, one user mentioned the depth of the analysis. One user highlighted as a limitation the lack of compliance with specific standards. One user highlighted that there are some tools which are not very up-to-date. Another comment indicated that the versions of the various languages (e.g., JavaScript) and frameworks (e.g., Bootstrap) that the tool is able to address could represent a limitation, and thus it should be clearly indicated. Another type of limitation regarded the occurrence of situations in which the results can be wrong (i.e., false positive or false negatives), or ambiguous. Seven users mentioned that tools should clearly indicate situations that can generate false positives/negatives in the evaluation or indicate when the evaluation could generate multiple interpretations. In this regard, one user mentioned that the ARC Toolkit issues warning for cases that may or may not be a problem depending on the context. Another theme regarded the lack of a clear declaration highlighting the need for manual checks (with possible guidance). Five users emphasized the fact that an automatic validation is never complete and exhaustive, therefore tools should clearly highlight this, also possibly providing guidance for carrying out manual checks. In addition, one user said: “An automatic evaluation is not enough to guarantee that a site is accessible. Heads up for manual checks would be appreciated; some tools do that.” Finally, as further aspects mentioned, one user suggested that it would be useful to know in advance the behaviour of the tool compared to a benchmark (in terms of false positive, false negatives, coverage), acknowledging that this would require to have a ‘normalized’ corpus and process to assess evaluation tools. One user highlighted that tools should be more explicit about their pricing options. One user would like to have more information about situations in which different tools return different results. One user mentioned that tools should mention the possible improvements. Another aspect mentioned by a user is that sometimes tools are too “code-based”.

Regarding the users’ ratings, as it can be seen from Table 4, since the median (Mdn) values are higher than mean (M) values, the data distribution is more concentrated on the right-side, corresponding to the higher scores. In addition, while the range (which gives a measurement of how spread out the entire data set is) is high (Min = 1, and Max = 5), the interquartile range (which gives the range of the middle half of a data set) is low (IQR = 1), which means that the middle half of the data shows little variability.

Lighthouse (see Figure 1) is an open-source, automated tool for improving the quality of web pages. It has audits for performance, accessibility, progressive web apps, SEO and more. It can be set up and used in a variety of ways, which makes it a good option for both technical and non-technical users. In the test we used the automatically available version in Chrome, with no setup or extensions to install, by opening Chrome Developer Tools panel and then choosing the Lighthouse tab at the top. One disadvantage of using Lighthouse in the browser is that only one page can be audited at a time, which makes the process cumbersome for sites with a large number of pages.

The Lighthouse tab in Chrome DevTools has a straightforward interface which allows users to choose to run just an accessibility audit or run additional audits (including, e.g., Performance, Progressive Web App), and it also allows the user to choose between running the audit on desktop or an emulated mobile device. Once those choices have been made, the user has to press the “Generate report” button to run the selected audit(s) directly in the browser. Lighthouse runs its audits against the currently focused page, then opens up a new tab directly showing a HTML-based report of the results. The report can also be saved in JSON, or it can be opened in the online Lighthouse Viewer (by submitting the JSON output of a Lighthouse report); in addition, users can also share their reports by creating secret GitHub Gists (the benefit of Gists is free version control), which can also be viewed in the viewer.

A feature of The Lighthouse accessibility audit report is a metric called Accessibility Score, which is generally shown at the top of the report. This score is a weighted average of all of the accessibility ‘audits’ Lighthouse runs on the current page, and it is based on Deque's axe user impact assessments (which Deque uses for its own axe browser's extension), which cover WCAG 2.0 and WCAG 2.1 for A and AA conformance levels. However, it is worth pointing out that within Lighthouse, the end user cannot directly find an explicit reference to the WCAG guidelines: the fact that Lighthouse covers WCAG 2.1 (A, AA) has to be implicitly deduced from the fact that it implements the axe-core rules (as reported in https://web.dev/accessibility-scoring/) which is a JavaScript library (used in the axe browser's extension) that covers WCAG 2.1 (A and AA). So, from this point of view, Lighthouse is not very transparent. Also, Lighthouse does not let the user know which WCAG rules are being violated. In addition, despite being based on axe's list, Lighthouse actually misses some issues that the axe extension is able to catch (i.e., it does not implement the full list of axe-core rules), so inconsistent results can be generated between axe and Lighthouse, which might confound users.

Lighthouse uses a simple 1–100 grading system for auditing pages, further divided into three sub-intervals associated with different colours: red colour applies to scores between 0 and 49, which are considered poor, 50–89 is associated with a yellow colour (needs improvements), 90–100 scores are associated with a green colour (good). It also shows where improvements can be made. However, it is worth emphasising that a score of 100 does not mean that the site is perfectly accessible, as Lighthouse uses the axe-core accessibility testing library with a custom (namely: partial) set of rules for its tests, and of course manual testing is still needed.

As mentioned before, the Lighthouse Accessibility score is a weighted average of the accessibility factors (or ‘audits’) considered by Lighthouse, and their weighting is based on axe user impact assessments (manual audits are of course not included in the score calculation). Each ‘accessibility audit’ is pass or fail, which means that a page does not get points for partially passing an accessibility audit. For example, if half the buttons have screen-reader friendly names, and half do not, you will not get “half” of the weighted average: you get a 0 because that accessibility criterion needs to be implemented correctly throughout the page.

As for the results provided by the tool, they can be grouped into: Failing Elements, Additional items to manually check, Passed audits, and Not applicable. The Failing elements are then further categorized into the accessibility criteria considered by Lighthouse (Navigation, ARIA, Names and Labels, Contrast, Tables and Lists, Best Practices, Audio and Video, and Internationalization and Localization).

For each violation found by Lighthouse, the tool provides: a short description of the issue, a “Learn more” link to the page on the web.dev website explaining the issue (thus, Lighthouse does not let the user know exactly which WCAG rules are being violated), and a list of the elements in the web page that fail that particular accessibility criterion (some of such elements are accompanied with a screenshot of the involved element shown within the page, so the user can visually locate it). In addition, it is also possible to see where the violation occurred in the HTML code.

Finally, as for its limitations, Lighthouse does not explicitly provide any indication about its limitations, apart from a general statement saying that only a subset of accessibility issues can be automatically detected, thereby manual testing is encouraged. Being available also as a browser extension (in Chrome and Firefox), it is able to validate dynamic content.

Regarding whether Lighthouse (Figure 1) provides information on the guidelines supported (Q1), 16 users (correctly) answered that the tool does not provide such info, 2 answered positively, even though only to some extent: both of them mentioned a limitation connected with checking ARIA standards, whose support is mentioned only after validating a specific page.

As for how the tool classifies the results of the assessment and whether it provides information on such classification (Q2, Q3), nine users correctly indicated the different types of results, which in Lighthouse are: ARIA, Navigation, Internationalization and localization, Additional items to manually check, Passed audits, Not applicable. It is worth noting that in Lighthouse errors do not have a category per se, but they are considered in the aforementioned categories, depending on the type of problem found. Four users recognized the categories just in a partial manner, while five users did not recognize the different ways in which the tool classifies the results at all.

When asked how many errors the tool identified in the considered Web page (Q4) and how many elements the tool indicated that it could not automatically check (Q5), i.e., the so-called ‘warnings’ or ‘cannotell’, 6 users correctly identified both the numbers of errors and warnings, 5 users did not identify them at all, 7 users identified them just in a partial manner.

Regarding whether the tool provides quantitative indicators of the accessibility and completeness of the evaluation (Q6), all participants answered affirmatively (and correctly), also saying that such indicators offer a succinct summary measure of the accessibility of the considered page. However, a few of them expressed some doubts about how they were concretely calculated.

As for the question on whether the results’ presentation is useful and comprehensible (Q8), the answers were yes (N = 4), in part (N = 7), and no (N = 7). The positive comments said that the presentation is quite immediate and easy to understand, as it presents a list of errors clearly differentiated into a number of macro-categories (passed, not applicable, fail), and it also offers the possibility to automatically refer to the relevant code portions (by clicking on the associated link). Those who were partially positive found that sometimes it was not easy to understand which part(s) of the page generate(s) errors; another user, while acknowledging that the tool highlights errors, it does not specify in what way the checks fail; another user complained that it was not easy to find immediately the total number of errors; another user highlighted the need to group the errors according to their type; another user said that sometimes the explanations are extremely short. Finally, a user suggested including an explicit reference to the accessibility standard(s) referred by the tool. The majority of the negative comments regarded the visualization of the results, which was found unclear, with a confusing layout, and more in general too concise, also lacking structured visualizations that include graphs. Three users complained about the fact that the tool does not allow users to understand where the errors/warnings are located within the page. One user complained that the visualization of results does not help in concretely solving the accessibility issues found.

Then the participants were asked to analyse one of the errors detected and indicate the associated information provided by the tool (Q9). The majority (N = 16) correctly found in the tool the information about the errors detected, two users did not. In addition, 14 users correctly provided the information that the tool offers to solve the found issue (Q10), while 4 users did not. Regarding whether such provided information was perceived as sufficient to solve the problem (Q11), 7 users found that the tool does not provide sufficient information. Seven users were overall satisfied about this aspect. The remaining 4 users expressed a ‘borderline’ point of view: while acknowledging that the tool provides some information, this was found sufficient to address just the simplest cases, while in other cases further detail is needed, especially by unskilled users.

Regarding whether the tool provides clear information on its limitations (Q12), 11 users incorrectly answered yes, the remaining 7 answered no: among them, 4 users explicitly declared not to have found it, the remaining 3 complained that this information is provided only in an implicit manner and further clarity should be added.

Among the aspects that users appreciated most (Q13), they mentioned the fact that the tool indicates what it cannot check, and emphasizes that additional manual tests should be done on specific elements (N = 3), the result presentation is immediate, concise and overall easy to use (N = 3) and also uses icons to categorize the type of results, the links available for further inspection (N = 3), the categorization of the errors according to the various concerned aspects (N = 2), the visualization of the global level of accessibility (N = 2), the fact that it lists the items that passed the automatic check, those that need further manual check, and those that are non-applicable.

Among the aspects that were not liked (Q14) the answers were varied as well. Three users complained about issues connected with standards: in particular, one complained that the tool refers only to ARIA standard, one complained about the lack of clarity on the specific standard referred, and another user declared to have not found any possibility to set the conformance level (“A”, “AA”, “AAA”). Three users complained about aspects related to problem resolution, in particular the need to give more information about how to concretely solve the errors. Moreover, aspects related to errors were mentioned by users. In particular, two users said that it was not clear which elements generate errors, nor were the referred criteria clear. Another user complained about the fact that Lighthouse does not clearly indicate the errors and warnings and also the associated descriptions are a bit difficult to understand. Another user highlighted the inability to have the results of the analysis shown directly in the page and within the code. Five users raised concerns associated with the usability, intuitiveness and the clarity of the tool and the provided visualizations. A couple of users pointed out that the presentation of results done with the accordion-like widgets was very minimal. Another user noted that if a user exploits a small monitor, it could be difficult to see the long list of results that the tool often provides, especially because the top part of the UI is occupied by the preview; three users raised concerns associated with the limitations of the validation, which they found unclear.

Regarding the additional features/characteristics that users would have liked to find in the tool (Q15), the participants mentioned: