A Review on Tools, Mechanics, Benefits, and Challenges of Gamified Software Testing

To motivate the need for a literature review, we utilize the decision table proposed by Garousi et al. [24], based on the guidelines by Adams et al. [3]. According to these authors, one or more positive answers to the question in the decision table suggest the inclusion of grey Literature in addition to White Literature in a review. The interested reader can find a detailed motivation for the positive answers to each question in the online Appendix B.

When defining the review questions, we identified three main goals for this review work:

We formulated a set of Review Questions for each of the defined goals to analyze the three identified aspects in-depth. The Review Questions are reported in Table 2.

To conduct the review, we applied the following steps:

The papers that resulted from the search merging the different digital libraries were a total of 1,221: 328 from IEEE Xplore, 151 from ACM Digital library, 396 from Springer Link, 231 from Science Direct, and 115 from Google Scholar (of which, one item was repeated twice). The search for grey literature was limited to the first 100 results from Google.

The total number of collected items was 1,321, counting both grey and white literature. After removing the duplicates, the papers remaining were 1,293. After removing duplicates, inclusion and exclusion criteria were applied; this operation has shrunk the number of resources in the pool to 70 units, 43 items of white and 27 of grey literature.

The following step was the backward snowballing: From the selected papers, a total of 1,562 sources came up, including duplicate papers both from the initial set and within the resulting pool. We applied duplicate removal and inclusion/exclusion criteria to this resulting group obtained through snowballing. This first snowballing process allowed us to add 16 results (14 white papers and 2 items of grey literature). Each title of the contribution found this way has been searched in the same starting digital libraries to assign the newly discovered publications in the correct repository, as shown in Table 3.

The same process was used for the forward snowballing, where the total number of retrieved studies was 1,281. After the filter was applied, we included the remaining 12 items (9 pieces of white literature and 3 of grey literature). The resulting set of literature has been subjected to quality assessment.

After applying the stages described in the previous sections, our final pool included 73 sources. The full list of sources is reported as online additional material in Appendix D. Table 3 breaks down the number of sources that were present in the pool after each of the review stages. We report the information about all contributions in a publicly available spreadsheet.² Our final pool comprised 50 white literature sources and 23 grey literature sources. The number of grey literature sources found is about half of the number of white literature sources. This can be considered as a first confirmation of the need to include such sources in a literature review.

The distribution of sources per year is reported in Figure 2. The chart discriminates between grey and white literature sources. For grey literature sources, we considered the “first published” parameter as the year of publication without taking into account further modifications of the web pages. We observe a steady increase in the number of sources starting from 2016 for both types of literature. The increase in the number of grey literature sources with the year of publication is an expected result, since the Effort Bounded strategy (i.e., including only the top N search engine hits [24]) for searching the Google engine tends to favor more recent sources. At the same time, older grey literature sources that are not permanently archived can become unavailable several years after publication. We cannot find an immediate rationale for the absence of grey literature sources from 2020 or for the decrease in the number of both types of sources in 2014–2015.

The distribution of the sources for contribution type is reported in Figure 3. Regarding white literature sources, we collected 35 works published in conference proceedings, 8 journal papers, and 7 works published in companion proceedings of conferences (i.e., workshop papers). Regarding grey literature, master theses were the most frequent type of contribution, with 9 sources. We also included one PhD dissertation in the pool and 3 preprints. These numbers testify that academia plays a fundamental role also in the production of grey literature sources about software testing gamification. Finally, we counted 5 blog posts, 1 documentation web page, 2 webinar presentations, 1 item from a question-and-answer website (namely, StackExchange), and 2 tech talks.

Figure 4 shows the number of sources per type of contributors. The sources were divided into three different categories: (i) academia, i.e., sources whose all authors were affiliated with universities or research institutions; (ii) industry, i.e., sources whose all authors were working in the industry; and (iii) collaboration, i.e., sources for which at least one author was affiliated with university or research institutions and at least one author was working for the industry. All-academic sources outnumbered all industrial sources (44 vs. 1 for WL, 13 vs. 10 for GL). Collaboration works were present only among WL sources (5 items).

From all the resulting sources, five main categories of contribution were found:

The difference between experiment reports and guidelines is that while the former argues on the result of one single case of application of a particular tool, the latter provides an analysis at a higher level of abstraction.

We assigned one or more categories (i.e., categories are not mutually exclusive) to each literature item by application of open coding. The bar plot in Figure 5 reports the number of sources that were assigned to each category. We observe that, for white Literature sources, the most common types of contribution were Experiment Reports (26 sources) and Tool presentations (25 sources), while the numbers of theoretical frameworks, guidelines, and discussion papers were limited. However, for grey Literature sources, we identify a prevalence of contributions that we flagged as Discussion (10 sources) followed by Tool Presentation (7 sources) and reports. This result suggests that the grey Literature sources about gamification in software testing are less technical than White Literature ones, and that no actual validation of the benefits of gamification is carried out outside peer-reviewed academic research efforts.

We divided the sources into four different methodological categories by utilizing a subset of the categorization provided by Petersen et al. [42]. We assigned a single category (the most formal applicable) to each source, i.e., research methodologies were considered mutually exclusive. The four research typologies that we considered are the following:

In Figure 6, we report the distribution of the sources from the final pool according to the type of research methodology adopted. The largest set was that of solution proposals (23 studies, 17 WL and 6 GL), followed by Descriptive studies (18 studies, 7 WL and 11 GL), Case Studies and Reports (17 studies, 12 WL and 5 GL), and Empirical Research (13 studies, 12 WL and 1 GL). These results confirm an expectable predominance of solution proposals and descriptive studies in grey literature, with a higher prevalence of empirical research (12 sources out of 48) in white Literature.

We referred to the traditional test automation pyramid to identify the testing levels covered by the gamification mechanics described in the sources. Therefore, we identify three different testing levels:

We identify in each source the testing levels to which the discussed gamification aspects were or could be applied. The categorization was not considered mutually exclusive, since a single tool or technique can cover different levels of the testing practice. For papers not mentioning any specific testing level, we considered the testing level as undefined.

In Figure 7, we report the distribution of the sources according to the mentioned testing level. We identify Unit testing as the most mentioned testing level in sources applying gamification to testing (41 mentions, 33 WL and 8 GL), followed by system-level testing (19 mentions, 12 WL and 7 GL), and integration (8 mentions, all from WL). 13 sources (5 WL, 8 GL) did not explicitly specify any testing level, and no information could be deduced by reading the implementation details provided. It is worth noting that unit testing is more mentioned in white literature, while grey literature is equally focused on system and unit testing. This result can be justified by the more industrially leant orientation of grey literature, which is more likely to discuss or evaluate system testing frameworks to be directly used by practitioners. 14 sources in the pool (6 from WL, 8 from GL) did not explicitly mention any testing level to which the tools, frameworks, or guidelines for gamification could be applied.

The second aspect of the testing discipline we analyzed is the testing phase that the gamification tool, mechanic, or framework supports.

We consider the following phases of the testing discipline:

We collected from all the sources the test phases to which the discussed gamification aspects were or could be applied. The categorization was not considered mutually exclusive, since it is possible that a single tool or methodology covers different phases of the testing process. For papers not mentioning any specific testing phase, we considered the testing phase as undefined.

In Figure 8, we report the distribution of the sources according to the mentioned testing phase. The majority of the literature items discussed the application of gamified mechanics in the phase of test creation (32 WL and 11 GL items), closely followed by test execution (22 WL and 19 GL items). We found a largely smaller number of items mentioning the phases of test design, maintenance, and reporting. 6 sources (4 WL, and 2 GL) did not explicitly mention any testing phase. Interestingly, no GL items discussed the phases of test design and test maintenance, suggesting a lesser interest from non-academic sources in these phases of the verification and validation process.

Throughout the analysis of all the selected sources, we found the mention of 11 different methodologies for testing. We did not consider the testing methodologies as mutually exclusive in the literature sources, as some items explicitly report the adoption of different testing methodologies during the use of the gamified environment. In Figure 9, we report the distribution of the sources according to the mentioned test methodology.

The methodologies that were mentioned the most in the selected sources were White-box testing (13 WL, 3 BL), Black-box testing (12 WL, 3 BL), Mutation testing (12 WL, 3 GL). Some methodologies to test non-functional properties were present: Performance testing (one source), Security testing (one source), Penetration testing (one source), Interoperability testing (one source), Usability testing (three sources). Over the whole set, there were 11 WL sources and 10 GL sources not mentioning any specific testing methodology. 21 sources (11 WL, 10 GL) did not explicitly state any testing methodology on which gamification was applied.

From the selected literature, it is evident how the focus of GL is less aimed towards developers or testers who define test scripts at the code level: The most mentioned testing methodologies in GL were, in fact, Exploratory testing (5 sources) and Capture & Replay (3 sources).

We report all the existing testing tools or frameworks that were adopted by the retrieved items in Table 4. We retrieved mentions of 18 different testing tools in our literature pool. For each testing tool, we report the tool name, a short description, a URI for tool retrieval, and the literature sources where it is mentioned.

What emerges from Table 4 is that the most used tool is by far JUnit, the most widely used tool to develop, execute, and report the results of unit test cases in Java. The tool was mentioned in 14 different literature sources. Other tools that were mentioned multiple times were Scout, an augmented tool for visual GUI testing described in white literature by Nass et al. (4 mentions), Major Mutation Framework, EvoSuite, and the code coverage tool JaCoCo (all with 2 mentions). Several tools were mentioned only in GL Sources, Bugzilla, GUnit, Nose, and Unittest. It is worth mentioning that over the 60% of the items (45 over 73) did not specify the tool used (either bespoke or already available).

A further categorization can be performed about the programming languages and domains addressed by the pool of studies. In Table 5, we report the target languages mentioned by the papers and the specific literature items mentioning them. According to our investigation, the most targeted language is Java, in accordance with the most used tool (JUnit), which is specifically used for unit testing in Java. Our findings agree with the study of Abdullahi et al. [2] that reports Java as the most used testing language. In fact, we observe that a consolidated infrastructure is often the starting point for the development of new tools; this explains why many gamified tools adopt Java as the target language. Additionally, StackSocial reports Python as the most used in universities to teach coding [49], for this reason, it appears natural to also consider Python for gamified testing in education. Although the interest in Python is rising, its actual usage spread in universities around the world is more recent mainly because academic courses are less prone to change a consolidated language than practitioners. It has to be underlined that the majority of the contributions did not specify any language (39 over 73). We did not assign any categorical value to all the studies that did not explicitly mention a language.

In Table 6, we report the target domains mentioned by the papers and the specific literature items mentioning them. We identified two main domains to which pieces of literature could be assigned: the web domain (6 sources) and the mobile domain (5 sources). One literature source mentioned both web and mobile domains. One additional paper was specifically tailored for the Internet of Things domain. The sparsity of the data also characterizes the testing domain: Only 11 items over a total of 73 (less than 20%) explicitly identified a domain for the gamified technique or tool. It can be assumed that most of the remaining sources describe approaches that are not domain-specific.

We discriminate between three different focuses for the sources that we consider:

Figure 10 reports the distribution of sources for each of the three categories, divided by grey literature and white literature. The majority of the sources deals with the practical aspect of the application of gamification to software testing, with an even distribution between grey literature and white literature. Educational-focused sources come principally from items of white literature. Unsurprisingly, we notice that grey literature provides contributions mostly for testing practice.

In Table 7, we report all the gamification mechanics proposed, adopted, or mentioned in the final pool of sources. We applied a coding procedure to the mechanics described in the sources by labelling each one with one of the mechanics codified in the Octalysis framework. In the table, we report the name of the mechanic, its definition, the synonyms with which the mechanic is also mentioned, and all the sources in which it is mentioned. We also report for each mechanic the Octalysis core driver to which it can be assigned. This way, we are able to provide a higher-level characterization of the gamification mechanics mentioned in software testing literature.

We measure a median of 10 citations per gamification mechanic. 10 mechanics had more mentions than the median. The most popular mechanic was Score, which was implemented or discussed in 45 different sources, followed by the highly correlated leaderboard mechanic (36 mentions), and graphical feedback (22 mentions). The less-mentioned mechanics were Epic Meaning (one mention), Punishment, and Auctions (two mentions each). The absence of punishment mechanics suggests that research and development in the gamified testing field have mostly focused on providing positive motivation to the users instead of negative disincentives.

In Figure 11, we report the number of mentions for each gamification core drive, according to the Octalysis Framework. From the graph, it is evident how the main focus of available gamification implementations for software testing is to provide Accomplishment to the users as a positive means of motivation (43 WL and 21 GL mentions). Conversely, only two WL sources implemented mechanics related to the Avoidance dimension, which is related to the enforcement of correct patterns by applying punishments and maluses to non-conforming users. Few mentions were also gathered by the gamification mechanics related to the Epic Meaning macro category of the Octalysis framework. We consider such a low number of mentions as an effect of the still prototypal nature of most of the described tools, which did not allow for the implementation of complex narratives.

In Table 8, we report all tools and/or frameworks mentioned or proposed in the selected literature. In the table, we report for each tool its name, a brief description, the adopted gamification mechanics, the testing methodology supported by the tool (if specified), and the literature item(s) where the tool or framework is mentioned. For tools that are not explicitly provided with a name, we report the name of the authors of the papers where the tool is first described. If the tool is mentioned multiple times in different literature items and different mechanics are mentioned, then we report in the table all the mechanics that are mentioned at least once in the set of papers mentioning the tool. As is evident from the table, we have found the description of 30 different tools and/or frameworks in our pool of literature items. Several tools were mentioned multiple times in the collected sources:

It is worth noticing that three different tools were mentioned only in GL sources:

No testing methodologies have been explicitly mentioned for several tools (i.e., Auction-based Bug Management, CleanGame, CodeMetropolis, DevRPG, GamiWare, GOAL, and Testable).

After the application of Open Coding, we came up with 35 different codes, i.e., categories of advantages discussed by the selected papers. The application of Axial Coding resulted in the identification of three main categories of advantages. In Table 9, we report the complete list of codes in each category and, for each code, its description and the sources where it is mentioned.

In Figure 12, we report the number of manuscripts that report at least one advantage for each of the main categories. It is evident that gamification is mainly evaluated in the selected sources in terms of the benefits that are provided to the tester in terms of user experience (32 WL and 14 GL sources). A lower overall amount of sources mentioned benefits related to effectiveness (21 WL, 10 GL) and efficiency (12 WL, 5 GL). The inclusion of grey Literature allowed the identification of one benefit related to user experience (Empowerment), which was not mentioned in white literature.

After the application of Open Coding, we came up with 23 different codes, i.e., categories of disadvantages discussed in the selected papers. The application of Axial Coding resulted in the identification of five main categories of drawbacks. In Table 10, we report the complete list of codes and for each code, the category, and the sources where it is mentioned.

In Figure 13, we report the number of manuscripts that report at least one drawback for each of the main categories. Bad User Experience proved to be overall the most mentioned concern in the selected set of sources, with 16 mentions in WL papers and 5 in GL. The less-mentioned issues were those related to a reduced effectiveness of the gamified techniques and technical issues related to the implementation of the designed mechanics. No specific drawbacks were mentioned only in grey literature sources.

After the application of Open Coding, we came up with 22 different codes, i.e., categories of challenges and future directions discussed in the selected papers. The application of Axial Coding resulted in the identification of four main categories of challenges and future directions. In Table 11, we report the complete list of codes and for each code, the category, and the sources where it is mentioned.

In Figure 14, we report the number of manuscripts that report at least one challenge or future research direction for each of the main categories. Evaluation is reported as the primary challenge and research direction in the selected literature (28 WL, 6 GL sources). The need for empirical or longitudinal evaluations was mentioned mostly in white literature sources. Fewer sources mentioned the need for design improvements (19 WL, 10 GL) or implementation improvements (18 WL, 5 GL). Two design-related needs (Add competition and Add cooperation) were mentioned in grey literature sources only.

The first goal of our study aimed at defining a mapping of all the elicited sources. The studies were classified according to the literature category, the type of contributor (industrial vs. academic), and the kind of contribution provided to the community.

Bibliometric Trends for Gamified Software Testing. By analyzing the mapping in Figure 2, we can state that gamification in software testing is a topic with a positive trend, presenting a growing number of publications over the years, from both white and grey literature. In the mapping, we see an exception with the year 2020: As a possible hypothesis for the missing grey literature items, we may mention the outbreak of the Covid-19 pandemic and the consequent shift in working modes and communication priorities in the development and testing communities.

Our mapping also allowed us to determine how the research community in this field is distributed. The collaboration graph, shown in Figure 15 (where, for the sake of clarity, we include only authors of at least two papers in our final pool) clearly shows that the different manuscripts analyzed have little to no author overlap, and therefore few collaborations are established in the field between different research groups and most prolific authors. What emerges are many isolated research groups providing autonomous contributions, with a handful of collaborations between different groups. This situation substantially differs from collaboration graphs that are reported for more mature communities (e.g., for the Graphical User Interface Testing Community [44]), confirming the relative immaturity of the field. The most significant research group is composed of G. Fraser and J. Rojas, respectively, with nine and eight items produced.

Types of Contributions. The results related to RQ1.1 denote that the main contributors to the state-of-the-art are researchers from academia, who contributed five times more than industry (57 vs. 11 items) as shown in Figure 4. Industrial contributions are directed toward grey literature, while academia leaned toward the publication of white literature items. Few collaborations have been found. These data denote the fact that industry may be far from applying gamification to software testing: Future research efforts should be directed toward the usage of gamified environments in industrial contexts to execute case studies with the final purpose of validating whether the observed benefits and drawbacks, still mostly analyzed in-vitro, also apply to practitioners.

More than half of the considered sources provided tool presentations or experience reports from the field. We observe a limited amount of literature items (both white and grey) providing guidelines or theoretical frameworks discussing the application of gamification to software testing, as shown in Figure 5. The effort so far has been devoted primarily to creating tools and assessing them in experimental settings. Academic literature currently lacks large-scale empirical studies able to confirm the available preliminary assessment, providing generalizable results. Current literature, especially in the educational setting, mostly provide experience reports that are gathered from ad hoc settings and are sometimes strongly contradicting (RQ1.2). Future research in this regard should provide evidence on how the results obtained in case studies and reports can be reproduced to achieve the claimed benefits, with practical suggestions on how to apply a successful gamified environment, and replication of experiments to actually demonstrate the positive impact of gamification. Future research could also focus on extracting common and universal sets of guidelines for developers of gamified environments for testing practice and education.

The second goal of our study aimed at identifying the characteristics of the software testing activities that are most commonly augmented with gamified components.

Gamified testing levels, phases, and methodologies. The results to review question 2.1 highlight that—currently—gamified techniques are applied mostly to Unit-testing tools, being used in more than half of the considered studies. Next are System and Integration testing tools, both with a significantly smaller number of documented applications. Some items, especially those dealing with software testing education, explicitly mentioned more than one testing level, others neither mentioned, nor intended, any level; this could be explained by the sources discussing a general approach that may be applied to different testing levels. Creating a gamified environment related to integration testing or even system testing appears to be more complex, because those testing levels require the incorporation of libraries, third-party code, or more difficult deployment for the solution than just the integration within an existing unit testing engine. All those additional aspects constitute a significant execution and building overhead, possibly driving away investigators. Future research should address those higher testing levels, rather than just focusing on unit testing and its variations.

We can identify five distinct process stages related to testing: design, creation, execution, reporting, and maintenance. As the results of our review question 2.2 suggest, gamified mechanics are mainly implemented in the phase of test creation and execution, while a limited number of applications of gamified mechanics have been provided for other software testing activities, e.g., test reporting, test maintenance, or test design. The two most frequent phases are the most operative ones, which are repeated several times during the iterative process. When test creation is automated, the creation phase involves the definition and implementation of test scripts, while in manual testing the creation typically overlaps with test execution in the exploration phases of the SUT. Given these premises, it seems natural that the effort has been directed primarily at these two phases. However, we believe that further study is needed especially for test maintenance and test reporting activities. Especially the former, in fact, has been identified by many literature items as crucial and costly in the testing pipeline. Therefore, we advocate further investigations in gamifying testing activities different than test creation and execution.

Regarding the most frequently mentioned test methodologies (RQ2.3), we find that gamified activities are most frequently applied in tasks that are either simple (such as mutation, black-box, and white-box testing at unit-level) or do not require significant programming skills (e.g., manual and Capture & Replay testing at the system level). The presented distribution may serve as another confirmation of the novelty of the gamified approach, mainly applied to traditional white-box and black-box testing activities. Mutation testing is an exception in this respect but, even if it has been mentioned in 15 papers, 9 of them are from the author pair Fraser and Rojas, dealing with the evolution of the same tool, i.e., Code Defenders. The importance of their work is so great that we can consider the two authors as the main investigators in the scope of this research work (as better explained in Section 5.2.1). Future research works should be directed towards other less exposed techniques, aiming at identifying other fertile ground for gamification to be applied.

Tools and languages for gamified testing. The results for RQ 2.4 show that gamification was applied mostly to open-source unit-testing tools. Besides the vast majority of the items (46 out of 73) not specifically mentioning any tool in their discussion, the data shows how widespread is the usage of JUnit. This result is highly justifiable, since JUnit can be considered a de facto standard test runner for any testing activity to be performed on SUTs written in Java. Another commonly mentioned tool in the elicited work is the Capture & Replay testing tool Scout. What emerges from the results is a highly fragmented situation: Apart from JUnit, each research group used a specific tool. The interpretations of this result can be manifold: (i) researchers that are dealing with low-level and simple testing activities can build upon JUnit to implement gamified mechanics; (ii) researchers dealing with higher-level and more complex testing activities (e.g., GUI testing or system-level testing) rely on building completely new testing environments to implement gamified mechanics. Therefore, we stress the need for research for generic, possibly open-source, gamification frameworks that would allow the introduction of gamification in more complex scenarios. The presence of easily implementable and generalizable gamification frameworks may also fuel comparative and quantitative empirical research across multiple settings and different SUTs to better assess the benefits and drawbacks of gamified practices.

Regarding languages (RQ 2.5), the vast majority of the collected studies provide solutions that adopt Java as the target programming language. This data is in accordance with the previous review question, highlighting JUnit as the most used tool, and with other studies affirming Java as the most-used language for test scripting [2]. Regarding the target testing domain, few data are available. Actually, only a few items provide specifications about the domain of application of the gamified testing tool: This data can be explained as the studies are mainly focused on unit testing; at this level, the code under test is agnostic of any domain and it can be used at low-level for any application context (e.g., desktop, mobile, or web). The studies explicitly reporting a domain suggest an almost even distribution between mobile and web, with the exception of a single IoT-related source. New contributors should consider the exploration of gamified testing in the embedded and IoT domain, which remains an unexplored topic for the practice.

Practice vs. Education. The results collected for RQ2.6 show a prevalence in white literature sources for the application of gamification in educational settings. Researchers’ interest in applying gamification in educational contexts is not new; indeed, there is a firm and shared belief in the literature that students (especially adolescents and pre-adolescents) are likely to enjoy a gamified environment due to their familiarity with games [7]. Industrial approaches to gamified software testing are almost equally distributed between white and grey literature, meaning that practitioners are more interested in the application of game elements to support the practical testing process, rather than using them in the testing learning process. Only a few are the proposed approaches that can be used both to teach how to test and to conduct testing in industrial settings. We look forward to future research works to explore the mixed approach, proposing tools, frameworks, and guidelines able to support both test education and practice.

The third goal of the study had the objective of providing a characterization of the most frequently adopted and evaluated gamification mechanics and tools in both white and grey literature.

Game mechanics and Gamification tools. By performing a meta-analysis on the collected data, before applying the synthesis process, we observed two main trends: First, the collected literature items do not adopt an unambiguous definition of game elements; in fact, in most of the literature items, there is no distinction between game dynamics and mechanics. Thus, we categorized all the mentioned game elements as game mechanics for our discussion. In future dissertations, we suggest adopting the more strict game elements classification provided by Robson et al. [43] to distinguish between game mechanics adopted and game dynamics stimulated.

Second, we observed that only a few items employed a standardized ad hoc method to build or to evaluate a gamification tool.³ We highlight the fact that, being a gamified tool highly dependent on the perception of its users, its evaluation should take into account the user experience as a whole (some examples are Octalysis [10] and GAMEX [19]). This implies the selection of the correct evaluation method for a fair assessment of the developed tool and the usage of frameworks helping in the selection of the right game elements to build a balanced and successful gamification environment.

After performing a mapping of gamified mechanics, we observe that few stood out significantly with respect to others. We also adopted the classification provided by the Octalysis framework to aggregate the found game mechanics; this allowed us to discover trends and lacks of existing gamification design. The main trend is to adopt elements based on the Accomplishment core drive, providing positive feedback to the user. Although this trend is not negative in itself, the negative aspect is the lack of a counterbalance in Avoidance elements, which make the game experience quite unbalanced in most cases. Epic Meaning core drive is also quite often neglected, with almost no narrative layer in the tools. . Since gamification is mainly designed to increase engagement and to make testing activities less stressful and redundant, we argue that the addition of black-hat or punishment-related aspects of gamification has not been yet considered by the authors developing gamified approaches, but, since there is no evidence of its uselessness, future research effort should try to incorporate them.

Even though the gamified approach can still be considered in its infancy for software testing research, we found 30 distinct tools in the collected literature. It is worth mentioning, however, that the level of maturity of the tools is highly variable and ranges from mature tools already empirically evaluated in multiple settings (e.g., CodeDefenders) to pure academic prototypes and demonstrators (RQ3.2).

Pros and Cons of Gamification. Many studies among the collected sources analyzed the benefits and drawbacks introduced by gamification in the testing practice. Among the benefits and drawbacks, we identified three factors that were mentioned in both directions: user experience, effectiveness, and efficiency. When discussing the drawbacks, the sources also mentioned difficulties in design and implementation, which were not mentioned explicitly among the benefits. Since the sources often provided little details on the analyzed case studies, to provide a general balance between pros and cons, we simply counted the number of sources that reported the given common factors as either benefits or drawbacks. Figure 16 summarizes graphically the number of mentions for each of the factors by type of literature. A better User Experience is the most reported benefit, with 96 claims in this regard, while only 24 papers mention bad user experience as a drawback of gamified testing. The same kind of proportion is also found regarding the effectiveness of a gamified environment (44 claims in favor, 12 against). By contrast, more discordant opinions are found regarding efficiency (20 claims in favor, 14 against).

These findings show that there is little disagreement about the better User Experience obtained when gamification is applied to software testing. Negative opinions can also be observed in the literature; this contrast should not discourage researchers, since, Gamification being a user-centered technique, it can be considered normal that the way it is perceived can vary in different studies, domains, and practices. A higher disagreement is found when considering the enhancement of effectiveness provided by gamification. We attribute this disagreement to the dependence of the measurement of effectiveness in the practical domain where gamified testing is applied (e.g., effectiveness might refer to bugs found, coverage reached, test cases produced, and so on). In this respect, we highlight the need for a unified metrics framework to assess effectiveness. Positive opinions overcome negative opinions also for efficiency—however, in this aspect, opinions are even more contrasting. We argue that building and maintaining a gamified environment typically constitutes a significant overhead in the software testing process, and it is reasonable that no unequivocal opinion can be found about such necessary overhead. Therefore, we highlight the need for further investigations into the efficiency provided by gamification in future research activities.

Future Challenges for Gamification. Among the future challenges for academic researchers and industry players in the field of software testing, there is a general consensus about the need for a more thorough evaluation and experimentation of gamified mechanics in real-world settings. These evaluations are required, since often the reported results refer to preliminary evaluation. Experts are also required to be involved in such validations, both to determine if the effects observed with students also apply to practitioners and to clarify if the results reached are considered valid by the testers’ community. The category Design Improvements denotes the fact that tools are often developed iteratively by learning from the errors without a systematic approach. Differently, Implementation Improvements show that, even if many attempts have positive results, there is room for features and variations of gamified software testing in other contexts and domains.

Future research and practical direction regarding this topic should include:

Comparing efforts from academia and industry in the field of gamified software testing can be performed in two ways: first, comparing white literature items (mostly associated with the academia) with grey literature items (mostly including industry-related content); second, comparing theoretical with practical dissertations.

The comparison between white and grey literature does not show notable differences, as from the results we can state that researchers and practitioners follow roughly the same trend. Regarding test phases, both WL and GL focus mostly on test creation and test execution; the test levels that are addressed the most are almost equally unit and system testing; also, the adopted gamification aspects follow a similar distribution.

The main difference that we observe between WL and GL items concerns the relative proportion of two mechanics categories: social influence and unpredictability; the former being relatively more common in WL and the latter in GL. Another minor difference that we highlight is in the testing methodologies adopted, where exploratory testing and capture and replay have been considered more for practical usage than for research. Considering Grey Literature in our final pool of literature items, we included three tools, one possible advantage of gamification, and two future challenges that were not signalled in white literature.

By analyzing the type of literature items discussing gamified software testing (either WL or GL), we notice that the main types of contributions consist of tool presentations and experiments. Literature items providing guidelines and frameworks are scarce and typically provide experience-based analyses and not empirical results. General discussions are mainly found in grey literature, hypothesizing possible benefits or providing clues on how to apply gamification to testing activities. All this evidence supports our conclusion that the application of gamification to testing is still in an immature phase without consolidated practice or well-rooted empirical evidence.

The results and discussion presented in this article expand the existing state-of-the-art by complementing the outcome of previous similar secondary studies. In particular, with respect to Mäntylä and Smolander [35], we included 50 more recent additional sources. This allowed us to find a larger set of game elements; we provided a deeper testing-related characterization by decomposing testing into seven dimensions. We also found a meta-analysis of the discussed findings and challenges, providing updated clues for those who will approach the topic in the future.

Comparing our work to De Jesus et al. [14], we included 58 additional literature items containing also grey literature, thanks to the multivocal nature of our study and the usage of both backward and forward snowballing activities, which were missing in Reference [14]. We expanded the authors’ testing characterization from three to seven testing dimensions (including the testing tools used, the target language, the domain, and the goals, whether practical or educational), providing a new point of view and new data to be analyzed. Our game characterization included 13 more game elements, plus a link to Octalysis’ core drives. Another difference is that they focused on the goals that each paper had in applying gamification to testing, while we considered the stated outcomes grouped in benefits and drawbacks, considering also the open challenges presented by authors.

Our results include several new tools and frameworks that were not covered by previous secondary studies. However, we notice that most of the contributors of such tools and frameworks have a practical focus and provide few empirical results and validations; as well as they lack theoretical discussion required to establish consolidated methodologies for gamified software testing. With the current manuscript, we expand the previously presented views on the currently available instrumentation for gamified software testing, hopefully providing inputs for future comparative investigations.

Threats to Construct validity for a Literature Review concern possible issues in reaching full coverage of the spectrum of all the studies related to the chosen topic. This study mitigated that threat by identifying five essential sources of white literature studies that we required to be present in the retrieved pool of sources coming from the used search string in the different repositories. Namely, we selected the following items of literature:

We decided to train our search string iteratively until the essential sources were found. This process required only one iteration. Despite its importance, one of the essential sources was not included in the final pool of sources due to the exclusion criterion allowing only primary studies in the selection. This does not represent a threat, as these studies were used to test the search strings.

Grey literature was also considered to include gamification tools, gamification frameworks, and related empirical evaluations that are not presented in peer-reviewed papers.

For both white and grey literature, we applied a reproducible methodology based on established guidelines. To broaden the research as much as possible, we included the most commonly used terms in the search strings, as well as the main synonyms already used in other SLRs, as seen in Section 2.4. However, it is still possible that some terms describing other relevant works in the literature may have been overlooked. Regarding grey literature, there is a possibility that literature items about relevant tools and frameworks have not been included in the analysis because of the inability to access the documents.

Threats to Internal validity are those related to the data extraction and synthesis phases of the Literature Review. All the primary sources resulting from the search strings application were read and evaluated by all authors and collaborators of this study to assess their quality. When in doubt about the decision to include a resource or not, the final decision was taken by a majority vote. The authors also applied inclusion and exclusion criteria and extraction the information to answer the Review Questions of the study. Hence, the validity of the study is threatened by possible errors in the authors’ judgment when examining the sources and/or misinterpretations of the original content of the papers. This threat was mitigated by multiple readings of the same sources and discussion among the authors about potential disagreements during the review phase.

Threats to External validity concern the generalizability of the findings of the Literature Review. We scoped our research to gamification applied to software testing. We also included in our search literature items in the general field of Software Engineering, but that showed clear applicability to software testing. We can not estimate the generalizability of our findings to the application of gamification to software engineering activities other than software testing. As well, very specific gamification tools and mechanics, testing tools, or domains can exhibit benefits and drawbacks that have been not found in our analysis.