FAIRSECO: An Extensible Framework for Impact Measurement of Research Software

Abstract

The growing usage of research software in the research community has highlighted the need to recognize and acknowledge the contributions made not only by researchers but also by Research Software Engineers. However, the existing methods for crediting research software and Research Software Engineers have proven to be insufficient. In response, we have developed FAIRSECO, an extensible open source framework with the objective of assessing the impact of research software in research through the evaluation of various factors. The FAIRSECO framework addresses two critical information needs: firstly, it provides potential users of research software with metrics related to software quality and FAIRness. Secondly, the framework provides information for those who wish to measure the success of a project by offering impact data. By exploring the quality and impact of research software, our aim is to ensure that Research Software Engineers receive the recognition they deserve for their valuable contributions.

I Introduction

The FAIR4RS Working Group [1] defines research software (RS) as source code files, algorithms, scripts, computational workflows and executables created during the research process or for a research purpose. In contrast, other types of software, such as operating systems, libraries, dependencies, and packages are not explicitly created for research but are employed in research activities.

Recent studies [2, 3] have shown that 33% of international research produces new code and 90-95% of UK and US researchers use and acknowledge RS as important for their research. In many research projects, Research Software Engineers (RSEs) closely collaborate with researchers to understand their challenges and to develop RS that helps to provide the answers to their research questions [4]. Reusing (parts of) RS is beneficial to the research community. It saves time and effort and encourages collaboration and good coding practices. By encouraging RS reuse, the developers increase their work’s impact and foster a collaborative environment that drives scientific progress [5].

Two factors contribute to the success of RS reuse. First, like research data, the RS should be made FAIR [6, 7]. Following the FAIR principles increases the potential for the RS to be found and reused by other researchers, as well as encouraging good coding practices and enabling software citation.

Second, when reusing RS, proper credit should be given to the developers. Similar to scholarly papers and data sets, RS should get recognition through citations. The current lack of consistent software citation makes it difficult to measure the reuse and impact of RS and creates challenges in giving credit to RSEs for their contributions [8]. Properly citing RS will assist the authors in obtaining financial support for further software development, thereby improving the RSEs career paths.

Many guidelines and tools exist that help developers improve the FAIRness, quality, and citeability of their code [9, 10], [1, 2, 3]. Currently, however, there is no single framework RSEs can use to get an overview of the FAIRness, quality and impact of their RS. Moreover, Gomez-Diaz and Recio [9] and Istrate et al. [11] argue that there is currently no sufficient method to evaluate the quality and impact of RS. To address this issue, we have created the FAIR Research Software Ecosystem (FAIRSECO) framework [4]. The FAIRSECO framework is designed to combine different RS metrics under one extensible framework. It combines data from many existing tools that provide information on license conflicts, dependencies, method-level code-reuse, bibliometric analysis, citation metadata, FAIRness aspects, code indexing, and Software Bill of Materials (SBOM) generation to generate a concise overview of the quality, FAIRness, and impact of RS. FAIRSECO then combines the output of these existing tools into two scores, one for quality and one for impact. This enables RSEs to quickly gain insight into the recognition their software already receives and provide suggestions on how they can improve the quality and potential impact of their software.

The idea of the FAIRSECO framework has been introduced in a conceptual short paper [12]. However, that short paper focuses only on impact measurement through method reuse. While method reuse is important, we have evolved our FAIR software ideas and significantly broadened our scope.

The contribution of current work is twofold:

• Integration of quality, FAIRness and impact measurement tools into a single extensible framework: The FAIRSECO framework consolidates a comprehensive collection of tools designed to measure various aspects of RS quality, FAIRness, and impact into a single, unified framework. This provides researchers with a convenient and efficient solution for measuring and assessing these aspects of their RS.

• Quantifying the quality and impact of RS based on key features: We provide a novel Quality Score ($S_{quality}$) for assessing the quality of RS based on key features, incorporating FAIRness, license violations, maintainability, and documentation as factors in the equation. By considering these aspects together, we provide a robust and objective method for systematically measuring and quantifying RS quality. Similarly, an Impact Score ($S_{impact}$) is introduced based on three factors: the number of citations, the number of reused methods, and the Quality Score.

We organize this paper as follows. First, we show what other tools are available and how they compare to the FAIRSECO framework in the related work Section II. This includes a brief description of each tool used by the FAIRSECO framework. Next, a full description is provided of the FAIRSECO framework in Section III. The FAIRSECO framework is then demonstrated in action in Section IV by its application to an existing RS project. The evaluation of five RS projects using FAIRSECO is presented in Section V. Finally, the paper concludes with a summary and discussion of the FAIRSECO framework infrastructure and a set of steps to take in the near future in Section VI.

II Related work

We observed a shortage of tools and frameworks RSEs could use to measure their impact on the RS ecosystem. Therefore, we aimed to build an accurate framework for evaluating software quality and measuring the effect of RS. To this end, we reviewed FAIRSECO framework and existing tools focusing on identification of license violation (license checking), examining the dependencies of a software project for outdated components or vulnerabilities (dependency checking), method level checking of code reuse (method-level checking), analysis of citation, impact related features (bibliometric analysis), checking presence of citation (citation file checking), checking compliance to the FAIR principles (FAIRness aspects), building the source code index (code indexing), and building a list of all the components and dependencies used in a software project (SBOM generation).

II-A Tools of FAIRSECO

TABLE I: Comparison Table: This table highlights a comparative analysis between FAIRSECO and other tools, emphasizing specific impact metrics.

Tools	Feature
	License checking	Dependency checking	Method level checking	Bibliometric Analysis	Citation file checking	FAIRness Aspect	Code indexing	SBOM Generation
howfairis	✗	✗	✗	✗	✔	✔	✗	✗
SearchSECO	✗	✗	✔	✗	✗	✗	✔	✗
Tortellini	✔	✗	✗	✗	✗	✗	✗	✗
Libraries.io	✔	✔	✗	✗	✗	✗	✗	✗
swg-graph	✗	✗	✗	✗	✗	✗	✔	✗
RSD	✗	✗	✗	✔	✔	✔	✗	✗
Depsy	✗	✗	✗	✔	✗	✗	✗	✗
GitHub	✗	✔	✗	✔	✔	✗	✔	✗
GrimoireLab	✗	✗	✗	✔	✗	✗	✔	✗
SIRGRID	✗	✔	✗	✗	✗	✗	✗	✗
SQAaaS	✔	✗	✗	✔	✔	✔	✗	✗
FAIRSECO	✔	✗	✔	✔	✔	✔	✔	✔

We integrated several tools and external data sources into the FAIRSECO framework, namely, the tools howfairis, Tortellini, and SearchSECO.

howfairis is a Python package to analyze a GitHub or GitLab repository’s compliance with the recommendations given on fair-software.eu  [1]. The howfairis generates a FAIRness report with the help of fairtally [5]. Based on this report, users identify areas where improvements are needed to make their repository more compliant.

The Netherlands eScience Center has developed a tool called Tortellini [3], a GitHub action that checks for any licensing issues in a given software, such as incompatible licenses in the software’s dependencies.

SearchSECO [13] is a hash-based index for code fragments that enables searching for source code at the method level in the global software ecosystem [14]. SearchSECO supports a number of languages and deals with multi-language projects.

Table I summarizes the functionality of each of the tools described above. Howfairis focuses on citation file checking and FAIRness aspect but lacks support for dependency checking and other tasks. Similarly, SearchSECO only performs method-level checking and code indexing, and Tortellini only supports license checking. Finally, FAIRSECO framework performs all listed tasks, except dependency checking. In particular, FARSECO performs license checking, method-level checking, citation file checking, FAIRness aspect checking, code indexing with the help of Howfairis, Tortellini, and SearchSECO. However, FAIRSECO is limited to the Github repositories.

II-B Related platforms

GitHub, the largest open-source repository, uses metrics such as stars, watchers, and forks (as a part of its Bibliometric Analysis) to measure the impact of software. The star system in GitHub functions similarly to the ’like’ button in social media platforms, indicating user interest or support for a project or being used as a bookmark. However, there is a lack of comprehensive and well-founded empirical research to determine the exact meaning and practical implications of ’starring a project’ in GitHub [15]. The credibility of the star metric on GitHub is questionable, as it is possible to create counterfeit or bot accounts that are used to give an excessive number of stars to a project. However, it performs dependency checking through dependency bots, and if citation file is present, it provides options for citing repository.

Libraries.io [6] is a platform designed to assist RSEs in discovering new open-source libraries, modules, and frameworks and keep track of the ones they rely on. Its primary objective is to enhance software quality by addressing three critical issues: discovery, maintainability, and sustainability. Libraries.io performs dependency and license checking but does not support the other tasks.

Depsy [16] measures impact of RS through software citations such as imports by other software and through references to software from papers. It is, however, limited to the RS available via the Python packages repository PyPI and the R packages repository CRAN.

The tooling and services known as Software Heritage Graph (swh-graph) [17] offer quick access to the graph representation of the Software Heritage Archive, an archive that attempts to collect as much software as possible. Currently, it can claim to have the largest collection of source code. These services are collectively called swh-graph and operate based on an in-memory, compressed representation of the Software Heritage Merkle DAG. swh-graph is limited to code indexing, as it does not perform any of the other listed tasks.

The Research Software Directory (RSD) [7] is designed to show the impact of RS on scientific community. Its primary objective is to promote RS reuse and encourage proper RS citation to ensure researchers and RSEs get credit for their work. The RSD, by default, is configured to gather RS data from various platforms such as GitHub, Gitlab, ORCID, Research Organization Registry, Zenodo, DataCite, and Crossref. The RSD performs bibliometric analysis, citation file checking, and FAIRness aspect checking.

Software development analytics platforms and toolsets such as GrimoireLab [18], Sigrid and SQAaaS [19] take a modular approach similar to FAIRSECO employing several tools that collect metadata to compile a quality report of software for their developers. However, none of these tools explicitly consider impact of RS.

SQAaaS focuses on improving RS quality by offering RSEs ready-to-use continuous integration and continuous development pipelines that align with the principles of Open Science [8]. It performs license checking, citation file checking, reports on quality of RS. It is integrated with FAIR data assessment tools which follows the FAIRsFAIR route [20].

Sigrid performs only dependency checking, and the rest of the tasks such as FAIRness aspect, citation file checking, biblometric analysis, license checking are not applicable because the scope of Sigrid is on industrial software and its compliance to the industrial standards.

GrimoireLab is an open-source toolkit focusing on data collection, indexing and storage in the GrimoireLab database for potential analysis of software development. It provides very powerful and flexible basis for analysis, but requires an additional efforts and expertise to extract and display relevant information.

Thus, FAIRSECO offers a unique combination in terms of the considered features for assessing impact and quality of RS.

III The FAIRSECO framework

Figure 1: FAIRSECO architecture consists of five components: 1) Data collection unit gathers data from different data sources, 2) Impact calculation performs an impact calculation, 3) Code analyzer analyzes the code of the RSE project, 4) Report generates final report related to Quality and Impact, and 5) Artifacts generates two artifacts, such as SBOM, and License results for the RS.

III-A The FAIRSECO Extensible Software Architecture

Figure 1 depicts the architecture of the FAIRSECO framework and its components. The framework consists of five primary components: a Code analyzer, Data collection unit, Impact calculation module, Report, and Artifacts module. All these components are interconnected with a FAIR artifact generator. The FAIR artifact generator will receive input from the Code analyzer, Data collection unit, and Impact calculation module and generates Reports and Artifacts as output.

The FAIRSECO integrates different sources of information and tools to support RSEs in becoming more FAIR, improving and measuring RS quality, and measuring the impact of their RS. The framework uses the RS’s repository such as GitHub as input for checking FAIRness and quality and uses other data sources for measuring impact and quality.

III-B Collecting data

The data collection unit consists of a crawler and an API call component to collect citation data and code reuse details from various sources such as Semantic scholar [9], OpenAlex [21], and SearchSECO.

Besides generating a report, the FAIRSECO framework also creates artifacts, such as:

• SBOM SBOM is a structured list of third-party software components and libraries included directly or indirectly in the code [10]. It contains information about open-source licenses and version numbers. FAIRSECO will produce an SBOM for the RS using the tool named SBOM.

• Tortellini Results A detailed report is generated specifically for identifying license violations within the RS. This report provides comprehensive information about the licenses of packages used in the RS, including the primary packages and their dependencies. It includes details such as each package’s specific license types and version information. By examining this report, RSEs identify any instances where the RS may violate license agreements, ensuring compliance with licensing requirements and addressing any potential legal or ethical concerns related to the software’s usage and distribution.

III-C Becoming more FAIR

The FAIRSECO framework assesses the FAIRness score ($S_{fair}$) of RS based on five recommendations for FAIR software [22], and it is calculated using howfairis tool. Explanations for these recommendations are given below:

1. Use a publicly accessible repository with version control: The initial recommendation focuses on verifying that the RS utilizes a publicly accessible repository with version control. This ensures widespread accessibility and encourages transparency. By employing a version control system, RSEs can easily monitor and manage modifications made to the RS, facilitating effective collaboration and traceability.

2. Add a license It is important to emphasize that an appropriate license must accompany the RS. Without a license, even if the RS is publicly available on platforms such as GitHub, the absence of legal permissions makes it impossible for anyone to use it.

3. Register your code in a community registry This recommendation suggests registering RS in a community registry such as Zenodo, CodeOcean, FigShare, Software Heritage Archive, etc. Registering RS makes it easier for others to find it, particularly through the use of search engines such as Google.

4. Enable citation of the software Citation is an integral part of scientific accountability and reproducibility. Citation helps RSEs be recognized for their work. Citation File Format (CFF) [2] is specifically designed to enable the citation of software.

5. Use a software quality checklist This recommendation proposes examining quality checklists that have been included on fair-software.eu

The $S_{fair}$ is calculated by combining these recommendations, the RS gets a maximum score of 5, and each recommendation contributes a maximum score of 1 (cf. [1], [5]). We will provide a more detailed explanation in the below subsections on how these scores are calculated.

III-D Measuring RS Quality

The code analyzer in the FAIRSECO framework architecture evaluates the quality of RS. It employs various techniques to analyze the RS, encompassing the evaluation of FAIRness [24], identification of license violations, examination of software maintainability, and checking the presence of documentation. Using $S_{quality}$, RSEs evaluate their project’s development and identify improvement areas. The following sections describe the main features contributing to $S_{quality}$.

III-D1 License violation or information

License violations have significant consequences for violators and open-source communities [25]. These violations damage the reputation of a project and discourage further contributions from the community, which have negative long-term effects [26].

We argue that researchers and RSEs should carefully review the license associated with any open-source software before using it and ensure that they agree to the terms and conditions of that license. This practice can foster the perpetual expansion and prosperity of the open-source ecosystem.

We utilized Tortellini for predicting license violations; the FAIRSECO framework helps users become more aware of potential license violations and the potential problems that arise.

III-D2 Maintainability

Software maintainability refers to the ease with which the operations, such as the addition of new features, obsolete code deletion, and error correction, are carried out [27]. It is an aspect of software development that consumes a significant portion of the overall project cost. However, measuring maintainability during the early stages of software development aids in better planning and optimal resource utilization [28].

The FAIRSECO framework analyzes open and closed issues on GitHub to assess maintainability. It calculates maintainability by calculating the percentage of issues that have been closed since the date on which the software was added to GitHub. RSEs utilize the maintainability score as a guideline to improve their development process and address current open issues more efficiently.

III-D3 Documentation

To be usable, open-source software needs to be sufficiently documented. This helps users understand the functionality and usability of the software. GitHub and Zenodo are platforms that typically include a README file and documentation for each software, giving users information about the software.

The FAIRSECO framework checks whether the RS is adequately documented to ensure users access the necessary documentation. Therefore, FAIRSECO assigns a maximum score of 100 if the RS includes both README and documentation.

III-D4 Reporting on Quality

The code analyzer generates reports that summarize the analysis results. These reports provide RSEs with actionable feedback on the quality of their code. The reports highlight areas that require improvemnt, encompassing aspects such as FAIRness, license information or violations, maintainability, and documentation. By reviewing these reports, RSEs make informed decisions to enhance the overall quality of their RS.

Finally, we have formulated an equation to determine the $S_{quality}$, incorporating factors such as FAIRness, license violations, maintainability, and documentation:

$$S_{quality}=\frac{S_{fair}\cdot W_{f}+S_{l}\cdot W_{l}+S_{m}\cdot W_{m}+S_{d}\cdot W_{d}}{W_{total}}$$ (1)

The all weights, namely, the FAIRness weight $W_{f}$, license weight $W_{l}$, maintainability weight $W_{m}$, and documentation weight $W_{d}$ are four distinct constant assigned to each factor. The total weight $W_{total}$ is calculated by summing up these weights $W_{total}=W_{f}+W_{l}+W_{m}+W_{d}$. Currently, $S_{quality}$ is determined using weighted criteria; however, future work will investigate the justification behind these weights. The scores used in (1) are calculated as follows:

• The $S_{fair}$ is calculated by multiplying the howfairis tool’s output (0-5) by 20.

• The license score $S_{l}$ is calculated using the license information provided by the Tortellini tool as

  $$S_{l}={F_{l}}^{\log_{2}(1+N_{l})}\cdot 100$$ (2)

  where $F_{l}$ is the fraction of licenses that are not violated, and $N_{l}$ is the total number of licenses.

• The maintainability score $S_{m}$ is calculated as follows:

  $$S_{m}=\frac{100\cdot C_{i}}{N_{i}}$$ (3)

  where $C_{i}$ is the number of closed issues, and $N_{i}$ is the total number of issues.

• The documentation score $S_{d}$ is determined based on the presence of README, and documentation files. The score is assigned a value of 50 if there is a documentation directory and another 50 if there is a README file. These scores are then combined to obtain the overall documentation score.

Thus, a score between 0 and 100 will be assigned to the RS based on Equation (1). The calculations described above represent our current FAIRSECO approach, considered a starting point. However, we plan to enhance the formulas by incorporating more detailed and higher-quality metrics. The framework is designed to be sufficiently generic to accommodate these expansions.

III-E Measuring RS Impact

One of the largest challenges in providing insight into RS impact is where and how to report this data. There are indicators such as GitHub stars, inclusions of a package in dependency trees, citations to articles, mentions in news items, the number of issues posted by non-community members, etc. Gathering and interpreting these data is challenging. For FAIRSECO, currently, we have implemented two impact measures: citations and code reuse. Furthermore, researchers would likely hesitate to use poor-quality RS that introduces bugs, performance and maintenance issues, etc., because of potentially erroneous results produced by the RS and higher costs for fixing its technical debt [29]. Thus, we consider the quality of RS as an additional factor that influences $S_{impact}$, along with citations and code reuse. This intuition is in line with recent efforts of policymakers regulating long-term plans for maintaining RS [30].

III-E1 Citation

In academic writing, the practice of citing relevant literature to establish connections with other works in the field is considered essential and common practice [31]. Recognizing the significance of this practice, FAIRSECO offers valuable assistance to its users by facilitating the rapid and efficient retrieval of citation data for their RS. This is accomplished by gathering information from APIs of the open catalogues of scientific publications (in particular, OpenAlex and Semantic Scholar), enabling users to conveniently obtain the necessary citations for their RS. Moreover, it collects information about the scientific fields in which the RS is cited. It generates a radar citation graph [32] illustrating how the RS is connected across different scientific fields. A screenshot is provided in Figure 3.

III-E2 Code Reuse

For this, the FAIRSECO framework utilizes SearchSECO: it feeds a repository link to SearchSECO, which then parses all the source code from the RS project and calculates a hash for an abstract version of each method in the work. Subsequently, SearchSECO looks in its database for occurrences of the method in the worldwide software ecosystem. If found, there are two possible outcomes:

1. (A fraction of) the RS is being reused by other parties, showing its impact. No further action is needed if the other software uses the RS’s intellectual property correctly.

2. The RS project is reusing other software. While this does not lead to impact, it does provide an opportunity to ensure that the code in the RS project is used correctly with correct references and that the reused methods do not contain any vulnerabilities, which is a feature of SearchSECO.

III-E3 Reporting on Impact

In addition to citations and code reuse, we include $S_{quality}$ as a factor influencing the Impact score. Similarly to the $S_{quality}$, the $S_{impact}$ is determined by applying constant weights to factors such as the number of citations $N_{c}$, code reuse in other projects ($N_{r}$) as well the $S_{quality}$ obtained from Equation (1):

$$S_{impact}=\frac{N_{c}\cdot W_{c}+N_{r}\cdot W_{r}+S_{quality}\cdot W_{q}}{W^{\prime}_{total}}$$ (4)

where citation weight $W_{c}$, reused weight $W_{r}$, quality weight $W_{q}$ are constants. $W^{\prime}_{total}$ is calculated by summing up these weights: $W^{\prime}_{total}=W_{c}+W_{r}+W_{q}$.

Note that these constant weights can be changed after an expert evaluation and the impact measurement metric can be extended with more elaborate metric.

IV The FAIRSECO framework in Action

This section overviews how RSEs utilize FAIRSECO to generate reports on FAIRness, Quality, and Impact. Generally, RSEs will want to use the FAIRSECO framework every couple of months or when they wish to provide a report on the impact of their repository. The history feature of the FAIRSECO infrastructure supports this, as the RSEs show the improvements they made (or did not make) regarding the impact and citation of their software. For a complete manual to operate the FAIRSECO GitHub Action, please see the documentation at README.

RSEs face challenges in monitoring the usage of their software, tracking academic papers that reference it, and identifying software built upon it. With the FAIRSECO framework, RSEs easily access information on license violations, impact, citation details, and the $S_{quality}$ of their RS. This eliminates the need to identify such details, thereby saving time and effort manually. The report generated by the FAIRSECO framework increases the chances of other researchers citing the work of RSEs.

To illustrate the functionality of the FAIRSECO framework in a RS context, we have chosen the Mcfly tool [11] from GitHub. The following are the pages or tabs within the FAIRSECO framework, each with its respective explanation:

Figure 2: Overview tab: This screenshot displays a summary of the FAIRSECO report, including the number of citations, $S_{fair}$, matching methods, $S_{quality}$, and several metrics from GitHub. For a more detailed understanding of the $S_{quality}$, please refer to Section IV

1) Overview tab: In Figure 2, we provided a screenshot of the “Overview” tab of the framework, which displays various details about the evaluated tool. These details include the repository title, owner, issue date, number of stars, watchers, and forks. Detailed explanations of each of these terms are given below:

1. Repository title: Repository title refers to the repository’s name, which is evaluated using the FAIRSECO framework.

2. Repository owner: The repository owner is the name of the individual or organization who owns the repository on GitHub. In case the repository has been forked, it shows the user’s name who forked the primary repository.

3. Date of issue: The issue date refers to when the repository was assessed using the FAIRSECO framework.

4. Repository stars: The Repository stars refer to the number of stars the repository receives on GitHub.

5. Repository watchers: The number of users who have added the repository to their watchlist on GitHub is added to the Repository watchers list.

6. Repository forks: The Repository forks refers to the number of times the repository has been forked on GitHub.

Moreover, the FAIRSECO framework gives further information, including citation counts, FAIRness value, method matching with other projects, and $S_{quality}$ for the evaluated repository. Detailed explanations of these terms are provided in the FAIRSECO framework’s next tab. Additionally, there is an option to download this report as a PDF.

2) Citation tab: An overview of the research papers that have cited RS is presented in the citation tab. The list of papers includes their title, author’s names, description indicating the source from which the paper is collected, publication date, DOI, and a link to the paper.

3) FAIRness: The report on FAIRness is generated by analyzing the results of the howfairis tool, and it is presented in the FAIRness tab. For this example tool, it has received a $S_{fair}$ of 4 out of 5, because it did not meet all the quality criteria outlined in the 5 recommendations for FAIR software.

4) License violation: The FAIRSECO employs the Tortellini tool to provide a comprehensive account of license violations, informing the original author of the repository or the user about any infractions. The license violations tab displays the information regarding license violations.

5) Impact: As shown in Figure 3, the Impact section of FAIRSECO displays various statistics about using the software in scientific research. It includes information about the frequency of citations the software has received across different fields, visually represented in a radar chart showcasing these fields’ distribution. Additionally, it provides information on the most significant scientific paper that cited the software based on the number of citations and the journal’s reputation.

Figure 3: Impact tab: The Impact tab screenshot presents an overview of the RS Mcfly impact. It has been cited a total of 4 times, with two of those citations coming from different fields. Additionally, the screenshot includes a radar graph that illustrates the software’s citation frequency across various fields. For a more detailed understanding of this tab, please refer to Section IV.

6) Quality Score ($S_{quality}$): The $S_{quality}$ tab is illustrated in Figure 4. The FAIRSECO assigns a $S_{quality}$ to each project and explains how this score was calculated. Four key metrics are considered, including the percentage of adherence to FAIRness principles, the extent to which the software complies with the license requirements, the percentage of closed issues on GitHub, and the presence of README and documentation files. This score doesn’t reflect the actual quality of a project’s code but rather indicates how well the project follows the best practices of software engineering. The example repository received a $S_{quality}$ of 92% out of 100 because it did not pass the quality checklist. Thus, it receives a FAIRness value of 4 out of 5.

Figure 4: Quality Score tab: This screenshot shows an overview of the generated FAIRSECO $S_{quality}$. For a complete explanation of the $S_{quality}$, see Section IV

.

7) Impact History: The impact history tab provides a historical overview of the impact of RS by displaying information from previous runs of the FAIRSECO, if available. This feature allows RSEs to track how the impact of their software has evolved. The overview section shows the changes in $S_{quality}$, number of citations, reused methods, and $S_{fair}$ since the last run. Ideally, RSEs will observe an increase in these metrics over time. The graph below visually represents this data, including runs before the most recent one.

TABLE II: Evaluation table: This table compares selected tools using the FAIRSECO framework. See Section III-C for the definition of R1-R5.

Tools	Feature
	R1	R2	R3	R4	R5	FAIRness	Licenses	Maintainability	Documentation	Quality Score
Kernel Tuner	✔	✔	✔	✔	✔	100%	100%	100%	50%	94%
Spec2Vec	✔	✔	✔	✔	✔	100%	100%	100%	50%	94%
ESMValTool	✔	✔	✔	✔	✗	80%	100%	100%	50%	86%
GPT index	✗	✔	✔	✔	✗	60%	100%	100%	100%	85%
CFF-Converter-Python	✔	✔	✔	✔	✔	100%	19%	100%	100%	78%

TABLE III: Impact Score Table: Impact Score table shows the influences of various factors on the overall Impact Score

Tools	Feature
	# citations	Reused in other Projects	Quality Score	Impact Score
Kernel Tuner	17	5	94%	60%
Spec2Vec	4	5	94%	56%
ESMValTool	40	4	86%	61%
GPT index	0	3	85%	50%
CFF-Converter-Python	0	5	78%	46%

V Evaluation

Table I compares five tools based on their performance concerning features such as Recommendation 1 (R1), Recommendation 2 (R2), Recommendation 3 (R3), Recommendation 4 (R4), Recommendation 5 (R5), license violation, maintainability, and documentation. The FAIRSECO framework calculates a $S_{quality}$ for each evaluated tool based on these features, also provided in the table. Each of these tools such as Kernel Tuner [12], Spec2Vec [13], ESMValTool [14], GPT index [15], and CFF-Converter-Python [16] were evaluated using the FAIRSECO tool. Among the five tools compared, Kernel Tuner and Spec2Vec met the FAIRness criteria and had a higher $S_{quality}$ than the other tools. Kernel Tuner stands out in its FAIRness as it is registered in Zenodo with a DOI, making it easily findable. Additionally, it is publicly accessible through GitHub and downloaded using standard protocols such as HTTPS. Moreover, Kernel Tuner has citation and license files, further enhancing its FAIRness. Therefore, Kernel Tuner scores a perfect 5 out of 5 regarding FAIRness criteria. According to our analysis, it does not violate any licenses. Kernel Tuner demonstrates effective maintenance by actively closing issues on GitHub, which is properly documented through documentation files and a README file. The ESMValTool does not meet the quality checklist mentioned in 5 recommendations for FAIR software and receives a lower $S_{quality}$ than Kernel Tuner and Spec2Vec. GPT index scored lower in the R1 and R5 categories but performed well in R3, R4 and R5. CFF-Converter-Python scored relatively low in the $S_{quality}$. Still, it performed well in all five features, scoring less due to issues with the version of the PyPI package, resulting in a license violation.

In summary, Kernel Tuner and Spec2Vec stand out as highly reliable and FAIR tools, since while ESMValTool, GPT index, and CFF-Converter-Python show strengths in specific areas but may require some attention to enhance their overall performance.

Table III gives the $S_{impact}$ for selected tools. As per Equation (4), $S_{impact}$ is calculated based on three factors: the number of citations, the number of methods reused, and the $S_{quality}$. For example, the tool Kernel Tuner has 17 citations, has been reused in 5 other projects, and has a $S_{quality}$ of 94%. Using Equation (4), the $S_{impact}$ for Kernel Tuner is 60%.

The FAIRSECO framework provides a comprehensive report on the quality and impact of RS on research. RSEs utilize this report to identify areas where improvements are needed and make necessary changes to their software.

VI Conclusion and Future Work

RS is becoming increasingly valued in the research ecosystem, leading to the evolution of international and national policy practices to reflect its importance. The FAIRSECO framework, with its collection of tools, is designed to report on the quality and impact of RS. FAIRSECO framework automatically measures the FAIRness of the RS using five recommendations for FAIR software. This is achieved by conducting several checks to ensure that the RS is registered in a community registry, making it available in a public repository, using a quality checklist, encouraging the use of the CFF to facilitate appropriate citation, and including a license file to enable reuse by other researchers.

To assess the quality of the software, the FAIRSECO framework calculates a Quality Score based on several factors, including FAIRness, license violation, maintainability, and documentation. It also generates an SBOM for the given repository. In addition, FAIRSECO provides insights into the software’s impact on the RS ecosystem by gathering citation data and tracking its reuse by other software. The Impact Portal generated by FAIRSECO enables RSEs to assess the success of their software and gain insights into its impact. By showing this, we hope to raise awareness that software is valuable research output.

From the FAIRSECO report, RSEs understand the areas they must focus on to make their RS available worldwide and improve its quality. This allows users to quickly find the RS they need for their purposes, which increases its popularity. When an RS gains more users, its impact on research helps RSEs receive the credit they deserve.

In the future, we plan to:

1. Evaluate with RSEs – As part of the design study, we plan to present the framework to RSEs and gather their feedback for evaluation, and further development.

2. Promote the FAIRSECO GitHub Action – We want to encourage RSEs to use the FAIRSECO framework. We will organize events for them and present the framework at various FAIR software events.

3. Create a Live Dashboard for FAIRSECO Impact – The current GitHub action is a manually triggered task. We imagine that RS projects need a continuous impact dashboard, but we will evaluate this first with the intended users.

4. Expand the functionality of the FAIRSECO tool to support other platforms – Currently, the framework is designed to process projects hosted on GitHub, primarily through a GitHub action. However, we aim to explore the potential of extending FAIRSECO’s capabilities to include repositories or tools hosted on various other platforms. This expansion will allow for a broader range of applications and make FAIRSECO more versatile and accessible to users across different development environments.

5. Expand documentation score – At present, FAIRSECO’s functionality is limited to checking the presence of documentation in the repository. However, in the future, our aim is to use Natural Language Processing techniques to verify complexity and extensiveness of the documentation.

6. Introduce a dependency score – Currently, our code reuse score is restricted to the results obtained from SearchSECO. To enhance the effectiveness of the code reuse analysis, we intend to also incorporate a dependency score by detecting which other libraries and tools reuse a particular RS. This additional score will strengthen the outcomes of the code reuse analysis significantly.

7. Score Selection and Weighting – There are different stakeholders in the development of a research software project, such as the scientists who need it, the funding agency, and the RSEs. Each of these is interested in different scores; the scientists want to know who uses the software in their network, the funding agency wants to know whether the software is developed so it does not suffer too much from so-called software rot, and. the RSEs wish to know how many open bugs there are and their severity. For this reason different scores are required to provide insight into the different qualities of a research software project. Furthermore, it could be that composite scores are needed to rapidly provide insight into the software’s qualities.

8. Expand the notion of impact – Ideally, a term impact would mean how the RS has influenced and advanced research that it was used in [33]. However, this is difficult to measure with the data available for RS. Most of the research uses quantitative impact metrics from bibliometric analysis, namely, the number of citations and code reuse (which applies to the software only). We have included these and a Quality Score as yet another factor influencing impact. In the future, we want to consider a broad notion of impact that includes both academic and socio-economical aspects, such as the number of grants and information regarding people hired to maintain the RS.

By extending FAIRSECO with these features, we strive to guide researchers and RSEs to adapt high quality standards for their RS, and provide more accurate insight into (potential) impact of their RS.

Acknowldegment

We thank the eScience Center for funding this Project (NLESC.CIT2021.002). We also thank the students who supported the software development of the FAIRSECO framework: A. Aydin, B. Hageman, B. Lankhorst, J. Hendriksen, Q. Donkers, R. Schouten, T. Bolhuis, & V. Bykova.