biogitflow: development workflow protocols for bioinformatics pipelines with git and GitLab

Introduction

The importance of best practices for bioinformatics analysis and software have been highlighted by many authors who proposed very valuable guidelines for better reproducibility and traceability (Georgeson et al., 2019; Hamburg & Mogyorodi, 2019; Noble, 2009; Sandve et al., 2013). However, reproducing an analysis is often a challenge (Kim et al., 2018). Therefore, guidelines have to be promoted by the computational labs and bioinformatics core facilities to federate the bioinformaticians across common practices for software development. This is especially essential when the bioinformatics pipeline is used in precision medicine to support the diagnostic and theranostic activities. Indeed, many hospitals worldwide use High-Throughput Sequencing in routine clinical practice to guide the therapeutic decision. This new era of genomic medicine is even promoted in healthcare systems at a large scale within several national initiatives, such as France, USA, UK or Australia (Stark et al., 2019).

This evolution has brought Bioinformatics at the forefront of the healthcare process with the bioinformatics pipeline being a fully integrated component of the clinical decision. Compliance with the healthcare laboratory accreditation standards and regulations is thus required for the development and exploitation of the bioinformatics pipeline. In this context, several authors (Hume et al., 2019; INCa, 2018; Matthijs et al., 2016; Roy et al., 2018) recommended in their guidelines that an appropriate code repository tool should be used to enforce version control. It ensures to track the different releases of the bioinformatics pipeline, their validation and the developers involved in their implementation. This must be integrated in a quality management process with standardized protocols approved for a diagnostic use.

There is a wide ecosystem of tools (see Perez-Riverol et al., 2016; Riesch et al., 2020; Zolkifli et al., 2018) that can support the implementation of such protocols. Among them, we capitalized on git for the version-control system as it became a very popular tool in the software engineering community and GitLab for the repository manager as it can be self-hosted. Both tools offer a large set of very powerful functionalities that can be used and combined in multiple ways thus increasing their usage complexity. It is thus mandatory to formalize through detailed guidelines how to use them on a daily basis for the development and deployment of the bioinformatics pipeline. Therefore, we implemented the set of protocols biogitflow (Kamoun et al., 2020) that describes step-by-step all the command lines and actions to be performed by the developers.

These protocols are mainly dedicated to bioinformaticians and software developers who are involved in the development, deployment and maintenance of bioinformatics pipelines that support routine production. For example, core facilities (such as sequencing platforms) generate, with a very a high-throughput, many data that have to be processed and delivered on the fly to the end-users. Short time-to-delivery and continuity of service under any circumstances is a challenge that can be tackled by promoting these protocols to pave the way towards better industrialized processes for the software component in the context of production, in particular in healthcare.

A comprehensive documentation available at https://biogitflow.readthedocs.io provides all the technical details. We introduce here the main concepts, steps and principles.

Methods

Results

Two use cases have been addressed with dedicated protocols that detail the different actions step-by-step. The first one is the nominal mode (Figure 1) in which a new feature is implemented to improve the bioinformatics pipeline based on requirements and expectations from the end-users who operate it for their daily clinical practice. The second one is the hotfix mode (Figure 2) in which there is a bug in the bioinformatics pipeline in the production environment that hampers the delivery of the results for the patients. The biogitflow documentation provides all the technical details, on how to configure the remote repository in GitLab to develop a new bioinformatics pipeline, how to use git and GitLab depending on the roles and permissions during the time frame of the development workflow for both use cases.

Figure 1. biogitflow protocol for the nominal mode.

This graphical synopsis provides an overview of the different steps of the development workflow when a new feature is implemented in the bioinformatics pipeline according to the role and permissions of the developer. It describes the different git actions that are performed on the different branches and the deployments in the different environments.

Figure 2. biogitflow protocol for the hotfix mode.

This graphical synopsis provides an overview of the different steps of the development workflow when a bug occurred in the production environment in the bioinformatics pipeline according to the role and permissions of the developer. It describes the different git actions that are performed on the different branches and the deployments in the different environments.

Whatever the use case, the bioinformatics pipeline is deployed in production and operated by the end-users once it has successfully passed all the testing. Whenever new patient data has to be analyzed to deliver results for diagnostic or theranostic purposes, the bioinformatics pipeline can be launched by the end-users only if the operational testing reproduces the results from the golden dataset, otherwise it is blocked by internal control mechanism. In this case, the developers investigate the reason of the failure and have to fix it using the hotfix mode.

As a real example, we provided a repository that is available at https://gitlab.com/biogitflow/biogitflow-template (it is recommended that the user registers on GitLab.com to benefit from all its functionalities). This repository has been created according to the guidelines from the section Create a new project in Gitlab from the biogitflow documentation. It contains the different protected and non-protected branches according to the branching model, protected tags, templates for the issues and merge requests, and a set of labels for the issues. The user can fork the repository in a personal workspace and apply the nominal mode (Figure 1) and hotfix mode (Figure 2) following the proposed protocols.

Discussion

While gitflow was a main source of inspiration, biogitflow differs in several ways as:

Writing these protocols required some feedback from the developers in order to decide what was the best way to capitalize on git and GitLab taking into account our internal constraints and organization. The need of formalization required by the laboratory accreditation agency such that a bioinformatics pipeline can be used in healthcare was a great catalyst to implement these protocols. It was a major step to improve our daily practice of software engineering towards better quality management and was a source of motivation to change our work habits. More precisely, the use of the biogitflow protocols contributed to the harmonization of the development practices across developers. Every git pipeline repository is now structured the same way that simplifies the skill transfer from one person to another. This is very convenient, especially when you have to maintain different pipeline repositories with different developers involved. Moreover, due to multiple deployment environments along with the branching model, the developer can really perceives a constructive and positive pressure as long as the deployment in the production (the danger zone) is getting close: this is a source of stimulation for better quality.

In order to promote these protocols, it was necessary to train the users involved in the development workflow to demonstrate to the laboratory accreditation agency that the technical protocols are known, understood and mastered by the developers. Therefore, an internal accreditation process of our developers was implemented. The training consists of a series of exercises that cover all the different use cases, roles and permissions of the protocols. The exercises are first realized in pair with the tutor and the trainee, then the trainee performs the exercises alone twice, and finally the trainee performs the exercises alone but in the presence of the tutor who ask additional questions to ensure that the tricky parts of the protocols are understood. To be endorsed, the trainee has to perform the exercises fluently in full autonomy. All the actions of the training process are tracked in a dedicated GitLab remote repository. Endorsement is valid for one year that can be extended if the developer still masters the protocols. This is assessed during a dedicated yearly interview with the tutor. The exercises are available on https://biogitflow.readthedocs.io.

As depicted in Figure 1 and Figure 2, the development workflow protocols contains many steps that are complex. This complexity arises from the reality of the numerous tasks to be performed to ensure the development, deployment and maintenance of bioinformatics pipelines in the context of production as mentioned in the introduction. Writing the documentation was clearly mandatory in order to provide a formal description of our work methods and promote them internally. Obviously, such protocols contains many manual operations that are prone to error and misapplication. This is the reason why, not only training is important, but also practicing on a weekly (or at least on a monthly) basis is crucial to remain fluent. To circumvent these pitfalls, these protocols would benefit from improved automation to reduce the number of manual steps and avoid possible mistakes. This will be the scope of further improvements that could capitalize of additional valuable functionalities such as git hooks, GitLab CI/CD (continuous integration / continuous deployment) and Operations features.

Conclusions

We described two protocols that are used on a daily practice to develop bioinformatics pipelines compliant with the accreditation standards for healthcare. Bioinformaticians and software developers involved in the development, deployment and maintenance of bioinformatics pipelines in the context of production will benefit from these protocols. While some choices were made to match our internal constraints and organization, the protocols can be easily transposed in other institutes as the main concepts, steps and principles hold in most of the contexts. Indeed, the protocols were internally motivated by the development of bioinformatics pipelines to support the use of genomics sequencing in clinical diagnostic and can be straightforwardly applied not only to genomics but also to any kind of omics approaches (proteomics, metabolomics, etc.), medical informatics or more generally any software development whether it is bioinformatics or not. According to the principle of the Deming wheel of continuous improvement in quality management (Swamidass, 2000), the protocols we described are intended to evolve in order to address future requirements, handle new risk management, integrate new tools or technical frameworks offering better efficiency of the overall process. We strongly encourage the promotion of such protocols in the context of production not only for the healthcare activities but also for research activities.

Data availability

Author contributions

CK, HDS, JR and PH conceived the protocols. PH wrote the protocols and the manuscript that were reviewed by AG, CK, EG, HDS and JR. PH supervised the study.