Introduction

In recent years, the scientific world has been witnessing a significant increase in scrutiny, once solely focused on the results, and now it also focuses on the related issues of reliability, safety and efficacy. Reproducibility of scientific results is now demanding care, in light of the increasing number of cases of irreproducibility and retractions of published articles, some of them because of obvious fraud, others simply due to lack of controls or poor practice [1, 2]. The problem of reproducibility has also risen to attention in major scientific literature, stimulating vigilance about this problem in the scientific research world [3, 4]. In order to guarantee reliable results and improved data consistency, while operating with reduced funding, scientists need to acquire a new culture of management, more tools and specific training, so as not to be left floundering in a field that, apparently, has little to do with scientific experimentation. Research management founded on a quality approach is emerging as an essential tool to ensure valuable, robust and dependable outcomes, within a framework of the best practice [2, 5]. The importance of the result for the final users, a central concept in the quality approach, easily leads to effectiveness—intended as a guarantee of results—prevention and safety, as well as to efficiency, with rigorous resource management and minimization of waste. This approach strengthens the reproducibility of any research process, reducing sources of error and variability: Errors if intercepted cause reworking, and if not, they generate inaccuracy and even wrong results. As far as variability is concerned, it can induce such inaccuracy as to risk the non-reproducibility of research results. Moreover, a lean, reliable experimentation process avoids waste of material and time, i.e., unfruitful expenditures, thus optimizing the use of funding. To face issues of this kind, quality disciplines have been widely used for decades in industrial and business fields, from which useful examples and directions can be derived.

Among them, failure mode and effect analysis (FMEA) is a systematic approach for identifying all possible failures in a design, in the manufacturing or assembly process, or in the product or service [2, 6]. FMEA was first developed by the US military in the 1940s and came into its full use when it was adopted by the National Aeronautics and Space Administration (NASA) in connection with manned space missions (mid-1960s). Widely practiced throughout the automotive, software, food services and many other industries, only in recent years FMEA has been also successfully applied in the health care as a proactive tool to improve patient safety and efficiency in hospitals [7, 8]. Today, FMEA is making its way as a tool for risk assessment of biopharmaceutical manufacturing processes, analytical procedures for screening drugs and more recently clinical trials [9,10,11]. However, there are still too few reports about this last use and even fewer data are available on the application of this methodology in non-regulated research laboratories.

As part of the Quality and Project Management OpenLab (qPMO) research network [12] of the National Research Council of Italy, we aimed at testing the validity and applicability of quality methodologies to basic science and then at developing custom research models of Quality Management. One of the four work packages of the project [12] was specifically devoted on the application of process FMEA (a.k.a. P-FMEA) as a powerful tool to examine potential causes of failures of a scientific protocol, evaluate risk management priorities and identify specific actions to mitigate the risks. Our working hypothesis is that FMEA application on Life Sciences research activities can be of great value, helping in keeping under control changes, manual operations and hazardous activities. FMEA focuses on process development and on the control of opportunities for error and represents a formal documentation which includes detailed description, risk assessment and information about necessary process controls. Therefore, risk analysis could be especially useful for processes developed in the laboratory that are likely to be transferred to exploitation.

In order to demonstrate the validity of FMEA analysis in a Life Sciences research laboratory, we applied this method to a specific experimental process within the framework of a joint project between Institute of Experimental Endocrinology and Oncology “G. Salvatore”(IEOS) and a biopharmaceutical company, aiming to develop oligonucleotide aptamers as innovative tools in cancer diagnosis and therapy. Aptamers are short, artificial, single-stranded oligonucleotides that can selectively bind to cell surface receptors overexpressed in cancer cells. The wide-ranging utility of aptamers derives both from their versatility, with high selectivity and sensitivity, and from their easy manufacturing and structural stability. The possibility to generate aptamers that bind and inhibit transmembrane receptors involved in cancer progression through a variant of the SELEX technology that makes use of living cells as complex targets, has been amply demonstrated [13,14,15]. We have chosen this process as an example of a complex multi-step process involving several procedures, equipment, technologies and human skills, also suitable for exploitation.

Our pilot application of FMEA analysis on a research activity shows the validity of this approach. FMEA application contributes to building a control framework for key laboratory protocols so as to guarantee better performance and improved reproducibility of results. We also propose “FMEA strip worksheets” as a new and useful tool to facilitate risk analysis in non-regulated research laboratories, performing a thorough evaluation of experimental procedures and processes. Successive phases of risk review and monitoring will be fundamental to determine the advantages of this approach at both medium and long term on the performance of the research process. This work is strictly linked to another work package of the qPMO research network, aimed at developing guidelines for research activities [14]. Both guidelines and FMEA analysis outputs merged in the Quality Management System (QMS) model for a research laboratory [12]. The various quality tools customized and/or developed by the qPMO network in order to support the research show homogeneity of principles and structures, so as to ease the application of more than one tool, or the switching from one to another. Our final aim is to support the spreading of quality culture and competence among scientists, the development and utilization of research-based Quality Management tools and, ultimately, the formation of a critical mass that can really deal with these topics without prejudice and analyze their impact on research performance and management.

Methods

We have applied the FMEA on a “pilot” process, developed for the selection of cell-specific aptamers, and it comprises three sub-processes: (1) RNA Aptamer Dephosphorylation and Extraction; (2) Aptamer Phosphorylation and Purification; and (3) Cell-binding Assay. The management of the analysis was performed according to the ISO 31000:2009 standard and the execution of the analysis according to IEC 60812 standard [17, 18]. Please note that the work was not updated following the new version of the ISO 31000 standard issued in 2018. The project scope was determined as “trialling the risk assessment approach to an experimental procedure of a scientific non-regulated research laboratory by means of the FMEA methodology,” and its goal was “to demonstrate the validity of FMEA on a research laboratory procedure”; then, the following steps, recommended by the ISO 31000:2009, were followed to manage the project:

  1. i.

    Establishing the external and internal contexts

External and internal contexts were determined, to identify the most suitable experimental process on which to perform the risk assessment, with regard to the importance of the research topic, the need for a more detailed process control and the meaningfulness of the analysis.

  1. ii.

    Establishing the context of the risk management process

The objective of the FMEA was to demonstrate its validity for a scientific protocol of basic research, thus producing valuable proposals for the improvement of research performance, process control and the overall working environment. The team for the FMEA analysis was composed by researchers and technologists belonging to the qPMO network [12], with specific molecular and cellular biology skills, trained on the job by a quality consultant. The IEC 60812 standard was kept as a main reference for the execution of the risk analysis.

The FMEA team at IEOS collected information about the “pilot” process from the researchers who had developed it and routinely applied it and performed preliminary risk analysis, while the other researchers of the qPMO network belonging to the other institutes reviewed and integrated the analysis according to their specific competence. All information regarding laboratory organization was also recorded. The pilot process was described using a flowchart, and the analysis was performed for each step (Fig. 1).

Fig. 1
figure 1

Flowchart of the pilot process under analysis. The process comprises three sub-processes: a RNA Aptamer Dephosphorylation and Extraction; b RNA Aptamer Phosphorylation and Purification; and c Cell-binding Assay. The double-sided boxes are predefined processes (series of activities no further detailed)

Full size image

Some terms used in the standard FMEA are ill-suited for a research laboratory, so the working team focused on the principles and the meaning of the terms to be used and substituted some of them with others more suitable to the application of FMEA in Life Sciences research laboratories, for instance “error” or “negative effect” instead of “failure” or “impact,” respectively; failure and error, however, were occasionally used as synonyms. Finally, the team was asked to evaluate the risk analysis tool itself—i.e., FMEA—with respect to usefulness and usability, and possibly to propose a solution for a more streamlined application in Life Sciences research.

  1. iii.

    Defining risk criteria

Each failure mode was ranked by its severity (S), its estimated frequency of occurrence (O) and the probability that the error would remain undetected (D); each parameter was determined on a scale from 1 to 10 (from lower to higher importance). This evaluation was performed by the IEOS team by means of the scheme reported in Table 1 and accordingly with IEC 60812 standard tables adapted for the research field and reviewed by the qPMO network. For each identified failure mode, a Risk Priority Number (RPN) was calculated by multiplying the values of S, O and D. Guidelines about FMEA [19] require that operations are assessed for actions by high values of severity and then of occurrence. RPN can be used to rank operations and to apply the Pareto “80-20” principle [20] to identify those deserving actions to reduce the associated risks. To simplify the first approach, we preferred to follow the previous method, using a RPN threshold indicating the level of risk considered affordable for the process. Using the chosen tables, the highest theoretically possible RPN value is 1000 (10 × 10 × 10) and the lowest possible value is 1 (Table 1). There is no best practice or rule of thumb for setting the RPN threshold limit; organizations also in healthcare environments, from clinical laboratory to pharmaceuticals industries [21, 22], use values between 40 and 150 depending on the point of development of their process or product/process requirements, the most frequently used value being around 100. As an example of a compromise between the two approaches, the carmaker Daimler [23] sets the RPN threshold from 10 to a maximum of 100, depending on the S value. Therefore, on the first attempt, we empirically set the maximum allowable RPN at 100, a generic value suitable for most applications, and also decided to submit to improvement actions the operations with the highest severity value. We then planned to assess the suitability of the chosen criterion once the analysis was completed.

Table 1 Tables for severity, occurrence and detection rating scale
Full size table
  1. iv.

    Risk identification, analysis and evaluation

According to the ISO 31000 standard, risk identification refers to the detection of possible errors, while risk analysis refers to the identification and assessment of consequences, causes and present controls, in our case by means of the three FMEA parameters S, O and D. For risk evaluation, RPN is calculated multiplying S, O and D and then compared with the predefined RPN threshold, in order to identify the opportunities for error deserving improvement actions.

The flowcharts of sub-processes (Fig. 1) were analyzed by means of a FMEA template (Fig. 2) by seeking every opportunity for error. The template is organized in rows, one for each process step, and in columns for FMEA characteristics and parameters. The elementary operations of the process are listed in the first column Process step or input. For each operation, one or more opportunities of error are recorded under the column Potential failure mode and the related effects are detailed in the column Potential Failure Effects, together with their assessment in terms of severity in column SEV. For each potential error, the cause(s) are identified and recorded in column Potential Causes and the related probability of occurrence in the column OCC. The actual controls to avoid undesired consequences, reduce the probability of errors and filter unsuitable results are listed in the Current controls column and their assessment (detection) in column DET. The RPN is then calculated as the product of S, O and D and inserted in the dedicated column RPN. An example of how the information is generated from the FMEA application on the first step of Fig. 1a is shown in Fig. 3.

Fig. 2
figure 2

FMEA template. The FMEA template is headed by general information about the process, the process team performing the analysis and the execution of the analysis

Full size image
Fig. 3
figure 3

Example of FMEA application on the first step (Dephosphorylation, double-sided boxes) of Fig. 1a

Full size image
  1. v.

    Risk treatment

For each opportunity for error whose RPN is above the threshold, one or more actions have been identified to lower S, O or D and these are listed in the column Action Recommended (Fig. 2). However, the person in charge (in the column Resp.) and the state of the action, together with forecasted and actual dates (in the column Action Taken), were not actually defined in the present paper, being beyond the purpose of the article. Therefore, also the successive risk treatment plans and monitoring and review foreseen by the ISO 31000:2009 standard are not reported in the manuscript. The reason for this is that the primary purpose of this article was to test the validity of FMEA risk analysis to basic scientific non-regulated research, giving a first hint on the FMEA application in a context distant from automotive, manufacturing and therapeutics, in which it is almost applied on a routine basis, and leaving the impact of the actions taken and the successive follow-up for next reports.

  1. vi.

    Recording the risk management process

The FMEA template (Fig. 2) is a comprehensive tool for recording information regarding these processes. Moreover, a scientific report and relative guidelines [14] have been delivered, disseminating the experience in applying this method to improve scientific protocols and their outcomes.

  1. vii.

    Comparison with the 2018 edition of ISO 31000

The work was completed before the issue of the 2018 edition of ISO 31000. We decided to keep the previous edition as a reference not to denature the original approach, which actually corresponds to the process of risk assessment outlined by the 2018 edition. In addition, the 2018 edition emphasizes the design of a risk assessment, i.e., criteria and directions to preparing, planning and conducing the risk assessment, tailoring it to the organization, obtaining proper commitment, assigning roles and responsibilities and arranging for appropriate communication. Those specific aspects acquire great importance when risk assessment is carried out in an organization with a clear target to manage and minimize; however—as outlined in the goals of the present work—this was outside the purpose of the study.

Results

We have performed FMEA analysis on a “pilot” process developed for the selection of cell-specific aptamers, divided into three sub-processes. For each sub-process, we designed a flowchart (Fig. 1) that was analyzed with the FMEA template (Fig. 2) seeking every opportunity for error and defining adequate improvement actions. The three sub-processes consist of 29 steps (elementary operations), including pipetting, centrifuging, measurements (spectrophotometer and β counter) and the use of other equipment. The results of this analysis are illustrated in Table 2. It is worth noting that the number of possible errors indicated by the RPN differs from the number of operations, because for each operation several errors can be identified.

Table 2 For each of the three sub-processes: Dephosphorylation and Extraction, Phosphorylation and Purification, and cell-binding assay, the number of total operations, the RPN below the threshold = 100 and the RPN greater than 100 are shown
Full size table

One hundred (100) opportunities for error were identified. For our analysis, we selected a non-automated procedure: Manual operations are more common in a Life Sciences research laboratory and, moreover, they are a source of frequent errors, as they cannot be easily controlled. It is then not surprising to find that the analysis produced RPN greater than 100 in more than 50% of the opportunities for error, involving more than 80% of the process operations. Applying the FMEA allowed to validate the threshold value = 100 for RPN, that was chosen based on experience, preliminary assessments and contribution of FMEA reviewers inside the qPMO network. The steps considered risky by experience were judged adequately weighted by the values of S, O and D and brought to attention by a higher RPN value, signaling the need for actions. This control confirmed the suitability of the scales for S, O and D and of the 100 threshold chosen for RPN. The actions identified covered most laboratory aspects, such as management of instrumentation and materials, and training of personnel.

These were categorized according to the four “M” suggested by K. Ishikawa for his fishbone diagram [24], which is the approach most frequently adopted in Quality Management to identify the possible causes of a problem and organize them in a causal chain. The primary causes can be classified according to the type of effect with reference to four standard categories: human resources (Manpower), equipment (Machines), protocols and procedures (Methods), and products and reagents (Materials) [16, 24]. In Table 3, we show the proposed actions used to lower the overall risk. Some improvement actions have been addressed to the specific protocol, but, interestingly, most actions emerging from the analysis refer to the overall organization of the laboratory and could be common to many cellular and molecular protocols regularly carried out in a biological research laboratory.

Table 3 The improvement actions identified to lower the RPN above the set threshold, acting on S, O and D, were classified according to the Ishikawa’s 4Ms: Machines, Manpower, Methods and Materials
Full size table

These results led us to define a set of standard FMEA analyses for main operations commonly carried out in a Life Sciences laboratory, e.g., pipetting, use of reagents and instruments—and mainly dedicated to the use of equipment and materials. Therefore, we created a series of “FMEA strip worksheets” ready for use, with precompiled FMEA rows populated with all information available for the specific operation. These FMEA strips can then be applied directly to specific scientific protocols with dedicated assessments and RPN and could be very useful for researchers aimed to approach Quality Management and risk analysis applied to basic science. Figure 4 shows an example of a FMEA strip that can be used in a generic protocol analysis. Most common errors are listed, followed by their causes and related occurrence (O); some controls, typical of a quality-managed environment, are considered already in place and assessed with related detection (D). Some suggestions for improving controls or avoiding mistakes have been also inserted in the column “actions” in the second part of the FMEA strip worksheet; some of them are drawn from the guidelines issued by the Working Group 1 (WP1) of the qPMO Network [12]. These FMEA strips are intended as a model, a simplified tool that scientists not familiar with FMEA can start to deal with. It is responsibility of the researcher to customize FMEA strips for the specific scientific process he/she wants to refer, identifying the possible impact and severity on the outputs of their protocol, evaluating the resulting RPNs (automatically calculated) and defining possible preventive actions to reduce the most significant ones. Addition of other operations and actions suitable for the specific process or context of researchers is also desirable, to continually improve risk analysis and its impact on research performance.

Fig. 4
figure 4

Example of FMEA strip worksheet. FMEA strips are precompiled portions of the FMEA template, designed for the most common laboratory process steps (column Process step or input), including already identified potential errors (column Potential Errors), their probable causes (column Potential Causes) and controls already in place (column Current controls), with a proposal for their assessment (respectively, columns OCC and DET). The effect of errors (column Potential Negative Effects) should be identified by the process team, together with their severity (column SEV). The column Action Recommended is prefilled with suggestions useful for further improvement of the potentially critical process steps. The column RPN is already populated with the formula for SxOxD. The columns for action plan (Resp. and Date) and the recalculated parameters S, O and D are left blank, to be compiled by the process team

Full size image

Finally, a comprehensive report on the risk assessment analysis performed was prepared and disseminated to the researchers involved in the development of aptamers and to the entire research institution. In addition, a guideline for the application of the FMEA to research laboratories, together with FMEA strips, is going to be published on www.quality4lab.cnr.it Web site, available to all researchers in the international scientific community.

Discussion and conclusions

Risk management is a process of identifying hazards associated with a product or process, estimating and evaluating the risks associated, controlling these risks and monitoring the effectiveness of the control. FMEA is a methodology, first developed for system engineers, that examines potential failures in products or processes and that can be used to evaluate risk management priorities for mitigating known threat vulnerabilities [25]. For automotive manufacturing, technical risk analyses are a mandatory part for QMS certification IATF 16949 and the FMEA method is used as a living tool throughout the product/process realization process [26]. Moreover, Quality Risk Management became an integral part for manufacturing and pharmaceutical industry [27,28,29], where FMEA is also the preferable method [30, 31]. Even though risk management is not formally required for non-regulated research, it began to capture the attention of researchers for the great possibility to identify and control errors and therefore increase research performance. However, due to the need to invest time and resources to acquire new competence in Quality Management and/or to hire external consultants, together to the spread prejudice that Quality Management can be a constrain to creativity and innovation, the process is still at the beginning. In 2012 as part of the qPMO network, we started a project aimed to test the application of quality tools and methodologies to non-regulated research, mainly in the field of Life Sciences, and to develop customized and friendly models for a broader use in our field [12, 16, 32, 33]. Our starting point was the belief that the successful translation of a management system from the engineering and manufacturing sector to non-regulated research strongly required to consider specific needs, objectives and expectations of this type of research as well as to create a scientific quality culture.

The aim of this work was to trial and test the validity of a specific quality and risk management tool, the FMEA, successfully used in several contexts, such as automotive and pharmaceuticals, for the risk assessment of a laboratory experimental procedure, in the specific field of non-regulated basic research. In particular, we applied FMEA to a specific experimental process in Life Sciences, the selection of aptamers, that is a multi-step process involving several procedures, equipment and human skills and likely to be transferred to exploitation. Risk management is composed of different steps that are risk assessment, control and review and monitoring. In this paper, we focus on risk assessment of a specific process, identifying opportunities for error and the sources of uncertainty that can be addressed with specific interventions, in order to improve its accuracy and precision. Some authors prefer to separate organizational aspects (i.e., activities common to many processes in the laboratory) from aspects specific of the process under analysis [6, 9]. We preferred instead to let the FMEA highlight potential failures of all kinds that could be easily ignored if attention is focused only on the detailed steps and possible errors of the specific protocols under analysis. This approach let us to identify several sources of variability that should be taken under control for a correct protocol management. As in a previous work regarding the drafting of guidelines [14], we chose to take advantage of the cause–effect Ishikawa diagram [24] that is widely used to identify possible primary and secondary causes of a problem, by subdividing them in different specifying categories, that all together take care of almost all aspects related to the problem itself. We classified the actions emerging from the risk analysis according to the 4Ms of Ishikawa that are Manpower, Materials, Methods and Machines. This classification partially agrees with the one presented by Freedman et al. [1] regarding major causes of irreproducibility of scientific results: biological reagents and reference materials, study design, data analysis and report, and laboratory protocols. With respect to this classification, applying the 4Ms of Ishikawa, we also included in the analysis the human factor and the use of equipment. The human factor has a specific importance for the pilot process herein analyzed: The aptamer process is manually performed by highly qualified personnel. Despite this, human errors can occur and also significantly affect the process, as some of their steps and procedures require a high degree of experience and a constant control of equipment and materials. Therefore, protocols must be designed very clearly, with the aim of preventing mistakes or at least reducing the associated risk, in order to limit undesired effects on the results. Risk analysis gives the opportunity to highlight the most delicate operations and suggests how to maintain manual operations under control, reducing the possibility of reworking due to errors. Many actions arising from the FMEA are broadly aimed and mostly related to the organization of the laboratory, based on support processes such as management of documentation, equipment, staff training and knowledge, and procurement. This is an apparently obvious, but often ignored conclusion, since in practice when there are problems with an experimental process, most researchers tend first to analyze the specific steps of their procedures and to design and perform control experiments to identify what was wrong, in terms of specific sample or reagent. Often, only when no problem has been identified, researcher might begin to consider more general causes of failures, such as calibration of common instruments. In this light, a FMEA can be an excellent and practical starting point for scientists to begin to consider earlier general causes of failures, preventing waste of time and money and, even better, to implement the management and the control of procedures, equipment and staff training of the laboratory. Moreover, such a quality tool, possibly combined with a quantitative approach that measures errors and uncertainty, accustoms the researcher to paying attention to the margins for error of any laboratory operation. If needed, this approach might support the implementation of a QMS for the entire laboratory.

However, FMEA has also some disadvantages; one central issue, that is a bias toward severity ratings, was bypassed in the updated FMEA version [26]. Other disadvantages include: (1) the requirement of a strong initial investment in time and resources; (2) strict dependence on contribution and experience of the team, since the same analysis performed by different working groups may lead to slightly different results; (3) compromise between completeness and practicality (if every issue ends up on an FMEA sheet, then FMEA analysis becomes very long and complex) [34]. To bypass some of these disadvantages, we propose FMEA strip worksheets to facilitate researchers in their approach to the quality culture. FMEA strip worksheets are a valid starting point for a first approach to this technique, streamlining its application and acting as an easily accessible repository of scientific and quality control knowledge. In our experience, this latter aspect is confirmed and emphasized, because in a research laboratory staff turnover is usually high, due to the presence of students, graduates, postgraduates and research fellows. Recording in a formal way the experience of all specialists who have been working on a protocol allows collecting multiple contributions, at the same time creating an internal standard and a basis for improvement and eventually innovation. This knowledge base can also be used for the training of new entries and students. For sure, FMEA strips are not thought as fixed tools to be applied passively by researchers, but, on the contrary, aim to help scientists to become familiar with FMEA as risk assessment tool. Researchers are invited to implement FMEA strips, adding other possible causes of failures that impact on their specific results and identifying relative preventive actions.

Finally, in light of the risk-based thinking introduced by the new ISO 9001:2015 standard, FMEA is a useful tool to perform risk analysis of operational processes under an ISO-certified Quality Management System, with a uniform and shared method, and an evidence-based approach. The application here illustrated refers to a basic research laboratory, which does not perform any assay service. Moreover,  a risk-based approach is strictly required by the ISO 17025:2017 standard dedicated to measurement laboratories, in order to improve uncertainty and thus the reliability and reproducibility of measurements. A natural progression of the FMEA is the generation of a control plan that summarizes all controls identified during the analysis: In such a plan, controls would be defined in finer detail, and persons in charge, equipment and methods would be identified and described for ease of application. A control plan has not been developed in this work because its sole purpose was of testing the validity of the FMEA on an experimental procedure, without measuring any outcome on the protocol itself, since it was not involved in the development of the aptamer generation process. Moreover, a control plan requires a full commitment of the responsible of the laboratory and sometimes of the institute, to translate identified actions in practice, as well as adequate allocation of time and resources for the control activities. However, a control plan would be an important contribution to the exploitation of the protocol, because it supplies a clear, formalized and thorough knowledge about how to maintain the process under control to the downstream team. Risk control, review and monitoring activities shall be also needed to assess the impact of the preventive actions. FMEA outcome should be maintained and updated by the team collecting experiences, improvements and any kind of evolution of the procedure analyzed, also involving people who contribute with different roles to the execution of the procedure. All together the activities of risk assessment, control, review and monitoring might have a strong impact in improving the control of the scientific process variability and the repeatability of results, as well as reducing effort and material needs for running the experimental procedures, thus eliminating sources of waste.

In summary, our experience shows that the FMEA risk assessment tool as applied to a scientific process in non-regulated research is in line with the current needs of research management models to raise effectiveness and efficiency, to enhance reproducibility, and to facilitate a rapid industrialization of research results.