A Survey of Data Quality Requirements That Matter in ML Development Pipelines

Long-standing definitions of data quality have viewed good-quality data as “data that are fit for use by data consumers” [74]. This has been accompanied by granular specifications of what makes a good-quality dataset, with essential dimensions such as accuracy, completeness, consistency and validity being just some of the 60 dimensions identified in the wider data management literature [13]. Practical applications of these dimensions typically focus on smaller subsets of the most relevant qualities, and can be found in the UK government’s data strategy,⁴ professional associations for information management such as AHIMA,⁵ and the requirements of open data and open science initiatives, where datasets should ideally be linked by the peer-reviewed code or publication that uses them.

It is worth noting that ML data tend to be managed differently depending on whether the system is within an academic or industry setting [52]. In academia, data management is typically contained within the projects of individuals or small teams, who are able to design and amend the data collection, storage and sharing systems at their discretion. Industry researchers, however, often rely on separate data collection, processing and storage systems that sit across multiple company functions, requiring formal data management guidelines to ensure consistency and coordination across teams.

Formal practices of this kind are sometimes established with the help of industry standards. For example, the International Organization for Standardization (ISO) standard ISO/IEC 25012 provides guidance on how to define the data quality characteristics that matter to an organisation. Defining these characteristics is a prerequisite to deciding how data quality can be evaluated in a practical sense. This latter task is addressed by the standard ISO/IEC 25024, which guides organisations in defining the data quality assurance criteria and ways of measuring them quantitatively. These data quality models are complemented by standards which recognise that organisations differ in their preparedness to define and execute data quality assurance. The ISO 8000-61 standard specifies the pure activities of enhancing data quality processes, whereas ISO 8000-62 defines ways to assess the maturity, or readiness, of organisations to implement these data quality tasks.

More recently, ISO has begun to develop the ISO/IEC 5259 standard which focuses on data quality for the fields of analytics and ML, as well as ISO/IEC DIS 8183 that addresses the AI data lifecycle framework. These newer standards address processes that can be employed by various stakeholders at different stages of the AI lifecycle, which differs from earlier data quality guidelines that tended to view quality as a uniform outcome that fulfils a pre-defined list of desired criteria. These standards are still under development, so there is value in publications that inform practitioners of how data quality applies to ML tasks.

Traditionally, data quality compliance has meant meeting the needs of the immediate data users (e.g., analysts or engineers who value clean machine-readable data). However, this singular focus can flatten the variety of uses and data quality requirements that are encountered at various stages of the ML development pipeline over the longer term [57, 70]. For instance, data quality aspects that are important to ML developers are likely to be different from what was important to upstream data subjects, who may have valued mechanisms for expressing consent and data usage preferences. Similarly, downstream users of trained ML algorithms, such as software developers and organisations, may have their own preferences for specific data qualities when procuring the system, including aspects such as security, provenance, legal compliance and the capacity to meet business goals in real-world contexts. It is therefore useful to consider data quality processes in ML as being less about obtaining a finished outcome and more about creating a dynamic artefact that is imbued with the potential to be improved and shaped by different stakeholders to meet their own requirements.

Many of the data quality issues that could reasonably concern the stakeholders mentioned previously can already be accommodated by the granular data quality specifications produced in the field of data management. For example, the list of 60 dimensions created by Black and van Nederpelt [13] includes qualities related to data accuracy, lineage, currency, coverage, legal compliance and usability. These dimensions are subsumed by higher-order characterisations that capture the intrinsic, contextual, accessibility and representational aspects of datasets [74].

Although the advent of data-centric technologies has been accompanied by a proliferation of updated data quality definitions and metrics tailored to fields such as big data [68] and linked data [77], contemporary authors continue to find value in existing data quality characterisations and conceptual structures. For example, in their “Data Quality in Use” model for big data, Merino et al. [47] draw on the canonical distinction of Wang and Strong [74] between the intrinsic, contextual, accessibility and representational aspects of datasets when using the preceding industry standards ISO/IEC 25012 and ISO/IEC 25024. Other efforts have been made to adapt traditional data quality management practices to specific fields. This includes the work of Kim et al. [41], who developed new frameworks for assessing and improving the maturity of IoT data quality processes based on the standards ISO 8000-61 and ISO 8000-62.

In the following, we will draw on the categorisation by Wang and Strong [74] of the intrinsic, contextual, accessibility and representational aspects of datasets to illustrate some of the ways in which previously established data quality categories already apply to ML problems.

Intrinsic data quality has traditionally been understood to reflect the extent to which data values conform to the actual or true values [74]; this includes specific requirements such as accuracy, provenance and cleanliness, the latter of which covers practices such as the addressing missing values and redundant cases. Besides the usual data qualities needed for statistical analysis (e.g., addressing missing data, anomalies), an intrinsic quality that is increasingly valued by ML practitioners and regulators relates to data lineage and traceability. For data that require multiple pre-processing steps or transactions between organisations, the origins of their features becomes important. Traceability makes it possible to interpret and audit the history that precedes the output of ML algorithms [33], but despite recent regulations on Explainable AI (XAI),⁶ traceability is not yet shortlisted in the data quality framework used by the UK government,⁷ suggesting that this data quality characteristic may need to be promoted in the context of ML.

Contextual data quality relates to the extent to which data are pertinent to the task of the data user [74]; this includes dimensions such as relevance, timeliness, completeness and appropriateness. An essential question that is considered here is the extent to which the sample of cases contained in the dataset diverges from the true distribution of cases that are likely to be encountered when the ML model is deployed. Possible sources of divergence may include historical time or geographic representation. For example, temporality has been flagged as a potential source of difficulty in textual data, where models trained on historical text corpora, such as Google News articles, have been found to reproduce past social stereotypes (e.g., the word “man” being associated with “computer programmer” and “woman” with “homemaker”) [14]. If left untreated, the use of such data in downstream applications (e.g., web search rankings, question retrieval) can perpetuate or amplify the biases that were and continue to be present in broader society. Other contextual biases have been detected in image data, with publicly available image corpora such as ImageNet and Open Images coming predominantly from amerocentric and eurocentric contexts [66]. Insufficient representation of some geographic regions, such as Asia or Africa, has meant that ML algorithms have less information to learn about these contexts. This results in solutions that perform poorly for under-represented groups (e.g., passport photo software that does not recognise the facial expressions of ethnic minorities, or electronic soap dispensers that do not respond to darker skin tones). These cases urge ML data practitioners to think critically about the context captured by their dataset and the degree to which it reflects the use case and lived experience of the end users.

Representational data quality refers to the extent to which data are presented in an intelligible and clear manner, including requirements such as being interpretable, easy to understand, and represented concisely and consistently [74]. In practical terms, these qualities can be implemented through practices such as standardisation and documentation. Standardisation refers to conventions for capturing information in a consistent manner, including machine-readable data structures and formats for capturing specific attributes (e.g., date, location, measurement error). This helps engineers to ingest datasets from multiple sources and build interoperable solutions. Documentation about the dataset provides an additional layer of descriptive information to support the creation of ML applications. For example, it can help engineers to understand where the dataset sits in relation to the physical world (e.g., the calibration of equipment, seasonality of data collection, contextual limitations) [64] so that the training data or model output can be transformed accordingly. It is worth highlighting that when the limitations of a dataset are made explicit in the documentation, this helps subsequent users to take the steps needed to improve the quality of the dataset for their specific use case. Some solutions even allow for the dataset to remain unchanged while the ML algorithm is tuned to produce more robust or socially equitable outcomes [14, 29].

The accessibility category refers to the extent to which data are available, obtainable and secure. The rise of big data and ML applications in recent decades has been accompanied by calls for publishing datasets in an open manner, as well as secure access mechanisms for restricted datasets, so that their value can be realised [75]. For ML stakeholders who work with personal or commercially sensitive data, advances in the accessibility of data have been tempered by security and legal precautions (e.g., compliance with GDPR and intellectual property rights).

The data quality concerns exemplified previously are already within the scope of the concepts and frameworks that have been established in data management literature, suggesting that this field already has a good grounding for defining the data quality dimensions that will continue to remain important to ML. What is new, however, is that ML development is characterised by complex configurations of datasets, data services and data handlers, which makes individuals more vulnerable to abstain from taking action due to the belief that data quality is somebody else’s problem [34]. This diffusion of responsibility can be addressed by providing clearer indicators about which data quality aspects are and are not out of scope of particular ML roles.

The struggle of clarifying which data quality requirements are important is not exclusive to ML. Even where detailed data quality standards and practices exist, organisations and/or practitioners have to specify which data quality characteristics are relevant to their use case and how to define them.

In a study of organisations that applied the ISO/IEC 25012 data quality standard, Gualo et al. [24] found that practitioners struggled to identify and describe the data quality rules that applied to their use case. The authors found that providing examples of what the requirements can look like helps to guide practitioners in clarifying their own rules.

Another challenge relates to information overload. Long lists of requirements have been found to deter practitioners from applying traditional standards, with Kim et al. [41] showing that there is value in simplified frameworks that are tailored to a specific use case or technology.

Both of the preceding challenges are encountered during the initial stage of planning and defining which data qualities to evaluate. In other words, they occur at the beginning of the data quality management process defined by the standard ISO 8000-61, as illustrated in Figure 1. Without the planning stage, it becomes harder for a practitioner to develop the right data quality rules and select the tools to enforce them.

Our work aims to support ML practitioners and data managers at the planning stage of their data quality journey. We identify a series of considerations that can help them to define their own requirements and data quality strategy. By understanding the requirements that exist, practitioners can be better positioned to select the most meaningful data quality control, assurance and improvement steps for their use case. Although the tasks of implementing specific data quality measures, evaluation criteria and tools for checking data quality are outside the scope of this review, we will mention examples where relevant.

Our goal in this article is twofold. Firstly, we want to inform practitioners of the data quality requirements and practices that exist and are meaningful in the field of ML. This will be done by synthesising recent academic literature and grouping the recommendations according to the dimensions of data quality that are already familiar to the field of data management. Secondly, to assist readers in selecting a smaller set of data quality practices that may apply to their use case, we map the recommendations onto specific stages and stakeholders in the ML development pipeline. In doing so, it is our hope to make it easier for organisations and individuals to prepare their data management routines for ML and to anticipate some of the scenarios that may arise at each stage of the ML development pipeline.

Our review aimed to identify and discuss the data quality requirements that are important to ML development, and how they differ from more established data management practices. For this purpose, we defined the following research questions:

The preceding questions deal with data quality management planning as opposed to implementation. This is a distinction that has previously been recognised in industry standards such as ISO 8000-61, as depicted in Figure 1. The planning stage (1) deals with the identification of data quality requirements and strategies for implementing them, whereas the implementation stages (2–4) are about translating these plans into practical rules and techniques for data quality control, assurance and improvement. This distinction between data quality planning and implementation informed the selection criteria of our review.

Our interest in data quality planning (as distinct from implementation) helped to limit the scope of our literature review and make the topic small enough to be discussed in a single paper. Specifically, our targeted articles dealt with philosophical or experiential perspectives on data quality frameworks, as opposed to articles that evaluated specific data management techniques or proposed new solutions for managing data quality. Our choice of research categories is highlighted in Table 1 alongside the other possible types of research as defined in the systematic mapping protocol of Petersen et al. [54].

Our inclusion criteria were as follows:

Our exclusion criteria were as follows:

There was some overlap between our inclusion and exclusion criteria. For example, many abstracts discussed conceptual frameworks in addition to validating specific techniques, developing new prototypes or sector-specific solutions. We included these article as long as the the main part of the abstract was generalisable (i.e., discussing data quality concepts that apply to general ML applications and not focusing only on a specific industry or solution).

Our literature search strategy consisted of three stages: (1) pre-selected articles that were already known to us, (2) automatic search on Google Scholar and selected conference proceedings, and (3) forward and backward snowballing to identify further articles.

Pre-Selected Articles. We began with a list of six articles [3, 23, 32, 34, 35, 58] related to data quality planning, and in particular documentation, that were already known to us based on our previous work with ML models.

Automatic Search. We used Google Scholar to search for articles whose title included keywords related to our research questions. Limiting the search only to titles helped to eliminate marginally relevant articles from the results. The results were then filtered by examining the titles and abstracts of the articles. Only those that met the selection criteria were retained.

We began by searching the entire Google Scholar corpus using the query “allintitle: “data quality” (“machine learning” OR “AI”).” This returned 185 results. We truncated our analysis after examining the first 30 results, as many of them did not meet our inclusion criteria. After examining the abstracts, seven articles were retained [12, 19, 21, 25, 27, 28, 63].

We then conducted searches inside the proceedings of two leading academic conferences in ML and HCI: ICML (International Conference on Machine Learning) and CHI (Conference on Human Factors in Computing Systems). This was done using Advanced searches in Google Scholar, where the “published in” box was filled with the name of each conference. We adapted the search query to each venue’s area of specialisation. For example, when searching through CHI proceedings, we used used a slightly more lenient query due to the smaller size of the search space: “allintitle: data (quality OR “machine learning” OR AI).” This returned 19 results, 9 of which met our inclusion criteria [2, 26, 31, 49, 56, 62, 64, 70, 73]. We also adapted the query for ICML, as the conference already specialises in ML. A search for “allintitle: “data quality” OR “data management”” returned 16 results, 1 of which was identified as relevant [38]. Table 2 summarises each of our search queries, the number of results returned by them, and the number of articles that were subsequently selected for our discussion.

We are aware that there may be other venues with relevant contributions that were not included in our selection.

Snowballing. After we started reading and reviewing the articles selected using the preceding techniques, we came across references to other articles that were relevant to our research questions. Eight articles were identified in this way [5, 9, 11, 36, 48, 53, 55, 57]. These articles were initially chosen based on the descriptions provided by authors who cited them, then assessed using our inclusion criteria. One further article [53] was identified using a forward search of articles that cited [64], as we were curious about the work of other authors who cited this paper. Our general approach to snowballing was informal. Due to time constraints, we did not conduct a systematic review of all possible forward and backward citations.

Although we sought to gather a representative sample of articles, it is important to acknowledge that the 32 articles reviewed here are only a small part of the growing number of articles related to data quality in ML that exist in reality.

After selecting the articles, we read them and extracted information that helped to answer our research questions. Relevant information was recorded for each paper using a spreadsheet with the following groups of columns:

Once we started reading the articles, we found that some of the authors’ comments and data quality requirements were targeted to specific stages in the ML development pipeline. For this reason, we added the following set of columns to organise our notes:

Information about each paper was coded using the 18 columns described previously. We reviewed this spreadsheet to synthesise common themes at the intersection of two dimensions: each stage of ML development vs. the four traditional categories of data quality. This is the structure we use to present our results.

Before presenting our results, we want to clarify their scope. Although the initial goal of our work was concerned with theoretical frameworks that can help to define and plan data quality requirements in ML, we also noted down any practical techniques mentioned by the authors. Many of our reviewed articles went beyond data quality “planning” to make recommendations on how data practitioners and managers should prepare their datasets for ML. We did not review these techniques in a systematic manner, as this would merit a separate review of its own. However, we included some of the techniques in our findings to illustrate how data quality plans can be translated into practical specifications, assurance techniques and solutions that apply during stages 2 through 4 of the process depicted in Figure 1.

Besides extending into practical techniques, many of our selected articles discussed topics that went beyond our original focus on data quality in the technical sense. Specifically, they overlapped with other related communities of research and practice in data management, such as data ethics, data justice and data feminism. These fields have historically been addressed by different communities, so the relations between them are not neatly delineated. Nonetheless, there is significant overlap which we attempt to illustrate in Figure 2. Rising [59] presented an understanding where justice is about situations and consequences, whereas ethics is about the actions that lead to consequences. In line with this, data ethics deals with the way that practitioners manage data to ensure privacy, fairness, accountability, security and environmental sustainability [6]. However, data justice addresses inequalities in the way people are represented and treated as a result of the data that they emit [69]. Data feminism traces the cause of such inequalities to the power relations present in society, and advocates for actions that support the political, social and economic equality of the sexes, including intersections across other social dimensions such as race and class, sexuality, ability, age, religion and geography [20].

As illustrated in Figure 2, each of these works highlights how systemic challenges in the lived experience of ordinary people are embedded in data, and their potential to be reinforced or mitigated through data-centric technologies. Although we did not explicitly search for these perspectives, and time and space constraints prevent us from covering them in the detail that they deserve, we encourage interested readers to investigate these topics separately.

Another scoping challenge which emerged during our review was related to the definition of data. Our initial intention was to focus on observational data (for training, testing and serving models), but this was later expanded due to the substantial attention that our reviewed articles dedicated to the quality of software systems, ML models and their accompanying documentation. Although there is some ambiguity among academics as to whether it is constructive to view software as data [39], we have included the aspects of ML model and documentation quality that emerged during our review. For instance, we found that training data quality can be mediated by software systems (e.g., for data maintenance, or for checking input or output data). Moreover, the inclusion of model and documentation quality helped to highlight the areas where ML model quality is dependent on good-quality training, testing or serving data, as well as metadata in the form of documentation. For these reasons, our discussion of data quality grew to include model, software and documentation quality.

The initial steps to ensuring data quality begin before data are collected. These steps include clarifying the use case for which the data are sought and investigating the operational and/or infrastructural requirements of gathering the data. These preparatory steps must be recorded in the dataset’s documentation, to inform current and/or future colleagues about the requirements of the use case.

It is common that the precise data quality requirements will not be known upfront, and new use cases may emerge as the model matures. This means that practitioners will likely need to return iteratively to the dataset design and data collection process [31]. In cases where additional data are required but cannot be collected iteratively, other methods are available to enhance the dataset, as we will discuss later. With this in mind, the preparatory steps described in the following should be viewed as a desirable rather than essential part of the data quality pipeline.

Once the data use case and operational requirements are in place, the process of data collection can start. The design decisions made in the previous step may be implemented in several ways, such as through software systems, annotator guidelines and labelling platforms. In the following, we discuss the ways in which documentation, standards and interfaces can support the acquisition of data that are high in quality.

Once the data have been collected, they typically undergo a process of checking and cleaning before being usable for an ML system. This stage of the ML development pipeline bears a large bulk of the activities related to data quality assurance. In the following, we discuss these tasks, which include pre-processing, validating the contextual coverage of the data, data quality metrics, user interfaces for inspecting data, dataset accessibility and maintenance over the longer term.

In many contexts, the previous data collection and preparation steps are likely to have been carried out by a person different to the one who builds the ML model. For this reason, the ML practitioner would ideally go back and check the dataset’s documentation to make sure that it meets their use case requirements. This can help them to avoid using the data for a purpose that may be morally or ethically objectionable to the original curators [53].

Once the dataset is confirmed to be suitable, the process of building ML can begin. Some of the initial data work may be similar to the data pre-processing stage mentioned earlier, but here the requirements will depend to a greater extent on the selected ML techniques and use case. Examples of possible tasks include feature selection, enrichment and sampling. We summarise these requirements next, followed by a discussion on data accessibility issues that accompany ML models.

In many documented cases of adverse ML outcomes, the issues with training data became apparent only after the solution was deployed in real-world contexts. To avoid this, ML practitioners and auditors can test the system for contextual bias and security issues before releasing the system. We discuss these considerations next.

Once ML models are trained and ready for deployment, the focus of data quality work shifts from internal operations on training and test data, and instead looks at assuring the quality of serving data that enter the system from the outside.

Mechanisms are needed to ensure that the serving data undergo the same preparation steps as the steps that were applied to the raw training data [57]. This can be especially challenging in settings where new data arrive continuously, and where they are used to retrain and deploy updated models. The latter case will require additional measures for preventing adversarial attacks such as data poisoning [52] or spam [57].

Other precautions also apply to models that do not ingest new training data. For example, proprietary models can be stolen by repeatedly querying the system (e.g., via a public prediction API) and monitoring the outputs to reverse engineer a substitute model [52]. Another similar risk relates to model inversion, where querying can be used to recover parts of a private training dataset, thereby breaking its confidentiality [52]. These risks are especially likely in models that report confidence values alongside their predictions.

To mitigate the preceding risks, ML developers should work closely with software engineers to ensure that public-facing systems built on top of the ML are robust against malicious attacks. Recent trends in ML have discussed the development of new engineering approaches such as federated learning to foster privacy, data protection and security [33]. Federated learning works by allowing devices to learn a shared prediction model collaboratively while keeping the training data securely on the user’s own computer.

Once a model is deployed, the focus on serving data should continue. At this stage, the work shifts to monitoring the properties of incoming data and ensuring that they are contextually similar to the data that the model was trained on. Polyzotis et al. [57] and Schelter et al. [65] propose analyses that can be used to detect training-serving skew in pre-defined variables. However, others note the difficulty in trying to establish which columns must be inspected, and what the required thresholds should be [63]. Chen et al. [19] suggest that the thresholds can be based on the expected distribution of a targeted population for relevant features (e.g., the usage frequency of a phrase, or the number of individuals with a particular skin tone).

Some monitoring activities can be automated and communicated to the users of ML systems via alerts. This may include data integrity checks, anomaly detection and performance metrics [52]. Additionally, the system can be designed to gather additional data about misuses or outliers while the model operates in the real world, providing DevOps engineers and ML developers with more information for mitigating security and performance issues in subsequent versions of the model.

In this final section of our results, we summarise the data quality requirements that matter within specific stakeholder roles. Knowledge of relevant responsibilities can help practitioners to understand and resolve the data quality issues that are within their capacity, and to articulate their own requirements to relevant colleagues:

An important caveat we would like to restate is that our findings are not exhaustive, and they capture only a small selection of recurring themes that came up during our review. We therefore encourage readers to remain open to other data quality requirements that may matter to them and their colleagues, bearing in mind that these may not have been covered here.

Earlier in this article, we noted that data quality frameworks and standards present practitioners with dozens of possible criteria to comply with. These are accompanied by a growing range of tools for data pre-processing, documentation and assurance. It is impossible for all of these requirements to be met, nor is it necessary. Previous studies that explored the application of data quality standards found that practitioners benefit from seeing examples of data quality requirements, as it helps to clarify their own needs [24]. It was also found that there is value in simplified data quality frameworks that are tailored to specific use cases or technologies [41].

Our review sought to assist ML practitioners who are trying to define their data quality requirements. Firstly, we synthesised previous literature to illustrate the common data quality requirements that can exist in ML. Secondly, by mapping these requirements to different stages of the ML pipeline, we provide a way for readers to see the requirements that are likely to precede and follow their specific task, and to discern which data quality outcomes to focus on in their role. This type of clarity is needed to prevent the diffusion of responsibility and to ensure that every stakeholder is proactive at mitigating data quality issues that are within their capacity.

Besides individuals who work directly with data, we anticipate that our review will be useful to coordinators of data innovation projects that involve multiple stakeholders. Our own experience of this includes a series of projects that emerged from a Public Private Partnership (PPP) between the European Commission and the Big Data Value Association (BDVA). These projects included the EuropeanDataIncubator (EDI), EuRopEAnincubatorfortrustedandsecuredatavalueCHains (REACH) and EUHubs4Data. Their goal was to facilitate data-driven innovation in startups and SMEs through collaboration between data providers, data users, business coaches and legal experts assembled from different geographic regions. The review provided in this article can help managers of similar initiatives to understand the data quality requirements of colleagues who are responsible for different parts of the data value chain, and to signpost participants to resources that will support their data quality practice and documentation.

Our review is limited by the relatively small sample of articles, which represent only a minor portion of the growing number of works emerging at the intersection of data management, ML and HCI. It is possible that the keywords and sources that were used during our search for articles, as well as the insights drawn from them, were influenced by the authors’ fields of expertise. For example, we did not elaborate greatly on concepts such as data ethics, data feminism and data justice, which relate to data quality but lie outside the technical focus adopted in our review.

Another limitation relates to the simplification of our findings for the purpose of this review. For example, the ML pipelines illustrated in Figures 4 and 5 (along with the sequence of stages in Table 3) use a linear sequence of data quality assurance steps that is unlikely to be structured like this in reality. Specifically, multi-dataset–multi-model and agile data iteration scenarios are more common than the waterfall-ish view used to report our findings. Moreover, much of our discussion focused on desirable or ideal scenarios rather than what is feasible. So we did not do justice to the important tradeoffs and negotiations that occur when some parts of data quality may need to be adapted or sacrificed in favour of practical requirements and business goals.

Our review focused mainly on defining the requirements of ML data as part of the “planning” stage of the data quality management process illustrated in Figure 1. We did not systematically review the literature on how these plans can be implemented through tools for data quality control, assurance and improvement. As the fields of ML and data quality research continue to grow, we envision a demand for reviews that are able to compile a list of the available data quality tools, and compare their overlaps, differences, and blind spots.

We also encourage organisations to look beyond the traditional view of data quality as a fixed outcome that meets a list of pre-defined criteria, and to consider a more dynamic perspective that specifies the particular data quality requirements that are valued within their part of the ML lifecycle. Movement in this direction is already under way in the standardisation community, with standards such as ISO/IEC 5259 and ISO/IEC DIS 8183 beginning to incorporate the ML lifecycle into data quality recommendations. In Figure 3, we illustrated how the findings of our review may be reconciled with the pipeline of the forthcoming ISO/IEC 5259 standard. We encourage future researchers to investigate the practical application of this standard by individuals and organisations, and to share their experiences with the wider community.