Fundamental Components and Principles of Supervised Machine Learning Workflows with Numerical and Categorical Data

Abstract

This paper offers a comprehensive examination of the process involved in developing and automating supervised end-to-end machine learning workflows for forecasting and classification purposes. It offers a complete overview of the components (i.e., feature engineering and model selection), principles (i.e., bias–variance decomposition, model complexity, overfitting, model sensitivity to feature assumptions and scaling, and output interpretability), models (i.e., neural networks and regression models), methods (i.e., cross-validation and data augmentation), metrics (i.e., Mean Squared Error and F1-score) and tools that rule most supervised learning applications with numerical and categorical data, as well as their integration, automation, and deployment. The end goal and contribution of this paper is the education and guidance of the non-AI expert academic community regarding complete and rigorous machine learning workflows and data science practices, from problem scoping to design and state-of-the-art automation tools, including basic principles and reasoning in the choice of methods. The paper delves into the critical stages of supervised machine learning workflow development, many of which are often omitted by researchers, and covers foundational concepts essential for understanding and optimizing a functional machine learning workflow, thereby offering a holistic view of task-specific application development for applied researchers who are non-AI experts. This paper may be of significant value to academic researchers developing and prototyping machine learning workflows for their own research or as customer-tailored solutions for government and industry partners.

1. Introduction

Several machine learning applications are implemented as a workflow, which starts with data collection and ends with a model evaluation and simulations or software development. Examples of fields that introduce custom machine learning workflow solutions include, but are not limited to, malware detection and classification [1], software development with adversarial attack classification [2], task fault prediction in workflows developed with cloud services [3], pipeline optimization [4], the classification of forest stand species via remote sensing [5], the detection of mechanical discontinuities in materials and the prediction of martensitic transformation peak temperature of alloys [6,7], the optimization of metabolic pathways and ranking of miRNAs retarding insulin gene transcription in human islets [8,9], large-scale crop yield forecasting [10], classification and forecasting in chemical engineering [11], predictive modeling in medicine [12], protein engineering/biophysical screening in pharmaceutical sciences [13], the forecasting of oil uptake in batter for food science [14], vegetation height classification, forecasting fractured coal seams, climate-related forecasting in environmental sciences [15,16,17], and energy systems controlled by occupancy detection or energy demand forecasting [18,19,20,21,22], as well as environmental impact estimation from commercial aviation and aerospace requirements engineering [23,24,25,26,27]. Specifically in Artificial Intelligence (AI), machine learning frameworks have been proposed for a variety of data-driven expert systems, such as recommendation systems [28], decision support systems [29], fault diagnosis [30,31], and crowdsourcing [32], as well as generic data science [33] and big data [34] applications. Machine learning workflows or workflow sub-modules are often automated, i.e., automated classification [35], automated machine learning (AutoML) in healthcare [36], aviation [37,38], biology [39], and agriculture [40,41].

Many researchers, including non-AI experts, develop machine learning workflows for a specific application as the previous references clearly suggest. Therefore, they could benefit from guidance on how to develop rigorous and functional end-to-end machine learning workflows, for their sustainable research or for a government or industry partner, without missing important components or making decisions without considering fundamental AI principles, and also from understanding the state-of-the-art automation tools that can optimize their pipeline design and parameterization. Related work, such as [42,43,44], addressed critical components of a machine learning workflow, but without a focus on supervised cases with numerical and categorical data, while they were not very thorough on the interconnection of the sub-modules, and lacked the explanation of the basic principles required for the development of the workflow. Moreover, the references of [42] are mostly websites and blogs, as opposed to scientific papers. On the other hand, relative books in pipeline development focus more on coding using a specific language and often lack important concept explanations, such as the bias–variance decomposition [45,46,47], or may focus only on a specific part of the workflow, such as feature engineering [48], lacking to provide a holistic overview of the entire workflow, the importance of every sub-module and their meaningful, other than random, integration. Other review papers focus entirely on the automation of a very specific machine learning area, such as time series forecasting automation [49], and lack the explanation of basic AI principles and workflow sub-module operation, interconnection and development for the education and guidance of a non-AI researcher, seeking to understand and complete their manual workflow development before automating. Last, there are some machine learning workflow automation papers [41,50] that focus on a very specific application, i.e., biology or computer networks, that are, however, relatively incomplete and missing important AI principles, as well as references of methods outside of what has been published for that specific application under consideration, making them appropriate to guide researchers only on the specific application and not on the broader spectrum of supervised learning with numerical and categorical data.

The contributions of this paper are threefold. First, this work introduces a methodology for the development and thorough understanding of end-to-end supervised machine learning workflows with numerical and categorical data. Second, this methodology is accompanied by all the workflow sub-modules, methods, principles, models, and algorithms, as well as their integration into the workflow, to guide and inform the non-AI researcher on developing rigorous pipelines. Third, it provides state-of-the-art tools for automated machine learning, including but not limited to automated feature engineering, architecture search, hyperparameter search, etc. A systematic and in-depth explanation of machine learning workflow sub-modules and interconnection allows researchers and non-AI professionals to fully comprehend, improve, and excel in machine learning workflow development and automation.

The rest of this paper is organized as follows. Section 2 provides an overview of the end-to-end machine learning workflow architecture. Section 3 presents the data-engineering sub-module, and Section 4 presents the machine learning sub-module, which includes models and algorithms, as well as training, validation, and evaluation methods. Section 5 analyzes the model deployment step, Section 6 presents the state-of-the-art automation methods in machine learning workflows and related coding practices, and Section 8 provides conclusions.

2. End-to-End Architecture of Machine Learning Workflows

The workflow of every machine learning (ML) project broadly consists of four stages: scoping of the problem, data engineering, modeling, and deployment of the model. In order to deploy and maintain machine learning models in production reliably and efficiently, a set of practices is necessary to automate the machine learning workflow while meeting business and regulatory requirements, known as Machine Learning Operations (MLOps) [51]. MLOps is considered to be the cross-section of Development Operations (DevOps), data engineering, and Machine Learning Engineering [51]. DevOps is another set of practices that combine software development (Dev) and IT operations (Ops) in order to develop useful software and maintain all data resources in data centers. The core principle of DevOps is automation, and therefore Continuous Integration Continuous Delivery or Deployment (CI/CD) is a very important component so that any software updates from developers are integrated and delivered to the latest software version [52]. CI/CD requires continuous development, continuous testing (passing a series of predefined tests; see Section 5 for more details), continuous integration, continuous deployment, and continuous monitoring. Expertise in all three areas shown in Figure 1 is required for a machine learning workflow to transition into a product.

A machine learning workflow begins with scoping the problem to be solved, examining the available resources, and deciding upon the feasibility of the project. Then, the objectives and requirements are set, and the project is initiated. As Figure 2 shows, data engineering, machine learning model engineering, and model development (production code) are the subsequent stages that will be continuously updated in an automated way every time new data arrive. This workflow will be incorporated into production code and be subject to testing (Section 5) before every new version deployment as well as continuous monitoring and maintenance throughout the entire lifecycle.

3. Data Engineering

This section provides a detailed explanation of the various data-engineering stages.

3.1. Data Pipeline: Extraction, Loading and Transformation

Data can be extracted and fed into a machine learning workflow from several sources, including databases (DBs), documents, sensors, and simulations as shown in Figure 2a. Loading and integrating these data can be challenging, and it is an ongoing process, even after the machine learning model has been deployed and an initial master database has been constructed (continuous integration). All these data are in raw format and require transformation (cleaning), i.e., treating missing values (i.e., imputation [53,54], complete entry removal, matrix completion [55,56,57,58,59] depending on the randomness in the missing data pattern or by producing the missing data with simulations [60]), correcting for data formats (mathematical representation of numbers, time stamps, inconsistent text format, size and resolution of images, etc.), de-duplicating [61] (with or without a unique key per real-world entry) or removing redundant data, dealing with contradicting data, removing outliers (point outliers, collective or contextual outliers), etc. Current industry practices for data engineering include ELT tools (Extract, Load, Transform) [62], which offer data extraction, data integration, and data transformation functions via some Application Programming Interface (API). ELT is currently preferred as opposed to the previously followed approach ETL (Extract, Transform, Load) [63] because the data are integrated and saved before any transformations occur (data lake implementation). This way, the same data can be accessed by several analysts, scientists, and engineers before any transformations take place, and the scalability of transformation scripts, as the data set increases, is no longer an issue. These transformations are version-controlled, and therefore historical transformations can be re-created by rolling back commits in ELT, while data and models can be easily tested. ELT has been growing quickly due to the following enablers: the development of modern data lakes, access to products that can load code and store data in data lakes, and the necessity to open the ELT process to anyone who has Structured Query Language (SQL) knowledge [64].

3.2. Feature Engineering

Feature engineering consists of feature generation, feature transformation, feature selection, and automated feature extraction for Deep Learning as shown in Figure 2b.

3.2.1. Feature Generation

Feature generation refers to the process of creating new features for the purpose of improving the success metric, which validates the machine learning model (Figure 2a), and therefore, it is a very important step in the machine learning workflow. To extract new features, data visualization and domain expertise [18,65] are necessary, but when unavailable or too complicated, Deep Learning neural network architectures can automatically extract features with additional layers which are trained to simulate the appropriate feature transformations that optimize training and generalization errors [66,67]. However, for a researcher with a deeper understanding of the underlying problem, feature engineering will result in producing features by generating, combining, or transforming existing features (feature transformation explained next) in the direction of machine learning model performance improvement, given an appropriate evaluation metric (Section 4.3). However, it is important to consider a priori a machine learning model in terms of feature learning capabilities. For example, linear regression models are able to learn sums and differences easily but a complex relationship between inputs and outputs would not be learned easily due to model linearity, and therefore, such a model could benefit from generated features that explicitly capture that complicated non-linear relationship [68], or by linearizing these relationships. On the other hand, it is wise to be careful regarding generated features such as ratios because they can be harder to learn by most machine learning models. Moreover, trees are more capable of capturing complex relationships between the inputs and outputs (numerical or categorical) but could still benefit from explicit features generated to capture those [69]. They can also benefit from group aggregate features, such as counts (features that count the total number of times a binary value is present), minimum and maximum feature values, and mean and standard deviation, while they are often used for feature selection, a by-product of the tree training [70]. However, trees can be sensitive to noisy inputs, so it is recommended to remove irrelevant input information, either by removing a feature or by removing a feature’s component (i.e., higher frequency noise) [71].

3.2.2. Feature Transformation

Feature transformation includes feature scaling and the application of deterministic transformations, and its significance in the machine learning workflow is explained here. Feature scaling is an important part of the machine learning workflow since models that are smooth functions of the input, such as linear regression, Logistic Regression (LR), Neural Networks (NNs), and other matrix-based models can be affected by the feature scale (values), given that the parameter vector which is updated via the Stochastic Gradient Descent method will be biased towards specific features [72]. Stochastic Gradient Descent algorithms also converge faster [73] with scaled feature values and with smaller derivatives during back-propagation (a by-product of feature scaling). Feature scaling is also necessary for Principal Component Analysis (PCA), used for dimensionality reduction, which may also be biased since the direction of maximum variance can be affected by large feature values. This issue has been bypassed by creating features with unit variance via scaling with the standard deviation [74]. On the other hand, distance-based algorithms, such as k-Nearest Neighbors (kNN), k-Means, and Support Vector Machines (SVMs) are mostly affected by the range of values because they use distances between data points to determine their similarity; therefore, imposing scaling methods that keep the range the same for all features, such as Min-Max, can provide improved results [75]. Moreover, tree-based supervised algorithms such as Classification and Regression Trees (CARTs), Random Forests (RFs), and Gradient Boosted Decision Trees [71], are not affected by either the scale or the range of the feature values [48] because most splitting criteria utilize one feature at a time, and monotonic feature transformations do not affect the order of the data points [71]. Hence, the same data point (threshold) will eventually be selected to split the specific node of the tree, whether the data points are scaled or not [48]. Last, graphical classifiers like Naive Bayes or Fisher Linear Discriminant Analysis (LDA) that rely on variable distributions also do not require feature scaling. The most common feature-scaling methods are summarized in

Table 1. Summary of the most common numerical feature-scaling methods.

Scaling Method	Scaled Feature	Scaling Effect	ML Algorithm/Model
Min-Max	${\mathbf{x}}^{\prime }=\frac{\mathbf{x}-{\mathbf{x}}^{\min }}{x^{\max }-x^{\min }}$	${\mathbf{x}}^{\prime }\in [0,1]$	k-Means, kNN, SVM
Standardization (z-score)	${\mathbf{x}}^{\prime }=\frac{\mathbf{x}-\overline{\mathbf{x}}}{\sigma }$	${\overline{\mathbf{x}}}^{\prime }=0$, $\sigma =1$	Linear/Logistic Regression, NN
l2-Normalization	${\mathbf{x}}^{\prime }=\frac{\mathbf{x}}{{\parallel \mathbf{x}\parallel }_2}$	$\parallel {\mathbf{x}}^{\prime }{\parallel }_2=1$	Vector Space Model

, where $\mathbf{x}\in {\mathbb{R}}^n$ is some feature. For an example of the effect of different scaling methods on different supervised classification models and algorithms, the reader is referred to [75], which provides performance index values for a heart disease data set using different scaling methods on different machine learning models and algorithms. It is up to the researcher to evaluate the impact of different scaling methods on the workflow and decide the best way to move forward based on Table 1 recommendations, as well as by following a similar approach to the heart disease example in [75].

To better understand the effect of scaling methods, it is worth mentioning that the principle behind any scaling method in Table 1 that divides by a normalization constant is that the feature is scaled without changing the shape of the single-feature distribution (see Figures 2–18 in [48]). For example, if the data distribution is not Gaussian before standardization, then it will not become Gaussian after. It will, however, have a mean of 0 and a standard deviation of 1 as mentioned in [48]. The interested researcher, however, should be careful in choosing a scaling method in terms of sparsity in the original features. For example, although Min-Max scaling bounds the scaled feature in the $[0,1]$ range, that is not the case with standardization and $l_2$-normalization. Min-Max and standardization both subtract a value from the original feature, which may or may not be zero. In the case where the original feature is sparse, doing so may result in a dense scaled feature if the subtracted value is not zero, and that may further burden a classifier model [48].

Feature transformation also includes the application of deterministic, usually invertible, transformations on numerical data. Following data visualization, transformations can be applied to improve feature interpretability (for example, removing high-frequency noise, introducing lag features, feature derivatives, etc. [49]) or conform with the assumptions of machine learning models, such as linear regression (linearity, residual independence, homoscedasticity, and residual normality). For example, in order for a researcher to use linear regression models, a linear relationship between the independent variable and the target variable is required, something not necessary if a neural network architecture was selected. If that relationship is visualized to be non-linear, then a transformation can be decided to improve the linearity between the transformed independent variable and the target [76]. A common family of such transformations is the power transformations (power monotonic functions), with special cases of the log transformation, the square root transformation, and the multiplicative inverse transformation (reciprocal transformation), all for non-zero data. The power transformations are parameterized by a non-negative value $\lambda$, which can be found via statistical estimation methods. When feature values vary in both the positive and negative range, other transformations, such as the multiplicative inverse transformation, the Yeo–Johnson transformation [77], and the inverse hyperbolic sine transformation can be applied. However, for transformations that assume non-negative data, constant offsets can be applied first to the feature values to shift all values in the positive reals. Another reason behind feature transformation is the principle of variance stabilization, which removes the dependency of a population variance from the population mean (for examples, the reader is referred to Figures 2–11 in [48]). Common variance stabilizing transformations are the Fisher transformation [78] for the sample correlation coefficient, the square root transformation or Anscombe transform [79] for Poisson data (count data), the Box–Cox transformation [80] for regression analysis, and the arcsine square root transformation or angular transformation for proportions (binomial data) [81].

Feature transformation is also required for categorical features for the same range and value size reasons as in the numerical data case above. The most common transformation techniques for categorical features are summarized in

Table 2. Summary of most common methods for encoding categorical features.

Encoding Method	Original Feature	Transformed Features	Result
Ordinal Encoding	string1, string2, …	1, 2, …	Nonordinal categorical data becomes ordinal
One-Hot Encoding	string1, string2, …	001, 010, …	k features for k categories, only one bit is “on”
Dummy Encoding	string1, string2, …	001, 010, …, (000)	$k-1$ features for k categories, reference category is 0
Effect Encoding	string1, string2, …	001, 010, …, (-1-1-1)	k features for k categories, reference category is −1

, and their advantages and disadvantages are stated here. Researchers should be cautious in using Ordinal Encoding because of its disadvantage in generating ordered numerical features when no order is present in the original categorical data, which may affect the machine learning algorithm or model [48]. On the other hand, One-Hot Encoding does not have that effect; it clearly assigns one value to each category, leaving the 0 category free for missing data. However, it has a sum of 1 amongst the k new features, which implies a linear dependency between those transformed features. That can lead to issues with training linear models since different linear combinations of the dependent features can produce the same predicted target value [48]. By removing one of the degrees of freedom, One-Hot Encoding becomes Dummy Encoding, where the reference category (missing category) is represented by all zeros, which does not allow for easy missing data representation. Moreover, in terms of interpretation, if a linear regression model is used, in the case of One-Hot Encoding, the intercept represents the target variable’s mean, while with Dummy Encoding, the intercept refers to the mean value of the target variable for the reference category [48]. Last, with Effect Coding, the resulting linear regression models are easier to interpret because the intercept represents the mean of the target variable. On the other hand, the dense −1 representation of the reference category can be expensive in terms of storage and computation. Since all the aforementioned encoding methods are not scalable for large data sets, scalability is only possible via feature compression. Feature compression is translated into either feature hashing (linear and kernel models) or bin counting (linear models and trees), where instead of the categorical feature, the conditional probability of the target variable under that value is used. The back-off method [48] or the count-min sketch [82] methods are available in the literature for rare categories in large data sets that were previously transformed with bin counting [48].

To summarize the feature transformation sub-module, the interested researcher is encouraged to also consider multivariate feature transformations, which transform several features at a time and may have superior performance in specific circumstances where, for example, the features are correlated [83] and high correlation can lead to feature redundancy. One such example is whitening transformations (linear) which lead to uncorrelated features with an identity covariance matrix [83]. The covariance matrix may first be expressed in a decomposed form via, for example, Cholesky decomposition [84], before the whitening transformation is applied. Other similar transformations include decorrelation, which removes the correlations but leaves variances intact [83]; standardization, which sets the variances to 1 but leaves the correlations intact; and coloring, which turns a random a vector of white random variables (with identity covariance) into a random vector with a pre-specified covariance matrix.

3.2.3. Feature Selection

Feature selection is another crucial sub-module of the machine learning workflow, often omitted for brevity or due to lack of expertise in this area. However, the lack of features can lead to machine learning model underfitting and redundancy to overfitting, and therefore proper feature selection is important [72], following the feature generation and transformation steps. Feature selection can remove redundant features that are irrelevant to the target variables (adding “noise”) by either selecting a subset of the features or by creating a lower-dimensional representation (embedding) of the feature set, which adequately summarizes the necessary information contained in the original feature space [48] but unfortunately without the physical interpretation of the original features (dimensionality reduction is traded for interpretability). Moreover, feature selection serves the purpose of dimensionality reduction [85], which can remove the computational burden from the machine learning model by removing linearly dependent features. A very common method of dimensionality reduction is PCA, which is also a whitening transformation in the sense that the transformed features are no longer correlated [48]. In PCA, the dimensionality k of the embedding is subject to user choice, but the operational cost of PCA can be very high for over a thousand features since it involves Singular Value Decomposition (SVD) of the original feature matrix [48]. A very interesting application of PCA is anomaly detection in time series data [86]. Since PCA projects the features to a linear subspace via linear projection, it is not ideal when the data lie in a nonlinear manifold. It is also not ideal when feature interpretability is necessary. In contrast, k-Means can perform feature selection when data lie in a nonlinear manifold, by clustering features together and selecting one feature to survive from each cluster (representative), while the rest are being discarded [48]. Another approach to dimensionality reduction is autoencoders [87], which, however, suffer from a lack of feature interpretability as well. Common unsupervised and supervised feature selection methods are summarized in Figure 3.

Unsupervised feature selection methods can be categorized as follows (a taxonomy is provided in Figure 1 of [88]). Attention is required to determine whether the method can directly incorporate categorical features or not.

• Filter methods [88,89,90]: In this category of methods, feature selection is a pre-processing step to the machine learning model training, and these methods are time-efficient:

  (a)

  Statistical/information-based: These methods maximize feature relevance by maximizing a dependence measure, such as variance, covariance, entropy [90], linear correlation [91], Laplacian score [92], and mutual information. Representative methods include Feature Selection with Feature Similarity (FSFS) [91] based on Maximal Information Compression Index (MICI) and Relevance Redundancy Feature Selection (RRFS) [93]. Fisher’s criterion [94] is only used in supervised learning.

  (b)

  Spectral/sparsity learning: These methods perform spectral analysis or combine spectral analysis with spectral learning. They find a trade-off between Goodness-of-Fit and a feature similarity measure. Representative methods include Multi-Cluster Feature Selection (MCFS) [95], Unsupervised Discriminative Feature Selection (UDFS) [96], and Non-negative Discriminative Feature Selection (NDFS) [89].

• Wrapper methods [88,89,90]: In this category, feature selection is intertwined with the machine learning model training and hence evaluated by the model performance. These methods are more accurate than filter methods but less time-efficient:

  (a)

  Sequential methods: These methods perform clustering on each feature subset and evaluate the clustering results based on some criterion. They can be based on Expectation Maximization or the Trace Criterion [97], or on the min/max of intra/inter-cluster variance [98] and a decision can be made based on a score that provides feature ranking. Another alternative is the Simplified Silhouette Sequential Forward Selection (SS-SFS) proposed in [99].

  (b)

  Iterative methods: Ref. [100] performs clustering and feature selection simultaneously by evaluating feature weights called feature saliences. Other iterative methods include Local Learning-based Clustering with Feature Selection (LLC-fs) [101], Embedded Unsupervised Feature Selection (EUFS) [102], and Dependence Guided Unsupervised Feature Selection (DGUFS) [103].

• Embedded methods: In this category, feature selection is part of the machine learning model training process [104].

Figure 3. Taxonomy of feature selection methods.

Figure 3. Taxonomy of feature selection methods.

Supervised feature selection methods include the deployment of a supervised machine learning model when target variables are available. After training a supervised machine learning model, a feature-ranking process is used to select the most important features that adequately describe the outputs (one or more target variables). Three main approaches in supervised feature selection are explained below. A more thorough overview of feature selection methods can be found in the book [104] and in chapter 19 of the book [105].

• Shrinkage-based methods [106]: Single-output or multi-output regression models with $L_1$- or $L_2$-regularization can be trained via k-fold cross-validation (CV) to optimize a shrinkage parameter $\lambda$ which trades-off model bias for variance. Penalization of the model weights with an $l_1$ norm is appropriate for feature selection because it can introduce feature sparsity (Lasso estimator [107]) when penalization with an $l_2$ norm (Ridge Regression [69]) does not force feature weights to zero. The combination of the two is called Elastic Net, which is useful when there is a group of features with high pairwise correlations [108]. Multi-output regression models perform better when outputs are correlated, i.e., when multi-task learning is desired, instead of independent task learning [109,110]. Multi-output models utilize an $l_{2,1}$ norm penalization term, which either includes or excludes a feature from the model for all outputs [111]. In the multi-output case, the average weight of a feature across all outputs is obtained, and then these average weights are normalized in the $[0,1]$ range (relative importance) with the min-max scaling method so that a rank of feature relative importance is derived [23].
• Tree-based methods: CART can be trained in a supervised sense and provide feature ranking as a byproduct of the training process [71] in single-output or multi-output Decision Trees (DTs) [112]. DTs are over-sensitive to the training set, irrelevant information, and noise; therefore, prior unsupervised feature selection is strongly encouraged via one of the methods proposed above. Moreover, DTs are known to overfit, and hence, ensembles of DTs [112], such as Bagging (bootstrap aggregation) [113], Boosted Trees [114] and Rotation Forests, are constructed to cope with overfitting. The RF, a characteristic example of Bagging, can generate diverse trees by bootstrap sampling and/or randomly selecting a subset of the features during learning [115,116]. Although an RF is faster and easier to train than a boosted tree [117,118,119], it is less accurate and sacrifices the intrinsic interpretability (explanation of output value and feature ranking) present in DTs [69]. In particular, feature selection happens inherently in single-output and multi-output DTs as the tree is being constructed since the splitting criteria used at each node select the feature which performs the most successful separation of the remaining examples [71]. Therefore, in RFs, feature ranking is either impurity-based, such as the Mean Decrease in Impurity (MDI), or permutation-based, such as Permutation Importance [115]. MDI is also known as Mean Decrease Gini or Gini Importance.
• Permutation Importance [115] is not only useful in RFs which have lost the inherent feature-ranking mechanism of the tree but in other supervised machine learning models as well. Permutation Importance is better than MDI because it is not computed on the training set but on the Out-of-Bag (OoB) sample and is, therefore, more useful to inform the feature importance for predictions [115]. Moreover, MDI significantly favors numerical features over categorical ones as well as high-cardinality categorical features (many categories) over low-cardinality ones [115], something that does not happen with Permutation Importance. The Permutation Importance of a feature is calculated as the difference between the original error and the average permuted error of this feature, over a number of specified repetitions [115]. The permuted error of each feature (the OoB error) occurs when that feature is permuted (shuffled). Permutation is a mechanism that breaks the relationship between that feature and the target variables, revealing the importance of a feature to the model training accuracy [120]. In trees and other supervised methods which use a feature-ranking approach to feature selection, the least-performing features in terms of relative importance can be excluded from the feature set.

3.2.4. Automated Feature Extraction

An important note on feature engineering is that as the number of features and samples increases, data engineering, feature engineering, and Machine Learning Engineering methods require scaling and modification to adjust to the new challenges. This concept was first introduced by Richard Bellman, as the Curse of Dimensionality (CoD) [121]. This principle is important to understand the need behind automated feature engineering.

CoD is associated with several challenges. First, the data representation in higher dimensions can be very sparse since the data tend to concentrate in lower-dimensional spaces, and therefore the sample size necessary for learning a statistical or machine learning model grows exponentially [122]. Second, the pairwise distances of the data points in higher-dimensional spaces increase, and they become more homogeneous; hence, observations may appear to be equally distanced from each other [123]. Two additional challenges related to CoD arise in the case of high-dimensional data clustering and anomaly detection. Those are the relevant attribute identification, i.e., the difficulty of describing the relevant quantitative locality of data instances in the high-dimensional space, and hubness, which refers to the tendency of high-dimensional data to contain data points that appear frequently in nearest neighbor clusters, known as hubs [124].

Table 3. Summary of most common automated feature engineering tools for high-dimensional data (FE = feature engineering).

Automated FE Tool	Operation	Tool Tested On	Developer	Paper
ExploreKit	Feature generation and ranking	DT, SVM, RF	UC Berkeley	[125]
One Button Machine	Feature discovery in relational DBs	RF, XGBOOST	IBM	[126]
AutoLearn	Feature generation and selection	kNN, LR, SVM, RF, Adaboost, NN, DT	IIIT	[127]
GeP Feature Construction	Feature generation from GeP on DTs	kNN, DT, Naive Bayes	Wellington University	[128]
Cognito	Feature generation and selection	N/A	IBM	[129]
RLFE	Feature generation and selection	RF	IBM	[130]
LFE	Feature transformation	LR, RF	IBM	[131]

Table 3. Summary of most common automated feature engineering tools for high-dimensional data (FE = feature engineering).

Automated FE Tool	Operation	Tool Tested On	Developer	Paper
ExploreKit	Feature generation and ranking	DT, SVM, RF	UC Berkeley	[125]
One Button Machine	Feature discovery in relational DBs	RF, XGBOOST	IBM	[126]
AutoLearn	Feature generation and selection	kNN, LR, SVM, RF, Adaboost, NN, DT	IIIT	[127]
GeP Feature Construction	Feature generation from GeP on DTs	kNN, DT, Naive Bayes	Wellington University	[128]
Cognito	Feature generation and selection	N/A	IBM	[129]
RLFE	Feature generation and selection	RF	IBM	[130]
LFE	Feature transformation	LR, RF	IBM	[131]

The last component of feature engineering is automated feature extraction, which removes human intervention. This component is very useful in high-dimensional data that suffer from CoD, big data, and Deep Learning. Common examples of high-dimensional big data are time series, images (Chapter 8 in [48]), video data [132], and finance applications [133,134], while extracted features are saved and organized in feature stores (structured data in Figure 2a) for accessibility and re-usability. Core approaches in automated feature extraction include Multi-Relational Decision Tree Learning (MRDTL) [135], Deep Feature Synthesis (DFS) [136] and methods/models such as Deep Neural Networks (DNNs) [137], Adaptive Linear Approximation (ALA), PCA, extraction of Best Fourier Coefficients (BFCs) and Best Daubechies Wavelet Coefficients (BDWs), and statistical moments [138]. Once the features are extracted, then feature selection can be performed via an unsupervised or supervised method, appropriate for the problem dimensions. Recursive Feature Elimination Support Vector Machines (REFSVMs) and RELIEFF algorithms are proposed in [138] for feature selection, and they can take into account feature interaction (correlation) for up to 500 features. For more features, the Pearson correlation coefficient is used to remove highly correlated features before any of the two aforementioned algorithms is applied. Other methods for automated feature extraction are Genetic Programming (GeP) [139], Reconstruction Independent Component Analysis (RICA) [140], which generates sparse representations of whitened data, Sparse Filtering [141], which minimizes an objective function via a standard limited memory Broyden–Fletcher–Goldfarb–Shanno (LBFGS) quasi-Newton optimizer [142] and Wavelet–Scattering networks [143,144,145,146], which apply wavelet and scattering filters to produce low-variance features from time series and image data in Deep Learning applications. Features extracted by Wavelet–Scattering networks are insensitive to input translations on an invariance scale in the 1-D case [145], and insensitive to rotations in the 2-D case. Invariance to input transformations, such as measurement noise and image rotation and translation, is desired so that the performance of the machine learning model is unaffected. The interested reader is also encouraged to review additional automated feature engineering tools, such as ExploreKit [125], One Button Machine [126], AutoLearn [127], GeP Feature Construction [128], Cognito [129], RLFE [130], and LFE [131] (Table 3).

4. Machine Learning Engineering

4.1. Models and Algorithms for Supervised Learning with Numerical and Categorical Data

Since the area of machine learning models and algorithms is very broad, it would be impossible to cover in one paper, and therefore, this section focuses only on some basic supervised models and algorithms for the purposes of representing this workflow sub-module and its integration with the others. Supervised learning is possible when target variable values are present in regression (continuous range target values) or classification problems (pre-specified target values indicative of a class). Machine learning models are characterized as parametric if they have a specific function form with parameters, whose values can be determined by a data set [122]. The most commonly used machine learning models and algorithms for supervised learning are summarized in

Table 4. Summary of common supervised machine learning models and standard algorithm complexity. Symbols: n—samples; p—features; s—support vectors; k—neighbors or trees in a model; d—nodes in a tree; q—maximum number of bins; and $\gamma$—constant. The star ★ indicates that complexity varies with architecture (neurons, layers, connections, and activation function [147]) and algorithm (type, epochs). Typically, Gradient Descent has a running complexity of $\mathcal{O}\left(n^{2}p\right)$.

ML Model/Algorithm	Parametric	Linear	Train, Test, Space Complexity	Paper
Ordinary Least Squares (OLS)	✓	✓	$\mathcal{O}(n{p}^2+p^3)$, $\mathcal{O}\left(p\right)$, $\mathcal{O}\left(p\right)$	[148]
Kernel Ridge Regression	✓	✓	$\mathcal{O}\left(n^3\right)$, -, $\mathcal{O}\left(n^2\right)$	[149]
Lasso Regression (LARS)	✓	✓	$\mathcal{O}(n{p}^2+p^3)$, -, -	[150]
Elastic Net	✓	✓	$\mathcal{O}(n{p}^2+p^3)$, -, -	[108]
Logistic Regression	✓	✓	$\mathcal{O}\left(np\right)$, $\mathcal{O}\left(p\right)$, $\mathcal{O}\left(p\right)$	[151]
GP Regression	✗	✗	$\mathcal{O}\left(n^3\right)$, -, $\mathcal{O}\left(n^2\right)$	[152]
Multi-Layer Perceptron	✓	✗	★	[153,154]
RNN/LSTM	✓	✗	★	-
CNN	✓	✗	★	-
Transformers	✓	✗	★	-
Radial Basis Function NN	✗	✗	★	-
DNN	✓	✗	★	-
Naive Bayes Classifier	✓	✓	$\mathcal{O}\left(np\right)$, $\mathcal{O}\left(p\right)$, $\mathcal{O}\left(nq\right)$	[155]
Bayesian Network	✗	✗	★	[156]
Bayesian Belief Network	✓	✗	★	-
SVM	✗	✓	$\mathcal{O}\left(n^2\right)$, $\mathcal{O}\left(sp\right)$, $\mathcal{O}\left(np\right)$	[157]
PCA	✗	✓	$\mathcal{O}(npmin(n,p)+p^3)$, -, $\mathcal{O}\left(np\right)$	[158]
kNN	✗	✗	$\mathcal{O}\left(knp\right)$, $\mathcal{O}\left(np\right)$, $\mathcal{O}\left(np\right)$	[159,160]
CART	✗	✗	$\mathcal{O}(n·logn·p)$, $\mathcal{O}\left(p\right)$, $\mathcal{O}\left(\mathrm{tree}\mathrm{depth}\right)$	[161]
RF	✗	✗	$\mathcal{O}(n·logn·pk)$, $\mathcal{O}\left(pk\right)$, $\mathcal{O}(\mathrm{tree}\mathrm{depth}·k)$	[162]
Gradient Boost Decision Tree	✗	✗	$\mathcal{O}(n·logn·pk)$, $\mathcal{O}(dk+\gamma )$, $\mathcal{O}(\mathrm{tree}\mathrm{depth}·k)$	[163]
LDA	✓	✓	$\mathcal{O}(npt+t^3)$, -, $\mathcal{O}(np+nt+pt)$, $t=min(n,p)$	[164]

. Note that DNN can be modeled as Gaussian Processes (GPs) [67] and hence are considered non-parametric as well. The choice of the problem model or algorithm is not trivial and highly depends on the scalability desired as well as the features collected (dimensionality, linearity, signal-to-noise ratio, probabilistic assumptions, correlation, etc.) as explained in the feature-engineering section. Often, researchers will experiment with multiple models/algorithms, but it is wise to narrow down the search for practical purposes. Table 4 summarizes basic criteria for guiding the choice of a model/algorithm, with each individual having additional assumptions that need to be taken into consideration.

4.2. Model Training and Validation

It is important to ensure that the machine learning model is not overfitting during the training process so that better model generalization is achieved in the testing phase and future use of this model. Common approaches to prevent model overfitting include validation, CV [165], data augmentation, removal of features, early stopping of training before optimality, regularization, and ensembling (Bootstrap [166], Bootstrap Aggregating, also known as Bagging [113,167] and Boosting [168]). Although CV and Bootstrap include efficient sample re-use [69], other validation methods can be analytical, such as the Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), Minimum Description Length (MDL), and Structural Risk Minimization (SRM) [69]. Different validation methods provide a different test error estimate, which can be a more or less unbiased estimate of the true underlying test error as mentioned by the authors in [69].

To explain the model validation principle and the model complexity choice better to applied researchers developing pipelines in their field, train and test error estimates are summarized in Figure 4a. The light blue and red lines correspond to the train and test error estimates for different data sets of equal size. The dark blue and red lines are the averages of the train and test error estimates of the light red and blue lines. The left half of Figure 4a (left of the red line minimum point) corresponds to higher error bias and lower error variance and is a sign of model underfitting due to the limited model’s capacity. The right half of the same figure refers to higher model complexity, with higher error variance and lower error bias. That is the region of overfitting and happens when random noise in the data is modeled. The middle part which corresponds to the minimum of the test error (red line), is the ideal model complexity so that the best model generalization is achieved. The test error estimate is further decomposed into its components in Figure 4b. In supervised learning, the test error estimate is calculated by comparing the model’s predictive capability with the ground truth. In unsupervised learning, model selection is often conducted by selecting the point of maximum absolute second derivative on the train error curve, also known as the elbow method [169], commonly used in k-Means. For other validation and evaluation metrics, including CV in unsupervised learning, the interested researcher is referred to the following papers [170,171,172].

All the errors in Figure 4a are estimates based on some data set. Since the bias–variance decomposition is the most fundamental principle in machine learning, this section will provide the proof behind the results shown in Figure 4a,b, and help the interested researcher understand the role this concept plays in every decision made across the pipeline, from choosing a metric to using CV, regularization, feature selection, etc. The test error is decomposed in Figure 4b, according to Equation (1). Assume for simplicity that $y=f\left(x\right)+ϵ$, where $y\in \mathbb{R}$ is the target variable, $f\left(x\right)\in \mathbb{R}$ is the machine learning model, $x\in \mathbb{R}$ is the model input, $ϵ\in \mathbb{R}$ is noise with mean $\mathbb{E}\left[ϵ\right]=0$ and variance ${\sigma }_{ϵ}^2$, and $\widehat{f}\left(x\right)\in \mathbb{R}$ is the fitted model to a specific training set. Then, the expected test error is decomposed as follows, with the help of the identity ${(a+b+c)}^2=a^2+b^2+c^2+2ab+2bc+2ca$:

$$\begin{array}{cc}\hfill & \mathrm{Expected}\mathrm{test}\mathrm{error}\left(x\right)=\hfill \\ \hfill =& \mathbb{E}\left[{(y-\widehat{f}\left(x\right))}^2\right]=\mathbb{E}\left[{(f\left(x\right)+ϵ-\widehat{f}\left(x\right))}^2\right]=\mathbb{E}\left[{(f\left(x\right)+ϵ-\widehat{f}\left(x\right)+\mathbb{E}\left[\widehat{f}\left(x\right)\right]-\mathbb{E}\left[\widehat{f}\left(x\right)\right])}^2\right]\hfill \\ \hfill =& \mathbb{E}\left[{ϵ}^2\right]+\mathbb{E}\left[{(f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])}^2\right]+\mathbb{E}\left[{(\mathbb{E}\left[\widehat{f}\left(x\right)\right]-\widehat{f}\left(x\right))}^2\right]+2\mathbb{E}\left[(f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])ϵ\right]+2\mathbb{E}\left[ϵ(\mathbb{E}\left[\widehat{f}\left(x\right)\right]-\widehat{f}\left(x\right))\right]\hfill \\ \hfill & +2\mathbb{E}\left[(\mathbb{E}\left[\widehat{f}\left(x\right)\right]-\widehat{f}\left(x\right))(f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])\right]\hfill \\ \hfill =& \mathbb{E}\left[{(ϵ-0)}^2\right]+{(f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])}^2+\mathbb{E}\left[{(\mathbb{E}\left[\widehat{f}\left(x\right)\right]-\widehat{f}\left(x\right))}^2\right]=\mathbb{E}\left[{(ϵ-\mathbb{E}\left[ϵ\right])}^2\right]+{(f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])}^2+\mathrm{Var}\left[\widehat{f}\left(x\right)\right]\hfill \\ \hfill =& {\sigma }_{ϵ}^2+{\mathrm{Bias}}^2\left[\widehat{f}\left(x\right)\right]+\mathrm{Var}\left[\widehat{f}\left(x\right)\right]=\mathrm{irreducible}\mathrm{error}+{\mathrm{Bias}}^2\left[\widehat{f}\left(x\right)\right]+\mathrm{Var}\left[\widehat{f}\left(x\right)\right],\hfill \end{array}$$

(1)

where the cross-terms are zero due to $f\left(x\right)$ being deterministic, i.e., $f\left(x\right)=\mathbb{E}\left[f\right(x\left)\right]$, the independence of $ϵ$ and $\widehat{f}\left(x\right)$ and the assumption that $\mathbb{E}\left[ϵ\right]=0$,

$$\begin{array}{cc}\hfill & \mathrm{Cross}-\mathrm{terms}=\hfill \\ \hfill & =2\mathbb{E}\left[(f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])ϵ\right]+2\mathbb{E}\left[ϵ(\mathbb{E}\left[\widehat{f}\left(x\right)\right]-\widehat{f}\left(x\right))\right]+2\mathbb{E}\left[(\mathbb{E}\left[\widehat{f}\left(x\right)\right]-\widehat{f}\left(x\right))(f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])\right]\hfill \\ \hfill & =2(f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])\mathbb{E}\left[ϵ\right]+2\mathbb{E}\left[ϵ\right]\mathbb{E}[\mathbb{E}\left[\widehat{f}\left(x\right)\right]-\widehat{f}\left(x\right)]+2\mathbb{E}[\mathbb{E}\left[\widehat{f}\left(x\right)\right]-\widehat{f}\left(x\right)](f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])\hfill \\ \hfill & =2(\mathbb{E}\left[\widehat{f}\left(x\right)\right]-\mathbb{E}\left[\widehat{f}\left(x\right)\right])(f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])\hfill \\ \hfill & =0.\hfill \end{array}$$

The first term in (1), corresponds to the variance of the target variable around its true mean since $\mathbb{E}\left[{ϵ}^2\right]=\mathbb{E}\left[{(ϵ-0)}^2\right]=\mathbb{E}\left[{(ϵ-\mathbb{E}\left[ϵ\right])}^2\right]=\mathbb{E}\left[{(y-f\left(x\right)-(\mathbb{E}\left[y\right]-\mathbb{E}\left[f\left(x\right)\right]))}^2\right]=\mathbb{E}\left[{(y-f\left(x\right)-\mathbb{E}\left[y\right]+\mathbb{E}\left[f\left(x\right)\right])}^2\right]=\mathbb{E}\left[{(y-f\left(x\right)-\mathbb{E}\left[y\right]+f\left(x\right))}^2\right]=\mathbb{E}[y-\mathbb{E}\left[y\right]]=\mathrm{Var}\left(y\right)$ and cannot be avoided. The second term, the squared bias, shows how the average of the estimate differs from the true mean, i.e., ${(f\left(x\right)-\mathbb{E}\left[\widehat{f}\left(x\right)\right])}^2={(\mathbb{E}\left[f\left(x\right)\right]-\mathbb{E}\left[\widehat{f}\left(x\right)\right])}^2$. The third term describes the variance of $\widehat{f}\left(x\right)$, i.e., the expected squared deviation of $\widehat{f}\left(x\right)$ from its mean [69].

Having observed the behavior of the expected test error on testing data in Figure 4b, it is suggested to perform model selection on the best bias–variance trade-off point. Another set, the validation set, that was not used for training, is used for model selection, and the model performance is then evaluated on the testing set as explained in Section 4.3. Common ways of splitting the data set are 70% or 50% for the train set, 20% or 25% for the validation set, and 10% or 25% for the test set [69]. When splitting includes random sampling, then the model may generalize better. However, it is advised to avoid random sampling (shuffling) in time series data because of the signal autocorrelation. Moreover, it is suggested to use stratified training data in the case of classification to ensure that all classes are equally represented in the training set so that the classifier is not biased towards a specific class. Although in the case of big data or Deep Learning, the previous splitting approach would be sufficient (Chapter 7 in [69]), in small data sets with a higher danger of overfitting, it would not. Hence, different forms of CV are suggested to prevent bias and overfitting by re-using the data set multiple times. CV is avoided in Deep Learning because training can become computationally intractable, and therefore overfitting is managed in different ways. In big data sets, following training, the model performance is evaluated on the validation set. The validation set is used to tune model hyperparameters and prevent overfitting by early stopping, regularization, etc. The model performance is tested on the testing set that was not used for training or validation.

In cases where the data set is small, some form of CV is utilized as a way to avoid overfitting to that particular data set (

Table 5. Summary of CV methods.

CV Category	Specific CV Method	Result
Exhaustive CV	Leave-p-out CV	$C_p^n=\frac{n!}{(n-p)!p!}$ models trained
	Leave-one-out CV	$C_1^n=n$ models trained
Non-Exhaustive CV	k-fold CV	k models trained
	Holdout	1 model trained
	Repeated Random Sub-Sampling	k models trained
	Validation (also known as Monte Carlo CV)
Nested CV	k*l-fold CV	$k·l$ models trained
	k-fold CV with validation and test set	k models trained with test set

). The first category of CV is Exhaustive CV. The data set of size n is split into a training set of size $n-p$ and a validation set of size p, in all possible ways. Every time, a model is fit and a validation error occurs. The average validation error across all validation sets is estimated for each hyperparameter value so that the optimal value can be selected at the minimum average error. Other variations of Exhaustive CV include the $p=1$ case, which is less computationally heavy than the p case, or $p=2$, which is an almost unbiased method for estimating the area under the Receiver Operating Characteristic (ROC) curve for binary classifiers [173].

The second category of CV includes the Non-Exhaustive methods, an approximation of the Exhaustive Leave-p-out CV, which is computationally tractable. The most popular method, k-fold CV, requires splitting the data set randomly in k sets, and keeping a different set for validation every time as shown in Figure 5. Again, the average error is calculated and utilized for hyperparameter estimation, by constructing a plot of average kfold CV error vs hyperparameter values. Popular choices for k are 5, 10, and 20. It is worth mentioning that the Leave-1-out CV has low bias and high variance, whereas the kfold CV with $k=5$ or $k=10$, is a better compromise [174,175]. The next type of Non-Exhaustive CV method is the Holdout method, which randomly assigns p points in the training set and $n-p$ points in the testing set. It involves a single iteration and may be considered the same as the simple validation approach [175,176]. The last Non-Exhaustive CV method is the Repeated Random Sub-Sampling Validation, also known as Monte Carlo CV, which is similar to the k-fold CV process, only with random validation sets, which means that some observations may never be part of the validation set [105].

The third category of CV methods is the Nested CV, where CV is used for hyperparameter tuning and error estimation simultaneously. Nested CV methods consider a testing set, in addition to the training and validation sets of the previous categories. The k*l-fold CV splits the data set into k sets and iteratively, each of the k sets is split into l sets, where $l-1$ are used for training and the other for validation. The inner sets are used to train the model and tune its hyperparameters and the outer is used for testing in order to provide an unbiased evaluation of the model fit. A special case of k*l-fold CV is the k-fold CV with validation and testing sets for $l=k-1$. One by one, a testing set is selected, and one by one, one of the remaining sets is used as a validation set while the other $k-2$ sets are used as training sets until all possible combinations have been evaluated.

4.3. Model Evaluation

Following the training and tuning of the model on the training and validation sets explained in Section 4.2 comes model evaluation. This step includes methods and metrics to evaluate that the model indeed performs adequately for the purpose for which it was developed. This step includes utilizing a training set based on which prediction is performed and the evaluation of performance indices. Moreover, evaluation refers to the verification of the model’s assumptions after the predicted values are collected. One such example is linear regression models, the assumptions of which are summarized below [177]:

• Linear relationship between each feature and each target variable: this assumption can be verified in the testing set by constructing scatter plots of each output vs each feature.
• Homoscedasticity, i.e., constant residual variance for all the values of a feature: this assumption can be verified by plotting the residuals vs. each feature in the testing set.
• Independence of residual observations, which is the same as the independence of target variable observations (commonly violated in time series data): this can be verified by checking that the autocorrelation of the residual observations is non-zero with the Durbin–Watson test since that would indicate sample dependence.
• Normality of the target variable observations: this can be verified by constructing QQ plots of the residuals against the theoretical normal distribution and observing the straightness of the produced line.

It is worth mentioning that if some of the above assumptions do not hold, feature transformations can be applied to satisfy them, such as the ones explained in Section 3.2 of this dissertation. For additional solutions, in order to conform with the linear regression model assumptions, the reader is referred to [177]. After model training, the assumptions above should also be verified on the testing set.

Supervised regression and classification models are not only evaluated based on their modeling assumptions but also based on performance metrics. Some commonly used performance indices in supervised learning are summarized in

Table 6. Summary of most common performance indices commonly used to evaluate the performance of regression and classification models.

Performance Index	Formula	Purpose
Mean Squared Error (MSE)	$\frac{{\sum }_{i=1}^N{(y^i-{\widehat{y}}^i)}^2}{N}$	Regression
Root Mean Squared Error (RMSE)	$\sqrt{\frac{{\sum }_{i=1}^N{(y^i-{\widehat{y}}^i)}^2}{N}}$	Regression
Mean Absolute Error (MAE)	${\sum }_{i=1}^N\frac{|y^i-{\widehat{y}}^i|}{N}$	Regression
Mean Absolute Percentage Error (MAPE)	${\sum }_{i=1}^N\frac{|y^i-{\widehat{y}}^i|}{N}·100$	Regression
Coefficient of Determination ($R^2$)	$1-\frac{{\sum }_{i=1}^N{(y^i-{\widehat{y}}^i)}^2}{{\sum }_{i=1}^N{(y^i-{\overline{y}}^i)}^2}$	Regression
Adjusted Coefficient of Determination (A-$R^2$)	$1-\left(\frac{N-1}{N-k-1}\right)(1-R^2)$	Regression
Confusion Matrix	TP, TN, FP, FN	Classification
Accuracy	$\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$	Classification
Balanced Accuracy	$\frac{\mathrm{Precision}+\mathrm{Recall}}{2}$	Classification
Misclassification	$\frac{\mathrm{FP}+\mathrm{FN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$	Classification
F1-Score	$\frac{2·\mathrm{Precision}·\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$	Classification
F-Score	$(1+{\beta }^2)\frac{\mathrm{Precision}·\mathrm{Recall}}{{\beta }^2\mathrm{Precision}+\mathrm{Recall}}$	Classification
Receiver Operating Characteristic (ROC)	Graphical	Classification
Area Under Curve (AUC)	Graphical	Classification
Total Operating Characteristic (TOC)	Graphical	Classification

, where the following shortcuts are used: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). TPs/TNs are when the model predicts an observation to belong/not belong to a class and it does/does not indeed belong to that class. Equivalently, FPs/FNs are when the model predicts that an observation belongs/does not belong to a class and it does not/does belong to that class. Note that N is the number of samples, k is the number of independent variables, $y^i$ is an observed value, $\widehat{y^i}$ is a predicted value, and $\overline{y}$ is the sample average of the observed data.

The choice of the evaluation metric is of great importance, as the wrong choice can lead to the optimization of an irrelevant objective or the introduction of bias as, for example, in the case of using accuracy instead of balanced accuracy when the data set is unbalanced in terms of number of data points from each class [178]. The interested researcher may look into [179] for an empirical analysis of different model performance criteria.

5. Model Deployment

The final phase in the machine learning workflow of Figure 2a is the model deployment where businesses aim to harness its value. As model deployment is, for the most part, a task for software developers in the industry, this section is introduced here for completion purposes. Often, non-AI expert academics work closely with the industry to develop customer-tailored ML workflows, which are then deployed by in-house software developers. However, on the academic side, researchers first test their ideas in prototype code with industry data, very often performing unit testing before sharing their code with the industry partner. A short overview of the model deployment steps to follow provides non-AI expert academics with an insight into how their prototype code can be developed more efficiently for integration with other code and extensive testing.

The model deployment process encompasses a series of steps undertaken before, during, and after the model is put into production, effectively facilitating the productization of the model. Model deployment steps include implementing best practices for deploying a model to production, which involves the standardization of code (TDD [180], Object-Oriented Programming (OOP) or Functional Programming paradigms, and the application of Design Patterns [181,182]), rigorous testing (bug identification, performance testing, integration testing, robustness testing, A/B testing [183], etc.), and ensuring the security (deploying in a private network or Virtual Private Cloud (VPC) [184]) and monitoring of the deployed ML model (for the model or data drifts, bugs, failures, etc.). Subsequently, the deployment undergoes continuous updates, which include the integration of new data, additional data resources, transformations, models, etc., into the workflow, with new model versions being continuously delivered and deployed into the product. The inherent complexity of large-scale systems necessitates the implementation of workflows with sound design principles and robust testing. The next subsections provide all the steps involved in transitioning a model into a production environment.

5.1. Testing

This subsection describes the different types of testing that are performed before a model is put into production.

5.1.1. Unit Testing

Testing is a very important component of a machine learning workflow deployment into production for industry purposes and begins with unit testing. Unit testing verifies that software subprograms function as expected in isolation [185] and focuses on scrutinizing components within the software stack rather than predicting the model’s output, given its inherent unpredictability. Various testing frameworks like Pytest [186], Unittest [187], JUnit [188], Mockito [189], etc., can help generate dummy data sets and test expected outputs. Multiple scripted scenarios, encompassing both success and failure cases, are executed in the programming language to assess functionality and identify bugs. A comprehensive suite of unit tests is run during each deployment to ensure the code behaves as intended.

5.1.2. Performance Testing

In addition to unit testing, performance testing is frequently employed to assess the time required for training and gather details about the hardware configurations involved in the training process. This information has been proven to be crucial for future development and testing endeavors. Performance tests also evaluate the model’s response time under normal and peak loads, ensuring adherence to the Service Level Agreement (SLA) [190] and preventing compromise in availability due to request timeouts. Collecting statistics on the ML model’s responsiveness under varying workloads aids in identifying potential issues and guarantees that the system performs within specified operational parameters.

5.1.3. Integration Testing

The next level of testing is integration testing, a vital component of CI/CD, which tests the integration of a new or updated ML model and the rest of the code within the software stack before promoting it to production. Integration testing verifies the interaction between software components [185] by testing them together based on functional threads or exhaustively all with all (also known as "big bang" testing). For example, an ML model that predicts the number of neighborhood houses sold must seamlessly interact with gateways, web servers, databases, and other networking layers [191]. Therefore, integration tests would be performed in this case to ensure that the interaction of the model with these services goes as expected. This rigorous integration testing helps identify and rectify potential issues, reducing the risk of disruptions in the live system.

5.1.4. System Testing

Most functional failures must have been identified during the unit and integration testing. System testing examines the behavior of the entire system for the purposes of system security, speed, accuracy, and reliability (non-functional requirements) and for proper interfacing with other applications, utilities, hardware devices, and the operating environment [185]. Installation testing is one form of system testing that may appear in software systems that host an ML model [185]. System and acceptance testing are tests used across software engineering, whether ML is involved or not, but they are not always present in the ML pipeline testing process.

5.1.5. Acceptance Testing

Acceptance testing checks the system behavior against the customer’s requirements, and the customers undertake, or specify, typical tasks to check that their requirements have been met or that the organization has identified these for the software’s target market. The developers of the system may or may not be involved in this testing level [185].

5.1.6. A/B Testing

Organizations derive value from making data-driven decisions regarding a model’s real-world performance. A/B testing [192] serves the understanding of the relationship between changes made to the model and the actual outcomes observed in user behavior. The model users are divided into the control group A and the treatment group B. The control group responds to the existing model and the treatment group to the new/modified model. User responses are recorded, and metrics are subsequently computed and compared. This comparison helps to assess the model’s performance before deploying it to the entire user base.

5.2. Model Deployment

After successfully completing various tests in the CI/CD pipeline, the approved changes are deployed to the production environment. The deployed model can be executed either through a scheduled process or triggered by certain events. Scheduled model execution, for example, includes generating an overnight report predicting sales for the next day, which runs to provide up-to-date predictions for business planning. Trigger-based execution, on the other hand, involves responding to specific events, such as user recommendations triggered by a user’s search keywords, in order to deliver real-time, personalized results, thus enhancing the user’s experience.

5.3. Monitoring and Maintenance

The consideration of model and data drift is of paramount importance during the protracted phases of model monitoring and maintenance. Model drift [193], driven by alterations in the statistical properties of the target variable over time, can result in a degradation of the model’s predictive performance. Simultaneously, data drift [194] pertains to variations in the distribution of input features, posing challenges to the model’s generalization capabilities. In the lifecycle of an ML model after deployment, sustained vigilance entails regular assessment of performance metrics, and juxtaposing them against benchmarks established during the model’s training phase. In the context of contemporary MLOps, the automation of drift detection and response mechanisms is increasingly pivotal to upholding model efficacy and alignment with evolving data dynamics.

5.4. Security Considerations

The infrastructure in which the ML model operates often faces threats, attacks, and risks that can jeopardize assets and availability. Therefore, it is imperative to assess and prepare for these scenarios by regularly applying software patches to known vulnerabilities. Organizations conduct thorough audits of open-source code for any vulnerabilities before incorporating it into production. A recommended strategy to mitigate risk is the creation of internally approved repositories for various versions of software used in building the model and the overall software stack. This centralized control over software versions ensures that only vetted and secure components are employed, reducing the likelihood of vulnerabilities compromising the system. If the ML model is exclusively serving customers within the organization, it is advisable to deploy the code in a private network or a VPC when hosted in the cloud. Adhering to the Principle of Least Privilege (PoLP) [195] is crucial when configuring both the software and ML model, regardless of whether the audience is internal or external. By implementing PoLP, access permissions are restricted to the minimum necessary for users or systems to perform their intended tasks. This helps reduce the potential attack surface and enhances overall security. In addition, monitoring usage patterns helps detect and mitigate any potential Distributed Denial-of-Service (DDoS) [196] attacks on the system. By closely observing network traffic and user interactions, abnormal patterns indicative of a DDoS attack can be identified and appropriate measures can be taken to ensure the continued availability and performance of the ML model.

To conclude, model production code is developed via TDD, which continuously creates tests to fail the system’s operation and then upgrades the system to pass the test [180]. This is a slow process, but it leads to fewer bugs and re-usable code through various levels of testing [185,197] as shown in Figure 2c. Testing, deployment, monitoring, maintenance, and security follow the development. For comprehensive insights into both static and dynamic testing, the book by Lewis [198] is highly recommended. For guidance on adhering to good coding practices, it is advised to consult the works of Martin [199] and Thomas et al. [200]. These references offer valuable resources for researchers seeking in-depth knowledge in the domains of coding and testing practices for software development.

6. Automation in Machine Learning Workflows

Section 3.2.4 analyzed the automation of feature extraction. This section focuses on the automation of the entire machine learning workflow and optimization of pipeline hyperparameters and is often referred to as AutoML. According to [201], AutoML makes decisions in a data-driven, objective, and automated way, with the user only providing the data, and AutoML automatically determining the best approach. AutoML is particularly useful for domain scientists who do not focus on learning in-depth machine learning practices, as well as big data and Deep Learning applications [201]. The automation of AI expert systems, i.e., AI software with task-specific machine learning workflow for decision-making in the place of a human, is also possible via AutoML. The focal points of automation encompass various facets such as data engineering, feature engineering, model selection, ensembling, algorithm selection, and hyperparameter optimization, as well as considerations of time, memory, and complexity constraints within the pipeline. Additionally, automation extends to the determination of evaluation and validation metrics. It is noteworthy that while certain aspects are more commonly automated, the spectrum of automated processes is diverse, catering to the nuanced requirements of the machine learning pipeline.

6.1. AutoML Methods

AutoML methods are utilized for tuning machine learning workflow hyperparameters, also known as Automated Hyperparameter Optimization (AHO), including those decided before training and those decided during training. For example, the number of layers and neurons in a NN is tuned before training, while the NN weights are tuned during training. Although hyperparameter optimization has been considered an art, AutoML aims to automate this part of the machine learning workflow to make machine learning more accessible to non-technical professionals [36], and help optimize the functionality of developed pipelines.

According to [201], AHO can be very useful because it reduces the human effort necessary for machine learning and improves machine learning algorithm performance [202,203] and the reproducibility of scientific results [204,205]. Hyperparameter optimization is a very challenging problem because (a) function evaluations can be extremely expensive for large models (i.e., DNN), large data sets, or very complex machine learning workflows, (b) the configuration space can be complex (continuous, discrete, conditional hyperparameters) and high-dimensional, (c) access to the loss function’s gradient with respect to the hyperparameters is often impossible, and (d) generalized optimization is not possible, as it depends on the data set size which varies [36]. The most common methods for hyperparameter optimization are summarized below.

• Black-box hyperparameter optimization:

  (a)

  Model-free black-box optimization methods include grid search in a finite range, which, however, suffers from the CoD and random search, where random search samples configurations at random until a certain budget for the search is exhausted [206]. This works better than grid search when some hyperparameters are much more important than others, which is very often the case [206]. Covariance Matrix Adaption Evolutionary Strategy (CMA-ES) [207], is one of the most competitive black-box optimization algorithms.

  (b)

  Bayesian optimization has gained interest due to DNN tuning for image classification [203,208], speech recognition [209] and neural language modeling [202]. For an in-depth introduction to Bayesian optimization, the interested reader is referred to [210,211]. Many recent advances in Bayesian optimization do not treat hyperparameter tuning as a black-box anymore, i.e., multi-fidelity hyperparameter turning, Bayesian optimization with meta-learning, and Bayesian optimization taking the pipeline structure into account [212,213].

• Multi-fidelity optimization: These methods are less costly than black-box optimization methods, which approximately assess the quality of hyperparameter settings. Multi-fidelity methods introduce heuristics inside an algorithm, using low-fidelity approximations of the actual loss function to reduce runtime. Such heuristics include hyperparameter tuning on a small data subset or feature subset and training for a few iterations by using CV or down-sampled images. Learning curve-based prediction for early stopping is used, as well as Bandit-based (successive halving [214] and Hyperband [215]) algorithms for algorithm selection based on low-fidelity algorithm approximations. Moreover, Bayesian Optimization Hyperband (BOHB) [216] combines Bayesian optimization and HyperBand to achieve a combination of strong anytime performance (quick improvements in the beginning by using low fidelities in HyperBand) and strong final performance (good performance in the long run by replacing HyperBand’s random search by Bayesian optimization). For adaptive fidelity options, see [36].

Another AutoML family of methods, besides AHO, is meta-learning. This family aims to systematize the developer’s experience based on which a new model is built. First, meta-data need to be collected that describe previously learned tasks and models and include algorithm configurations and hyperparameter values, workflow architectures, model evaluations (accuracy and training time), optimal model parameters, and meta-features. Second, learning from meta-data needs to take place in order to extract knowledge that can guide future optimal models and tasks. Examples of such methods are transfer learning [217] and few-shot learning [218]. For more related content on learning from model evaluations, task properties, and prior models, please see [36]. A very popular example of AutoML and the aforementioned methods is neural architecture search (NAS), which aims to automate the discovery of new NN architectures [219]. Core approaches to NAS can be found in

Table 7. Overview of main NAS methods in AutoML.

Method	Approach to Speed-Up	Paper
Lower fidelity estimates	Less epochs, data subsets, downscaled models/data, etc.	[215,216,220,221,222,223]
Learning curve extrapolation	Performance extrapolated after few epochs	[224,225,226,227]
Weight inheritance/network morphisms	Models warm-started with inherited weights	[228,229,230,231,232]
One-Shot models/weight sharing	One-shot model’s weights shared across architectures	[233,234,235,236,237,238]

.

The scalability of AHO is an open-ended research topic that seems to concern multi-fidelity optimization, gradient-based methods, and meta-learning methods [36]. Parallel computing is expected to play an important role in the scalability of AHO, with parallel Bayesian optimization already implemented [239]. With the only exception being DNN [203,208,210,216,240,241,242,243], Bayesian optimization has not yet been applied successfully to data sets larger than a few thousand data points. For solutions related to overfitting, generalization, and arbitrary pipeline sizes, the reader is referred to [36]. State-of-the-art approaches to scalable AutoML with data privacy include federated learning [244].

6.2. AutoML Systems

AutoML systems, also known as pipeline optimizers, are popular software that try to automate the machine learning workflow. An overview of available AutoML systems is provided in

Table 8. Overview of available AutoML systems (DE = data engineering, FE = feature engineering).

Software	Problem Automated	AutoML Method	Paper
Auto-WEKA	CASH	Bayesian optimization	[248]
Auto-WEKA 2.0	CASH with parallel runs	Bayesian optimization	[249]
Hyperopt-Sklearn	Space search of random hyperparameters	Bayesian otimization	[250]
Auto-Sklearn	Improved CASH with algorithm ensembles	Bayesian optimization	[251,252]
TPOT	Classification with FE	GeP	[253]
Auto-Net	Automates DNN tuning	Bayesian optimization	[36]
Auto-Net 2.0	Automates DNN tuning	BOHB	[36]
Automatic Statistician	Automates data science	Various	[36]
AutoPytorch	Algo. selection, ensemble constr., hyperpar. tuning	Bayesian opt., meta-learn.	[254]
AutoKeras	NAS, hyperpar. tuning in DNN	Bayesian opt. guides network morphism	[255]
NNI	NAS, hyperpar. tuning, model compression, FE	One-shot modes, etc.	[256,257]
TPOT	Hyperpar. tuning, model selection	GeP	[253]
AutoGluon	Hyperpar. tuning	-	[258]
H2O	DE, FE, hyperpar. tuning, ensemble model selection	Random grid search, Bayesian opt.	[259]
FEDOT	Hyperparameter tuning	Evolutionary algorithms	-
Auto-Sklearn 2	Model selection	Meta-learning, bandit strategy	[252]
FLAML	Algorithm selection, hyperpar. tuning	Search strategies	[260]
AutoGluon-TS	Ensemble constr. for time-series forecasting	Probabilistic time-series	[261]
AutoTS	Time-series data analysis	Various	[262]

. For a complete overview of machine learning algorithms and coding languages supported by these systems, the reader is referred to [245]. Semi-automated pipeline optimizers AutoComplete [246] and PennAI are also mentioned in [245]. A performance comparison for AutoML systems is provided in [247].

7. A Supervised Classification Workflow Example

An example of supervised classification machine learning workflow, based on this paper’s methodology, is presented in Figure 6 and describes the approach taken by the IARPA-funded researchers in [18] to classify a room as occupied or empty (binary classification) from one CO₂ and one temperature sensor measurement. Applications include energy efficiency, indoor air quality, emergency evacuation, and other applications. The data streams collected from the two sensors were saved locally, where the data-engineering code treated the incoming values for missing data and timestamp consistency. Target variable values (occupancy class 0 or 1) were collected from the humans involved in the experiment, for model training and validation. Following that, a new feature, the HVAC state, was generated from the application of a domain-expertise deterministic transformation on the CO₂ and temperature data, which helps enrich the input–output correlations by introducing additional information to this poor-input experiment (limited sensors and limited data set challenge). Additional feature transformation took place by locally smoothing high-frequency noise on the CO₂ data with a Savitky–Golay (FIR) filter [263], and feature extraction by producing the numerical derivatives of the smoothed CO₂ signal and lagged inputs in real-time. All the input invariant features were utilized for training a supervised binary classifier, thus skipping any feature selection, in this feature-poor application, where all features were proved highly important. Although the automated feature extraction methods proposed in this paper may have revealed better features, or the extraction of the same features with less human labor, this opportunity was not taken advantage of in the [18] project, and will remain as future work for the authors. A feed-forward neural network was trained on all the aforementioned features, with its architecture optimized manually via the tracking of the training, validation, and testing errors, according to the bias–variance decomposition principle analyzed in this paper as opposed to the AutoML methods now available and presented in Section 6. The model evaluation took place via several classification metrics, including accuracy, balanced accuracy, F1-score, and custom application-related metrics (success rate, average detection delay, etc.). Model deployment is missing from this academic project. A rigorous methodology similar to the one presented in this paper was followed in [18], and resulted in highly accurate and mathematically rigorous results.

8. Discussion

This work provides an all-inclusive supervised machine learning workflow development and automation methodology for numerical and categorical data, along with an adequate literature review and associated industry practices. The machine learning workflow is analyzed from the early project brainstorming and scoping stage to data engineering, feature engineering, machine learning model engineering, workflow automation, and model deployment. Guidance is provided to the non-AI expert academic on how to develop and integrate the pieces of a rigorous, complete, functional, and optimized machine learning workflow for their application, without missing important sub-modules, whether it is for their own research or for a government or industry partner. Important AI principles (i.e., bias–variance decomposition, curse of dimensionality, model complexity, overfitting, model sensitivity to feature assumptions and scaling, output interpretability, etc.) are explained, and their utilization in making important algorithm or tuning choices in the workflow is emphasized. State-of-the-art tools on feature extraction automation, neural architecture search, model selection, hyperparameter tuning, algorithm selection, model compression, etc., are provided and explained (i.e., Bayesian optimization, Genetic Programming, and random grid search) for the optimization of the machine learning workflow under development.