A generalized decision tree ensemble based on the NeuralNetworks architecture: Distributed Gradient Boosting Forest (DGBF)

Abstract

Tree ensemble algorithms as RandomForest and GradientBoosting are currently the dominant methods for modeling discrete or tabular data, however, they are unable to perform a hierarchical representation learning from raw data as NeuralNetworks does thanks to its multi-layered structure, which is a key feature for DeepLearning problems and modeling unstructured data. This limitation is due to the fact that tree algorithms can not be trained with back-propagation because of their mathematical nature. However, in this work, we demonstrate that the mathematical formulation of bagging and boosting can be combined together to define a graph-structured-tree-ensemble algorithm with a distributed representation learning process between trees naturally (without using back-propagation). We call this novel approach Distributed Gradient Boosting Forest (DGBF) and we demonstrate that both RandomForest and GradientBoosting can be expressed as particular graph architectures of DGBT. Finally, we see that the distributed learning outperforms both RandomForest and GradientBoosting in 7 out of 9 datasets.

Appendices

Appendix A

Appendix B background in CART, RandomForest and GradientBoosting bias

1.1 B.1 CART bias

Bias type 1

During the learning process, node regions are computed iteratively until an end condition is reached. Because of its exhaustive nature, the predictions tend to be biased producing high variance predictions overfitting the dataset $\{x_i\}$. This bias can be defined as

$$\begin{aligned} Bias_1(x) = E[Y \mid X=x]-E[y_i \mid x_i \in R_j] \mid _{x \in R_j} . \end{aligned}$$

(16)

Bias type 2

During the learning process, the splitting thresholds to produce the nodes are computed using only the middle point between two contiguous points from the dataset. No other value can be a threshold candidate. This produces a bias in the prediction due to two reasons. First because of lack of learning capacity in the underpopulated areas and second producing a high variance in the overpopulated areas. This can be visually understood in Fig. 6.

1.2 B.2 Ensemble algorithms

Ensemble algorithms are able to solve the biased prediction of CARTs by combining the predictions of many models trained separately. There are different ensemble algorithms but the main two are GradientBoosting and RandomForest.

RandomForest-Bagging

One way of combining the prediction of many trees to reduce bias is to aggregate the results by the mean. Given a dataset, the learning process of a tree is deterministic, so to make multiple trees produce different predictions, each of them is trained in a different bootstrap subsample of the dataset. This technique is called $bagging$ [8].

$$\begin{aligned} F(x) = \frac{1}{n_{trees}} \sum _{j=0}^{n_{trees}} h_j(x) , \end{aligned}$$

(17)

where $h_j(x)$ is trained to minimize the loss function, $L(y,f(x))$, over a $bagging$ subsample $\{x\}_j$

$$\begin{aligned} h_j(x) = \underset{h}{\mathrm {arg\ min}}\ L(y_i, h(x_i)) / x_i \in \{x\}_j . \end{aligned}$$

(18)

This technique is based on the central limit theorem from statistics where we expect that the variance manifest of the CART biases can be reduced by averaging over enough tree predictions. All the trees learn in parallel, and this kind of learning is called "horizontal learning".

GradientBoosting-Boosting

In contrast, in GradientBoosting each tree does not try to learn over a different dataset sample to minimize the loss, but it follows a "stage-wise" optimization approach, where each tree is added to the ensemble to reduce the global loss from the previous trees. The final prediction is the sum of all the trees in the ensemble. This process is called $boosting$ [9]

$$\begin{aligned} F(x) = \sum _{m=0}^M h_m(x) , \end{aligned}$$

(19)

where $M$ is the number of trees of the ensemble. To reduce the loss function from the previous trees, $L(y,F_{t-1}(x))$, each of those trees is fitted with the gradient of the loss function from the previous predictors:

$$\begin{aligned} h_m(x)= & {} \rho _m g_m (x) , \end{aligned}$$

(20)

$$\begin{aligned} g_m(x)= & {} E_y \left[ \frac{\partial L(y,F(x)}{\partial F(x)}\right] _{F=F_{m-1}} , \nonumber \\ \rho _m= & {} \underset{\rho '}{\mathrm {arg\ min}}\ E_{y,x} \left[ L(y,F_{m-1}(x)-{\rho '} g_m(x))\right] \end{aligned}$$

(21)

In the previous formula, the value of $E_y[.]$ can only be computed knowing the density function $P(y \mid x)$. In real-world problems, we never have the density function, consequently, the $boosting$ is approximated using finite data by assuming regularity in the $P(y \mid x)$ distribution. To do so, each tree is trained with pseudo-responses that are computed (in the case of the $RMSE$ loss) as the difference between the target and the predictions of a current ensemble of trees (residual errors).

$$\begin{aligned} h_m(x) \simeq \underset{h'}{\mathrm {arg\ min}}\ E \left[ L(y-F_{m-1}(x),h'(x))\right] , \end{aligned}$$

(22)

While CART tries to reduce the loss during the training process by splitting leaf nodes into children nodes, the $boosting$ algorithm tries to reduce the loss from each tree by adding another tree. However, the $boosting$ algorithm generalizes better than the optimization from trees because the trees in each state, optimize the loss function using the entire dataset and not only the subsample from the previous node

1.3 B.3 Bagging and boosting bias

Both, $bagging$ and $boosting$, are different ensemble techniques that rely on different mathematical approaches. $Bagging$ relies on generalization by reducing the variance by averaging the prediction from multiple predictors trained to predict the same response. Meanwhile, $boosting$ relies on reducing the global training loss from the previous tree by training a new tree over the pseudo-responses. Even though these techniques are used separately, the two approaches are not incompatible.

GradientBoosting Bias

In the finite data approximation, we suppose that training over the pseudo-responses is representative of the gradient. However, computing the pseudo-responses over the same training data from one tree to another produces biased pseudo-responses. It is demonstrated in [10] that the bias induced by the finite data approximation is:

$$\begin{aligned} Bias_{GB}(x)\equiv & {} E[F_{t-1}(x')]_{x'=x}\nonumber \\{} & {} -E[F_{t-1} (x') \mid x'=x_k]_{x'=x} , \end{aligned}$$

(23)

RandomForest Bias

The reduction of bias from CART using the RandomForest ensemble is based on the reduction of the prediction variance by averaging the predictions from a high number of trees. However, averaging over all the trees does not ensure the convergence to a global minimum of the loss function everywhere (i.e. in all the areas of the feature space). This is due to the robustness of the decision trees to learn outlier regions, which is lost by averaging over the trees which have not been trained with the outliers.

$$\begin{aligned} Bias_{RF}(x) \equiv E \left[ h(x) \mid x \in R_j \right] -E \left[ h'(x) \mid x \in R'_j \right] , \end{aligned}$$

(24)

where $h(x)$ and $h'(x)$ are trees trained with different subsamples. If these subsamples have different statistics (for instance, because of a small outlier region), the ensemble is not flexible enough to learn this characteristic behavior of that region.