On the Robustness of Decision Tree Learning Under Label Noise

Abstract

In most practical problems of classifier learning, the training data suffers from label noise. Most theoretical results on robustness to label noise involve either estimation of noise rates or non-convex optimization. Further, none of these results are applicable to standard decision tree learning algorithms. This paper presents some theoretical analysis to show that, under some assumptions, many popular decision tree learning algorithms are inherently robust to label noise. We also present some sample complexity results which provide some bounds on the sample size for the robustness to hold with a high probability. Through extensive simulations we illustrate this robustness.

A Proof Sketch of Lemmas 1, 2

Let $n^+ ({\tilde{n}}^+)$ and $n^- ({\tilde{n}}^- )$ denote the positive and negative samples at the node under noise-free case (noisy case). Taking positive class as majority, we note $\rho = (n^+ - n^-)/n$. Using Hoeffding bound it is easy to show $\Pr [{\tilde{n}}^+ - {\tilde{n}}^- < 0]\le \exp \left( -\frac{\rho ^2 n (1 - 2 \eta )^2}{2}\right)$. This gives bound for samples needed as $n > \frac{2}{\rho ^2 (1 - 2 \eta )^2} \ln (\frac{1}{\delta })$, completing proof of Lemma 1.

Let $n, n_l, n_r$ be the number of samples at $v, v_l, v_r$ and recall $n_l=an$ and $n_r=(1-a)n$. Recall that ${\tilde{p}}, {\tilde{p}}_l, {\tilde{p}}_r$ are fraction of positive samples at $v, v_l, v_r$ and $p^{\eta }, p^{\eta }_l, p^{\eta }_r$ are their large sample values. Then, using Hoeffding bounds we get (with $\epsilon _1=\epsilon$, $\epsilon _2=\epsilon /\sqrt{a}$ and $\epsilon _3=\epsilon /\sqrt{1-a}$),

$$\begin{aligned} \Pr \left[ \big (|\tilde{p}-p^{\eta }|\ge \epsilon _1 \big ) \cup \big (|\tilde{p}_l-p_l^{\eta }|\ge \epsilon _2\big ) \cup \big (|\tilde{p}_r-p^{\eta }_r|\ge \epsilon _3\big )\right] \le 6e^{-2n\epsilon ^2} \end{aligned}$$

(5)

When this event happens, with some algebraic manipulation, one can show for Gini impurity, $|\hat{\text {gain}}_{\text {Gini}}^{\eta }(f)-\text {gain}_{\text {Gini}}^{\eta }(f)|\le 6(1-2\eta )\epsilon$ where $\hat{\text {gain}}_{\text {Gini}}^{\eta }$ is the random Gini-gain under noise with sample size n and $\text {gain}_{\text {Gini}}^{\eta }$ is its large sample limit. This gives us the bound as needed in Lemma 2. We can prove the lemma for other criteria also similarly.