Calculating Shannon Information of Data Augmentation Strategies I recently caught Andrew Ng's 2021 talk on MLOps (MLOps: From Model-centric to Data-centric AI). At 26:40, he talks about calculating the effectiveness of cleaning your data (training examples) vs. collecting new examples. Apparently this is possible to calculate using Shannon Information.
As an example, Ng notes that "[if you have] 500 pictures of iguanas and if 12% of the labels are noisy (incorrectly labelled), then from a Shannon Information calculation, you can show that [cleaning up the noisy data and collecting another 500 new examples] are about equally effective.".
Does anyone know how exactly this might be calculated?
I've tried to find more information on this but without much luck. Intuitively, it seems related to Shannon's Limit but the exact application is unclear to me. Thanks.

The approaches each address bias or variance, but not both: adding more biased data solely reduces variance, and cleaning up the original data solely reduces bias. So a natural way to analyze Ng's claim is through the bias-variance decomposition of the cross-entropy loss.
Assumptions
Reduce the problem to a toy problem, which makes these simplifications:

*

*The ground truth labels are i.i.d. Typically, labels are only assumed to be independent, as there's usually an input $x$ which makes them non-identically distributed.

*The model is just the arithmetic mean of i.i.d. training labels. Knowing the form of the model will make it easy to compute its mean and variance at an input. Here we'll specifically assume it's just the arithmetic mean b/c that's super easy to work w/.

*The labeling errors are all false positives or all false negatives. We'll assume this to make the math easy.

To dodge (1) and (2), repeat the analysis below for a logistic regression instead.
Definitions

*

*$Y \sim \text{Bernoulli}(\mu)$ is a ground truth label.

*$\mathcal{T}^\delta_N$ denotes a set of $N$ i.i.d. training examples sampled from a $\text{Bernoulli}(\mu + \delta)$ distribution. $\delta$ modulates bias.

*$\hat{Y}(\mathcal{T}^\delta_N)$ is the arithmetic mean of (potentially biased) labels in $\mathcal{T}^\delta_N$. This is our prediction for every test input.

Analysis
The bias-variance decomposition of the average cross-entropy (not to be confused with the cross-entropy when trained on a specific training set, which I think is mathematically harder to work w/) is adapted from equation (3) in this paper1:
$$
\begin{equation}
\text{E} \big[ H(Y, \hat{Y}(\mathcal{T}^\delta_N)) \big] =
\underbrace{D_{\text{KL}} \big( Y \: || \: \text{E} \big[ \hat{Y}(\mathcal{T}^\delta_N) \big] \big)}_{\text{Bias}^2}
+
\underbrace{\text{E} \big[ D_{\text{KL}} \big( \text{E} \big[ \hat{Y}(\mathcal{T}^\delta_N) \big] \: || \: \hat{Y}(\mathcal{T}^\delta_N) \big) \big]}_{\text{Variance}}.
\end{equation}
$$
Some notes before moving on:

*

*The expectations are taken over training sets, $\mathcal{T}^\delta_N$.

*I'm abusing notation for the cross-entropy $H$ and KL-divergence $D_{\text{KL}}$ by inputting random variables instead of their PMFs.

*This formula looks complicated b/c I don't want to omit the $\mathcal{T}^\delta_N$ dependence. If you're already familiar with the bias-variance decomposition for MSE, the formula should pass the eye-test.

It's easy to show that
$$
\begin{align}
\text{Bias}^2 = \mu \log\frac{\mu}{\mu + \delta} + (1-\mu) \log\frac{1-\mu}{1-\mu-\delta}.
\end{align}
$$
The $\text{Variance}$ term can't be computed exactly, but if $N$ is large, the delta method will give a really good approximation. I'll leave out the algebra (lmk if you'd like to see it), but it turns out that
$$
\begin{align}
\text{Variance} \approx \frac{1}{2N}.
\end{align}
$$
Is Ng's claim valid?
Under the major simplifications above, Ng appears to not go far enough! The average cross-entropy of cleaning data is approximately $\frac{1}{2(500)} = 0.001$. The average cross-entropy of adding more biased data as $N \rightarrow \infty$ is just the bias term
$$
\mu \log\frac{\mu}{\mu + 0.12} + (1-\mu) \log\frac{1-\mu}{1-\mu-0.12}
$$
which is greater than $0.01$ for all $\mu \in [0,1]$, i.e., no amount of additional training data will outperform cleaning the original, smaller dataset. So Ng's claim is too conservative under these simplifying assumptions and specific parameters. If you'd like parameters where there's roughly an equivalence, use, e.g., these ones. Notice that the bias, $\delta$, had to be reduced by almost 10x! Then again, the arithmetic mean is one of the least variant models there is, so don't take these results too seriously.
How should Ng's advice be interpreted?
Cleaning training data can be more beneficial than adding training data. IMO, it's important to point this out b/c in ML circles, it can feel like "get more training data" is repeated ad-nauseum. Implicitly, he's also arguing for exploring data to get a sense of the magnitude of labeling errors. Even in benchmark datasets, labeling errors are pervasive2. And in ML competitions, fixing labeling errors is a fine and dandy way to improve performance.
That being said, definitely don't confuse Ng's numbers to be rules of thumb. In the real world, the simplest and most reliable way to decide whether cleaning data is more impactful than gathering more biased data is to simulate. That's b/c the decision (exhaustively) depends on the amount of labeling error (more error => cleaning existing labels may help a lot) and the model complexity (more complex => gathering more data may help a lot), which are both super variable in real ML problems.
The financial costs of each approach must also be incorporated. Consider that cleaning data may be less expensive if the trained model can predict some of the labeling errors, e.g., where the model's predicted probability is high but the label is negative.
(btw, I'm the same person from reddit )
References

*

*Yang, Zitong, et al. "Rethinking bias-variance trade-off for generalization of neural networks." International Conference on Machine Learning. PMLR, 2020.

*Northcutt, Curtis G., Anish Athalye, and Jonas Mueller. "Pervasive label errors in test sets destabilize machine learning benchmarks." arXiv preprint arXiv:2103.14749 (2021).

