Answers.Body: <p>So, first of all, let me specify that we deal with a finite set of <span class="math-container">$x,y\sim p(x,y)$</span>, and not <span class="math-container">$p(x,y)$</span> directly, which is definitely a whole another different story</p>
  <p>Why, well, because we then have an <a href="https://en.wikipedia.org/wiki/Empirical_risk_minimization#Properties" rel="nofollow noreferrer">approximation error</a>, which breaks your assumption <span class="math-container">$p_{train}(x) = p_{test}(x)$</span>. If that was not the case, then you could have just used a giant polynomial, overfit on the training set, and you would have the best possible model imaginable.</p>
  <p>Instead, that's not the case, and we indeed don't like overfitting. Instead, we use data augmentation to avoid it (if you fix the size of the model, you can overfit up to a particular regime of data).</p>
  <p>A final note: usually data augmentation is done in such a way that <span class="math-container">$p_{train}(x^{augmented})\gt 0$</span>, in other words, that the produced sample is still likely to be observed under the training distribution</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-04T16:39:53.763Z
Answers.LastActivityDate: 2024-08-04T16:39:53.763Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>Your intuition about the need for data augmentation transformations to be consistent with the data generating distribution is correct. For your digits problem example, if your test data has no or very few cropped, blurred and randomly rotated images then data augmentation may not be beneficial and can even degrade performance similar to your noted case of training with ImageNet.</p>
  <p>However, in reality you may have very few training and test data availible or your ML model is intended to deploy for production facing real world data which may be very different from your available training and test data with possibly some other usual training challenges such as imbalanced data as discussed in a recent <a href="https://ai.stackexchange.com/questions/46373/dealing-with-likely-class-imbalance-for-unlabeled-test-data/">post</a>. A famous data augmentation example is the <a href="https://en.wikipedia.org/wiki/U-Net" rel="nofollow noreferrer">U-Net</a> CNN developed for biomedical image segmentation, where there're not many patient training samples available for some rare diseases. Based on some domain knowledge some deformed transformations were applied to synthetically augment training data which was required by its deep learning model with huge number of parameters to avoid overfitting.</p>
  <p>In summary facing with the usual reality of potential unknown test or production data distribution, data augmentation reduces variance of the model to prevent overfitting by tradeoff some bias and effectively samples from a lower-dimensional manifold of a usually high-dimensional data space consistent with the well known <a href="https://en.wikipedia.org/wiki/Manifold_hypothesis" rel="nofollow noreferrer">manifold hypothesis</a> of ML. From PAC statistical learning perspective, data augmentation can help lower the <a href="https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_dimension" rel="nofollow noreferrer">VC dimension</a> or <a href="https://en.wikipedia.org/wiki/Rademacher_complexity" rel="nofollow noreferrer">Rademacher complexity</a> of the hypothesis class thus improving the <a href="https://en.wikipedia.org/wiki/Empirical_risk_minimization#Properties" rel="nofollow noreferrer">generalization bound</a> <span class="math-container">${\displaystyle \mathbb {P} \left(L(\phi _{n})-L(\phi ^{*})\right)}$</span> provable by the <a href="https://en.wikipedia.org/wiki/Hoeffding%27s_inequality" rel="nofollow noreferrer">Hoeffding's inequality</a> in probability theory.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-04T22:47:43.143Z
Answers.LastActivityDate: 2024-08-04T22:47:43.143Z
Answers.UserDisplayName: cinch
Answers.Body: <p>Data augmentation helps resolve overfitting model by introducing more variability to the training data.</p>
  <p>If your model does not have overfitting problem, data augmentation may not be needed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-05T02:28:00.150Z
Answers.LastActivityDate: 2024-08-05T02:29:33.983Z
Answers.LastEditDate: 2024-08-05T02:29:33.983Z
Answers.UserDisplayName: Biku
Body: <p>It is common practice when training deep learning algorithms to augment the training data. For example, in computer vision an image might be flipped, rotated, cropped etc., adding new training samples and effectively increasing the training set size.</p>
  <p>In general, models trained with data augmentation perform better than models that didn't include data augmentation during training. The better performance is commonly attributed to the fact that the model was trained on a more diverse set of examples, effectively making it more robust.</p>
  <p>However, I fail to see how this can be explained on a theoretical basis. In machine learning, we typically assume that both the training and test set come from the same distribution <span class="math-container">$p(x, y)$</span> i.e <span class="math-container">$p(x, y) = p_\text{train}(x, y) = p_\text{test}(x, y)$</span>.</p>
  <p><strong>By augmenting the training set, it is not guaranteed that the new distribution <span class="math-container">$p_\text{train}'(x, y)$</span> will match the original <span class="math-container">$p_\text{train}(x, y)$</span> and as such, <span class="math-container">$p(x, y)$</span>. The only way that this can happen is if the applied transformations are consistent with <span class="math-container">$p(x, y)$</span>.</strong></p>
  <p>Is the following correct?</p>
  <blockquote>
  <p>Data augmentation works because we artificially increase the dataset
  with examples that &quot;respect&quot; the data generating distribution.</p>
  </blockquote>
  <p><strong>An example</strong></p>
  <p>Suppose we train a neural network for classifying digits. If all images in the dataset are in the same scale, crystal-clear and with a fixed orientation, is there any reason to augment by cropping, blurring and randomly rotating the training images?</p>
  <p>If the above reasoning is correct, then augmentation shouldn't be applied at all. I suspect this is also the reason when training in a dataset such as ImageNet, no one applies <span class="math-container">$180^\circ$</span> rotations, simply because the dataset doesn't include images upside-down.</p>
  <p>Can someone provide a clear, mathematical justification for data augmentation?</p>
  
Tags: <machine-learning><deep-learning><datasets><computational-learning-theory><data-augmentation>
Title: Theoretical justification for data augmentation
UserDisplayName: Antonios Sarikas
ThreadText: # Theoretical justification for data augmentation
  
  (Asked by: username_0 on 2024-08-04)
  
  It is common practice when training deep learning algorithms to augment the training data. For example, in computer vision an image might be flipped, rotated, cropped etc., adding new training samples and effectively increasing the training set size.
  
  In general, models trained with data augmentation perform better than models that didn't include data augmentation during training. The better performance is commonly attributed to the fact that the model was trained on a more diverse set of examples, effectively making it more robust.
  
  However, I fail to see how this can be explained on a theoretical basis. In machine learning, we typically assume that both the training and test set come from the same distribution $p(x, y)$ i.e $p(x, y) = p\_\text{train}(x, y) = p\_\text{test}(x, y)$.
  
  **By augmenting the training set, it is not guaranteed that the new distribution $p\_\text{train}'(x, y)$ will match the original $p\_\text{train}(x, y)$ and as such, $p(x, y)$. The only way that this can happen is if the applied transformations are consistent with $p(x, y)$.**
  
  Is the following correct?
  
  > Data augmentation works because we artificially increase the dataset
  > with examples that "respect" the data generating distribution.
  
  **An example**
  
  Suppose we train a neural network for classifying digits. If all images in the dataset are in the same scale, crystal-clear and with a fixed orientation, is there any reason to augment by cropping, blurring and randomly rotating the training images?
  
  If the above reasoning is correct, then augmentation shouldn't be applied at all. I suspect this is also the reason when training in a dataset such as ImageNet, no one applies $180^\circ$ rotations, simply because the dataset doesn't include images upside-down.
  
  Can someone provide a clear, mathematical justification for data augmentation?
  
  **Tags:** machine-learning, deep-learning, datasets, computational-learning-theory, data-augmentation
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2024-08-04. Score: 0
  
  So, first of all, let me specify that we deal with a finite set of $x,y\sim p(x,y)$, and not $p(x,y)$ directly, which is definitely a whole another different story
  
  Why, well, because we then have an [approximation error](https://en.wikipedia.org/wiki/Empirical_risk_minimization#Properties), which breaks your assumption $p\_{train}(x) = p\_{test}(x)$. If that was not the case, then you could have just used a giant polynomial, overfit on the training set, and you would have the best possible model imaginable.
  
  Instead, that's not the case, and we indeed don't like overfitting. Instead, we use data augmentation to avoid it (if you fix the size of the model, you can overfit up to a particular regime of data).
  
  A final note: usually data augmentation is done in such a way that $p\_{train}(x^{augmented})\gt 0$, in other words, that the produced sample is still likely to be observed under the training distribution
  
  
  ---
  
  ### Answer by username_2 on 2024-08-04. Score: 0
  
  Your intuition about the need for data augmentation transformations to be consistent with the data generating distribution is correct. For your digits problem example, if your test data has no or very few cropped, blurred and randomly rotated images then data augmentation may not be beneficial and can even degrade performance similar to your noted case of training with ImageNet.
  
  However, in reality you may have very few training and test data availible or your ML model is intended to deploy for production facing real world data which may be very different from your available training and test data with possibly some other usual training challenges such as imbalanced data as discussed in a recent [post](https://ai.stackexchange.com/questions/46373/dealing-with-likely-class-imbalance-for-unlabeled-test-data/). A famous data augmentation example is the [U-Net](https://en.wikipedia.org/wiki/U-Net) CNN developed for biomedical image segmentation, where there're not many patient training samples available for some rare diseases. Based on some domain knowledge some deformed transformations were applied to synthetically augment training data which was required by its deep learning model with huge number of parameters to avoid overfitting.
  
  In summary facing with the usual reality of potential unknown test or production data distribution, data augmentation reduces variance of the model to prevent overfitting by tradeoff some bias and effectively samples from a lower-dimensional manifold of a usually high-dimensional data space consistent with the well known [manifold hypothesis](https://en.wikipedia.org/wiki/Manifold_hypothesis) of ML. From PAC statistical learning perspective, data augmentation can help lower the [VC dimension](https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_dimension) or [Rademacher complexity](https://en.wikipedia.org/wiki/Rademacher_complexity) of the hypothesis class thus improving the [generalization bound](https://en.wikipedia.org/wiki/Empirical_risk_minimization#Properties) ${\displaystyle \mathbb {P} \left(L(\phi \_{n})-L(\phi ^{\*})\right)}$ provable by the [Hoeffding's inequality](https://en.wikipedia.org/wiki/Hoeffding%27s_inequality) in probability theory.
  
  
  ---
  
  ### Answer by username_3 on 2024-08-05. Score: 0
  
  Data augmentation helps resolve overfitting model by introducing more variability to the training data.
  
  If your model does not have overfitting problem, data augmentation may not be needed.
  
  
  ---
  
  