Answers.Body: <p>It would be better to run experiments on this. But, here's answer analytically.</p>
  <p>The models will be different.</p>
  <p>With augmentation, the network starts to learn to combat the noise too.</p>
  <p>With late onset augmentation, the network will start to deviate from its original solution to combat noise.</p>
  <p>Comparing this to the ball rolling down the hill, in case of augmentation, the ball will roll but will face a bit of friction (the noise due to augmentation). On the other hand, with late onset augmentation, it would be like the ball reached the bottom of hill, and now it has to face an uphill battle to model the noise. The ball will need to go a bit up the hill against gravity and friction due to noise.</p>
  <p>I don't think it would be a good thing to do. But, running experiments is the only way we can confirm.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-17T04:55:28.817Z
Answers.LastActivityDate: 2021-02-17T04:55:28.817Z
Answers.UserDisplayName: Abhishek Verma
Answers.Body: <p>There is no guarantee that the two approaches will converge to the same minimum loss or the same set of model weights.  All  of that depends on how the training is performed.  The order in which training images are presented is a part of the training process, as is the optimizer that is used.</p>
  <p>However, I do believe that the loss function landscape (and global minimum) should be the same (or at least similar) in both scenarios.  The loss landscape is dependent on the model architecture and the dataset, which are the same in both scenarios.  What is different is how the training images are presented, but I don't think that effects the actual shape of the loss landscape as much (just how we move over that landscape).  This, of course, will dictate the final loss and state of the model weights, which may or may not be the same.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-10T23:00:29.137Z
Answers.LastActivityDate: 2022-11-10T23:06:24.667Z
Answers.LastEditDate: 2022-11-10T23:06:24.667Z
Answers.UserDisplayName: Snehal Patel
Body: <p>If I train a U-Net model for image segmentation (e.g. medical images) and start training until it converges and then add augmentation - can i expect similar results as if i train with augmentation from the beginning ?</p>
  <p><a href="https://i.sstatic.net/ITA5H.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ITA5H.png" alt="enter image description here" /></a></p>
  
Tags: <u-net><generalization><data-augmentation>
Title: Late Onset Augmentation
UserDisplayName: Samuel Peterson
ThreadText: # Late Onset Augmentation
  
  (Asked by: username_0 on 2021-02-16)
  
  If I train a U-Net model for image segmentation (e.g. medical images) and start training until it converges and then add augmentation - can i expect similar results as if i train with augmentation from the beginning ?
  
  [![enter image description here](https://i.sstatic.net/ITA5H.png)](https://i.sstatic.net/ITA5H.png)
  
  **Tags:** u-net, generalization, data-augmentation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-02-17. Score: 0
  
  It would be better to run experiments on this. But, here's answer analytically.
  
  The models will be different.
  
  With augmentation, the network starts to learn to combat the noise too.
  
  With late onset augmentation, the network will start to deviate from its original solution to combat noise.
  
  Comparing this to the ball rolling down the hill, in case of augmentation, the ball will roll but will face a bit of friction (the noise due to augmentation). On the other hand, with late onset augmentation, it would be like the ball reached the bottom of hill, and now it has to face an uphill battle to model the noise. The ball will need to go a bit up the hill against gravity and friction due to noise.
  
  I don't think it would be a good thing to do. But, running experiments is the only way we can confirm.
  
  
  ---
  
  ### Answer by username_2 on 2022-11-10. Score: 0
  
  There is no guarantee that the two approaches will converge to the same minimum loss or the same set of model weights. All of that depends on how the training is performed. The order in which training images are presented is a part of the training process, as is the optimizer that is used.
  
  However, I do believe that the loss function landscape (and global minimum) should be the same (or at least similar) in both scenarios. The loss landscape is dependent on the model architecture and the dataset, which are the same in both scenarios. What is different is how the training images are presented, but I don't think that effects the actual shape of the loss landscape as much (just how we move over that landscape). This, of course, will dictate the final loss and state of the model weights, which may or may not be the same.
  
  
  ---
  
  