Answers.Body: <p>I think the closest approach to what is described in the papers you linked is <a href="https://arxiv.org/pdf/1508.06576.pdf" rel="nofollow noreferrer">Neural Style Transfer</a>. But I see also some (potential) misunderstanding regarding the full batch training, so let me elaborate a bit.</p>
  <p>In the papers you linked and the neural style transfer one a model is trained on a single image. Not a single image at the time, literally on a single image. So I wouldn't call it full batch training, since there's no batch at all, and there can't be one. But I guess this is open to interpretation and it depends on how you define a batch in the first place.</p>
  <p><em>it would be useful for me to come up with a list of applications/models where training a neural network in full batch training is preferable with respect to using multi-batches</em></p>
  <p>Aside from these examples (i.e. overfitting or fine tuning on a single training instance) I doubt you'll find papers where people use full batch training. Not only full batch training comes with all sort of computational issues (like keeping in memory all gradients for an entire dataset) but even from a mathematical point of view mini batch and stochastic gradient descent works better than full batch and (classic) gradient descent. I did link some references about this topic in another <a href="https://ai.stackexchange.com/questions/34532/in-general-is-stochastic-gradient-descent-a-superior-algorithm-compared-to-g/34536#34536">answer</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-10T19:14:54.423Z
Answers.Text: Thank you for your answer and reference! It looks interesting. A little comment: while it is true that the COIN paper trains on a single image, it is encoded as W x H different datapoints, one per pixel. This is why I called it full batch. More generally, yes, I'm interested in all of these very edge cases used in practice.
Answers.UserDisplayName: Alfred
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-10T14:41:23.633Z
Answers.LastActivityDate: 2022-10-10T14:41:23.633Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that it is possible to train the network on one datapoint at the time. It is quite rare, on the other hand, to see research papers, or state of the art models, that are trained on a whole dataset simultaneously, i.e., in full-batch training.</p>
  <p>For a research project, it would be useful for me to come up with a list of applications/models where training a neural network in full batch training is preferable with respect to using multi-batches. An example I have found is the <a href="https://arxiv.org/pdf/2103.03123.pdf" rel="nofollow noreferrer">COIN</a> and <a href="https://arxiv.org/pdf/2201.12904.pdf" rel="nofollow noreferrer">COIN++</a> papers, that train on full batch training in order to memorise datapoints.</p>
  <p><strong>Question:</strong></p>
  <p>Do you know other applications where performing full-batch training is preferable to use mini-batches? Which ones?</p>
  
Tags: <deep-neural-networks><batch-size>
Title: Training a neural network in full batch training
UserDisplayName: Alfred
ThreadText: # Training a neural network in full batch training
  
  (Asked by: username_0 on 2022-10-09)
  
  It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that it is possible to train the network on one datapoint at the time. It is quite rare, on the other hand, to see research papers, or state of the art models, that are trained on a whole dataset simultaneously, i.e., in full-batch training.
  
  For a research project, it would be useful for me to come up with a list of applications/models where training a neural network in full batch training is preferable with respect to using multi-batches. An example I have found is the [COIN](https://arxiv.org/pdf/2103.03123.pdf) and [COIN++](https://arxiv.org/pdf/2201.12904.pdf) papers, that train on full batch training in order to memorise datapoints.
  
  **Question:**
  
  Do you know other applications where performing full-batch training is preferable to use mini-batches? Which ones?
  
  **Tags:** deep-neural-networks, batch-size
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-10. Score: 1
  
  I think the closest approach to what is described in the papers you linked is [Neural Style Transfer](https://arxiv.org/pdf/1508.06576.pdf). But I see also some (potential) misunderstanding regarding the full batch training, so let me elaborate a bit.
  
  In the papers you linked and the neural style transfer one a model is trained on a single image. Not a single image at the time, literally on a single image. So I wouldn't call it full batch training, since there's no batch at all, and there can't be one. But I guess this is open to interpretation and it depends on how you define a batch in the first place.
  
  *it would be useful for me to come up with a list of applications/models where training a neural network in full batch training is preferable with respect to using multi-batches*
  
  Aside from these examples (i.e. overfitting or fine tuning on a single training instance) I doubt you'll find papers where people use full batch training. Not only full batch training comes with all sort of computational issues (like keeping in memory all gradients for an entire dataset) but even from a mathematical point of view mini batch and stochastic gradient descent works better than full batch and (classic) gradient descent. I did link some references about this topic in another [answer](https://ai.stackexchange.com/questions/34532/in-general-is-stochastic-gradient-descent-a-superior-algorithm-compared-to-g/34536#34536).
  
  > **Comments:**
  > **username_0**: Thank you for your answer and reference! It looks interesting. A little comment: while it is true that the COIN paper trains on a single image, it is encoded as W x H different datapoints, one per pixel. This is why I called it full batch. More generally, yes, I'm interested in all of these very edge cases used in practice.  
  
  ---
  
  