Answers.Body: <p>After more research, I found that a batch size of 1 is quite common in deep-learning image processing use-cases where there are high memory/GPU requirements for model training. In fact, it gives better results at times.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-19T16:09:28.800Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-19T10:47:59.620Z
Answers.LastActivityDate: 2022-07-19T10:47:59.620Z
Answers.UserDisplayName: A_C
Body: <p>I am training a very deep neural network (Panoptic-DeepLab) with a ResNet34 backbone on Google Colab on CityScapes dataset for Panoptic Segmentation, and noticed that, with a big crop size, the batch size has to be decreased to 1 image per batch, otherwise CUDA out of memory issues start to occur. While I know that this can create skewness in the training and it will likely be very hard to attain good convergence, can I ask this question in general to the experts: how valid is a batch size of 1 generally considered in image-based processing? The images in consideration can be considered large (high resolution). The optimizer used is Adam alongwith a warm up polynomial learning rate (with base around 0.00005), and 90k iterations.</p>
  <p>(I understand that it would possibly be a good idea to try out a smaller crop size and bigger batch size, but would like to know the feedback from the community anyway)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-01T08:12:51.810Z
Comments.Text: You can do gradient accumulation to artificially use a higher batch size even if only one image can fit in the memory.
Comments.UserDisplayName: Lelouch
Tags: <deep-learning><training><image-processing><image-segmentation><memory>
Title: Is batch size of 1 a valid choice for a very deep neural network with high memory requirement?
UserDisplayName: A_C
ThreadText: # Is batch size of 1 a valid choice for a very deep neural network with high memory requirement?
  
  (Asked by: username_0 on 2022-07-10)
  
  I am training a very deep neural network (Panoptic-DeepLab) with a ResNet34 backbone on Google Colab on CityScapes dataset for Panoptic Segmentation, and noticed that, with a big crop size, the batch size has to be decreased to 1 image per batch, otherwise CUDA out of memory issues start to occur. While I know that this can create skewness in the training and it will likely be very hard to attain good convergence, can I ask this question in general to the experts: how valid is a batch size of 1 generally considered in image-based processing? The images in consideration can be considered large (high resolution). The optimizer used is Adam alongwith a warm up polynomial learning rate (with base around 0.00005), and 90k iterations.
  
  (I understand that it would possibly be a good idea to try out a smaller crop size and bigger batch size, but would like to know the feedback from the community anyway)
  
  **Tags:** deep-learning, training, image-processing, image-segmentation, memory
  > **Comments:**
  > **username_1**: You can do gradient accumulation to artificially use a higher batch size even if only one image can fit in the memory.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-07-19. Score: 1 | Accepted answer
  
  After more research, I found that a batch size of 1 is quite common in deep-learning image processing use-cases where there are high memory/GPU requirements for model training. In fact, it gives better results at times.
  
  > **Comments:**
  > **username_2**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  