Body: <p>I have a dataset with an input size of 155x155, with the output being 155 x 1 with a 3-4 layer neural net being used for regression. With such a small sample size, should I use full batch gradient descent (so all 155 samples) or use mini batch/stochastic gradient descent. I have read that using smaller mini batch sizes allows better generalisation, but as the batch size is very very small computationally it shouldn't be a burden to use BGD.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-19T15:15:52.363Z
Comments.Text: Input size of 155x155 is rather ambiguous i feel, it could mean a input with dimension 155x155 and not the size of the training set. If the number of samples is small as in your case, I believe that a BGD should converge faster than a SGD. However, in your case where you have so few samples, the time to converge shouldn't be that much significant. FYI: Some Mini batch sizes range up to 256 samples
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-19T15:17:10.577Z
Comments.Text: Thanks for your answer! To clarify, the training set size is 155x155 not the dimension.
Comments.UserDisplayName: Max
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-19T15:18:25.397Z
Comments.Text: You mean (n_samples, n_features) = (155, 155) right?
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-19T15:24:21.390Z
Comments.Text: Yes, exactly right!
Comments.UserDisplayName: Max
Tags: <neural-networks><gradient-descent>
Title: Should I use batch gradient descent when I have a small sample size?
UserDisplayName: Max
ThreadText: # Should I use batch gradient descent when I have a small sample size?
  
  (Asked by: username_0 on 2021-02-19)
  
  I have a dataset with an input size of 155x155, with the output being 155 x 1 with a 3-4 layer neural net being used for regression. With such a small sample size, should I use full batch gradient descent (so all 155 samples) or use mini batch/stochastic gradient descent. I have read that using smaller mini batch sizes allows better generalisation, but as the batch size is very very small computationally it shouldn't be a burden to use BGD.
  
  **Tags:** neural-networks, gradient-descent
  
  > **Comments:**
  > **username_1**: Input size of 155x155 is rather ambiguous i feel, it could mean a input with dimension 155x155 and not the size of the training set. If the number of samples is small as in your case, I believe that a BGD should converge faster than a SGD. However, in your case where you have so few samples, the time to converge shouldn't be that much significant. FYI: Some Mini batch sizes range up to 256 samples  
  > **username_0**: Thanks for your answer! To clarify, the training set size is 155x155 not the dimension.  
  > **username_1**: You mean (n_samples, n_features) = (155, 155) right?  
  > **username_0**: Yes, exactly right!  
  
  ---
  
  ## 0 answers
  
  