Question: I am about to train a big LSTM network with 2-3 million articles and am struggling with Memory Errors (I use AWS EC2 g2x2large).

I found out that one solution is to reduce the `batch_size`. However, I am not sure if this parameter is only related to memory efficiency issues or if it will effect my results. As a matter of fact, I also noticed that `batch_size` used in examples is usually as a power of two, which I don't understand either.

I don't mind if my network takes longer to train, but I would like to know if reducing the `batch_size` will decrease the quality of my predictions.

Thanks.

Answer: 
I feel the accepted answer is possibly wrong. There are variants in *Gradient Descent Algorithms*.

1. **Vanilla Gradient Descent**: Here the Gradient is being calculated on all the data points at a single shot and the average is taken. Hence we have a smoother version of the gradient takes longer time to learn.
2. **Stochastic Gradient Descent** : Here one-data point at a time hence the gradient is aggressive (noisy gradients) hence there is going to be lot of oscillations ( we use Momentum parameters - e.g Nesterov to control this). So there is a chance that your oscillations can make the algorithm not reach a local minimum.(diverge).
3. **Mini-Batch Gradient Descent**: Which takes the perks of both the previous ones averages gradients of a small batch. Hence not too aggressive like SGD and allows Online Learning which Vanilla GD never allowed.

The smaller the Mini-Batch the better would be the performance of your model (not always) and of course it has got to do with your epochs too faster learning. If you are training on large dataset you want faster convergence with good performance hence we pick Batch-GD's. 

SGD had fixed learning parameter hence we start other Adaptive Optimizers like Adam, AdaDelta, RMS Prop etc which changes the learning parameter based on the history of Gradients.
Oddly enough, I found that larger batch sizes with keras require more epochs to converge.

For example, the output of [this script](https://github.com/shadiakiki1986/test-ml/blob/master/t4-LSTM%20stateful/test_temporal_regression.py) based on keras' [integration test](https://github.com/fchollet/keras/blob/master/tests/integration_tests/test_temporal_data_tasks.py#L78) is

```
epochs 15   , batch size 16   , layer type Dense: final loss 0.56, seconds 1.46
epochs 15   , batch size 160  , layer type Dense: final loss 1.27, seconds 0.30
epochs 150  , batch size 160  , layer type Dense: final loss 0.55, seconds 1.74

```

Related

* Keras [issue 4708](https://github.com/fchollet/keras/issues/4708): the user turned out to be using `BatchNormalization`, which affected the results.
* [This tutorial](https://machinelearningmastery.com/tune-lstm-hyperparameters-keras-time-series-forecasting/) on LSTM, section **Tuning the Batch Size**
* search results for [keras lstm batch size effect on result](https://duckduckgo.com/?q=keras+lstm+batch+size+effect+on+result&t=canonical&ia=web)
* [My Neural Network isn't working! What should I do?](http://theorangeduck.com/page/neural-network-not-working?imm_mid=0f6562&cmp=em-data-na-na-newsltr_20170920), point 5 (You Used a too Large Batch Size) discusses exactly this

> 
> Using too large a batch size can have a negative effect on the
>  accuracy of your network during training since it reduces the
>  stochasticity of the gradient descent.
> 
> 
> 

Edit: most of the times, increasing `batch_size` is desired to speed up computation, but there are other simpler ways to do this, like using data types of a smaller footprint via the `dtype` argument, whether in *keras* or *tensorflow*, e.g. `float32` instead of `float64`