that should be fine. I use batch sizes of 128 with 4 layers LSTM and larger model. My guess is you might want to accumulate loss.data to running loss, instead of loss. Maybe you are already doing that?