I tried with loss =0 at the beginning of every loop. I am accumulating loss by adding minibatch loss to the running_loss Variable. My current code is not able to handle large batch size (40 here). It works fine upto a batch size of 20. My model is 3 LSTM lsyers with 1024 units and 1945 output units and 1800 input units. My system config is Nvidia- 1080Ti GPU with 11 GB GPU memory and 32 RAM. I think model is not too big to handle batch size of 40. There must be something sub-optimal in the code