Weer you able to find out why this is happening? it is very strange since I call optim.zero_grad() inside the training loop.  Could it be because cuda is not purging the data from dataloader in every loop and getting accumulated?