In your training code, do you wrap everything that does not need to be included in the computation graph with with torch.no_grad()? I experienced a similar issue and it was because my evaluation code was being included in the comp. graph .