Solved by ptrblck in post #5 
              
              
                Yes, this should be the case, if you didn’t wrap the code block in a with torch.no_grad() block. 
Which model are you using? Also, how are you checking the memory usage? Could you print it using torch.cuda.memory_allocated() in the loop?