Solved by ptrblck in post #4 
              
              
                Your script might be already hitting OOM issues and would call empty_cache internally. 
You can check it via torch.cuda.memory_stats(). 
If you see that OOMs were detected, lower the batch size as suggested.