Your script might be already hitting OOM issues and would call empty_cache internally.
You can check it via torch.cuda.memory_stats() 40.
If you see that OOMs were detected, lower the batch size as suggested.