You might be hitting out of memory errors, which should automatically try to clear the memory cache and retry the last memory allocation.
torch.cuda.memory_summary() should give you information, if you were running into OOM issues.