If you are running out of memory on a device, PyTorch will clear the cache and try to reallocate the memory. There is unfortunately not much you can check besides torch.cuda.memory_summary().
You could try to del unnecessary tensors early, so that you might get potentially more memory once you hit the OOM issue.