torch.cuda.empty_cache() is called after the tensors were deleted.
Could you try to delete loader in the exception first, then empty the cache and see if you can recreate the loader using DataLoader2?
How did you create your DataLoader? Do you push all data onto the GPU?