its because of fragmentation, if you’re using like 90% device memory, it will fail to find big contiguous free blocks.
you can try to explicitly do python’s garbage collection and torch.cuda.empty_cache(), but this only helps in some cases. another thing is to try to avoid allocating tensors of varying sizes (e.g. varying batch sizes).