Ok, thanks I have not looked at torch.cuda.memory_allocated.
This is off-topic, but do you by chance know what happens when cublas fails to allocate memory (or whether it uses pytorchs allocator)? I suspect that the caching allocator consumes all the memory and then cublas fails when i use it in optimizer.step() because the memory requirement after backward.step() is indeed very small.