You’re right @ptrblck, I got an OOM error even when using a large negative value instead of gloat("-inf"). The tensor I am manipulating in that code snippet has dimensions 32x60x10x16. I don’t know if that’s big in PyTorch terms.
To answer your question, my training script does not consume all of my GPU memory from the beginning, it slowly increases the memory usage with time. It usually starts with 1 GB and goes up until it reaches 3.5 GB. My GPU has 8 GB of memory.
The thing is, because processing is actually low (training a network usually uses 5~20% of my GPU processing), I was being greedy and training multiple networks at the same time in the same GPU. So it’s not like a single process actually consumed all my GPU memory.
I now implemented a few measures to limit GPU memory usage, such as deleting tensors once they are used (especially the big ones), limiting the scope of some tensors with the use of functions and calling torch.cuda.empty_cache() periodically to free memory to other processes. I am still trying to figure out if these things are making a difference.