ixaxaar:

Its weird given there are other training sessions running on the two GPUs, which are not OOM-ing


These other sessions likely have cached blocks (we use a caching memory allocator).
When you initially do a CUDA call, it’ll create a cuda context and a THC context on the primary GPU (GPU0), and for that i think it needs 200 MB or so. That’s right at the edge of how much memory you have left.