I’m trying to train two nets simultaneously on my machine. When starting the second training script, I got this strange error
RuntimeError: CUDA out of memory. Tried to allocate 136.00 MiB (GPU 0; 12.00 GiB total capacity; 4.51 GiB already allocated; 130.87 MiB free; 164.77 MiB cached)
If I have 12.00 GiB total, and 4.51 GiB allocated, I should have 7.49 GiB free. Then why does allocating merely 136 MiB exhaust the memory? 136 MiB < 7.49 GiB
Also the size of my model checkpoint is less thank 200 MB. What’s a good way to figure out why I’m using up this memory?