I think that might actually be the problem, but can’t confirm quite yet.
I have a nvidia quadro P2000 (which notoriously has low memory) , using nvidia-smi just gives N/A for the gpu-usage, as I don’t have windows display device model (WDDM), and using the -dmi flag doesnt work as my actual display is linked to the gpu. I have researched trying to find ways to measure gpu usage for code running, but haven’t quite found a way (also gpustat doesnt work, as _curses is not supported on windows for python 3.7). Any suggestions? Or I just look at task-manager…
As an aside, I am starting to dislike DL training on windows…
Using the following code:
print('Allocated:', torch.cuda.memory_allocated(0)/1024.0**3, 'GB')
print('Cached:   ', torch.cuda.memory_cached(0)/1024.0**3, 'GB')

I get, the following after the crash
#before training:
Allocated: 0.00029277801513671875 GB
Cached:    0.001953125 GB

# training, and after failure
Allocated: 0.002089977264404297 GB
Cached:    0.060546875 GB

This is well below the max memory of my GPU