On my computer (Windows 10) nvidia-smi does not work correctly (https://stackoverflow.com/questions/44227767/nvidia-smi-does-not-display-memory-usage/44228331 7)
However I can use the task manager to see that the first python process is taking about 7 Gigs.

gpu_usage.PNG1096×249 19 KB

After printing torch.cuda.memory_allocated(device)/1e9 after every .to(device) call in my script, I’m seeing that everything remains under 0.4 until I call my net.forward() at which time I get the error, so I think you are right.
This would imply about a 20x overhead on top of just the weights + net input to reach the 7 gigs reported by task manager. This is surprising that the overhead would be that large.
But back to my original question, I had a thought on why the numbers do not add up. When saying already allocated, the error must only be reporting only the amount allocated by the current process, not the amount of GPU memory allocated period.