pytorch_noob:

2.76 GiB already allocated;


This is the issue - it seems there is some other process holding on to the gpu memory. Sometimes, when I force kill torch process on the server, I get this error. Try restarting the system once.