Solved by Fisher in post #2 
              
              
                Ok, I found a solution that works for me: 
On startup I measure the free memory on the GPU, take 80% of that, create a variable this big and put it on GPU. Directly after doing that, I override it with a small value. 
While the process is running, the GPU has still 80% memory blocked and pytorch is …