Solved by Dan_Erez in post #10 
              
              
                So after some more debugging I found that if I switch to pytorch cpu the ram stays stable. 
So it looks like it was a pytorch related bug after all. None the less I loved your keep it simple stupid debugging methods. I think i’ll hang them up on the wall. 
Thanks, 
Dan