Solved by Ron_Katzir in post #6 
              
              
                Referencing corresponding bug: Pytorch autograd sometimes fails with CUDNN_STATUS_INTERNAL_ERROR · Issue #52263 · pytorch/pytorch · GitHub 
As stated should be fixed with torch 1.8 and cudnn 8.1. 
I’m testing it and will update.