Hi,
I had similar issue and spent several days to figure out.
In my case, I turned on torch.backend.cudnn.benchmark and it keeps evaluate on each batch which also causing memory fluctuating.

  
    
    
    What does torch.backends.cudnn.benchmark do? 
  
  
    whats the difference when setting it True or False?
  


Hope this will help.