Thanks, actually I met the problem mentioned here:

  
    
    
    Out of memory when optimizer.zero_grad() is called 
  
  
    The track back is the same, though.
  


When I tried to clear the grad after the exception, another Exception was raised when calling zero_grad().