there are some patches that went into master branch that reduce memory usage of nn.LSTM, etc.
Can you try that?

  
      
      GitHub
  
  
    

pytorch/pytorch 39

pytorch - Tensors and Dynamic neural networks in Python  with strong GPU acceleration