I got the following warning message when I use LSTM with nn.DataParallel.
RuntimeWarning: RNN module weights are not part of single contiguous chunk of memory.
This means they need to be compacted at every call, possibly greatly increasing memory usage.
To compact weights again call flatten_parameters().

I found the error is gone when I put self.lstm.flatten_parameters() at the top of forward function, but I wonder why do we need it.
Why is the weight of RNN non-contiguous on memory when we use nn.DataParallel?
And also I found the error would be gone if we replace DataParallel with DistributedDataParallel, then why isn’t the weight non-contiguous in latter case?

  
      github.com/pytorch/pytorch
  
  
    



Issue: Multi-GPU autograd error with Pytorch 0.4 24


	
	opened by erogol
	on 2018-04-30
	
	
	


After updating pytorch 0.4 I am getting the following error when I try to train my model here: https://github.com/mozilla/TTS with multi-gpus....


 	todo


  
  
    
    
  
  


I found some similar questions but none of them had the answer.

  
    
    
    What does flatten_parameters() do? 
  
  
    I saw many Pytorch examples using flatten_parameters in the forward function of the RNN 
self.rnn.flatten_parameters()

I saw this RNNBase and it is written that it 

Resets parameter data pointer so that they can use faster code paths 

What does that mean?
  



  
    
    
    Do we need to call flatten_parameters() in LSTM only if we are using Multi-GPUs? nlp
  
  
    I am trying to understand the use case for flatten_parameters(), when do we use it and what does it do? Is it used only if we are running our model on multiple GPUs?