I’m not sure if others would find these useful, even if only as a sort for beginner tutorial/helper, but I’ve written two quick and simple (as-is) functions to convert to and from a ModuleList of LSTMCells and an LSTM object.
There’s a few assumptions being made here (namely that the state dictionaries will always keep the same names).
def convert_lstm_to_lstm_cells(lstm):
	lstm_cells = nn.ModuleList([nn.LSTMCell( lstm.input_size , lstm.hidden_size )] +
							   ([nn.LSTMCell( lstm.hidden_size , lstm.hidden_size)] * (lstm.num_layers - 1)))
	
	key_names = lstm_cells[0].state_dict().keys()
	source = lstm.state_dict()
	for i in range( lstm.num_layers ):
		new_dict = OrderedDict( [(k, source["%s_l%d" % (k, i)]) for k in key_names] )
		lstm_cells[i].load_state_dict( new_dict )
	
	return lstm_cells

def convert_lstm_cells_to_lstm(lstm_cells):
	lstm = nn.LSTM( lstm_cells[0].input_size , lstm_cells[0].hidden_size, len(lstm_cells) )
	
	key_names = lstm_cells[0].state_dict().keys()
	lstm_dict = OrderedDict()
	for i, lstm_cell in enumerate(lstm_cells):
		source = lstm_cell.state_dict()
		new_dict = OrderedDict( [("%s_l%d" % (k, i), source[k]) for k in key_names] )
		lstm_dict = OrderedDict(list(lstm_dict.items()) + list(new_dict.items()))
	lstm.load_state_dict( lstm_dict )
	
	return lstm

The need for such a conversion came about because of issues like the ones in the following posts:

  
    
    
    Minimal code snippet that seems to cause a memory leak on GPU code only 
  
  
    I ended up changing my architecture specifically to work around this issue. 
For inference mode, I simply use volatile=True and this solves everything. 
Otherwise, you can move your model using .cpu() and .cuda() calls. 
E.g. if seq is a class module, then you can do: 
seq.cpu()
output = seq(input)
seq.cuda()

Furthermore, you can call pin_memory() on the input tensors to facilitate transfers back and forth. 
It’s very hacky, and in the end, I found an alternate way of training my model.
  



  
    
    
    Using PackedSequence with LSTMCell 
  
  
    Thanks for the run-down James. 
I’m actually developing a solution that runs through streaming data, and so originally I had designed my module with LSTMCell's. But as you mention, I can see now that the advantages of LSTM layers far outweigh the drawbacks. 
Also, since my real time stream is much slower than my processing power, there’s really no harm in simply passing data columns one by one to the LSTM and caching the final hidden/cell states for my next call. This is a small price to pay for…
  


Criticisms and comments are welcome.