Same issues here. Trying to use DataParallel with a LSTM model.
RuntimeError: Expected hidden[0] size (1, 2500, 50), got (1, 10000, 50)
I can see from the shape mismatch what the general problem is. The hidden is being created for the entire model input (10000 in my case) where dataparallel is dividing that input by GPU count (4 in my case) to spread the load. Maybe we can also wrap the hidden input tensor with dataparallel so its also distributed correctly?