I tried to remove the embeddings and put them on cpu. Now I have the same error on LSTM, it seems to me that nn.DataParallel moves things wrongly from one gpu to another
RuntimeError: Input and parameter tensors are not at the same device, found input tensor at cuda:0 and parameter tensor at cuda:1