My network is a seq2seq net with three inputs:

Word embeddings for the input sequence
Word embeddings for the output sequence
Image-like inputs for each token in the output sequence

When I try to keep any of these three on CPU before the forward pass with DataParallel it complains that the tensors aren’t cuda tensors.
E.g., here’s a snippet of my input encoder:
# Tensor containing sequence word type indices
torch_indices: torch.Tensor = torch.zeros((len(examples), max(seq_lens)), dtype=torch.long)
for idx, (sequence, sequence_length) in enumerate(zip(batch_indices, seq_lens)):
    torch_indices[idx, :sequence_length] = torch.tensor(sequence, dtype=torch.long)

# Now embed things
batch_embeddings: torch.Tensor = self._embedder(torch_indices.to(DEVICE))

len(examples) gets my my batch size, max(seq_lens) gets me the maximum sequence length in the batch, and I iterate over indexed sequences (batch_indices) and modify values of the indices tensor indicating the indices of word types. I then put it on the device (in the case of single-GPU, this will be DEVICE=torch.device('cuda'); in the case of CPU, this will be DEVICE=torch.device('cpu'); and when I have more than one GPU this is by default also DEVICE=torch.device('cpu'). Perhaps I shouldn’t use to (and explicitly place on CPU) at all when using DataParallel?