This is the model I am using:
class DecoderRNN(nn.Module):
  def __init__(self, embed_size,vocab_size, hidden_size, num_layers=1):       
      super(DecoderRNN, self).__init__()                                      
      self.embed = nn.Embedding(vocab_size, embed_size)                       
      self.linear = torch.nn.Linear(2048,embed_size)                          
      self.bn = nn.BatchNorm1d(embed_size, momentum=0.01)                     
      self.gru = nn.GRU(embed_size, hidden_size, num_layers, batch_first=True)
                                                                 
  def forward(self, features, captions, lengths):                                         
      pdb.set_trace()                                                         
      features = self.linear(features)                                        
      embeddings = self.embed(captions)                                       
      pdb.set_trace()                                                         
      embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)          
      packed = pack_padded_sequence(embeddings, lengths, batch_first=True)◀   
      hiddens, _ = self.gru(packed)                                           
      outputs = self.linear(hiddens[0])                                       
      return outputs    

However after forward pass through embedding, all tensors on cuda are messed up. The tensors which are not cuda tensors seems to be fine. I get the following error. The following should clarify 
(Pdb) type(features.data)
<class ‘torch.cuda.FloatTensor’>
(Pdb) features.data
2.9174  1.9323  0.8640  …   0.1553  0.9829  0.8675
[torch.cuda.FloatTensor of size 1x2048 (GPU 0)]
(Pdb) aa = self.embed(captions)
(Pdb) aa.data
THCudaCheck FAIL file=/py/conda-bld/pytorch_1493669264383/work/torch/lib/THC/generic/THCTensorCopy.c line=65 error=59 : device-side assert triggered
*** RuntimeError: cuda runtime error (59) : device-side assert triggered at /py/conda-bld/pytorch_1493669264383/work/torch/lib/THC/generic/THCTensorCopy.c:65
(Pdb) aa.data.contiguous()
*** RuntimeError: cuda runtime error (59) : device-side assert triggered at /py/conda-bld/pytorch_1493669264383/work/torch/lib/THC/generic/THCTensorCopy.c:65
(Pdb) features
*** RuntimeError: cuda runtime error (59) : device-side assert triggered at /py/conda-bld/pytorch_1493669264383/work/torch/lib/THC/generic/THCTensorCopy.c:65
(Pdb) lengths
5332
[torch.LongTensor of size 1]