Turns out I didn’t know how to assign variables to cuda or cpu and things got all out of whack.  Turns out my packed padded sequence was not assigned to a cuda device.  Here is a simple example I used to test things that helped me figure it out.
import torch 
import torch.nn.utils.rnn as rnn_utils 
a = torch.Tensor([[1], [2], [3]]) 
b = torch.Tensor([[4], [5]]) 
c = torch.Tensor([[6]]) 
d = torch.Tensor([[7],[8],[9],[10]])
batch = [a,b,c,d]
padded= rnn_utils.pad_sequence(batch, batch_first=True)
sorted_batch_lengths = [len(x) for x in padded]
packed = rnn_utils.pack_padded_sequence(padded, sorted_batch_lengths, batch_first=True, enforce_sorted=False).cuda()
lstm = torch.nn.LSTM(input_size=1, hidden_size=3, batch_first=True).cuda()
lstm(packed)

Hopefully this will help someone else out in the future so you don’t spend months stuck.