The network works now after changing the input to decoder_test with .cuda() 
encoder_test = EncoderRNN(10, 10, 2) # I, H , L
decoder_test = AttnDecoderRNN('general', 10, 10, 2) # A, H, O, L


if USE_CUDA:
    encoder_hidden = encoder_test.init_hidden().cuda()
    word_inputs = Variable(torch.LongTensor([1, 2, 3]).cuda())
else:
    encoder_hidden = encoder_test.init_hidden()
    word_inputs = Variable(torch.LongTensor([1, 2, 3]))
encoder_outputs, encoder_hidden = encoder_test(word_inputs, encoder_hidden)
decoder_attns = torch.zeros(1, 3, 3)
decoder_hidden = encoder_hidden

if USE_CUDA:
    decoder_context = Variable(torch.zeros(1, decoder_test.hidden_size)).cuda()
else:
    decoder_context = Variable(torch.zeros(1, decoder_test.hidden_size))

decoder_output, decoder_context, decoder_hidden, decoder_attn = decoder_test(word_inputs[0], decoder_context, decoder_hidden, encoder_outputs)
print(decoder_output)
print(decoder_hidden)
print(decoder_attn)

But it’s still good to know why the TypeError appears when I did is_cuda() on pyTorch Variable objects.