Hm, I’m relatively new to Pytorch so this is just a guess, but the line
hs = tuple([h.squeeze(0) for h in hs])
Looks like backprop probably stops here, as you’re leaving tensors for a tuple, which, unless there’s magic happening that isn’t obvious, doesn’t have a .grad_fn or .grad attribute… So how would the encoder get any gradients to train on?