Reshaping itself isn’t a problem for backprop, but a break in the chain of torch.Tensor / torch.Function will break the graph into fragments, and so I’m guessing only the decoder half is getting trained.
It’s hard to know what the shape of the tensors are from eyeballing the code, but I’m guessing some application of hs.view(?, ?) will do the trick.