I’m coming back to say that I’ve still haven’t figured out what’s wrong although I now have more clues.
Following the seq2seq translation tutorial I’ve removed the batches entirely and iterated both on the inputs and outputs in a for loop as such:
for x_idx in range(max_x_sequence):
       encoder_output, encoder_hidden = encoder(x[x_idx].view(1, 1, -1), encoder_hidden)

decoder_hidden = encoder_hidden
sequence_total_loss = 0

for seq_idx in range(max_y_sequence):
       output, decoder_hidden = decoder(input, decoder_hidden)
       sequence_total_loss += criterion(output, y[seq_idx].view(1, -1))

What I’ve figured out is that the gradients propagated back to the encoder are tiny and I am guessing this is the reason my network never converges.

gf1510×474 200 KB

These are the absolute means of the gradients of each layer in the model and the bottom lines are the means of the gradients reaching the encoder.
I took it one step further and removed the encoder entirely. i.e. I’ve passed the encoder_hidden initial variable to the decoder without getting it though the forward pass of the encoder. This is the result in the training loss:

tl1512×496 46.2 KB

The green line is a run with the encoder present, while the blue line is a run without the encoder. This validated my initial finding on the gradients not modifying the encoder weights at all.
I’ve made many iterations over the days like changing the loss from MSE to RMSE to increase the loss so that the gradients would be bigger, or scaling up the features and/or the y values symmetrically on both train and validation
So now I’m completely stuck. I would be most grateful for any ideas on what could be wrong.