Thanks a ton. I am sure this took effort - fiddling with parameters, activations.
It’s hard for me to understand why we aren’t learning something very useful here! I calculated the MSE loss between (input, output) and between (pred, output). The MSE is lesser for (input, output)! Which means we’re aren’t denoising well enough, infact- not modifying the input would have resulted in lower MSE.
Would increasing the number of training samples help? Or, should I make the problem tougher to ensure that the network can learn a better function than identity?
Should we really have two RNNs as is usually done in Seq2seq tasks - one being an encoder, the other being a decoder?
I also feel that this trivial example would work substantially better if we could incorporate the notion of attention.