I couldn’t really find anything wrong. If it weren’t for your tf results, I wouldn’t be surprised if this architecture can’t overfit this model. Basically you are asking for 10 log_2 100 bits entropy to be encoded in a hidden space of 50-dimensional hypercube with only a 1 layer rnn. (I don’t think the yin helps much as it is always one step behind). I’d expect it to not overfit.
That said, if tf can do it, pytorch should also be able to do it.
Sorry for not being helpful.