yotam:

I tried replacing the Transformer with a feedforward net and it works easily.


I’m not sure I understand the use case completely, but were you able to predict the next number in a sequence of random numbers without any signal using a feed-forward model?
If so, I would check for a data leak, since as @mMagmer described, I would expect all models to fail predicting random numbers (unless the PRNG is giving you some information about the sequence).