Thank you for your responses @tom and @AjayTalati! Currently, I am starting with making a LSTM (and later a GRU) baseline for sequential and permutated MNIST. The current problem is that the loss refuses to go down and the accuracy remains fixed always at 11.35%. My code can be found here: https://gist.github.com/c0nn3r/f1d6a11d18ba700cf6e0fcc446f84377 142. If anyone would be willing to take a look, I would really appreciate it.
Best,
Conner