You could try to get matching results using a few layers (or blocks) and then scaling it up.
E.g. I couldn’t find a lot about how the stacked LSTM layers work, i.e. how is their hidden and state tensor defined, so this would be my starting point.