From my experience, this is actually pretty typical behaviour for an LSTM training curve. My intuition for this phenomena is as follows:

the LSTM learns to predict the “average” signal by setting the bias parameters (first plateau)
the LSTM figures out how to use the (forward and recurrent) weight parameters to actually solve the task (second plateau)

This can be interpreted as a saddle point (where the function is much steeper in bias dimensions than in the other dimensions).