This task is hard for RNNs, correct rule would actually only need one step lookback to distinguish half-cycles and otherwise would just memorize step differentials by last position in post-RNN layers.
But that rule is elusive, especially with an oversized network and unlimited lookback. So, the L-BFGS optimizer manages to find a better solution, by being more thorough and not using mini-batches. Training with usual optimizers is more fragile, easily picking step transition rules that don’t generalize to >1000 events (nb: LSTM’s update&forget gates don’t sum to 1, so “amount of information” is not auto-balanced, unlike GRU)
Switching to an optimizer without momentum, tuning it and adding lr scheduler may help, but perhaps it would be better to start from some other example instead.