I realized that possibly my comparison was a bit unfair. I was comparing my recurrent neural network with a feedforward neural network, so some of the computational complexity was vastly different (apart from the ode vs. no-dynamics comparison that is a bit unfair). I will compare again to a regular RNN in pytorch (non-LSTM). Am I correct that the nn.RNN is a regular vanilla RNN? In the description it says it is an Elman-RNN, which is not exactly true since it is without context neurons.