Thank you for the answer !
Error seems to be in the 1e-6 range right after the RNN layers indeed.
Are you aware of any way to attenuate this behaviour ? In my actual code, after a few layers, this error is more in the 1e-4 range, which causes non reproducibility problems during inference