Thank you for your help. I’ve spend some more time with my models and what I’ve found out is that nn.BatchNorm1d (PyTorch) works differently than layers.BatchNormalization (Keras). There are some discussions about that on the internet but not conclusive. I can’t work out how to make them behave the same.
In the end I’ve replaced nn.BatchNorm1d with nn.LayerNorm and the agent learns flawlessly.
I would really like to know what’s wrong with nn.BatchNorm1d in PyTorch. Until I find out I’ve got to avoid it.