Actually, I think it turned out that the problem was in Keras… I have reported an issue there and they have removed batch normalization from the example in question, because it prevented the model from learning.
Please see this thread: DDPG example uses BatchNormalization incorrectly · Issue #198 · keras-team/keras-io · GitHub.
And about your question, batch norm normalizes differently during training and inference. During training stats of current batch are used, during inference, moving stats of all seen batches are used.
Training:
(batch - mean(batch)) / (var(batch) + epsilon) * gamma + beta
Inference:
(batch - self.moving_mean) / (self.moving_var + epsilon) * gamma + beta
Source: BatchNormalization layer
In my view, PyTorch docs say the same, although maybe in a bit less concise form
..Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation...
Source: BatchNorm1d — PyTorch 1.8.1 documentation 1
In my project I also removed batch norm, and layer norm too. Although layer norm worked well, but pure MLP without any normalization worked too so why complicate. I’ve read somewhere that Layer norm in general behaves better than BatchNorm