A bit of a late reply, but:
this kind of normalization made me struggle for quite some time. Using it, my DQN applied on cartpole could not learn properly. So you definitely need to be very careful.
I will from now on either not normalize at all or freeze the update of the normalization parameters after some initial steps.