Hi, bit of a late response, but can’t help myself 
From what I have been reading lately, your hypothesis is correct, policy gradients generally get hurt by batchnorm.  As I understand it, figuring out a way to normalize deep policy networks under batchnorm is an open research problem.
If you look at a lot of the RL research, a lot of it seems to be about normalizing the rewards and value functions so that they don’t cause overly large changes to the policy or value function, so it kinda makes sense that adding on normalization that was designed mostly for supervised learning could cause issues.