Batch norm uses some internal buffers to keep track of running mean and variance per channel. I am not sure why it could be happening. One thing might be to instead of using batch norm layer before and after just use only conv layer. Just a guess