Usually you initialize the weights close to zero using a random distribution as was done for the conv layers.
The weight and bias in BatchNorm work as the rescaling parameters gamma and beta from the original paper 4.
Since BatchNorm uses the batch statistics (mean and std) to normalize the activations, their values should be close to zero with a stddev of 1.
After the normalization gamma and beta might “rescale” the activations again, i.e. the normalization might be undone. However, if you start with gamma=1 and beta=0, the first batch(es) will be normalized using their stats.