The mean and std are updated in each forward pass. The momentum terms defines how of the new value will be added to (1 - momentum) of the old “running” stats.
From the docs:

This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x̂_new=(1−momentum)×x̂ +momemtum×x_t, where x̂  is the estimated statistic and x_t is the new observed value.

Since this implementation is a moving average, using single samples might be problematic, since the running stats will change a lot.
Usually, BatchNorm yields good results using a batch size of ~ 64.