I mean, if the model is trained on synthesized data with multiple steps / multiple epochs, and we just backprop like this forward and backward example, then the gradient computation would be incorrect. Because after multiple steps / multiple epochs, the buffer value (running mean/var) is updated. Since the buffer value is not in the computation graph, the updated buffer value is used during backprop instead of real buffer value, especially when computing gradients of starting steps/epochs.
Also, I do hope they can be updated, otherwise the bn layer is not fully functioning.