nn.Parameter and register_parameter should be equivalent. Turns out that I should use Parameter instead of Variable in above code.
I set affine to false because b, beta and gamma will be all trainable. So the formulation is equivalent but with less params except for gradient values.