Hi,
Check this post,

  
    
    
    [SOLVED] Register_parameter vs register_buffer vs nn.Parameter autograd
  
  
    I was looking at the code of batchnorm 
    def __init__(self, num_features, eps=1e-5, momentum=0.1, affine=True,
                 track_running_stats=True):
        super(_BatchNorm, self).__init__()
        self.num_features = num_features
        self.eps = eps
        self.momentum = momentum
        self.affine = affine
        self.track_running_stats = track_running_stats
        if self.affine:
            self.weight = Parameter(torch.Tensor(num_features))
            self.bias = Parame…
  


With respect to batch normalization the best you can do is to read the paper:

  
      
      arXiv.org
  
  
    

Batch Normalization: Accelerating Deep Network Training by Reducing Internal... 12

Training Deep Neural Networks is complicated by the fact that the
distribution of each layer's inputs changes during training, as the parameters
of the previous layers change. This slows down the training by requiring lower
learning rates and careful...