Hi, I’ve tried to train a CNN after merged the BatchNorm layers into Convolution layers. After merge bn, the BactchNorm layer is remained and the parameters are reset to make the output same as the input, the code is as below:
           child.eval()
           child.running_mean = child.running_mean.new_full(
               child.running_mean.shape, 0)
           child.running_var = child.running_var.new_full(
               child.running_var.shape, 1)
           if child.weight is not None:
               child.weight.data = child.weight.data.new_full(
                   child.weight.shape, 1)
           if child.bias is not None:
               child.bias.data = child.bias.data.new_full(child.bias.shape, 0)
           child.track_running_stats = False
           child.momentum = 0
           child.reset_running_stats()
           child.eps = 0

“child” is referred to a BatchNorm layer. But unfortunately, the BatchNorm layer still have output different from the input after the reset. It’s weird.
The torch version is 1.6.0, CUDA10.2, run in the docker.