Even for nn.BatchNorm2d grad_check gives me error which makes me confused:
RuntimeError: Jacobian mismatch for output 0 with respect to input 0,
numerical:tensor([[ 1.1027, -0.0149, -0.0242,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  1.1325, -0.0233,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0000, -0.0298,  1.1139,  ...,  0.0000,  0.0000,  0.0000],
        ...,
        [ 0.0000,  0.0000,  0.0000,  ...,  1.0431, -0.0596, -0.0298],
        [ 0.0000,  0.0000,  0.0000,  ..., -0.0298,  0.9537, -0.0298],
        [ 0.0000,  0.0000,  0.0000,  ..., -0.0298, -0.0596,  1.0431]])
analytical:tensor([[ 1.1032e+00, -1.8262e-02, -2.3532e-02,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [-1.8262e-02,  1.1137e+00, -2.3129e-02,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [-2.3532e-02, -2.3129e-02,  1.1157e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        ...,
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  1.0088e+00,
          4.6197e-05, -2.8593e-02],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  4.6197e-05,
          1.0017e+00, -1.2919e-02],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -2.8593e-02,
         -1.2919e-02,  1.0241e+00]])