How do you check that. I don’t seem to see that when running on CPU:
In [1]: import torch

In [2]: a = torch.rand(2, 3, 50, 50)

In [3]: b = torch.nn.BatchNorm2d(3)

In [4]: scaling = torch.rand(1, 3, 1, 1)

In [5]: %timeit b(a)
115 µs ± 652 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [6]: %timeit a * scaling
5.15 µs ± 661 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)