I can verify that the output from batch normalization is not autocasted to float32, unlike that from layer normalization, which is right according to the documentation, only layer_norm is in the “autocast to float32” list.
I still cannot agree that this is the expected behavior.
Pytorch version: 1.6.0
with autocast():
    net = nn.BatchNorm1d(4).cuda()
    x = torch.randn(2, 4).half().cuda()
    y = net(x)
    print(y.dtype)  # torch.float16

with autocast():
    net = nn.LayerNorm(4).cuda()
    x = torch.randn(2, 4).half().cuda()
    y = net(x)
    print(y.dtype)  # torch.float32