Why does PyTorch uses three different kernels for backward (four when elementwise affine is True) for LayerNorm backward. NVIDIA Apex seems to use only a single kernel or two when elementwise affine is True. Are there some edge cases Apex does not deal with and PyTorch does ?.
Also how is the scale and bias here (pytorch/layer_norm_kernel.cu at master · pytorch/pytorch · GitHub 2) different from gamma and beta. Isn’t gamma and beta the scale and bias parameters for layernorm?.