Geometry and Dynamics of LayerNorm
<|reference_start|>Geometry and Dynamics of LayerNorm: A technical note aiming to offer deeper intuition for the LayerNorm function common in deep neural networks. LayerNorm is defined relative to a distinguished 'neural' basis, but it does more than just normalize the corresponding vector elements. Rather, it implements a composition -- of linear projection, nonlinear scaling, and then affine transformation -- on input activation vectors. We develop both a new mathematical expression and geometric intuition, to make the net effect more transparent. We emphasize that, when LayerNorm acts on an N-dimensional vector space, all outcomes of LayerNorm lie within the intersection of an (N-1)-dimensional hyperplane and the interior of an N-dimensional hyperellipsoid. This intersection is the interior of an (N-1)-dimensional hyperellipsoid, and typical inputs are mapped near its surface. We find the direction and length of the principal axes of this (N-1)-dimensional hyperellipsoid via the eigen-decomposition of a simply constructed matrix.<|reference_end|>
@article{riechers2024geometry,
    title={Geometry and Dynamics of LayerNorm},
    author={Paul M. Riechers},
    journal={arXiv preprint arXiv:2405.04134},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.04134},
    primaryClass={cs.LG}
}