Basic math related to computation and memory usage for transformers
Nice article, though I feel something went amiss with this part:$$ \begin{align}\text{Total Memory}{\text{Training}} = \text{memory}{\text{model}}+\text{memory}{\text{optimizer}}+\text{memory}{\text{activations}}+\text{memory}_{\text{gradients}}\end{align} $$