Basic math related to computation and memory usage for transformers
Nice article, though I feel something went amiss with this part:<p>$$ \begin{align<i>}\text{Total Memory}{\text{Training}} = \text{memory}{\text{model}}+\text{memory}{\text{optimizer}}+\text{memory}{\text{activations}}+\text{memory}_{\text{gradients}}\end{align</i>} $$