ature, it is known that neural networks outperform other classical learning methods, including the
corresponding (ﬁnite-width) neural tangent kernels (Allen-Zhu et al., 2019; Li & Liang, 2018) when
the underlying concept class has certain low-rank structure (Ghorbani et al., 2020; Allen-Zhu & Li,
2019; Allen-Zhu & Li, 2020a). Another theoretical result in Allen-Zhu & Li (2020b) suggests that
low-rank adaptations can be useful for adversarial training. In sum, we believe that our proposed
low-rank adaptation update is well-motivated by the literature.
LORA