Group and Shuffle: Efficient Structured Orthogonal Parametrization
<|reference_start|>Group and Shuffle: Efficient Structured Orthogonal Parametrization: The increasing size of neural networks has led to a growing demand for methods of efficient fine-tuning. Recently, an orthogonal fine-tuning paradigm was introduced that uses orthogonal matrices for adapting the weights of a pretrained model. In this paper, we introduce a new class of structured matrices, which unifies and generalizes structured classes from previous works. We examine properties of this class and build a structured orthogonal parametrization upon it. We then use this parametrization to modify the orthogonal fine-tuning framework, improving parameter and computational efficiency. We empirically validate our method on different domains, including adapting of text-to-image diffusion models and downstream task fine-tuning in language modeling. Additionally, we adapt our construction for orthogonal convolutions and conduct experiments with 1-Lipschitz neural networks.<|reference_end|>
@article{gorbunov2024group,
    title={Group and Shuffle: Efficient Structured Orthogonal Parametrization},
    author={Mikhail Gorbunov, Nikolay Yudin, Vera Soboleva, Aibek Alanov, Alexey
  Naumov, Maxim Rakhuba},
    journal={arXiv preprint arXiv:2406.10019},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.10019},
    primaryClass={cs.LG cs.AI cs.CL cs.CV cs.NA math.NA}
}