FORML: A Riemannian Hessian-free Method for Meta-learning on Stiefel Manifolds
<|reference_start|>FORML: A Riemannian Hessian-free Method for Meta-learning on Stiefel Manifolds: Meta-learning problem is usually formulated as a bi-level optimization in which the task-specific and the meta-parameters are updated in the inner and outer loops of optimization, respectively. However, performing the optimization in the Riemannian space, where the parameters and meta-parameters are located on Riemannian manifolds is computationally intensive. Unlike the Euclidean methods, the Riemannian backpropagation needs computing the second-order derivatives that include backward computations through the Riemannian operators such as retraction and orthogonal projection. This paper introduces a Hessian-free approach that uses a first-order approximation of derivatives on the Stiefel manifold. Our method significantly reduces the computational load and memory footprint. We show how using a Stiefel fully-connected layer that enforces orthogonality constraint on the parameters of the last classification layer as the head of the backbone network, strengthens the representation reuse of the gradient-based meta-learning methods. Our experimental results across various few-shot learning datasets, demonstrate the superiority of our proposed method compared to the state-of-the-art methods, especially MAML, its Euclidean counterpart.<|reference_end|>
@article{tabealhojeh2024forml:,
    title={FORML: A Riemannian Hessian-free Method for Meta-learning on Stiefel
  Manifolds},
    author={Hadi Tabealhojeh, Soumava Kumar Roy, Peyman Adibi and Hossein
  Karshenas},
    journal={arXiv preprint arXiv:2402.18605},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.18605},
    primaryClass={cs.LG}
}