Relative Representations: Topological and Geometric Perspectives
<|reference_start|>Relative Representations: Topological and Geometric Perspectives: Relative representations are an established approach to zero-shot model stitching, consisting of a non-trainable transformation of the latent space of a deep neural network. Based on insights of topological and geometric nature, we propose two improvements to relative representations. First, we introduce a normalization procedure in the relative transformation, resulting in invariance to non-isotropic rescalings and permutations. The latter coincides with the symmetries in parameter space induced by common activation functions. Second, we propose to deploy topological densification when fine-tuning relative representations, a topological regularization loss encouraging clustering within classes. We provide an empirical investigation on a natural language task, where both the proposed variations yield improved performance on zero-shot model stitching.<|reference_end|>
@article{garcía-castellanos2024relative,
    title={Relative Representations: Topological and Geometric Perspectives},
    author={Alejandro Garc'ia-Castellanos, Giovanni Luca Marchetti, Danica
  Kragic, Martina Scolamiero},
    journal={arXiv preprint arXiv:2409.10967},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.10967},
    primaryClass={cs.LG}
}