title: Knowledge Composition using Task Vectors with Learned Anisotropic
    Scaling
summary: Pre-trained models produce strong generic representations that can be adapted
  via fine-tuning. The learned weight difference relative to the pre-trained
  model, known as a task vector, characterises the direction and stride of
  fine-tuning. The significance of task vectors is such that simple arithmetic
  operations on them can be used to combine diverse representations from
  different domains. This paper builds on these properties of task vectors and
  aims to answer (1) whether components of task vectors, particularly parameter
  blocks, exhibit similar characteristics, and (2) how such blocks can be used to
  enhance knowledge composition and transfer. To this end, we introduce aTLAS, an
  algorithm that linearly combines parameter blocks with different learned
  coefficients, resulting in anisotropic scaling at the task vector level. We
  show that such linear combinations explicitly exploit the low intrinsic
  dimensionality of pre-trained models, with only a few coefficients being the
  learnable parameters. Furthermore, composition of parameter blocks leverages
  the already learned representations, thereby reducing the dependency on large
  amounts of data. We demonstrate the effectiveness of our method in task
  arithmetic, few-shot recognition and test-time adaptation, with supervised or
  unsupervised objectives. In particular, we show that (1) learned anisotropic
  scaling allows task vectors to be more disentangled, causing less interference
  in composition; (2) task vector composition excels with scarce or no labeled
  data and is less prone to domain shift, thus leading to better
  generalisability; (3) mixing the most informative parameter blocks across
  different task vectors prior to training can reduce the memory footprint and
  improve the flexibility of knowledge transfer. Moreover, we show the potential
  of aTLAS as a PEFT method, particularly with less data, and demonstrate that
  its scalibility.
paper_authors._id: 668dd783a697b69d668feaed
paper_authors.name: Frederic Z. Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T08:19:56.816Z
paper_authors._id: 668ca1b3ed63008dfa687990
paper_authors.avatarUrl: /avatars/b3ae0e7ff60c60db605288c9cf5ae6f3.svg
paper_authors.fullname: Fred Zhang
paper_authors.type: user
paper_authors.user: fredzzhang
paper_authors._id: 668dd783a697b69d668feaee
paper_authors.name: Paul Albert
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T08:19:54.961Z
paper_authors._id: 65f26ed18404ac0e4cfe7d83
paper_authors.avatarUrl: /avatars/9c72744836f86a8e355a45700b10e393.svg
paper_authors.fullname: Paul Albert
paper_authors.type: user
paper_authors.user: PAlbert31
paper_authors._id: 668dd783a697b69d668feaef
paper_authors.name: Cristian Rodriguez-Opazo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T10:20:24.326Z
paper_authors._id: 66456873dd82c5792f4fd53a
paper_authors.avatarUrl: /avatars/0dcdc7243a81920e7448230541590195.svg
paper_authors.fullname: Cristian Rodriguez-Opazo
paper_authors.type: user
paper_authors.user: crodriguezo
paper_authors._id: 668dd783a697b69d668feaf0
paper_authors.name: Anton van den Hengel
paper_authors._id: 668dd783a697b69d668feaf1
paper_authors.name: Ehsan Abbasnejad
paper_title: Knowledge Composition using Task Vectors with Learned Anisotropic
    Scaling
paper_summary: Pre-trained models produce strong generic representations that can be adapted
  via fine-tuning. The learned weight difference relative to the pre-trained
  model, known as a task vector, characterises the direction and stride of
  fine-tuning. The significance of task vectors is such that simple arithmetic
  operations on them can be used to combine diverse representations from
  different domains. This paper builds on these properties of task vectors and
  aims to answer (1) whether components of task vectors, particularly parameter
  blocks, exhibit similar characteristics, and (2) how such blocks can be used to
  enhance knowledge composition and transfer. To this end, we introduce aTLAS, an
  algorithm that linearly combines parameter blocks with different learned
  coefficients, resulting in anisotropic scaling at the task vector level. We
  show that such linear combinations explicitly exploit the low intrinsic
  dimensionality of pre-trained models, with only a few coefficients being the
  learnable parameters. Furthermore, composition of parameter blocks leverages
  the already learned representations, thereby reducing the dependency on large
  amounts of data. We demonstrate the effectiveness of our method in task
  arithmetic, few-shot recognition and test-time adaptation, with supervised or
  unsupervised objectives. In particular, we show that (1) learned anisotropic
  scaling allows task vectors to be more disentangled, causing less interference
  in composition; (2) task vector composition excels with scarce or no labeled
  data and is less prone to domain shift, thus leading to better
  generalisability; (3) mixing the most informative parameter blocks across
  different task vectors prior to training can reduce the memory footprint and
  improve the flexibility of knowledge transfer. Moreover, we show the potential
  of aTLAS as a PEFT method, particularly with less data, and demonstrate that
  its scalibility.
paper_ai_summary: aTLAS algorithm leverages parameter blocks from pre-trained models to enhance knowledge composition and transfer, achieving better generalization and flexibility with limited data.
paper_ai_keywords: pre-trained models fine-tuning task vector parameter blocks anisotropic scaling linear combinations intrinsic dimensionality few-shot recognition test-time adaptation disentangled domain shift knowledge transfer PEFT scalibility