title: TransAgent: Transfer Vision-Language Foundation Models with
    Heterogeneous Agent Collaboration
summary: Vision-language foundation models (such as CLIP) have recently shown their
  power in transfer learning, owing to large-scale image-text pre-training.
  However, target domain data in the downstream tasks can be highly different
  from the pre-training phase, which makes it hard for such a single model to
  generalize well. Alternatively, there exists a wide range of expert models that
  contain diversified vision and/or language knowledge pre-trained on different
  modalities, tasks, networks, and datasets. Unfortunately, these models are
  "isolated agents" with heterogeneous structures, and how to integrate their
  knowledge for generalizing CLIP-like models has not been fully explored. To
  bridge this gap, we propose a general and concise TransAgent framework, which
  transports the knowledge of the isolated agents in a unified manner, and
  effectively guides CLIP to generalize with multi-source knowledge distillation.
  With such a distinct framework, we flexibly collaborate with 11 heterogeneous
  agents to empower vision-language foundation models, without further cost in
  the inference phase. Finally, our TransAgent achieves state-of-the-art
  performance on 11 visual recognition datasets. Under the same low-shot setting,
  it outperforms the popular CoOp with around 10% on average, and 20% on EuroSAT
  which contains large domain shifts.
paper_authors._id: 6711c90dfe58fd40dde8cc17
paper_authors.name: Yiwei Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:12:28.191Z
paper_authors._id: 6711c8a16019f85716ef2f34
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/W8f4Tbwx3LjTWoK39oR3s.png
paper_authors.fullname: Yiwei Guo
paper_authors.type: user
paper_authors.user: markywg
paper_authors._id: 6711c90dfe58fd40dde8cc18
paper_authors.name: Shaobin Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:02:47.949Z
paper_authors._id: 644d3c3ad1945fe77a15ce35
paper_authors.avatarUrl: /avatars/61ca3587eed853f69df89f1e0d02bb03.svg
paper_authors.fullname: shaobin zhuang
paper_authors.type: user
paper_authors.user: greysunshine
paper_authors._id: 6711c90dfe58fd40dde8cc19
paper_authors.name: Kunchang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:02:53.724Z
paper_authors._id: 61fb81006374891646732f37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
paper_authors.fullname: Kunchang Li
paper_authors.type: user
paper_authors.user: Andy1621
paper_authors._id: 6711c90dfe58fd40dde8cc1a
paper_authors.name: Yu Qiao
paper_authors._id: 6711c90dfe58fd40dde8cc1b
paper_authors.name: Yali Wang
paper_title: TransAgent: Transfer Vision-Language Foundation Models with
    Heterogeneous Agent Collaboration
paper_summary: Vision-language foundation models (such as CLIP) have recently shown their
  power in transfer learning, owing to large-scale image-text pre-training.
  However, target domain data in the downstream tasks can be highly different
  from the pre-training phase, which makes it hard for such a single model to
  generalize well. Alternatively, there exists a wide range of expert models that
  contain diversified vision and/or language knowledge pre-trained on different
  modalities, tasks, networks, and datasets. Unfortunately, these models are
  "isolated agents" with heterogeneous structures, and how to integrate their
  knowledge for generalizing CLIP-like models has not been fully explored. To
  bridge this gap, we propose a general and concise TransAgent framework, which
  transports the knowledge of the isolated agents in a unified manner, and
  effectively guides CLIP to generalize with multi-source knowledge distillation.
  With such a distinct framework, we flexibly collaborate with 11 heterogeneous
  agents to empower vision-language foundation models, without further cost in
  the inference phase. Finally, our TransAgent achieves state-of-the-art
  performance on 11 visual recognition datasets. Under the same low-shot setting,
  it outperforms the popular CoOp with around 10% on average, and 20% on EuroSAT
  which contains large domain shifts.
paper_ai_summary: TransAgent framework integrates multiple heterogeneous models to enhance vision-language foundation models' generalization, achieving state-of-the-art performance without additional inference cost.
paper_ai_keywords: vision-language foundation models CLIP transfer learning image-text pre-training expert models heterogeneous structures knowledge distillation low-shot setting CoOp EuroSAT