TransAgent: Transfer Vision-Language Foundation Models with
  Heterogeneous Agent Collaboration
Vision-language foundation models (such as CLIP) have recently shown their
power in transfer learning, owing to large-scale image-text pre-training.
However, target domain data in the downstream tasks can be highly different
from the pre-training phase, which makes it hard for such a single model to
generalize well. Alternatively, there exists a wide range of expert models that
contain diversified vision and/or language knowledge pre-trained on different
modalities, tasks, networks, and datasets. Unfortunately, these models are
"isolated agents" with heterogeneous structures, and how to integrate their
knowledge for generalizing CLIP-like models has not been fully explored. To
bridge this gap, we propose a general and concise TransAgent framework, which
transports the knowledge of the isolated agents in a unified manner, and
effectively guides CLIP to generalize with multi-source knowledge distillation.
With such a distinct framework, we flexibly collaborate with 11 heterogeneous
agents to empower vision-language foundation models, without further cost in
the inference phase. Finally, our TransAgent achieves state-of-the-art
performance on 11 visual recognition datasets. Under the same low-shot setting,
it outperforms the popular CoOp with around 10% on average, and 20% on EuroSAT
which contains large domain shifts.
6711c90dfe58fd40dde8cc17
Yiwei Guo
claimed_verified
2024-10-18T08:12:28.191Z
6711c8a16019f85716ef2f34
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/W8f4Tbwx3LjTWoK39oR3s.png
Yiwei Guo
user
markywg
6711c90dfe58fd40dde8cc18
Shaobin Zhuang
admin_assigned
2024-10-18T13:02:47.949Z
644d3c3ad1945fe77a15ce35
/avatars/61ca3587eed853f69df89f1e0d02bb03.svg
shaobin zhuang
user
greysunshine
6711c90dfe58fd40dde8cc19
Kunchang Li
admin_assigned
2024-10-18T13:02:53.724Z
61fb81006374891646732f37
https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
Kunchang Li
user
Andy1621
6711c90dfe58fd40dde8cc1a
Yu Qiao
6711c90dfe58fd40dde8cc1b
Yali Wang
TransAgent: Transfer Vision-Language Foundation Models with
  Heterogeneous Agent Collaboration
Vision-language foundation models (such as CLIP) have recently shown their
power in transfer learning, owing to large-scale image-text pre-training.
However, target domain data in the downstream tasks can be highly different
from the pre-training phase, which makes it hard for such a single model to
generalize well. Alternatively, there exists a wide range of expert models that
contain diversified vision and/or language knowledge pre-trained on different
modalities, tasks, networks, and datasets. Unfortunately, these models are
"isolated agents" with heterogeneous structures, and how to integrate their
knowledge for generalizing CLIP-like models has not been fully explored. To
bridge this gap, we propose a general and concise TransAgent framework, which
transports the knowledge of the isolated agents in a unified manner, and
effectively guides CLIP to generalize with multi-source knowledge distillation.
With such a distinct framework, we flexibly collaborate with 11 heterogeneous
agents to empower vision-language foundation models, without further cost in
the inference phase. Finally, our TransAgent achieves state-of-the-art
performance on 11 visual recognition datasets. Under the same low-shot setting,
it outperforms the popular CoOp with around 10% on average, and 20% on EuroSAT
which contains large domain shifts.
TransAgent framework integrates multiple heterogeneous models to enhance vision-language foundation models' generalization, achieving state-of-the-art performance without additional inference cost.
vision-language foundation models CLIP transfer learning image-text pre-training expert models heterogeneous structures knowledge distillation low-shot setting CoOp EuroSAT