title: Language-Image Alignment with Fixed Text Encoders
summary: Currently, the most dominant approach to establishing language-image
  alignment is to pre-train text and image encoders jointly through contrastive
  learning, such as CLIP and its variants. In this work, we question whether such
  a costly joint training is necessary. In particular, we investigate if a
  pre-trained fixed large language model (LLM) offers a good enough text encoder
  to guide visual representation learning. That is, we propose to learn
  Language-Image alignment with a Fixed Text encoder (LIFT) from an LLM by
  training only the image encoder. Somewhat surprisingly, through comprehensive
  benchmarking and ablation studies, we find that this much simplified framework
  LIFT is highly effective and it outperforms CLIP in most scenarios that involve
  compositional understanding and long captions, while achieving considerable
  gains in computational efficiency. Our work takes a first step towards
  systematically exploring how text embeddings from LLMs can guide visual
  learning and suggests an alternative design choice for learning
  language-aligned visual representations.
paper_authors._id: 68413c8eb64ba498925da6a8
paper_authors.name: Jingfeng Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T07:26:26.842Z
paper_authors._id: 65d45fbf9f087171b805c428
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d45fbf9f087171b805c428/bgxwcn2p_D9qEa5vynSxV.jpeg
paper_authors.fullname: Jingfeng Yang
paper_authors.type: user
paper_authors.user: JingfengY
paper_authors._id: 68413c8eb64ba498925da6a9
paper_authors.name: Ziyang Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:41:57.098Z
paper_authors._id: 64ea89932ca4ff1d53b77548
paper_authors.avatarUrl: /avatars/ce3df67ba3ea3197ebf74fbe5e2c0e48.svg
paper_authors.fullname: Ziyang Wu
paper_authors.type: user
paper_authors.user: robinwuzy
paper_authors._id: 68413c8eb64ba498925da6aa
paper_authors.name: Yue Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T13:34:00.155Z
paper_authors._id: 673c1b71223fa8d98458d7ce
paper_authors.avatarUrl: /avatars/dd1d71ae55ac4e4460f690f777823605.svg
paper_authors.fullname: Yue Zhao
paper_authors.type: user
paper_authors.user: zyzk-1337
paper_authors._id: 68413c8eb64ba498925da6ab
paper_authors.name: Yi Ma
paper_title: Language-Image Alignment with Fixed Text Encoders
paper_summary: Currently, the most dominant approach to establishing language-image
  alignment is to pre-train text and image encoders jointly through contrastive
  learning, such as CLIP and its variants. In this work, we question whether such
  a costly joint training is necessary. In particular, we investigate if a
  pre-trained fixed large language model (LLM) offers a good enough text encoder
  to guide visual representation learning. That is, we propose to learn
  Language-Image alignment with a Fixed Text encoder (LIFT) from an LLM by
  training only the image encoder. Somewhat surprisingly, through comprehensive
  benchmarking and ablation studies, we find that this much simplified framework
  LIFT is highly effective and it outperforms CLIP in most scenarios that involve
  compositional understanding and long captions, while achieving considerable
  gains in computational efficiency. Our work takes a first step towards
  systematically exploring how text embeddings from LLMs can guide visual
  learning and suggests an alternative design choice for learning
  language-aligned visual representations.
paper_ai_summary: Learning Language-Image alignment with a Fixed Text encoder (LIFT) using pre-trained large language models effectively guides visual representation learning, outperforming joint training methods like CLIP in compositional understanding and long captions.
paper_ai_keywords: contrastive learning CLIP pre-trained fixed large language model LLM Language-Image alignment LIFT image encoder compositional understanding long captions