title: Three Towers: Flexible Contrastive Learning with Pretrained Image Models
summary: We introduce Three Towers (3T), a flexible method to improve the contrastive
  learning of vision-language models by incorporating pretrained image
  classifiers. While contrastive models are usually trained from scratch, LiT
  (Zhai et al., 2022) has recently shown performance gains from using pretrained
  classifier embeddings. However, LiT directly replaces the image tower with the
  frozen embeddings, excluding any potential benefits of contrastively training
  the image tower. With 3T, we propose a more flexible strategy that allows the
  image tower to benefit from both pretrained embeddings and contrastive
  training. To achieve this, we introduce a third tower that contains the frozen
  pretrained embeddings, and we encourage alignment between this third tower and
  the main image-text towers. Empirically, 3T consistently improves over LiT and
  the CLIP-style from-scratch baseline for retrieval tasks. For classification,
  3T reliably improves over the from-scratch baseline, and while it underperforms
  relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k
  and Places365 pretraining.
paper_authors._id: 64741f210da364bd0d00d0fb
paper_authors.name: Jannik Kossen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T15:40:37.533Z
paper_authors._id: 64df30f2e672e4d2c57049e8
paper_authors.avatarUrl: /avatars/8c9503e43f403d655b11ea5c6e23ae79.svg
paper_authors.fullname: Jannik
paper_authors.type: user
paper_authors.user: jlko
paper_authors._id: 64741f210da364bd0d00d0fc
paper_authors.name: Mark Collier
paper_authors._id: 64741f210da364bd0d00d0fd
paper_authors.name: Basil Mustafa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-31T09:13:52.460Z
paper_authors._id: 63cfcad6e23b90128c66685c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674562206311-noauth.png
paper_authors.fullname: Basil Mustafa
paper_authors.type: user
paper_authors.user: BasilMustafa
paper_authors._id: 64741f210da364bd0d00d0fe
paper_authors.name: Xiao Wang
paper_authors._id: 64741f210da364bd0d00d0ff
paper_authors.name: Xiaohua Zhai
paper_authors._id: 64741f210da364bd0d00d100
paper_authors.name: Lucas Beyer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-31T09:14:24.187Z
paper_authors._id: 642d334ff65714b4585f2de4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642d334ff65714b4585f2de4/gxBynq5KyoUP0VlAQD3-w.jpeg
paper_authors.fullname: Lucas Beyer
paper_authors.type: user
paper_authors.user: giffmana
paper_authors._id: 64741f210da364bd0d00d101
paper_authors.name: Andreas Steiner
paper_authors._id: 64741f210da364bd0d00d102
paper_authors.name: Jesse Berent
paper_authors._id: 64741f210da364bd0d00d103
paper_authors.name: Rodolphe Jenatton
paper_authors._id: 64741f210da364bd0d00d104
paper_authors.name: Efi Kokiopoulou
paper_title: Three Towers: Flexible Contrastive Learning with Pretrained Image Models
paper_summary: We introduce Three Towers (3T), a flexible method to improve the contrastive
  learning of vision-language models by incorporating pretrained image
  classifiers. While contrastive models are usually trained from scratch, LiT
  (Zhai et al., 2022) has recently shown performance gains from using pretrained
  classifier embeddings. However, LiT directly replaces the image tower with the
  frozen embeddings, excluding any potential benefits of contrastively training
  the image tower. With 3T, we propose a more flexible strategy that allows the
  image tower to benefit from both pretrained embeddings and contrastive
  training. To achieve this, we introduce a third tower that contains the frozen
  pretrained embeddings, and we encourage alignment between this third tower and
  the main image-text towers. Empirically, 3T consistently improves over LiT and
  the CLIP-style from-scratch baseline for retrieval tasks. For classification,
  3T reliably improves over the from-scratch baseline, and while it underperforms
  relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k
  and Places365 pretraining.
paper_ai_keywords: contrastive learning vision-language models pretrained image classifiers image tower pretrained embeddings contrastively training three towers main image-text towers retrieval tasks classification JFT-pretrained models ImageNet-21k Places365 pretraining