title: TiC-CLIP: Continual Training of CLIP Models
summary: Keeping large foundation models up to date on latest data is inherently
  expensive. To avoid the prohibitive costs of constantly retraining, it is
  imperative to continually train these models. This problem is exacerbated by
  the lack of any large scale continual learning benchmarks or baselines. We
  introduce the first set of web-scale Time-Continual (TiC) benchmarks for
  training vision-language models: TiC-DataCompt, TiC-YFCC, and TiC-RedCaps with
  over 12.7B timestamped image-text pairs spanning 9 years (2014--2022). We first
  use our benchmarks to curate various dynamic evaluations to measure temporal
  robustness of existing models. We show OpenAI's CLIP (trained on data up to
  2020) loses approx 8% zero-shot accuracy on our curated retrieval task from
  2021--2022 compared with more recently trained models in OpenCLIP repository.
  We then study how to efficiently train models on time-continuous data. We
  demonstrate that a simple rehearsal-based approach that continues training from
  the last checkpoint and replays old data reduces compute by 2.5times when
  compared to the standard practice of retraining from scratch.
paper_authors._id: 6539d78ad1ca3238d0e5c0d0
paper_authors.name: Saurabh Garg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:42:17.117Z
paper_authors._id: 64a8a87e4a702899835e9b45
paper_authors.avatarUrl: /avatars/b6c60f63337a78a1bf3943885f4d7a18.svg
paper_authors.fullname: Saurabh Garg
paper_authors.type: user
paper_authors.user: saurabhgarg
paper_authors._id: 6539d78ad1ca3238d0e5c0d1
paper_authors.name: Mehrdad Farajtabar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:42:24.788Z
paper_authors._id: 62d188a6d50433c35eb14020
paper_authors.avatarUrl: /avatars/fc3d149a27bc8396010a2a02148e8ca0.svg
paper_authors.fullname: Mehrdad Farajtabar
paper_authors.type: user
paper_authors.user: mfarajtabar
paper_authors._id: 6539d78ad1ca3238d0e5c0d2
paper_authors.name: Hadi Pouransari
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-19T09:35:43.776Z
paper_authors._id: 653a8e65c75dc136bfb5b0f8
paper_authors.avatarUrl: /avatars/ef252794236ce0fe2debf12773c95bb2.svg
paper_authors.fullname: Hadi Pouransari
paper_authors.type: user
paper_authors.user: hpouransari
paper_authors._id: 6539d78ad1ca3238d0e5c0d3
paper_authors.name: Raviteja Vemulapalli
paper_authors._id: 6539d78ad1ca3238d0e5c0d4
paper_authors.name: Sachin Mehta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:42:37.752Z
paper_authors._id: 64f792652003abb61bf5c1b8
paper_authors.avatarUrl: /avatars/9721810de3519ece50b059c154baa9ca.svg
paper_authors.fullname: Sachin Mehta
paper_authors.type: user
paper_authors.user: sacmehta
paper_authors._id: 6539d78ad1ca3238d0e5c0d5
paper_authors.name: Oncel Tuzel
paper_authors._id: 6539d78ad1ca3238d0e5c0d6
paper_authors.name: Vaishaal Shankar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:42:47.789Z
paper_authors._id: 63118add64939fabc0108b28
paper_authors.avatarUrl: /avatars/0dfbd5d7f1111ec8798e3a9bcb5f7e77.svg
paper_authors.fullname: vaishaal shankar
paper_authors.type: user
paper_authors.user: vaishaal
paper_authors._id: 6539d78ad1ca3238d0e5c0d7
paper_authors.name: Fartash Faghri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T15:29:14.687Z
paper_authors._id: 65f8945a3ba58880f9794a11
paper_authors.avatarUrl: /avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
paper_authors.fullname: Fartash Faghri
paper_authors.type: user
paper_authors.user: fartashf
paper_title: TiC-CLIP: Continual Training of CLIP Models
paper_summary: Keeping large foundation models up to date on latest data is inherently
  expensive. To avoid the prohibitive costs of constantly retraining, it is
  imperative to continually train these models. This problem is exacerbated by
  the lack of any large scale continual learning benchmarks or baselines. We
  introduce the first set of web-scale Time-Continual (TiC) benchmarks for
  training vision-language models: TiC-DataCompt, TiC-YFCC, and TiC-RedCaps with
  over 12.7B timestamped image-text pairs spanning 9 years (2014--2022). We first
  use our benchmarks to curate various dynamic evaluations to measure temporal
  robustness of existing models. We show OpenAI's CLIP (trained on data up to
  2020) loses approx 8% zero-shot accuracy on our curated retrieval task from
  2021--2022 compared with more recently trained models in OpenCLIP repository.
  We then study how to efficiently train models on time-continuous data. We
  demonstrate that a simple rehearsal-based approach that continues training from
  the last checkpoint and replays old data reduces compute by 2.5times when
  compared to the standard practice of retraining from scratch.
paper_ai_keywords: Time-Continual (TiC) benchmarks vision-language models TiC-DataCompt TiC-YFCC TiC-RedCaps timestamped image-text pairs temporal robustness OpenAI's CLIP zero-shot accuracy retrieval task OpenCLIP repository rehearsal-based approach