TiC-CLIP: Continual Training of CLIP Models
Keeping large foundation models up to date on latest data is inherently
expensive. To avoid the prohibitive costs of constantly retraining, it is
imperative to continually train these models. This problem is exacerbated by
the lack of any large scale continual learning benchmarks or baselines. We
introduce the first set of web-scale Time-Continual (TiC) benchmarks for
training vision-language models: TiC-DataCompt, TiC-YFCC, and TiC-RedCaps with
over 12.7B timestamped image-text pairs spanning 9 years (2014--2022). We first
use our benchmarks to curate various dynamic evaluations to measure temporal
robustness of existing models. We show OpenAI's CLIP (trained on data up to
2020) loses approx 8% zero-shot accuracy on our curated retrieval task from
2021--2022 compared with more recently trained models in OpenCLIP repository.
We then study how to efficiently train models on time-continuous data. We
demonstrate that a simple rehearsal-based approach that continues training from
the last checkpoint and replays old data reduces compute by 2.5times when
compared to the standard practice of retraining from scratch.
6539d78ad1ca3238d0e5c0d0
Saurabh Garg
admin_assigned
2023-10-26T08:42:17.117Z
64a8a87e4a702899835e9b45
/avatars/b6c60f63337a78a1bf3943885f4d7a18.svg
Saurabh Garg
user
saurabhgarg
6539d78ad1ca3238d0e5c0d1
Mehrdad Farajtabar
admin_assigned
2023-10-26T08:42:24.788Z
62d188a6d50433c35eb14020
/avatars/fc3d149a27bc8396010a2a02148e8ca0.svg
Mehrdad Farajtabar
user
mfarajtabar
6539d78ad1ca3238d0e5c0d2
Hadi Pouransari
claimed_verified
2024-02-19T09:35:43.776Z
653a8e65c75dc136bfb5b0f8
/avatars/ef252794236ce0fe2debf12773c95bb2.svg
Hadi Pouransari
user
hpouransari
6539d78ad1ca3238d0e5c0d3
Raviteja Vemulapalli
6539d78ad1ca3238d0e5c0d4
Sachin Mehta
admin_assigned
2023-10-26T08:42:37.752Z
64f792652003abb61bf5c1b8
/avatars/9721810de3519ece50b059c154baa9ca.svg
Sachin Mehta
user
sacmehta
6539d78ad1ca3238d0e5c0d5
Oncel Tuzel
6539d78ad1ca3238d0e5c0d6
Vaishaal Shankar
admin_assigned
2023-10-26T08:42:47.789Z
63118add64939fabc0108b28
/avatars/0dfbd5d7f1111ec8798e3a9bcb5f7e77.svg
vaishaal shankar
user
vaishaal
6539d78ad1ca3238d0e5c0d7
Fartash Faghri
claimed_verified
2024-04-26T15:29:14.687Z
65f8945a3ba58880f9794a11
/avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
Fartash Faghri
user
fartashf
TiC-CLIP: Continual Training of CLIP Models
Keeping large foundation models up to date on latest data is inherently
expensive. To avoid the prohibitive costs of constantly retraining, it is
imperative to continually train these models. This problem is exacerbated by
the lack of any large scale continual learning benchmarks or baselines. We
introduce the first set of web-scale Time-Continual (TiC) benchmarks for
training vision-language models: TiC-DataCompt, TiC-YFCC, and TiC-RedCaps with
over 12.7B timestamped image-text pairs spanning 9 years (2014--2022). We first
use our benchmarks to curate various dynamic evaluations to measure temporal
robustness of existing models. We show OpenAI's CLIP (trained on data up to
2020) loses approx 8% zero-shot accuracy on our curated retrieval task from
2021--2022 compared with more recently trained models in OpenCLIP repository.
We then study how to efficiently train models on time-continuous data. We
demonstrate that a simple rehearsal-based approach that continues training from
the last checkpoint and replays old data reduces compute by 2.5times when
compared to the standard practice of retraining from scratch.
Web-scale Time-Continual (TiC) benchmarks are introduced to evaluate and improve the temporal robustness of vision-language models through efficient continual learning methods.
time-continual (TiC) benchmarks vision-language models temporal robustness CLIP OpenCLIP rehearsal-based approach continual learning compute efficiency