Finetuned Multimodal Language Models Are High-Quality Image-Text Data
  Filters
We propose a novel framework for filtering image-text data by leveraging
fine-tuned Multimodal Language Models (MLMs). Our approach outperforms
predominant filtering methods (e.g., CLIPScore) via integrating the recent
advances in MLMs. We design four distinct yet complementary metrics to
holistically measure the quality of image-text data. A new pipeline is
established to construct high-quality instruction data for fine-tuning MLMs as
data filters. Comparing with CLIPScore, our MLM filters produce more precise
and comprehensive scores that directly improve the quality of filtered data and
boost the performance of pre-trained models. We achieve significant
improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2)
and various downstream tasks. Our MLM filter can generalize to different models
and tasks, and be used as a drop-in replacement for CLIPScore. An additional
ablation study is provided to verify our design choices for the MLM filter.
65e7d8237c1853c8c170c329
Weizhi Wang
claimed_verified
2024-03-06T09:27:47.811Z
63d34004b734eaa4d4faeccf
https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/aH7F-xn4PMQjHHcvL-0vs.jpeg
Weizhi Wang
user
weizhiwang
65e7d8237c1853c8c170c32a
Khalil Mrini
admin_assigned
2024-03-06T10:00:18.453Z
62608429ffe8827cb1d86984
https://cdn-avatars.huggingface.co/v1/production/uploads/62608429ffe8827cb1d86984/bXN3V5hARj5qyxv4giyWu.jpeg
Khalil Mrini
user
KhalilMrini
65e7d8237c1853c8c170c32b
Linjie Yang
65e7d8237c1853c8c170c32c
Sateesh Kumar
65e7d8237c1853c8c170c32d
Yu Tian
65e7d8237c1853c8c170c32e
Xifeng Yan
admin_assigned
2024-03-06T10:00:53.540Z
65cd4785c40ab294321d610e
/avatars/3f0053aa2b3d90a10b60ab24cf575fd5.svg
Xifeng Yan
user
windwest
65e7d8237c1853c8c170c32f
Heng Wang
Finetuned Multimodal Language Models Are High-Quality Image-Text Data
  Filters
We propose a novel framework for filtering image-text data by leveraging
fine-tuned Multimodal Language Models (MLMs). Our approach outperforms
predominant filtering methods (e.g., CLIPScore) via integrating the recent
advances in MLMs. We design four distinct yet complementary metrics to
holistically measure the quality of image-text data. A new pipeline is
established to construct high-quality instruction data for fine-tuning MLMs as
data filters. Comparing with CLIPScore, our MLM filters produce more precise
and comprehensive scores that directly improve the quality of filtered data and
boost the performance of pre-trained models. We achieve significant
improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2)
and various downstream tasks. Our MLM filter can generalize to different models
and tasks, and be used as a drop-in replacement for CLIPScore. An additional
ablation study is provided to verify our design choices for the MLM filter.
A new framework using fine-tuned Multimodal Language Models (MLMs) improves the precision of image-text data filtering compared to CLIPScore, enhancing model performance across different foundation models and tasks.
Multimodal Language Models MLM CLIPScore CLIP BLIP2