title: Finetuned Multimodal Language Models Are High-Quality Image-Text Data
    Filters
summary: We propose a novel framework for filtering image-text data by leveraging
  fine-tuned Multimodal Language Models (MLMs). Our approach outperforms
  predominant filtering methods (e.g., CLIPScore) via integrating the recent
  advances in MLMs. We design four distinct yet complementary metrics to
  holistically measure the quality of image-text data. A new pipeline is
  established to construct high-quality instruction data for fine-tuning MLMs as
  data filters. Comparing with CLIPScore, our MLM filters produce more precise
  and comprehensive scores that directly improve the quality of filtered data and
  boost the performance of pre-trained models. We achieve significant
  improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2)
  and various downstream tasks. Our MLM filter can generalize to different models
  and tasks, and be used as a drop-in replacement for CLIPScore. An additional
  ablation study is provided to verify our design choices for the MLM filter.
paper_authors._id: 65e7d8237c1853c8c170c329
paper_authors.name: Weizhi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-06T09:27:47.811Z
paper_authors._id: 63d34004b734eaa4d4faeccf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/aH7F-xn4PMQjHHcvL-0vs.jpeg
paper_authors.fullname: Weizhi Wang
paper_authors.type: user
paper_authors.user: weizhiwang
paper_authors._id: 65e7d8237c1853c8c170c32a
paper_authors.name: Khalil Mrini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:00:18.453Z
paper_authors._id: 62608429ffe8827cb1d86984
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62608429ffe8827cb1d86984/bXN3V5hARj5qyxv4giyWu.jpeg
paper_authors.fullname: Khalil Mrini
paper_authors.type: user
paper_authors.user: KhalilMrini
paper_authors._id: 65e7d8237c1853c8c170c32b
paper_authors.name: Linjie Yang
paper_authors._id: 65e7d8237c1853c8c170c32c
paper_authors.name: Sateesh Kumar
paper_authors._id: 65e7d8237c1853c8c170c32d
paper_authors.name: Yu Tian
paper_authors._id: 65e7d8237c1853c8c170c32e
paper_authors.name: Xifeng Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:00:53.540Z
paper_authors._id: 65cd4785c40ab294321d610e
paper_authors.avatarUrl: /avatars/3f0053aa2b3d90a10b60ab24cf575fd5.svg
paper_authors.fullname: Xifeng Yan
paper_authors.type: user
paper_authors.user: windwest
paper_authors._id: 65e7d8237c1853c8c170c32f
paper_authors.name: Heng Wang
paper_title: Finetuned Multimodal Language Models Are High-Quality Image-Text Data
    Filters
paper_summary: We propose a novel framework for filtering image-text data by leveraging
  fine-tuned Multimodal Language Models (MLMs). Our approach outperforms
  predominant filtering methods (e.g., CLIPScore) via integrating the recent
  advances in MLMs. We design four distinct yet complementary metrics to
  holistically measure the quality of image-text data. A new pipeline is
  established to construct high-quality instruction data for fine-tuning MLMs as
  data filters. Comparing with CLIPScore, our MLM filters produce more precise
  and comprehensive scores that directly improve the quality of filtered data and
  boost the performance of pre-trained models. We achieve significant
  improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2)
  and various downstream tasks. Our MLM filter can generalize to different models
  and tasks, and be used as a drop-in replacement for CLIPScore. An additional
  ablation study is provided to verify our design choices for the MLM filter.
paper_ai_keywords: fine-tuned Multimodal Language Models (MLMs) CLIPScore synthesis Multimodal Language Models (MLMs) image-text data high-quality instruction data CLIP