title: ReferEverything: Towards Segmenting Everything We Can Speak of in Videos
summary: We present REM, a framework for segmenting a wide range of concepts in video
  that can be described through natural language. Our method capitalizes on
  visual-language representations learned by video diffusion models on
  Internet-scale datasets. A key insight of our approach is preserving as much of
  the generative model's original representation as possible, while fine-tuning
  it on narrow-domain Referral Object Segmentation datasets. As a result, our
  framework can accurately segment and track rare and unseen objects, despite
  being trained on object masks from a limited set of categories. Additionally,
  it can generalize to non-object dynamic concepts, such as waves crashing in the
  ocean, as demonstrated in our newly introduced benchmark for Referral Video
  Process Segmentation (Ref-VPS). Our experiments show that REM performs on par
  with state-of-the-art approaches on in-domain datasets, like Ref-DAVIS, while
  outperforming them by up to twelve points in terms of region similarity on
  out-of-domain data, leveraging the power of Internet-scale pre-training.
paper_authors._id: 6722e182440fd7dab3f28195
paper_authors.name: Anurag Bagchi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-31T09:54:38.370Z
paper_authors._id: 661dfccdd8427174a890910b
paper_authors.avatarUrl: /avatars/acf4310a159013ae440edd30632ce543.svg
paper_authors.fullname: Anurag Bagchi
paper_authors.type: user
paper_authors.user: anuragba
paper_authors._id: 6722e182440fd7dab3f28196
paper_authors.name: Zhipeng Bao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-31T09:54:35.841Z
paper_authors._id: 65cedb3e6712e2451ee14ac2
paper_authors.avatarUrl: /avatars/c34c8c71e7a45180bbf1684ad5b3f23c.svg
paper_authors.fullname: Zhipeng Bao
paper_authors.type: user
paper_authors.user: bzp15
paper_authors._id: 6722e182440fd7dab3f28197
paper_authors.name: Yu-Xiong Wang
paper_authors._id: 6722e182440fd7dab3f28198
paper_authors.name: Pavel Tokmakov
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-31T01:57:00.271Z
paper_authors._id: 65aeab2201b6f61e24323339
paper_authors.avatarUrl: /avatars/2c119173c0871f59df108f9bb4471954.svg
paper_authors.fullname: Pavel Tokmakov
paper_authors.type: user
paper_authors.user: ptokmakov
paper_authors._id: 6722e182440fd7dab3f28199
paper_authors.name: Martial Hebert
paper_title: ReferEverything: Towards Segmenting Everything We Can Speak of in Videos
paper_summary: We present REM, a framework for segmenting a wide range of concepts in video
  that can be described through natural language. Our method capitalizes on
  visual-language representations learned by video diffusion models on
  Internet-scale datasets. A key insight of our approach is preserving as much of
  the generative model's original representation as possible, while fine-tuning
  it on narrow-domain Referral Object Segmentation datasets. As a result, our
  framework can accurately segment and track rare and unseen objects, despite
  being trained on object masks from a limited set of categories. Additionally,
  it can generalize to non-object dynamic concepts, such as waves crashing in the
  ocean, as demonstrated in our newly introduced benchmark for Referral Video
  Process Segmentation (Ref-VPS). Our experiments show that REM performs on par
  with state-of-the-art approaches on in-domain datasets, like Ref-DAVIS, while
  outperforming them by up to twelve points in terms of region similarity on
  out-of-domain data, leveraging the power of Internet-scale pre-training.
paper_ai_summary: REM, a video segmentation framework using visual-language models, excels in segmenting and tracking unseen and dynamic concepts by leveraging Internet-scale pre-training.
paper_ai_keywords: visual-language representations video diffusion models Internet-scale datasets fine-tuning Referral Object Segmentation region similarity segmenting tracking rare objects unseen objects dynamic concepts waves crashing Referral Video Process Segmentation Ref-DAVIS