ReferEverything: Towards Segmenting Everything We Can Speak of in Videos
We present REM, a framework for segmenting a wide range of concepts in video
that can be described through natural language. Our method capitalizes on
visual-language representations learned by video diffusion models on
Internet-scale datasets. A key insight of our approach is preserving as much of
the generative model's original representation as possible, while fine-tuning
it on narrow-domain Referral Object Segmentation datasets. As a result, our
framework can accurately segment and track rare and unseen objects, despite
being trained on object masks from a limited set of categories. Additionally,
it can generalize to non-object dynamic concepts, such as waves crashing in the
ocean, as demonstrated in our newly introduced benchmark for Referral Video
Process Segmentation (Ref-VPS). Our experiments show that REM performs on par
with state-of-the-art approaches on in-domain datasets, like Ref-DAVIS, while
outperforming them by up to twelve points in terms of region similarity on
out-of-domain data, leveraging the power of Internet-scale pre-training.
6722e182440fd7dab3f28195
Anurag Bagchi
claimed_verified
2024-10-31T09:54:38.370Z
661dfccdd8427174a890910b
/avatars/acf4310a159013ae440edd30632ce543.svg
Anurag Bagchi
user
anuragba
6722e182440fd7dab3f28196
Zhipeng Bao
claimed_verified
2024-10-31T09:54:35.841Z
65cedb3e6712e2451ee14ac2
/avatars/c34c8c71e7a45180bbf1684ad5b3f23c.svg
Zhipeng Bao
user
bzp15
6722e182440fd7dab3f28197
Yu-Xiong Wang
6722e182440fd7dab3f28198
Pavel Tokmakov
extracted_confirmed
2024-10-31T01:57:00.271Z
65aeab2201b6f61e24323339
/avatars/2c119173c0871f59df108f9bb4471954.svg
Pavel Tokmakov
user
ptokmakov
6722e182440fd7dab3f28199
Martial Hebert
ReferEverything: Towards Segmenting Everything We Can Speak of in Videos
We present REM, a framework for segmenting a wide range of concepts in video
that can be described through natural language. Our method capitalizes on
visual-language representations learned by video diffusion models on
Internet-scale datasets. A key insight of our approach is preserving as much of
the generative model's original representation as possible, while fine-tuning
it on narrow-domain Referral Object Segmentation datasets. As a result, our
framework can accurately segment and track rare and unseen objects, despite
being trained on object masks from a limited set of categories. Additionally,
it can generalize to non-object dynamic concepts, such as waves crashing in the
ocean, as demonstrated in our newly introduced benchmark for Referral Video
Process Segmentation (Ref-VPS). Our experiments show that REM performs on par
with state-of-the-art approaches on in-domain datasets, like Ref-DAVIS, while
outperforming them by up to twelve points in terms of region similarity on
out-of-domain data, leveraging the power of Internet-scale pre-training.
REM, a video segmentation framework using visual-language models, excels in segmenting and tracking unseen and dynamic concepts by leveraging Internet-scale pre-training.
visual-language representations video diffusion models Internet-scale datasets fine-tuning Referral Object Segmentation region similarity segmenting tracking rare objects unseen objects dynamic concepts waves crashing Referral Video Process Segmentation Ref-DAVIS