Transferable and Principled Efficiency for Open-Vocabulary Segmentation
Recent success of pre-trained foundation vision-language models makes
Open-Vocabulary Segmentation (OVS) possible. Despite the promising performance,
this approach introduces heavy computational overheads for two challenges: 1)
large model sizes of the backbone; 2) expensive costs during the fine-tuning.
These challenges hinder this OVS strategy from being widely applicable and
affordable in real-world scenarios. Although traditional methods such as model
compression and efficient fine-tuning can address these challenges, they often
rely on heuristics. This means that their solutions cannot be easily
transferred and necessitate re-training on different models, which comes at a
cost. In the context of efficient OVS, we target achieving performance that is
comparable to or even better than prior OVS works based on large
vision-language foundation models, by utilizing smaller models that incur lower
training costs. The core strategy is to make our efficiency principled and thus
seamlessly transferable from one OVS framework to others without further
customization. Comprehensive experiments on diverse OVS benchmarks demonstrate
our superior trade-off between segmentation accuracy and computation costs over
previous works. Our code is available on https://github.com/Xujxyang/OpenTrans
66188bdfbfea2f9d4c9ce05e
Jingxuan Xu
66188bdfbfea2f9d4c9ce05f
Wuyang Chen
admin_assigned
2024-04-18T12:34:16.822Z
6489d8f967e8a51448487cbb
/avatars/731732fb3405542b95f1ce0bc13db059.svg
Wuyang Chen
user
chenwydj
66188bdfbfea2f9d4c9ce060
Yao Zhao
66188bdfbfea2f9d4c9ce061
Yunchao Wei
Transferable and Principled Efficiency for Open-Vocabulary Segmentation
Recent success of pre-trained foundation vision-language models makes
Open-Vocabulary Segmentation (OVS) possible. Despite the promising performance,
this approach introduces heavy computational overheads for two challenges: 1)
large model sizes of the backbone; 2) expensive costs during the fine-tuning.
These challenges hinder this OVS strategy from being widely applicable and
affordable in real-world scenarios. Although traditional methods such as model
compression and efficient fine-tuning can address these challenges, they often
rely on heuristics. This means that their solutions cannot be easily
transferred and necessitate re-training on different models, which comes at a
cost. In the context of efficient OVS, we target achieving performance that is
comparable to or even better than prior OVS works based on large
vision-language foundation models, by utilizing smaller models that incur lower
training costs. The core strategy is to make our efficiency principled and thus
seamlessly transferable from one OVS framework to others without further
customization. Comprehensive experiments on diverse OVS benchmarks demonstrate
our superior trade-off between segmentation accuracy and computation costs over
previous works. Our code is available on https://github.com/Xujxyang/OpenTrans
The approach leverages smaller vision-language models to achieve efficient Open-Vocabulary Segmentation with comparable or better performance and lower computational costs compared to larger models.
pre-trained foundation vision-language models Open-Vocabulary Segmentation large model sizes fine-tuning model compression efficient fine-tuning efficiency OVS benchmarks segmentation accuracy computation costs