DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model
Observing the close relationship among panoptic, semantic and instance
segmentation tasks, we propose to train a universal multi-dataset multi-task
segmentation model: DaTaSeg.We use a shared representation (mask proposals with
class predictions) for all tasks. To tackle task discrepancy, we adopt
different merge operations and post-processing for different tasks. We also
leverage weak-supervision, allowing our segmentation model to benefit from
cheaper bounding box annotations. To share knowledge across datasets, we use
text embeddings from the same semantic embedding space as classifiers and share
all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO
panoptic, and Objects365 detection datasets. DaTaSeg improves performance on
all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE
semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised
knowledge transfer on ADE panoptic and Objects365 instance segmentation.
Experiments show DaTaSeg scales with the number of training datasets and
enables open-vocabulary segmentation through direct transfer. In addition, we
annotate an Objects365 instance segmentation set of 1,000 images and will
release it as a public benchmark.
647d5d321b6ecd15f4c04d94
Xiuye Gu
647d5d321b6ecd15f4c04d95
Yin Cui
claimed_verified
2025-03-24T08:57:00.078Z
649f05367b57fab3a5b27c8b
https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
Yin Cui
user
richardaecn
647d5d321b6ecd15f4c04d96
Jonathan Huang
647d5d321b6ecd15f4c04d97
Abdullah Rashwan
647d5d321b6ecd15f4c04d98
Xuan Yang
647d5d321b6ecd15f4c04d99
Xingyi Zhou
admin_assigned
2023-06-07T10:28:19.317Z
629ee11846b4826be2d94676
/avatars/95ab04122ad626d74bf8a65d7b2bf48a.svg
Xingyi Zhou
user
zhouxy
647d5d321b6ecd15f4c04d9a
Golnaz Ghiasi
647d5d321b6ecd15f4c04d9b
Weicheng Kuo
647d5d321b6ecd15f4c04d9c
Huizhong Chen
647d5d321b6ecd15f4c04d9d
Liang-Chieh Chen
647d5d321b6ecd15f4c04d9e
David A Ross
DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model
Observing the close relationship among panoptic, semantic and instance
segmentation tasks, we propose to train a universal multi-dataset multi-task
segmentation model: DaTaSeg.We use a shared representation (mask proposals with
class predictions) for all tasks. To tackle task discrepancy, we adopt
different merge operations and post-processing for different tasks. We also
leverage weak-supervision, allowing our segmentation model to benefit from
cheaper bounding box annotations. To share knowledge across datasets, we use
text embeddings from the same semantic embedding space as classifiers and share
all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO
panoptic, and Objects365 detection datasets. DaTaSeg improves performance on
all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE
semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised
knowledge transfer on ADE panoptic and Objects365 instance segmentation.
Experiments show DaTaSeg scales with the number of training datasets and
enables open-vocabulary segmentation through direct transfer. In addition, we
annotate an Objects365 instance segmentation set of 1,000 images and will
release it as a public benchmark.
A universal multi-dataset multi-task segmentation model, DaTaSeg, improves performance across semantic, panoptic, and instance segmentation tasks using shared representations and weak supervision.
panoptic segmentation semantic segmentation instance segmentation mask proposals class predictions task discrepancy merge operations post-processing weak-supervision bounding box annotations text embeddings semantic embedding space classifiers open-vocabulary segmentation