title: DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model
summary: Observing the close relationship among panoptic, semantic and instance
  segmentation tasks, we propose to train a universal multi-dataset multi-task
  segmentation model: DaTaSeg.We use a shared representation (mask proposals with
  class predictions) for all tasks. To tackle task discrepancy, we adopt
  different merge operations and post-processing for different tasks. We also
  leverage weak-supervision, allowing our segmentation model to benefit from
  cheaper bounding box annotations. To share knowledge across datasets, we use
  text embeddings from the same semantic embedding space as classifiers and share
  all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO
  panoptic, and Objects365 detection datasets. DaTaSeg improves performance on
  all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE
  semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised
  knowledge transfer on ADE panoptic and Objects365 instance segmentation.
  Experiments show DaTaSeg scales with the number of training datasets and
  enables open-vocabulary segmentation through direct transfer. In addition, we
  annotate an Objects365 instance segmentation set of 1,000 images and will
  release it as a public benchmark.
paper_authors._id: 647d5d321b6ecd15f4c04d94
paper_authors.name: Xiuye Gu
paper_authors._id: 647d5d321b6ecd15f4c04d95
paper_authors.name: Yin Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:57:00.078Z
paper_authors._id: 649f05367b57fab3a5b27c8b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
paper_authors.fullname: Yin Cui
paper_authors.type: user
paper_authors.user: richardaecn
paper_authors._id: 647d5d321b6ecd15f4c04d96
paper_authors.name: Jonathan Huang
paper_authors._id: 647d5d321b6ecd15f4c04d97
paper_authors.name: Abdullah Rashwan
paper_authors._id: 647d5d321b6ecd15f4c04d98
paper_authors.name: Xuan Yang
paper_authors._id: 647d5d321b6ecd15f4c04d99
paper_authors.name: Xingyi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T10:28:19.317Z
paper_authors._id: 629ee11846b4826be2d94676
paper_authors.avatarUrl: /avatars/95ab04122ad626d74bf8a65d7b2bf48a.svg
paper_authors.fullname: Xingyi Zhou
paper_authors.type: user
paper_authors.user: zhouxy
paper_authors._id: 647d5d321b6ecd15f4c04d9a
paper_authors.name: Golnaz Ghiasi
paper_authors._id: 647d5d321b6ecd15f4c04d9b
paper_authors.name: Weicheng Kuo
paper_authors._id: 647d5d321b6ecd15f4c04d9c
paper_authors.name: Huizhong Chen
paper_authors._id: 647d5d321b6ecd15f4c04d9d
paper_authors.name: Liang-Chieh Chen
paper_authors._id: 647d5d321b6ecd15f4c04d9e
paper_authors.name: David A Ross
paper_title: DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model
paper_summary: Observing the close relationship among panoptic, semantic and instance
  segmentation tasks, we propose to train a universal multi-dataset multi-task
  segmentation model: DaTaSeg.We use a shared representation (mask proposals with
  class predictions) for all tasks. To tackle task discrepancy, we adopt
  different merge operations and post-processing for different tasks. We also
  leverage weak-supervision, allowing our segmentation model to benefit from
  cheaper bounding box annotations. To share knowledge across datasets, we use
  text embeddings from the same semantic embedding space as classifiers and share
  all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO
  panoptic, and Objects365 detection datasets. DaTaSeg improves performance on
  all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE
  semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised
  knowledge transfer on ADE panoptic and Objects365 instance segmentation.
  Experiments show DaTaSeg scales with the number of training datasets and
  enables open-vocabulary segmentation through direct transfer. In addition, we
  annotate an Objects365 instance segmentation set of 1,000 images and will
  release it as a public benchmark.
paper_ai_summary: A universal multi-dataset multi-task segmentation model, DaTaSeg, improves performance across semantic, panoptic, and instance segmentation tasks using shared representations and weak supervision.
paper_ai_keywords: panoptic segmentation semantic segmentation instance segmentation mask proposals class predictions task discrepancy merge operations post-processing weak-supervision bounding box annotations text embeddings semantic embedding space classifiers open-vocabulary segmentation