title: DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks
summary: Our primary goal here is to create a good, generalist perception model that
  can tackle multiple tasks, within limits on computational resources and
  training data. To achieve this, we resort to text-to-image diffusion models
  pre-trained on billions of images. Our exhaustive evaluation metrics
  demonstrate that DICEPTION effectively tackles multiple perception tasks,
  achieving performance on par with state-of-the-art models. We achieve results
  on par with SAM-vit-h using only 0.06% of their data (e.g., 600K vs. 1B
  pixel-level annotated images). Inspired by Wang et al., DICEPTION formulates
  the outputs of various perception tasks using color encoding; and we show that
  the strategy of assigning random colors to different instances is highly
  effective in both entity segmentation and semantic segmentation. Unifying
  various perception tasks as conditional image generation enables us to fully
  leverage pre-trained text-to-image models. Thus, DICEPTION can be efficiently
  trained at a cost of orders of magnitude lower, compared to conventional models
  that were trained from scratch. When adapting our model to other tasks, it only
  requires fine-tuning on as few as 50 images and 1% of its parameters. DICEPTION
  provides valuable insights and a more promising solution for visual generalist
  models.
paper_authors._id: 67bd3285ac4a596a43b53205
paper_authors.name: Canyu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:40:20.829Z
paper_authors._id: 646efd223dd912a539e0bd46
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EOFAv5xvOgJOzuDgh4nSb.png
paper_authors.fullname: Canyu Zhao
paper_authors.type: user
paper_authors.user: Canyu
paper_authors._id: 67bd3285ac4a596a43b53206
paper_authors.name: Mingyu Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:38:01.503Z
paper_authors._id: 652e25d2e647b0ee0a024f26
paper_authors.avatarUrl: /avatars/b5c65cf6c8d0ddc9b8ef0226e0295d56.svg
paper_authors.fullname: Mingyu Liu
paper_authors.type: user
paper_authors.user: MingyuLiu
paper_authors._id: 67bd3285ac4a596a43b53207
paper_authors.name: Huanyi Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:40:18.731Z
paper_authors._id: 64d60375d7e30889c65e8cf4
paper_authors.avatarUrl: /avatars/640f7c570fc45194557ce7931bdfe87f.svg
paper_authors.fullname: Huanyi Zheng
paper_authors.type: user
paper_authors.user: zhyya
paper_authors._id: 67bd3285ac4a596a43b53208
paper_authors.name: Muzhi Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:40:11.968Z
paper_authors._id: 632179745fc60c44fd91fc33
paper_authors.avatarUrl: /avatars/37d4fefbcc19f091dccffefec9706de2.svg
paper_authors.fullname: zhumuzhi
paper_authors.type: user
paper_authors.user: Z-MU-Z
paper_authors._id: 67bd3285ac4a596a43b53209
paper_authors.name: Zhiyue Zhao
paper_authors._id: 67bd3285ac4a596a43b5320a
paper_authors.name: Hao Chen
paper_authors._id: 67bd3285ac4a596a43b5320b
paper_authors.name: Tong He
paper_authors._id: 67bd3285ac4a596a43b5320c
paper_authors.name: Chunhua Shen
paper_title: DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks
paper_summary: Our primary goal here is to create a good, generalist perception model that
  can tackle multiple tasks, within limits on computational resources and
  training data. To achieve this, we resort to text-to-image diffusion models
  pre-trained on billions of images. Our exhaustive evaluation metrics
  demonstrate that DICEPTION effectively tackles multiple perception tasks,
  achieving performance on par with state-of-the-art models. We achieve results
  on par with SAM-vit-h using only 0.06% of their data (e.g., 600K vs. 1B
  pixel-level annotated images). Inspired by Wang et al., DICEPTION formulates
  the outputs of various perception tasks using color encoding; and we show that
  the strategy of assigning random colors to different instances is highly
  effective in both entity segmentation and semantic segmentation. Unifying
  various perception tasks as conditional image generation enables us to fully
  leverage pre-trained text-to-image models. Thus, DICEPTION can be efficiently
  trained at a cost of orders of magnitude lower, compared to conventional models
  that were trained from scratch. When adapting our model to other tasks, it only
  requires fine-tuning on as few as 50 images and 1% of its parameters. DICEPTION
  provides valuable insights and a more promising solution for visual generalist
  models.
paper_ai_summary: DICEPTION, a text-to-image diffusion model, achieves state-of-the-art performance in multiple perception tasks using minimal data and computational resources by leveraging color encoding and conditional image generation.
paper_ai_keywords: text-to-image diffusion models perception tasks color encoding conditional image generation