title: InstructDiffusion: A Generalist Modeling Interface for Vision Tasks
summary: We present InstructDiffusion, a unifying and generic framework for aligning
  computer vision tasks with human instructions. Unlike existing approaches that
  integrate prior knowledge and pre-define the output space (e.g., categories and
  coordinates) for each vision task, we cast diverse vision tasks into a
  human-intuitive image-manipulating process whose output space is a flexible and
  interactive pixel space. Concretely, the model is built upon the diffusion
  process and is trained to predict pixels according to user instructions, such
  as encircling the man's left shoulder in red or applying a blue mask to the
  left car. InstructDiffusion could handle a variety of vision tasks, including
  understanding tasks (such as segmentation and keypoint detection) and
  generative tasks (such as editing and enhancement). It even exhibits the
  ability to handle unseen tasks and outperforms prior methods on novel datasets.
  This represents a significant step towards a generalist modeling interface for
  vision tasks, advancing artificial general intelligence in the field of
  computer vision.
paper_authors._id: 64faae404c8924c4fe615cd1
paper_authors.name: Zigang Geng
paper_authors._id: 64faae404c8924c4fe615cd2
paper_authors.name: Binxin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T12:03:20.824Z
paper_authors._id: 6323401d75bf010a73d97595
paper_authors.avatarUrl: /avatars/0dcc991c4f45d3d910947928ea8f8824.svg
paper_authors.fullname: Binxin Yang
paper_authors.type: user
paper_authors.user: tennyson
paper_authors._id: 64faae404c8924c4fe615cd3
paper_authors.name: Tiankai Hang
paper_authors._id: 64faae404c8924c4fe615cd4
paper_authors.name: Chen Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T12:09:01.055Z
paper_authors._id: 64e0debc02fa032de402feb0
paper_authors.avatarUrl: /avatars/6c28a13216f7ae4c047cf6ea9846f6e3.svg
paper_authors.fullname: Chen Li
paper_authors.type: user
paper_authors.user: LC-Edward
paper_authors._id: 64faae404c8924c4fe615cd5
paper_authors.name: Shuyang Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T08:21:50.193Z
paper_authors._id: 64c38fcf573c5a427e12cd37
paper_authors.avatarUrl: /avatars/2b9de06f29147ed2c212e920afba0eaf.svg
paper_authors.fullname: cientgu
paper_authors.type: user
paper_authors.user: cientgu
paper_authors._id: 64faae404c8924c4fe615cd6
paper_authors.name: Ting Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-11T06:53:23.453Z
paper_authors._id: 6350ffa18445bbe32e926360
paper_authors.avatarUrl: /avatars/bc67d114191be7b16b463731a84c7e05.svg
paper_authors.fullname: Ting Zhang
paper_authors.type: user
paper_authors.user: hellozt
paper_authors._id: 64faae404c8924c4fe615cd7
paper_authors.name: Jianmin Bao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T12:06:09.380Z
paper_authors._id: 6481e690f9ed842838a2b106
paper_authors.avatarUrl: /avatars/e89a3c8366df504a95dc08a1a412bf3d.svg
paper_authors.fullname: Jianmin Bao
paper_authors.type: user
paper_authors.user: jianmin-ustc
paper_authors._id: 64faae404c8924c4fe615cd8
paper_authors.name: Zheng Zhang
paper_authors._id: 64faae404c8924c4fe615cd9
paper_authors.name: Han Hu
paper_authors._id: 64faae404c8924c4fe615cda
paper_authors.name: Dong Chen
paper_authors._id: 64faae404c8924c4fe615cdb
paper_authors.name: Baining Guo
paper_title: InstructDiffusion: A Generalist Modeling Interface for Vision Tasks
paper_summary: We present InstructDiffusion, a unifying and generic framework for aligning
  computer vision tasks with human instructions. Unlike existing approaches that
  integrate prior knowledge and pre-define the output space (e.g., categories and
  coordinates) for each vision task, we cast diverse vision tasks into a
  human-intuitive image-manipulating process whose output space is a flexible and
  interactive pixel space. Concretely, the model is built upon the diffusion
  process and is trained to predict pixels according to user instructions, such
  as encircling the man's left shoulder in red or applying a blue mask to the
  left car. InstructDiffusion could handle a variety of vision tasks, including
  understanding tasks (such as segmentation and keypoint detection) and
  generative tasks (such as editing and enhancement). It even exhibits the
  ability to handle unseen tasks and outperforms prior methods on novel datasets.
  This represents a significant step towards a generalist modeling interface for
  vision tasks, advancing artificial general intelligence in the field of
  computer vision.
paper_ai_summary: InstructDiffusion is a diffusion-based framework that interprets human instructions to manipulate images, achieving versatility across vision tasks and novel datasets.
paper_ai_keywords: InstructDiffusion diffusion process image-manipulating process pixel space segmentation keypoint detection editing enhancement