InstructDiffusion: A Generalist Modeling Interface for Vision Tasks
We present InstructDiffusion, a unifying and generic framework for aligning
computer vision tasks with human instructions. Unlike existing approaches that
integrate prior knowledge and pre-define the output space (e.g., categories and
coordinates) for each vision task, we cast diverse vision tasks into a
human-intuitive image-manipulating process whose output space is a flexible and
interactive pixel space. Concretely, the model is built upon the diffusion
process and is trained to predict pixels according to user instructions, such
as encircling the man's left shoulder in red or applying a blue mask to the
left car. InstructDiffusion could handle a variety of vision tasks, including
understanding tasks (such as segmentation and keypoint detection) and
generative tasks (such as editing and enhancement). It even exhibits the
ability to handle unseen tasks and outperforms prior methods on novel datasets.
This represents a significant step towards a generalist modeling interface for
vision tasks, advancing artificial general intelligence in the field of
computer vision.
64faae404c8924c4fe615cd1
Zigang Geng
64faae404c8924c4fe615cd2
Binxin Yang
admin_assigned
2023-09-08T12:03:20.824Z
6323401d75bf010a73d97595
/avatars/0dcc991c4f45d3d910947928ea8f8824.svg
Binxin Yang
user
tennyson
64faae404c8924c4fe615cd3
Tiankai Hang
64faae404c8924c4fe615cd4
Chen Li
admin_assigned
2023-09-08T12:09:01.055Z
64e0debc02fa032de402feb0
/avatars/6c28a13216f7ae4c047cf6ea9846f6e3.svg
Chen Li
user
LC-Edward
64faae404c8924c4fe615cd5
Shuyang Gu
claimed_verified
2024-07-10T08:21:50.193Z
64c38fcf573c5a427e12cd37
/avatars/2b9de06f29147ed2c212e920afba0eaf.svg
cientgu
user
cientgu
64faae404c8924c4fe615cd6
Ting Zhang
claimed_verified
2023-09-11T06:53:23.453Z
6350ffa18445bbe32e926360
/avatars/bc67d114191be7b16b463731a84c7e05.svg
Ting Zhang
user
hellozt
64faae404c8924c4fe615cd7
Jianmin Bao
admin_assigned
2023-09-08T12:06:09.380Z
6481e690f9ed842838a2b106
/avatars/e89a3c8366df504a95dc08a1a412bf3d.svg
Jianmin Bao
user
jianmin-ustc
64faae404c8924c4fe615cd8
Zheng Zhang
64faae404c8924c4fe615cd9
Han Hu
64faae404c8924c4fe615cda
Dong Chen
64faae404c8924c4fe615cdb
Baining Guo
InstructDiffusion: A Generalist Modeling Interface for Vision Tasks
We present InstructDiffusion, a unifying and generic framework for aligning
computer vision tasks with human instructions. Unlike existing approaches that
integrate prior knowledge and pre-define the output space (e.g., categories and
coordinates) for each vision task, we cast diverse vision tasks into a
human-intuitive image-manipulating process whose output space is a flexible and
interactive pixel space. Concretely, the model is built upon the diffusion
process and is trained to predict pixels according to user instructions, such
as encircling the man's left shoulder in red or applying a blue mask to the
left car. InstructDiffusion could handle a variety of vision tasks, including
understanding tasks (such as segmentation and keypoint detection) and
generative tasks (such as editing and enhancement). It even exhibits the
ability to handle unseen tasks and outperforms prior methods on novel datasets.
This represents a significant step towards a generalist modeling interface for
vision tasks, advancing artificial general intelligence in the field of
computer vision.
InstructDiffusion is a diffusion-based framework that interprets human instructions to manipulate images, achieving versatility across vision tasks and novel datasets.
InstructDiffusion diffusion process image-manipulating process pixel space segmentation keypoint detection editing enhancement