PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language
  Instructions
This paper presents a versatile image-to-image visual assistant, PixWizard,
designed for image generation, manipulation, and translation based on free-from
language instructions. To this end, we tackle a variety of vision tasks into a
unified image-text-to-image generation framework and curate an Omni
Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction
templates in natural language, we comprehensively include a large set of
diverse vision tasks such as text-to-image generation, image restoration, image
grounding, dense image prediction, image editing, controllable generation,
inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers
(DiT) as our foundation model and extend its capabilities with a flexible any
resolution mechanism, enabling the model to dynamically process images based on
the aspect ratio of the input, closely aligning with human perceptual
processes. The model also incorporates structure-aware and semantic-aware
guidance to facilitate effective fusion of information from the input image.
Our experiments demonstrate that PixWizard not only shows impressive generative
and understanding abilities for images with diverse resolutions but also
exhibits promising generalization capabilities with unseen tasks and human
instructions. The code and related resources are available at
https://github.com/AFeng-x/PixWizard
66f25e35aef962b2379c5812
Weifeng Lin
claimed_verified
2024-09-24T07:52:29.813Z
66026c9068d519ed32519e9c
/avatars/8fa051312c713772e5b8ba65989ff7f5.svg
Weifeng Lin
user
Afeng-x
66f25e35aef962b2379c5813
Xinyu Wei
66f25e35aef962b2379c5814
Renrui Zhang
66f25e35aef962b2379c5815
Le Zhuo
66f25e35aef962b2379c5816
Shitian Zhao
claimed_verified
2024-09-24T07:52:26.352Z
62c66504031996c36c86976a
https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
steve z
user
stzhao
66f25e35aef962b2379c5817
Siyuan Huang
claimed_verified
2024-09-24T07:52:17.942Z
634e4120038b5879133552f5
/avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
Siyuan
user
SiyuanH
66f25e35aef962b2379c5818
Junlin Xie
66f25e35aef962b2379c5819
Yu Qiao
66f25e35aef962b2379c581a
Peng Gao
66f25e35aef962b2379c581b
Hongsheng Li
PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language
  Instructions
This paper presents a versatile image-to-image visual assistant, PixWizard,
designed for image generation, manipulation, and translation based on free-from
language instructions. To this end, we tackle a variety of vision tasks into a
unified image-text-to-image generation framework and curate an Omni
Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction
templates in natural language, we comprehensively include a large set of
diverse vision tasks such as text-to-image generation, image restoration, image
grounding, dense image prediction, image editing, controllable generation,
inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers
(DiT) as our foundation model and extend its capabilities with a flexible any
resolution mechanism, enabling the model to dynamically process images based on
the aspect ratio of the input, closely aligning with human perceptual
processes. The model also incorporates structure-aware and semantic-aware
guidance to facilitate effective fusion of information from the input image.
Our experiments demonstrate that PixWizard not only shows impressive generative
and understanding abilities for images with diverse resolutions but also
exhibits promising generalization capabilities with unseen tasks and human
instructions. The code and related resources are available at
https://github.com/AFeng-x/PixWizard
PixWizard, a versatile visual assistant, uses Diffusion Transformers to perform diverse image-to-image tasks with flexible resolution handling and instruction-based guidance.
Diffusion Transformers any resolution mechanism structure-aware semantic-aware