title: PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language
    Instructions
summary: This paper presents a versatile image-to-image visual assistant, PixWizard,
  designed for image generation, manipulation, and translation based on free-from
  language instructions. To this end, we tackle a variety of vision tasks into a
  unified image-text-to-image generation framework and curate an Omni
  Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction
  templates in natural language, we comprehensively include a large set of
  diverse vision tasks such as text-to-image generation, image restoration, image
  grounding, dense image prediction, image editing, controllable generation,
  inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers
  (DiT) as our foundation model and extend its capabilities with a flexible any
  resolution mechanism, enabling the model to dynamically process images based on
  the aspect ratio of the input, closely aligning with human perceptual
  processes. The model also incorporates structure-aware and semantic-aware
  guidance to facilitate effective fusion of information from the input image.
  Our experiments demonstrate that PixWizard not only shows impressive generative
  and understanding abilities for images with diverse resolutions but also
  exhibits promising generalization capabilities with unseen tasks and human
  instructions. The code and related resources are available at
  https://github.com/AFeng-x/PixWizard
paper_authors._id: 66f25e35aef962b2379c5812
paper_authors.name: Weifeng Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:29.813Z
paper_authors._id: 66026c9068d519ed32519e9c
paper_authors.avatarUrl: /avatars/8fa051312c713772e5b8ba65989ff7f5.svg
paper_authors.fullname: Weifeng Lin
paper_authors.type: user
paper_authors.user: Afeng-x
paper_authors._id: 66f25e35aef962b2379c5813
paper_authors.name: Xinyu Wei
paper_authors._id: 66f25e35aef962b2379c5814
paper_authors.name: Renrui Zhang
paper_authors._id: 66f25e35aef962b2379c5815
paper_authors.name: Le Zhuo
paper_authors._id: 66f25e35aef962b2379c5816
paper_authors.name: Shitian Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:26.352Z
paper_authors._id: 62c66504031996c36c86976a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
paper_authors.fullname: steve z
paper_authors.type: user
paper_authors.user: stzhao
paper_authors._id: 66f25e35aef962b2379c5817
paper_authors.name: Siyuan Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:17.942Z
paper_authors._id: 634e4120038b5879133552f5
paper_authors.avatarUrl: /avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
paper_authors.fullname: Siyuan
paper_authors.type: user
paper_authors.user: SiyuanH
paper_authors._id: 66f25e35aef962b2379c5818
paper_authors.name: Junlin Xie
paper_authors._id: 66f25e35aef962b2379c5819
paper_authors.name: Yu Qiao
paper_authors._id: 66f25e35aef962b2379c581a
paper_authors.name: Peng Gao
paper_authors._id: 66f25e35aef962b2379c581b
paper_authors.name: Hongsheng Li
paper_title: PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language
    Instructions
paper_summary: This paper presents a versatile image-to-image visual assistant, PixWizard,
  designed for image generation, manipulation, and translation based on free-from
  language instructions. To this end, we tackle a variety of vision tasks into a
  unified image-text-to-image generation framework and curate an Omni
  Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction
  templates in natural language, we comprehensively include a large set of
  diverse vision tasks such as text-to-image generation, image restoration, image
  grounding, dense image prediction, image editing, controllable generation,
  inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers
  (DiT) as our foundation model and extend its capabilities with a flexible any
  resolution mechanism, enabling the model to dynamically process images based on
  the aspect ratio of the input, closely aligning with human perceptual
  processes. The model also incorporates structure-aware and semantic-aware
  guidance to facilitate effective fusion of information from the input image.
  Our experiments demonstrate that PixWizard not only shows impressive generative
  and understanding abilities for images with diverse resolutions but also
  exhibits promising generalization capabilities with unseen tasks and human
  instructions. The code and related resources are available at
  https://github.com/AFeng-x/PixWizard
paper_ai_keywords: PixWizard image-to-image visual assistant Omni Pixel-to-Pixel Instruction-Tuning Dataset text-to-image generation image restoration image grounding dense image prediction image editing controllable generation inpainting/outpainting Diffusion Transformers (DiT) flexible any resolution mechanism structure-aware guidance semantic-aware guidance generative and understanding abilities generalization capabilities