title: TIP: Text-Driven Image Processing with Semantic and Restoration
    Instructions
summary: Text-driven diffusion models have become increasingly popular for various
  image editing tasks, including inpainting, stylization, and object replacement.
  However, it still remains an open research problem to adopt this
  language-vision paradigm for more fine-level image processing tasks, such as
  denoising, super-resolution, deblurring, and compression artifact removal. In
  this paper, we develop TIP, a Text-driven Image Processing framework that
  leverages natural language as a user-friendly interface to control the image
  restoration process. We consider the capacity of text information in two
  dimensions. First, we use content-related prompts to enhance the semantic
  alignment, effectively alleviating identity ambiguity in the restoration
  outcomes. Second, our approach is the first framework that supports fine-level
  instruction through language-based quantitative specification of the
  restoration strength, without the need for explicit task-specific design. In
  addition, we introduce a novel fusion mechanism that augments the existing
  ControlNet architecture by learning to rescale the generative prior, thereby
  achieving better restoration fidelity. Our extensive experiments demonstrate
  the superior restoration performance of TIP compared to the state of the arts,
  alongside offering the flexibility of text-based control over the restoration
  effects.
paper_authors._id: 6582636b38f102154fd8559f
paper_authors.name: Chenyang Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:49:21.736Z
paper_authors._id: 642157a201c62c1e41fc526a
paper_authors.avatarUrl: /avatars/73ae831a73947a04468b1e87d17da24d.svg
paper_authors.fullname: chenyangqi
paper_authors.type: user
paper_authors.user: chenyangqi
paper_authors._id: 6582636b38f102154fd855a0
paper_authors.name: Zhengzhong Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:49:29.219Z
paper_authors._id: 62548d5fef3debb2ddf91217
paper_authors.avatarUrl: /avatars/14975b45568f9c399c92c3986b6ce83e.svg
paper_authors.fullname: Zhengzhong Tu
paper_authors.type: user
paper_authors.user: vztu
paper_authors._id: 6582636b38f102154fd855a1
paper_authors.name: Keren Ye
paper_authors._id: 6582636b38f102154fd855a2
paper_authors.name: Mauricio Delbracio
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:49:46.348Z
paper_authors._id: 651c262b497e4d326780c18a
paper_authors.avatarUrl: /avatars/65d50faaa22c5bbd982b5652a530da2c.svg
paper_authors.fullname: Mauricio Delbracio
paper_authors.type: user
paper_authors.user: mdelbra
paper_authors._id: 6582636b38f102154fd855a3
paper_authors.name: Peyman Milanfar
paper_authors._id: 6582636b38f102154fd855a4
paper_authors.name: Qifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:49:56.366Z
paper_authors._id: 6467b121e7a6a374fd19b44b
paper_authors.avatarUrl: /avatars/3f2874d58986d651aef55e3408b05700.svg
paper_authors.fullname: Qifeng Chen
paper_authors.type: user
paper_authors.user: cqf
paper_authors._id: 6582636b38f102154fd855a5
paper_authors.name: Hossein Talebi
paper_title: TIP: Text-Driven Image Processing with Semantic and Restoration
    Instructions
paper_summary: Text-driven diffusion models have become increasingly popular for various
  image editing tasks, including inpainting, stylization, and object replacement.
  However, it still remains an open research problem to adopt this
  language-vision paradigm for more fine-level image processing tasks, such as
  denoising, super-resolution, deblurring, and compression artifact removal. In
  this paper, we develop TIP, a Text-driven Image Processing framework that
  leverages natural language as a user-friendly interface to control the image
  restoration process. We consider the capacity of text information in two
  dimensions. First, we use content-related prompts to enhance the semantic
  alignment, effectively alleviating identity ambiguity in the restoration
  outcomes. Second, our approach is the first framework that supports fine-level
  instruction through language-based quantitative specification of the
  restoration strength, without the need for explicit task-specific design. In
  addition, we introduce a novel fusion mechanism that augments the existing
  ControlNet architecture by learning to rescale the generative prior, thereby
  achieving better restoration fidelity. Our extensive experiments demonstrate
  the superior restoration performance of TIP compared to the state of the arts,
  alongside offering the flexibility of text-based control over the restoration
  effects.
paper_ai_summary: TIP is a Text-driven Image Processing framework that uses natural language to control image restoration, offering superior performance and flexibility through semantic alignment and quantitative specification of restoration strength.
paper_ai_keywords: text-driven diffusion models inpainting stylization object replacement denoising super-resolution deblurring compression artifact removal semantic alignment restoration strength ControlNet generative prior restoration fidelity