TIP: Text-Driven Image Processing with Semantic and Restoration
  Instructions
Text-driven diffusion models have become increasingly popular for various
image editing tasks, including inpainting, stylization, and object replacement.
However, it still remains an open research problem to adopt this
language-vision paradigm for more fine-level image processing tasks, such as
denoising, super-resolution, deblurring, and compression artifact removal. In
this paper, we develop TIP, a Text-driven Image Processing framework that
leverages natural language as a user-friendly interface to control the image
restoration process. We consider the capacity of text information in two
dimensions. First, we use content-related prompts to enhance the semantic
alignment, effectively alleviating identity ambiguity in the restoration
outcomes. Second, our approach is the first framework that supports fine-level
instruction through language-based quantitative specification of the
restoration strength, without the need for explicit task-specific design. In
addition, we introduce a novel fusion mechanism that augments the existing
ControlNet architecture by learning to rescale the generative prior, thereby
achieving better restoration fidelity. Our extensive experiments demonstrate
the superior restoration performance of TIP compared to the state of the arts,
alongside offering the flexibility of text-based control over the restoration
effects.
6582636b38f102154fd8559f
Chenyang Qi
admin_assigned
2023-12-20T09:49:21.736Z
642157a201c62c1e41fc526a
/avatars/73ae831a73947a04468b1e87d17da24d.svg
chenyangqi
user
chenyangqi
6582636b38f102154fd855a0
Zhengzhong Tu
admin_assigned
2023-12-20T09:49:29.219Z
62548d5fef3debb2ddf91217
/avatars/14975b45568f9c399c92c3986b6ce83e.svg
Zhengzhong Tu
user
vztu
6582636b38f102154fd855a1
Keren Ye
6582636b38f102154fd855a2
Mauricio Delbracio
admin_assigned
2023-12-20T09:49:46.348Z
651c262b497e4d326780c18a
/avatars/65d50faaa22c5bbd982b5652a530da2c.svg
Mauricio Delbracio
user
mdelbra
6582636b38f102154fd855a3
Peyman Milanfar
6582636b38f102154fd855a4
Qifeng Chen
admin_assigned
2023-12-20T09:49:56.366Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
6582636b38f102154fd855a5
Hossein Talebi
TIP: Text-Driven Image Processing with Semantic and Restoration
  Instructions
Text-driven diffusion models have become increasingly popular for various
image editing tasks, including inpainting, stylization, and object replacement.
However, it still remains an open research problem to adopt this
language-vision paradigm for more fine-level image processing tasks, such as
denoising, super-resolution, deblurring, and compression artifact removal. In
this paper, we develop TIP, a Text-driven Image Processing framework that
leverages natural language as a user-friendly interface to control the image
restoration process. We consider the capacity of text information in two
dimensions. First, we use content-related prompts to enhance the semantic
alignment, effectively alleviating identity ambiguity in the restoration
outcomes. Second, our approach is the first framework that supports fine-level
instruction through language-based quantitative specification of the
restoration strength, without the need for explicit task-specific design. In
addition, we introduce a novel fusion mechanism that augments the existing
ControlNet architecture by learning to rescale the generative prior, thereby
achieving better restoration fidelity. Our extensive experiments demonstrate
the superior restoration performance of TIP compared to the state of the arts,
alongside offering the flexibility of text-based control over the restoration
effects.
TIP is a Text-driven Image Processing framework that uses natural language to control image restoration, offering superior performance and flexibility through semantic alignment and quantitative specification of restoration strength.
text-driven diffusion models inpainting stylization object replacement denoising super-resolution deblurring compression artifact removal semantic alignment restoration strength ControlNet generative prior restoration fidelity