Text-Aware Image Restoration with Diffusion Models
Image restoration aims to recover degraded images. However, existing
diffusion-based restoration methods, despite great success in natural image
restoration, often struggle to faithfully reconstruct textual regions in
degraded images. Those methods frequently generate plausible but incorrect
text-like patterns, a phenomenon we refer to as text-image hallucination. In
this paper, we introduce Text-Aware Image Restoration (TAIR), a novel
restoration task that requires the simultaneous recovery of visual contents and
textual fidelity. To tackle this task, we present SA-Text, a large-scale
benchmark of 100K high-quality scene images densely annotated with diverse and
complex text instances. Furthermore, we propose a multi-task diffusion
framework, called TeReDiff, that integrates internal features from diffusion
models into a text-spotting module, enabling both components to benefit from
joint training. This allows for the extraction of rich text representations,
which are utilized as prompts in subsequent denoising steps. Extensive
experiments demonstrate that our approach consistently outperforms
state-of-the-art restoration methods, achieving significant gains in text
recognition accuracy. See our project page: https://cvlab-kaist.github.io/TAIR/
684ae204dbd21a9cc27b0fba
Jaewon Min
claimed_verified
2025-06-13T07:41:25.024Z
66012e9c9e1cf5eb41ee0c4c
/avatars/b07240cb86315b9e33d14677e02e4024.svg
Jaewon Min
user
Min-Jaewon
684ae204dbd21a9cc27b0fbb
Jin Hyeon Kim
claimed_verified
2025-06-13T07:41:22.776Z
65ec3449a69aaabb431db0da
/avatars/d7b507be0175a61a8fc21176eea45001.svg
Jin Hyeon Kim
user
jinlovespho
684ae204dbd21a9cc27b0fbc
Paul Hyunbin Cho
claimed_verified
2025-06-13T07:41:20.327Z
6752b6315281c3cae4b0783f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/xmcyVEl2xBhk3G5_7dmpz.png
Paul Hyunbin Cho
user
paulcho98
684ae204dbd21a9cc27b0fbd
Jaeeun Lee
claimed_verified
2025-06-13T14:59:38.559Z
644be3e922d211df644416e9
/avatars/6bebbda7a7c16992eba64dc489eaeca5.svg
Jaeeun Lee
user
alicia10
684ae204dbd21a9cc27b0fbe
Jihye Park
684ae204dbd21a9cc27b0fbf
Minkyu Park
684ae204dbd21a9cc27b0fc0
Sangpil Kim
684ae204dbd21a9cc27b0fc1
Hyunhee Park
684ae204dbd21a9cc27b0fc2
Seungryong Kim
Text-Aware Image Restoration with Diffusion Models
Image restoration aims to recover degraded images. However, existing
diffusion-based restoration methods, despite great success in natural image
restoration, often struggle to faithfully reconstruct textual regions in
degraded images. Those methods frequently generate plausible but incorrect
text-like patterns, a phenomenon we refer to as text-image hallucination. In
this paper, we introduce Text-Aware Image Restoration (TAIR), a novel
restoration task that requires the simultaneous recovery of visual contents and
textual fidelity. To tackle this task, we present SA-Text, a large-scale
benchmark of 100K high-quality scene images densely annotated with diverse and
complex text instances. Furthermore, we propose a multi-task diffusion
framework, called TeReDiff, that integrates internal features from diffusion
models into a text-spotting module, enabling both components to benefit from
joint training. This allows for the extraction of rich text representations,
which are utilized as prompts in subsequent denoising steps. Extensive
experiments demonstrate that our approach consistently outperforms
state-of-the-art restoration methods, achieving significant gains in text
recognition accuracy. See our project page: https://cvlab-kaist.github.io/TAIR/
The proposed Text-Aware Image Restoration (TAIR) system integrates a multi-task diffusion framework with a text-spotting module to enhance both image recovery and textual fidelity, outperforming existing diffusion-based methods.
diffusion-based restoration text-image hallucination Text-Aware Image Restoration (TAIR) SA-Text multi-task diffusion framework TeReDiff text-spotting module text recognition accuracy