title: PixelHacker: Image Inpainting with Structural and Semantic Consistency
summary: Image inpainting is a fundamental research area between image editing and
  image generation. Recent state-of-the-art (SOTA) methods have explored novel
  attention mechanisms, lightweight architectures, and context-aware modeling,
  demonstrating impressive performance. However, they often struggle with complex
  structure (e.g., texture, shape, spatial relations) and semantics (e.g., color
  consistency, object restoration, and logical correctness), leading to artifacts
  and inappropriate generation. To address this challenge, we design a simple yet
  effective inpainting paradigm called latent categories guidance, and further
  propose a diffusion-based model named PixelHacker. Specifically, we first
  construct a large dataset containing 14 million image-mask pairs by annotating
  foreground and background (potential 116 and 21 categories, respectively).
  Then, we encode potential foreground and background representations separately
  through two fixed-size embeddings, and intermittently inject these features
  into the denoising process via linear attention. Finally, by pre-training on
  our dataset and fine-tuning on open-source benchmarks, we obtain PixelHacker.
  Extensive experiments show that PixelHacker comprehensively outperforms the
  SOTA on a wide range of datasets (Places2, CelebA-HQ, and FFHQ) and exhibits
  remarkable consistency in both structure and semantics. Project page at
  https://hustvl.github.io/PixelHacker.
paper_authors._id: 6814e35a19162d7749852c4b
paper_authors.name: Ziyang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-04T10:04:40.638Z
paper_authors._id: 633d4630e1aec4b8b33ad5b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633d4630e1aec4b8b33ad5b8/owFqnyCMW8yuo9VcjA1hx.jpeg
paper_authors.fullname: Ziyang Xu
paper_authors.type: user
paper_authors.user: Uyoung
paper_authors._id: 6814e35a19162d7749852c4c
paper_authors.name: Kangsheng Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T16:53:10.183Z
paper_authors._id: 66d0ba2420e64fc408564bbb
paper_authors.avatarUrl: /avatars/355566cfb6347750a5faab7c87fcb79c.svg
paper_authors.fullname: Anduin Dan
paper_authors.type: user
paper_authors.user: AnduinD
paper_authors._id: 6814e35a19162d7749852c4d
paper_authors.name: Xiaolei Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:32:16.185Z
paper_authors._id: 627a34dac488a8ce15a2dc4a
paper_authors.avatarUrl: /avatars/61aecef507dea6620fe5574493f83595.svg
paper_authors.fullname: ShenXiaolei
paper_authors.type: user
paper_authors.user: SmileTAT
paper_authors._id: 6814e35a19162d7749852c4e
paper_authors.name: Zhifeng Ding
paper_authors._id: 6814e35a19162d7749852c4f
paper_authors.name: Wenyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:32:31.210Z
paper_authors._id: 66c2e7fc934e2f07753542ac
paper_authors.avatarUrl: /avatars/f6fa3f94435cf1c1d06daa6c925d07d0.svg
paper_authors.fullname: LWY
paper_authors.type: user
paper_authors.user: wenyuliu
paper_authors._id: 6814e35a19162d7749852c50
paper_authors.name: Xiaohu Ruan
paper_authors._id: 6814e35a19162d7749852c51
paper_authors.name: Xiaoxin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:32:55.012Z
paper_authors._id: 65389a669c474315d7425f96
paper_authors.avatarUrl: /avatars/2fa3828ca489cfe1948129a0eccf264f.svg
paper_authors.fullname: chenxiaoxin
paper_authors.type: user
paper_authors.user: steelozazala
paper_authors._id: 6814e35a19162d7749852c52
paper_authors.name: Xinggang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T07:33:01.396Z
paper_authors._id: 62600de6d47e3dbae32ce1ce
paper_authors.avatarUrl: /avatars/a536417cfec6e10ac415091bd1829426.svg
paper_authors.fullname: Xinggang Wang
paper_authors.type: user
paper_authors.user: xinggangw
paper_title: PixelHacker: Image Inpainting with Structural and Semantic Consistency
paper_summary: Image inpainting is a fundamental research area between image editing and
  image generation. Recent state-of-the-art (SOTA) methods have explored novel
  attention mechanisms, lightweight architectures, and context-aware modeling,
  demonstrating impressive performance. However, they often struggle with complex
  structure (e.g., texture, shape, spatial relations) and semantics (e.g., color
  consistency, object restoration, and logical correctness), leading to artifacts
  and inappropriate generation. To address this challenge, we design a simple yet
  effective inpainting paradigm called latent categories guidance, and further
  propose a diffusion-based model named PixelHacker. Specifically, we first
  construct a large dataset containing 14 million image-mask pairs by annotating
  foreground and background (potential 116 and 21 categories, respectively).
  Then, we encode potential foreground and background representations separately
  through two fixed-size embeddings, and intermittently inject these features
  into the denoising process via linear attention. Finally, by pre-training on
  our dataset and fine-tuning on open-source benchmarks, we obtain PixelHacker.
  Extensive experiments show that PixelHacker comprehensively outperforms the
  SOTA on a wide range of datasets (Places2, CelebA-HQ, and FFHQ) and exhibits
  remarkable consistency in both structure and semantics. Project page at
  https://hustvl.github.io/PixelHacker.
paper_ai_summary: PixelHacker, a diffusion-based model using latent categories guidance and linear attention, enhances image inpainting by improving structural and semantic consistency across various datasets.
paper_ai_keywords: latent categories guidance diffusion-based model PixelHacker linear attention image-mask pairs fixed-size embeddings denoising process Places2 CelebA-HQ FFHQ