title: VLMs Can Aggregate Scattered Training Patches
summary: One way to mitigate risks in vision-language models (VLMs) is to remove
  dangerous samples in their training data. However, such data moderation can be
  easily bypassed when harmful images are split into small, benign-looking
  patches, scattered across many training samples. VLMs may then learn to piece
  these fragments together during training and generate harmful responses at
  inference, either from full images or text references. For instance, if trained
  on image patches from a bloody scene paired with the descriptions "safe," VLMs
  may later describe, the full image or a text reference to the scene, as "safe."
  We define the core ability of VLMs enabling this attack as visual
  stitching -- the ability to integrate visual information spread across
  multiple training samples that share the same textual descriptions. In our
  work, we first demonstrate visual stitching abilities in common open-source
  VLMs on three datasets where each image is labeled with a unique synthetic ID:
  we split each (image, ID) pair into {(patch,
  ID)} pairs at different granularity for finetuning, and we find that
  tuned models can verbalize the correct IDs from full images or text reference.
  Building on this, we simulate the adversarial data poisoning scenario mentioned
  above by using patches from dangerous images and replacing IDs with text
  descriptions like ``safe'' or ``unsafe'', demonstrating how harmful content can
  evade moderation in patches and later be reconstructed through visual
  stitching, posing serious VLM safety risks. Code is available at
  https://github.com/ZHZisZZ/visual-stitching.
paper_authors._id: 684134ca20ff8abcccb11302
paper_authors.name: Zhanhui Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T13:01:32.367Z
paper_authors._id: 642e5a7ba0b65dce1f87a7a2
paper_authors.avatarUrl: /avatars/3ae01c9330a47e98fac9f1eb0ba94073.svg
paper_authors.fullname: Zhanhui Zhou
paper_authors.type: user
paper_authors.user: ZHZisZZ
paper_authors._id: 684134ca20ff8abcccb11303
paper_authors.name: Lingjie Chen
paper_authors._id: 684134ca20ff8abcccb11304
paper_authors.name: Chao Yang
paper_authors._id: 684134ca20ff8abcccb11305
paper_authors.name: Chaochao Lu
paper_title: VLMs Can Aggregate Scattered Training Patches
paper_summary: One way to mitigate risks in vision-language models (VLMs) is to remove
  dangerous samples in their training data. However, such data moderation can be
  easily bypassed when harmful images are split into small, benign-looking
  patches, scattered across many training samples. VLMs may then learn to piece
  these fragments together during training and generate harmful responses at
  inference, either from full images or text references. For instance, if trained
  on image patches from a bloody scene paired with the descriptions "safe," VLMs
  may later describe, the full image or a text reference to the scene, as "safe."
  We define the core ability of VLMs enabling this attack as visual
  stitching -- the ability to integrate visual information spread across
  multiple training samples that share the same textual descriptions. In our
  work, we first demonstrate visual stitching abilities in common open-source
  VLMs on three datasets where each image is labeled with a unique synthetic ID:
  we split each (image, ID) pair into {(patch,
  ID)} pairs at different granularity for finetuning, and we find that
  tuned models can verbalize the correct IDs from full images or text reference.
  Building on this, we simulate the adversarial data poisoning scenario mentioned
  above by using patches from dangerous images and replacing IDs with text
  descriptions like ``safe'' or ``unsafe'', demonstrating how harmful content can
  evade moderation in patches and later be reconstructed through visual
  stitching, posing serious VLM safety risks. Code is available at
  https://github.com/ZHZisZZ/visual-stitching.
paper_ai_summary: VLMs exhibit visual stitching, an ability to integrate fragmented visual information, which enables harmful content to evade data moderation and be reconstructed during inference.
paper_ai_keywords: vision-language models VLMs visual stitching data moderation adversarial data poisoning image patches textual descriptions inference