title: Negative Token Merging: Image-based Adversarial Feature Guidance
summary: Text-based adversarial guidance using a negative prompt has emerged as a
  widely adopted approach to push the output features away from undesired
  concepts. While useful, performing adversarial guidance using text alone can be
  insufficient to capture complex visual concepts and avoid undesired visual
  elements like copyrighted characters. In this paper, for the first time we
  explore an alternate modality in this direction by performing adversarial
  guidance directly using visual features from a reference image or other images
  in a batch. In particular, we introduce negative token merging (NegToMe), a
  simple but effective training-free approach which performs adversarial guidance
  by selectively pushing apart matching semantic features (between reference and
  output generation) during the reverse diffusion process. When used w.r.t. other
  images in the same batch, we observe that NegToMe significantly increases
  output diversity (racial, gender, visual) without sacrificing output image
  quality. Similarly, when used w.r.t. a reference copyrighted asset, NegToMe
  helps reduce visual similarity with copyrighted content by 34.57%. NegToMe is
  simple to implement using just few-lines of code, uses only marginally higher
  (<4%) inference times and generalizes to different diffusion architectures like
  Flux, which do not natively support the use of a separate negative prompt. Code
  is available at https://negtome.github.io
paper_authors._id: 674e7c274b7915defe769665
paper_authors.name: Jaskirat Singh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:19:25.770Z
paper_authors._id: 60cc389a0844fb1605fef405
paper_authors.avatarUrl: /avatars/ec11f85735e0525439e8821cf6d12e53.svg
paper_authors.fullname: Jaskirat Singh
paper_authors.type: user
paper_authors.user: jsingh
paper_authors._id: 674e7c274b7915defe769666
paper_authors.name: Lindsey Li
paper_authors._id: 674e7c274b7915defe769667
paper_authors.name: Weijia Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:20:41.627Z
paper_authors._id: 6400f2ed568dbe30c9161e47
paper_authors.avatarUrl: /avatars/c55938df5bce82b5d96e592a1ec36a8b.svg
paper_authors.fullname: Weijia Shi
paper_authors.type: user
paper_authors.user: swj0419
paper_authors._id: 674e7c274b7915defe769668
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:20:47.646Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_authors._id: 674e7c274b7915defe769669
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:20:53.798Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_authors._id: 674e7c274b7915defe76966a
paper_authors.name: Pang Wei Koh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:21:00.057Z
paper_authors._id: 641b4263abfce26bcf7b27de
paper_authors.avatarUrl: /avatars/e91b4205e4f74b0dd8c333c23203a924.svg
paper_authors.fullname: Pang Wei Koh
paper_authors.type: user
paper_authors.user: pangwei
paper_authors._id: 674e7c274b7915defe76966b
paper_authors.name: Michael F. Cohen
paper_authors._id: 674e7c274b7915defe76966c
paper_authors.name: Stephen Gould
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:21:29.537Z
paper_authors._id: 63114d9c1f7088709e49b44c
paper_authors.avatarUrl: /avatars/8515edaef9c3f46ccd4c1e2efa763c98.svg
paper_authors.fullname: Stephen Gould
paper_authors.type: user
paper_authors.user: sgould
paper_authors._id: 674e7c274b7915defe76966d
paper_authors.name: Liang Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:21:55.807Z
paper_authors._id: 666351ebd86c026caa135e5c
paper_authors.avatarUrl: /avatars/50a37f7e999f660c69f518b71577eb7d.svg
paper_authors.fullname: Liang Zheng
paper_authors.type: user
paper_authors.user: liangzheng06
paper_authors._id: 674e7c274b7915defe76966e
paper_authors.name: Luke Zettlemoyer
paper_title: Negative Token Merging: Image-based Adversarial Feature Guidance
paper_summary: Text-based adversarial guidance using a negative prompt has emerged as a
  widely adopted approach to push the output features away from undesired
  concepts. While useful, performing adversarial guidance using text alone can be
  insufficient to capture complex visual concepts and avoid undesired visual
  elements like copyrighted characters. In this paper, for the first time we
  explore an alternate modality in this direction by performing adversarial
  guidance directly using visual features from a reference image or other images
  in a batch. In particular, we introduce negative token merging (NegToMe), a
  simple but effective training-free approach which performs adversarial guidance
  by selectively pushing apart matching semantic features (between reference and
  output generation) during the reverse diffusion process. When used w.r.t. other
  images in the same batch, we observe that NegToMe significantly increases
  output diversity (racial, gender, visual) without sacrificing output image
  quality. Similarly, when used w.r.t. a reference copyrighted asset, NegToMe
  helps reduce visual similarity with copyrighted content by 34.57%. NegToMe is
  simple to implement using just few-lines of code, uses only marginally higher
  (<4%) inference times and generalizes to different diffusion architectures like
  Flux, which do not natively support the use of a separate negative prompt. Code
  is available at https://negtome.github.io
paper_ai_keywords: text-based adversarial guidance negative prompt adversarial guidance visual features reference image negative token merging (NegToMe) semantic features reverse diffusion process output image quality visual similarity Flux diffusion architectures