Negative Token Merging: Image-based Adversarial Feature Guidance
Text-based adversarial guidance using a negative prompt has emerged as a
widely adopted approach to push the output features away from undesired
concepts. While useful, performing adversarial guidance using text alone can be
insufficient to capture complex visual concepts and avoid undesired visual
elements like copyrighted characters. In this paper, for the first time we
explore an alternate modality in this direction by performing adversarial
guidance directly using visual features from a reference image or other images
in a batch. In particular, we introduce negative token merging (NegToMe), a
simple but effective training-free approach which performs adversarial guidance
by selectively pushing apart matching semantic features (between reference and
output generation) during the reverse diffusion process. When used w.r.t. other
images in the same batch, we observe that NegToMe significantly increases
output diversity (racial, gender, visual) without sacrificing output image
quality. Similarly, when used w.r.t. a reference copyrighted asset, NegToMe
helps reduce visual similarity with copyrighted content by 34.57%. NegToMe is
simple to implement using just few-lines of code, uses only marginally higher
(<4%) inference times and generalizes to different diffusion architectures like
Flux, which do not natively support the use of a separate negative prompt. Code
is available at https://negtome.github.io
674e7c274b7915defe769665
Jaskirat Singh
admin_assigned
2024-12-06T17:19:25.770Z
60cc389a0844fb1605fef405
/avatars/ec11f85735e0525439e8821cf6d12e53.svg
Jaskirat Singh
user
jsingh
674e7c274b7915defe769666
Lindsey Li
674e7c274b7915defe769667
Weijia Shi
admin_assigned
2024-12-06T17:20:41.627Z
6400f2ed568dbe30c9161e47
/avatars/c55938df5bce82b5d96e592a1ec36a8b.svg
Weijia Shi
user
swj0419
674e7c274b7915defe769668
Ranjay Krishna
admin_assigned
2024-12-06T17:20:47.646Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
674e7c274b7915defe769669
Yejin Choi
admin_assigned
2024-12-06T17:20:53.798Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
674e7c274b7915defe76966a
Pang Wei Koh
admin_assigned
2024-12-06T17:21:00.057Z
641b4263abfce26bcf7b27de
/avatars/e91b4205e4f74b0dd8c333c23203a924.svg
Pang Wei Koh
user
pangwei
674e7c274b7915defe76966b
Michael F. Cohen
674e7c274b7915defe76966c
Stephen Gould
admin_assigned
2024-12-06T17:21:29.537Z
63114d9c1f7088709e49b44c
/avatars/8515edaef9c3f46ccd4c1e2efa763c98.svg
Stephen Gould
user
sgould
674e7c274b7915defe76966d
Liang Zheng
admin_assigned
2024-12-06T17:21:55.807Z
666351ebd86c026caa135e5c
/avatars/50a37f7e999f660c69f518b71577eb7d.svg
Liang Zheng
user
liangzheng06
674e7c274b7915defe76966e
Luke Zettlemoyer
Negative Token Merging: Image-based Adversarial Feature Guidance
Text-based adversarial guidance using a negative prompt has emerged as a
widely adopted approach to push the output features away from undesired
concepts. While useful, performing adversarial guidance using text alone can be
insufficient to capture complex visual concepts and avoid undesired visual
elements like copyrighted characters. In this paper, for the first time we
explore an alternate modality in this direction by performing adversarial
guidance directly using visual features from a reference image or other images
in a batch. In particular, we introduce negative token merging (NegToMe), a
simple but effective training-free approach which performs adversarial guidance
by selectively pushing apart matching semantic features (between reference and
output generation) during the reverse diffusion process. When used w.r.t. other
images in the same batch, we observe that NegToMe significantly increases
output diversity (racial, gender, visual) without sacrificing output image
quality. Similarly, when used w.r.t. a reference copyrighted asset, NegToMe
helps reduce visual similarity with copyrighted content by 34.57%. NegToMe is
simple to implement using just few-lines of code, uses only marginally higher
(<4%) inference times and generalizes to different diffusion architectures like
Flux, which do not natively support the use of a separate negative prompt. Code
is available at https://negtome.github.io
Adversarial guidance using visual features from reference images enhances diversity and reduces visual similarity to copyrighted assets during reverse diffusion processes.
negative prompt adversarial guidance visual features reference image negative token merging NegToMe reverse diffusion process output diversity output image quality Flux