Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models
Text-to-image (T2I) diffusion models rely on encoded prompts to guide the
image generation process. Typically, these prompts are extended to a fixed
length by adding padding tokens before text encoding. Despite being a default
practice, the influence of padding tokens on the image generation process has
not been investigated. In this work, we conduct the first in-depth analysis of
the role padding tokens play in T2I models. We develop two causal techniques to
analyze how information is encoded in the representation of tokens across
different components of the T2I pipeline. Using these techniques, we
investigate when and how padding tokens impact the image generation process.
Our findings reveal three distinct scenarios: padding tokens may affect the
model's output during text encoding, during the diffusion process, or be
effectively ignored. Moreover, we identify key relationships between these
scenarios and the model's architecture (cross or self-attention) and its
training process (frozen or trained text encoder). These insights contribute to
a deeper understanding of the mechanisms of padding tokens, potentially
informing future model design and training practices in T2I systems.
67876c0783ec692b4570e0af
Michael Toker
admin_assigned
2025-01-15T16:45:04.686Z
61c865e3d3702a3bbf50bc04
https://cdn-avatars.huggingface.co/v1/production/uploads/61c865e3d3702a3bbf50bc04/eGgcyeUnOHz0hyap5vhvr.jpeg
Michael Toker
user
tokeron
67876c0783ec692b4570e0b0
Ido Galil
admin_assigned
2025-01-15T16:45:37.404Z
66867535e44c9931f329ca43
/avatars/ec6fc05b9f596f1ce38ec061d3d1d230.svg
Ido Galil
user
IdoGalilNvidia
67876c0783ec692b4570e0b1
Hadas Orgad
admin_assigned
2025-01-15T16:45:44.092Z
60f82853c53e95176a7c6d45
/avatars/16f6ea944a014af6ebe60499f3460784.svg
Hadas Orgad
user
hadasor
67876c0783ec692b4570e0b2
Rinon Gal
admin_assigned
2025-01-15T16:45:49.560Z
627c1360f19c5eb46d55ba05
https://cdn-avatars.huggingface.co/v1/production/uploads/1652712871747-627c1360f19c5eb46d55ba05.jpeg
Rinon Gal
user
rinong
67876c0783ec692b4570e0b3
Yoad Tewel
admin_assigned
2025-01-15T16:46:02.818Z
6201838f305742efd5a6a0e3
/avatars/4e47ab9ef1a4b9f59756a6d8c90a970f.svg
Yoad Tewel
user
YoadTew
67876c0783ec692b4570e0b4
Gal Chechik
admin_assigned
2025-01-15T16:46:09.446Z
6493393f357b252af72196c5
https://cdn-avatars.huggingface.co/v1/production/uploads/6493393f357b252af72196c5/EWSy18XRcMRa_4XMM3Fu-.jpeg
Gal Chechik
user
galchechik
67876c0783ec692b4570e0b5
Yonatan Belinkov
admin_assigned
2025-01-15T16:46:15.360Z
614c57f1ee44bcfe57b366d6
/avatars/186a9aed84681246f48ed2a012c50def.svg
Yonatan Belinkov
user
belinkov
Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models
Text-to-image (T2I) diffusion models rely on encoded prompts to guide the
image generation process. Typically, these prompts are extended to a fixed
length by adding padding tokens before text encoding. Despite being a default
practice, the influence of padding tokens on the image generation process has
not been investigated. In this work, we conduct the first in-depth analysis of
the role padding tokens play in T2I models. We develop two causal techniques to
analyze how information is encoded in the representation of tokens across
different components of the T2I pipeline. Using these techniques, we
investigate when and how padding tokens impact the image generation process.
Our findings reveal three distinct scenarios: padding tokens may affect the
model's output during text encoding, during the diffusion process, or be
effectively ignored. Moreover, we identify key relationships between these
scenarios and the model's architecture (cross or self-attention) and its
training process (frozen or trained text encoder). These insights contribute to
a deeper understanding of the mechanisms of padding tokens, potentially
informing future model design and training practices in T2I systems.
An analysis of padding tokens' influence in T2I diffusion models identifies scenarios where they impact image generation and relationships with architecture and training process.
diffusion models text-to-image (T2I) padding tokens causal techniques token representation text encoding diffusion process cross-attention self-attention text encoder