FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion
  Model
Nature is infinitely resolution-free. In the context of this
reality, existing diffusion models, such as Diffusion Transformers, often face
challenges when processing image resolutions outside of their trained domain.
To address this limitation, we conceptualize images as sequences of tokens with
dynamic sizes, rather than traditional methods that perceive images as
fixed-resolution grids. This perspective enables a flexible training strategy
that seamlessly accommodates various aspect ratios during both training and
inference, thus promoting resolution generalization and eliminating biases
introduced by image cropping. On this basis, we present the Flexible
Vision Transformer (FiT), a transformer architecture specifically designed for
generating images with unrestricted resolutions and aspect ratios. We
further upgrade the FiT to FiTv2 with several innovative designs, includingthe
Query-Key vector normalization, the AdaLN-LoRA module, a rectified flow
scheduler, and a Logit-Normal sampler. Enhanced by a meticulously adjusted
network structure, FiTv2 exhibits 2times convergence speed of FiT. When
incorporating advanced training-free extrapolation techniques, FiTv2
demonstrates remarkable adaptability in both resolution extrapolation and
diverse resolution generation. Additionally, our exploration of the scalability
of the FiTv2 model reveals that larger models exhibit better computational
efficiency. Furthermore, we introduce an efficient post-training strategy to
adapt a pre-trained model for the high-resolution generation. Comprehensive
experiments demonstrate the exceptional performance of FiTv2 across a broad
range of resolutions. We have released all the codes and models at
https://github.com/whlzy/FiT to promote the exploration of diffusion
transformer models for arbitrary-resolution image generation.
6715dc1d634b87d9f032c08a
ZiDong Wang
claimed_verified
2024-10-22T08:01:20.362Z
64b7aa374df206a3ed1947d2
/avatars/a7c7e703ccf8824259fc5a8a90a25746.svg
wzd
user
GoodEnough
6715dc1d634b87d9f032c08b
Zeyu Lu
claimed_verified
2024-10-21T16:13:35.701Z
635626a8ec32331b227f407b
https://cdn-avatars.huggingface.co/v1/production/uploads/635626a8ec32331b227f407b/KRkAEN3eXN_mYzzJQ_8dO.jpeg
LuZeyu
user
whlzy
6715dc1d634b87d9f032c08c
Di Huang
6715dc1d634b87d9f032c08d
Cai Zhou
6715dc1d634b87d9f032c08e
Wanli Ouyang
6715dc1d634b87d9f032c08f
and Lei Bai
FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion
  Model
Nature is infinitely resolution-free. In the context of this
reality, existing diffusion models, such as Diffusion Transformers, often face
challenges when processing image resolutions outside of their trained domain.
To address this limitation, we conceptualize images as sequences of tokens with
dynamic sizes, rather than traditional methods that perceive images as
fixed-resolution grids. This perspective enables a flexible training strategy
that seamlessly accommodates various aspect ratios during both training and
inference, thus promoting resolution generalization and eliminating biases
introduced by image cropping. On this basis, we present the Flexible
Vision Transformer (FiT), a transformer architecture specifically designed for
generating images with unrestricted resolutions and aspect ratios. We
further upgrade the FiT to FiTv2 with several innovative designs, includingthe
Query-Key vector normalization, the AdaLN-LoRA module, a rectified flow
scheduler, and a Logit-Normal sampler. Enhanced by a meticulously adjusted
network structure, FiTv2 exhibits 2times convergence speed of FiT. When
incorporating advanced training-free extrapolation techniques, FiTv2
demonstrates remarkable adaptability in both resolution extrapolation and
diverse resolution generation. Additionally, our exploration of the scalability
of the FiTv2 model reveals that larger models exhibit better computational
efficiency. Furthermore, we introduce an efficient post-training strategy to
adapt a pre-trained model for the high-resolution generation. Comprehensive
experiments demonstrate the exceptional performance of FiTv2 across a broad
range of resolutions. We have released all the codes and models at
https://github.com/whlzy/FiT to promote the exploration of diffusion
transformer models for arbitrary-resolution image generation.
Flexible Vision Transformers (FiT and FiTv2) enable unrestricted resolution and aspect ratio image generation through innovative modular design, improving flexibility and performance across various resolutions.
diffusion models diffusion transformers Flexible Vision Transformer FiT FiTv2 Query-Key vector normalization AdaLN-LoRA module rectified flow scheduler Logit-Normal sampler resolution extrapolation high-resolution generation computational efficiency post-training strategy