title: FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion
    Model
summary: Nature is infinitely resolution-free. In the context of this
  reality, existing diffusion models, such as Diffusion Transformers, often face
  challenges when processing image resolutions outside of their trained domain.
  To address this limitation, we conceptualize images as sequences of tokens with
  dynamic sizes, rather than traditional methods that perceive images as
  fixed-resolution grids. This perspective enables a flexible training strategy
  that seamlessly accommodates various aspect ratios during both training and
  inference, thus promoting resolution generalization and eliminating biases
  introduced by image cropping. On this basis, we present the Flexible
  Vision Transformer (FiT), a transformer architecture specifically designed for
  generating images with unrestricted resolutions and aspect ratios. We
  further upgrade the FiT to FiTv2 with several innovative designs, includingthe
  Query-Key vector normalization, the AdaLN-LoRA module, a rectified flow
  scheduler, and a Logit-Normal sampler. Enhanced by a meticulously adjusted
  network structure, FiTv2 exhibits 2times convergence speed of FiT. When
  incorporating advanced training-free extrapolation techniques, FiTv2
  demonstrates remarkable adaptability in both resolution extrapolation and
  diverse resolution generation. Additionally, our exploration of the scalability
  of the FiTv2 model reveals that larger models exhibit better computational
  efficiency. Furthermore, we introduce an efficient post-training strategy to
  adapt a pre-trained model for the high-resolution generation. Comprehensive
  experiments demonstrate the exceptional performance of FiTv2 across a broad
  range of resolutions. We have released all the codes and models at
  https://github.com/whlzy/FiT to promote the exploration of diffusion
  transformer models for arbitrary-resolution image generation.
paper_authors._id: 6715dc1d634b87d9f032c08a
paper_authors.name: ZiDong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:01:20.362Z
paper_authors._id: 64b7aa374df206a3ed1947d2
paper_authors.avatarUrl: /avatars/a7c7e703ccf8824259fc5a8a90a25746.svg
paper_authors.fullname: wzd
paper_authors.type: user
paper_authors.user: GoodEnough
paper_authors._id: 6715dc1d634b87d9f032c08b
paper_authors.name: Zeyu Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T16:13:35.701Z
paper_authors._id: 635626a8ec32331b227f407b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635626a8ec32331b227f407b/KRkAEN3eXN_mYzzJQ_8dO.jpeg
paper_authors.fullname: LuZeyu
paper_authors.type: user
paper_authors.user: whlzy
paper_authors._id: 6715dc1d634b87d9f032c08c
paper_authors.name: Di Huang
paper_authors._id: 6715dc1d634b87d9f032c08d
paper_authors.name: Cai Zhou
paper_authors._id: 6715dc1d634b87d9f032c08e
paper_authors.name: Wanli Ouyang
paper_authors._id: 6715dc1d634b87d9f032c08f
paper_authors.name: and Lei Bai
paper_title: FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion
    Model
paper_summary: Nature is infinitely resolution-free. In the context of this
  reality, existing diffusion models, such as Diffusion Transformers, often face
  challenges when processing image resolutions outside of their trained domain.
  To address this limitation, we conceptualize images as sequences of tokens with
  dynamic sizes, rather than traditional methods that perceive images as
  fixed-resolution grids. This perspective enables a flexible training strategy
  that seamlessly accommodates various aspect ratios during both training and
  inference, thus promoting resolution generalization and eliminating biases
  introduced by image cropping. On this basis, we present the Flexible
  Vision Transformer (FiT), a transformer architecture specifically designed for
  generating images with unrestricted resolutions and aspect ratios. We
  further upgrade the FiT to FiTv2 with several innovative designs, includingthe
  Query-Key vector normalization, the AdaLN-LoRA module, a rectified flow
  scheduler, and a Logit-Normal sampler. Enhanced by a meticulously adjusted
  network structure, FiTv2 exhibits 2times convergence speed of FiT. When
  incorporating advanced training-free extrapolation techniques, FiTv2
  demonstrates remarkable adaptability in both resolution extrapolation and
  diverse resolution generation. Additionally, our exploration of the scalability
  of the FiTv2 model reveals that larger models exhibit better computational
  efficiency. Furthermore, we introduce an efficient post-training strategy to
  adapt a pre-trained model for the high-resolution generation. Comprehensive
  experiments demonstrate the exceptional performance of FiTv2 across a broad
  range of resolutions. We have released all the codes and models at
  https://github.com/whlzy/FiT to promote the exploration of diffusion
  transformer models for arbitrary-resolution image generation.
paper_ai_summary: Flexible Vision Transformers (FiT and FiTv2) enable unrestricted resolution and aspect ratio image generation through innovative modular design, improving flexibility and performance across various resolutions.
paper_ai_keywords: diffusion models diffusion transformers Flexible Vision Transformer FiT FiTv2 Query-Key vector normalization AdaLN-LoRA module rectified flow scheduler Logit-Normal sampler resolution extrapolation high-resolution generation computational efficiency post-training strategy