title: FiT: Flexible Vision Transformer for Diffusion Model
summary: Nature is infinitely resolution-free. In the context of this reality,
  existing diffusion models, such as Diffusion Transformers, often face
  challenges when processing image resolutions outside of their trained domain.
  To overcome this limitation, we present the Flexible Vision Transformer (FiT),
  a transformer architecture specifically designed for generating images with
  unrestricted resolutions and aspect ratios. Unlike traditional methods that
  perceive images as static-resolution grids, FiT conceptualizes images as
  sequences of dynamically-sized tokens. This perspective enables a flexible
  training strategy that effortlessly adapts to diverse aspect ratios during both
  training and inference phases, thus promoting resolution generalization and
  eliminating biases induced by image cropping. Enhanced by a meticulously
  adjusted network structure and the integration of training-free extrapolation
  techniques, FiT exhibits remarkable flexibility in resolution extrapolation
  generation. Comprehensive experiments demonstrate the exceptional performance
  of FiT across a broad range of resolutions, showcasing its effectiveness both
  within and beyond its training resolution distribution. Repository available at
  https://github.com/whlzy/FiT.
paper_authors._id: 65d4455bf08891235afc6390
paper_authors.name: Zeyu Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:53:20.006Z
paper_authors._id: 6522ab765e7247c29172067e
paper_authors.avatarUrl: /avatars/ea8d64a50d55456bd034166170a67748.svg
paper_authors.fullname: Zeyu Lu
paper_authors.type: user
paper_authors.user: zeyulu
paper_authors._id: 65d4455bf08891235afc6391
paper_authors.name: Zidong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T16:13:40.719Z
paper_authors._id: 64b7aa374df206a3ed1947d2
paper_authors.avatarUrl: /avatars/a7c7e703ccf8824259fc5a8a90a25746.svg
paper_authors.fullname: wzd
paper_authors.type: user
paper_authors.user: GoodEnough
paper_authors._id: 65d4455bf08891235afc6392
paper_authors.name: Di Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T11:02:17.512Z
paper_authors._id: 646dd0654ad7f907279e4e96
paper_authors.avatarUrl: /avatars/23e3e39f8ffae2f31d6b64cdbb44d47a.svg
paper_authors.fullname: dihuang
paper_authors.type: user
paper_authors.user: Kellan3327
paper_authors._id: 65d4455bf08891235afc6393
paper_authors.name: Chengyue Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:54:03.704Z
paper_authors._id: 617526c9de8feb54b0ce45ad
paper_authors.avatarUrl: /avatars/7faf8c6f71fc318a0113d780d376c381.svg
paper_authors.fullname: Wu Chengyue
paper_authors.type: user
paper_authors.user: WuChengyue
paper_authors._id: 65d4455bf08891235afc6394
paper_authors.name: Xihui Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-03T18:29:12.877Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_authors._id: 65d4455bf08891235afc6395
paper_authors.name: Wanli Ouyang
paper_authors._id: 65d4455bf08891235afc6396
paper_authors.name: Lei Bai
paper_title: FiT: Flexible Vision Transformer for Diffusion Model
paper_summary: Nature is infinitely resolution-free. In the context of this reality,
  existing diffusion models, such as Diffusion Transformers, often face
  challenges when processing image resolutions outside of their trained domain.
  To overcome this limitation, we present the Flexible Vision Transformer (FiT),
  a transformer architecture specifically designed for generating images with
  unrestricted resolutions and aspect ratios. Unlike traditional methods that
  perceive images as static-resolution grids, FiT conceptualizes images as
  sequences of dynamically-sized tokens. This perspective enables a flexible
  training strategy that effortlessly adapts to diverse aspect ratios during both
  training and inference phases, thus promoting resolution generalization and
  eliminating biases induced by image cropping. Enhanced by a meticulously
  adjusted network structure and the integration of training-free extrapolation
  techniques, FiT exhibits remarkable flexibility in resolution extrapolation
  generation. Comprehensive experiments demonstrate the exceptional performance
  of FiT across a broad range of resolutions, showcasing its effectiveness both
  within and beyond its training resolution distribution. Repository available at
  https://github.com/whlzy/FiT.
paper_ai_summary: The Flexible Vision Transformer adapts to varied image resolutions and aspect ratios through dynamic tokenization and extrapolation techniques, outperforming traditional methods.
paper_ai_keywords: diffusion models Diffusion Transformers Flexible Vision Transformer transformer architecture dynamically-sized tokens resolution generalization image cropping training-free extrapolation resolution extrapolation generation