FiT: Flexible Vision Transformer for Diffusion Model
Nature is infinitely resolution-free. In the context of this reality,
existing diffusion models, such as Diffusion Transformers, often face
challenges when processing image resolutions outside of their trained domain.
To overcome this limitation, we present the Flexible Vision Transformer (FiT),
a transformer architecture specifically designed for generating images with
unrestricted resolutions and aspect ratios. Unlike traditional methods that
perceive images as static-resolution grids, FiT conceptualizes images as
sequences of dynamically-sized tokens. This perspective enables a flexible
training strategy that effortlessly adapts to diverse aspect ratios during both
training and inference phases, thus promoting resolution generalization and
eliminating biases induced by image cropping. Enhanced by a meticulously
adjusted network structure and the integration of training-free extrapolation
techniques, FiT exhibits remarkable flexibility in resolution extrapolation
generation. Comprehensive experiments demonstrate the exceptional performance
of FiT across a broad range of resolutions, showcasing its effectiveness both
within and beyond its training resolution distribution. Repository available at
https://github.com/whlzy/FiT.
65d4455bf08891235afc6390
Zeyu Lu
admin_assigned
2024-02-20T09:53:20.006Z
6522ab765e7247c29172067e
/avatars/ea8d64a50d55456bd034166170a67748.svg
Zeyu Lu
user
zeyulu
65d4455bf08891235afc6391
Zidong Wang
claimed_verified
2024-10-21T16:13:40.719Z
64b7aa374df206a3ed1947d2
/avatars/a7c7e703ccf8824259fc5a8a90a25746.svg
wzd
user
GoodEnough
65d4455bf08891235afc6392
Di Huang
claimed_verified
2024-02-20T11:02:17.512Z
646dd0654ad7f907279e4e96
/avatars/23e3e39f8ffae2f31d6b64cdbb44d47a.svg
dihuang
user
Kellan3327
65d4455bf08891235afc6393
Chengyue Wu
admin_assigned
2024-02-20T09:54:03.704Z
617526c9de8feb54b0ce45ad
/avatars/7faf8c6f71fc318a0113d780d376c381.svg
Wu Chengyue
user
WuChengyue
65d4455bf08891235afc6394
Xihui Liu
claimed_verified
2024-06-03T18:29:12.877Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
65d4455bf08891235afc6395
Wanli Ouyang
65d4455bf08891235afc6396
Lei Bai
FiT: Flexible Vision Transformer for Diffusion Model
Nature is infinitely resolution-free. In the context of this reality,
existing diffusion models, such as Diffusion Transformers, often face
challenges when processing image resolutions outside of their trained domain.
To overcome this limitation, we present the Flexible Vision Transformer (FiT),
a transformer architecture specifically designed for generating images with
unrestricted resolutions and aspect ratios. Unlike traditional methods that
perceive images as static-resolution grids, FiT conceptualizes images as
sequences of dynamically-sized tokens. This perspective enables a flexible
training strategy that effortlessly adapts to diverse aspect ratios during both
training and inference phases, thus promoting resolution generalization and
eliminating biases induced by image cropping. Enhanced by a meticulously
adjusted network structure and the integration of training-free extrapolation
techniques, FiT exhibits remarkable flexibility in resolution extrapolation
generation. Comprehensive experiments demonstrate the exceptional performance
of FiT across a broad range of resolutions, showcasing its effectiveness both
within and beyond its training resolution distribution. Repository available at
https://github.com/whlzy/FiT.
The Flexible Vision Transformer adapts to varied image resolutions and aspect ratios through dynamic tokenization and extrapolation techniques, outperforming traditional methods.
diffusion models Diffusion Transformers Flexible Vision Transformer transformer architecture dynamically-sized tokens resolution generalization image cropping training-free extrapolation resolution extrapolation generation