GIVT: Generative Infinite-Vocabulary Transformers
We introduce generative infinite-vocabulary transformers (GIVT) which
generate vector sequences with real-valued entries, instead of discrete tokens
from a finite vocabulary. To this end, we propose two surprisingly simple
modifications to decoder-only transformers: 1) at the input, we replace the
finite-vocabulary lookup table with a linear projection of the input vectors;
and 2) at the output, we replace the logits prediction (usually mapped to a
categorical distribution) with the parameters of a multivariate Gaussian
mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT,
where transformers are used to model the discrete latent sequences of a VQ-VAE,
we use GIVT to model the unquantized real-valued latent sequences of a VAE.
When applying GIVT to class-conditional image generation with iterative masked
modeling, we show competitive results with MaskGIT, while our approach
outperforms both VQ-GAN and MaskGIT when using it for causal modeling. Finally,
we obtain competitive results outside of image generation when applying our
approach to panoptic segmentation and depth estimation with a VAE-based variant
of the UViM framework.
656e8e29b075b63c909d255b
Michael Tschannen
admin_assigned
2023-12-05T09:23:45.665Z
6489893e1ec8356ba5bb9777
/avatars/54354c1e5774cadd1d83d42054e9d96b.svg
Michael Tschannen
user
mitsch
656e8e29b075b63c909d255c
Cian Eastwood
656e8e29b075b63c909d255d
Fabian Mentzer
GIVT: Generative Infinite-Vocabulary Transformers
We introduce generative infinite-vocabulary transformers (GIVT) which
generate vector sequences with real-valued entries, instead of discrete tokens
from a finite vocabulary. To this end, we propose two surprisingly simple
modifications to decoder-only transformers: 1) at the input, we replace the
finite-vocabulary lookup table with a linear projection of the input vectors;
and 2) at the output, we replace the logits prediction (usually mapped to a
categorical distribution) with the parameters of a multivariate Gaussian
mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT,
where transformers are used to model the discrete latent sequences of a VQ-VAE,
we use GIVT to model the unquantized real-valued latent sequences of a VAE.
When applying GIVT to class-conditional image generation with iterative masked
modeling, we show competitive results with MaskGIT, while our approach
outperforms both VQ-GAN and MaskGIT when using it for causal modeling. Finally,
we obtain competitive results outside of image generation when applying our
approach to panoptic segmentation and depth estimation with a VAE-based variant
of the UViM framework.
Generative infinite-vocabulary transformers model unquantized continuous latent sequences for tasks including image generation, panoptic segmentation, and depth estimation.
generative infinite-vocabulary transformers GIVT decoder-only transformers linear projection multivariate Gaussian mixture model VQ-GAN MaskGIT VQ-VAE VAE class-conditional image generation iterative masked modeling causal modeling panoptic segmentation depth estimation UViM framework