title: GIVT: Generative Infinite-Vocabulary Transformers
summary: We introduce generative infinite-vocabulary transformers (GIVT) which
  generate vector sequences with real-valued entries, instead of discrete tokens
  from a finite vocabulary. To this end, we propose two surprisingly simple
  modifications to decoder-only transformers: 1) at the input, we replace the
  finite-vocabulary lookup table with a linear projection of the input vectors;
  and 2) at the output, we replace the logits prediction (usually mapped to a
  categorical distribution) with the parameters of a multivariate Gaussian
  mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT,
  where transformers are used to model the discrete latent sequences of a VQ-VAE,
  we use GIVT to model the unquantized real-valued latent sequences of a VAE.
  When applying GIVT to class-conditional image generation with iterative masked
  modeling, we show competitive results with MaskGIT, while our approach
  outperforms both VQ-GAN and MaskGIT when using it for causal modeling. Finally,
  we obtain competitive results outside of image generation when applying our
  approach to panoptic segmentation and depth estimation with a VAE-based variant
  of the UViM framework.
paper_authors._id: 656e8e29b075b63c909d255b
paper_authors.name: Michael Tschannen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T09:23:45.665Z
paper_authors._id: 6489893e1ec8356ba5bb9777
paper_authors.avatarUrl: /avatars/54354c1e5774cadd1d83d42054e9d96b.svg
paper_authors.fullname: Michael Tschannen
paper_authors.type: user
paper_authors.user: mitsch
paper_authors._id: 656e8e29b075b63c909d255c
paper_authors.name: Cian Eastwood
paper_authors._id: 656e8e29b075b63c909d255d
paper_authors.name: Fabian Mentzer
paper_title: GIVT: Generative Infinite-Vocabulary Transformers
paper_summary: We introduce generative infinite-vocabulary transformers (GIVT) which
  generate vector sequences with real-valued entries, instead of discrete tokens
  from a finite vocabulary. To this end, we propose two surprisingly simple
  modifications to decoder-only transformers: 1) at the input, we replace the
  finite-vocabulary lookup table with a linear projection of the input vectors;
  and 2) at the output, we replace the logits prediction (usually mapped to a
  categorical distribution) with the parameters of a multivariate Gaussian
  mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT,
  where transformers are used to model the discrete latent sequences of a VQ-VAE,
  we use GIVT to model the unquantized real-valued latent sequences of a VAE.
  When applying GIVT to class-conditional image generation with iterative masked
  modeling, we show competitive results with MaskGIT, while our approach
  outperforms both VQ-GAN and MaskGIT when using it for causal modeling. Finally,
  we obtain competitive results outside of image generation when applying our
  approach to panoptic segmentation and depth estimation with a VAE-based variant
  of the UViM framework.
paper_ai_summary: Generative infinite-vocabulary transformers model unquantized continuous latent sequences for tasks including image generation, panoptic segmentation, and depth estimation.
paper_ai_keywords: generative infinite-vocabulary transformers GIVT decoder-only transformers linear projection multivariate Gaussian mixture model VQ-GAN MaskGIT VQ-VAE VAE class-conditional image generation iterative masked modeling causal modeling panoptic segmentation depth estimation UViM framework