title: Democratizing Text-to-Image Masked Generative Models with Compact
    Text-Aware One-Dimensional Tokens
summary: Image tokenizers form the foundation of modern text-to-image generative
  models but are notoriously difficult to train. Furthermore, most existing
  text-to-image models rely on large-scale, high-quality private datasets, making
  them challenging to replicate. In this work, we introduce Text-Aware
  Transformer-based 1-Dimensional Tokenizer (TA-TiTok), an efficient and powerful
  image tokenizer that can utilize either discrete or continuous 1-dimensional
  tokens. TA-TiTok uniquely integrates textual information during the tokenizer
  decoding stage (i.e., de-tokenization), accelerating convergence and enhancing
  performance. TA-TiTok also benefits from a simplified, yet effective, one-stage
  training process, eliminating the need for the complex two-stage distillation
  used in previous 1-dimensional tokenizers. This design allows for seamless
  scalability to large datasets. Building on this, we introduce a family of
  text-to-image Masked Generative Models (MaskGen), trained exclusively on open
  data while achieving comparable performance to models trained on private data.
  We aim to release both the efficient, strong TA-TiTok tokenizers and the
  open-data, open-weight MaskGen models to promote broader access and democratize
  the field of text-to-image masked generative models.
paper_authors._id: 678734168c1e7b6c4a6e5ff9
paper_authors.name: Dongwon Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-16T08:31:43.848Z
paper_authors._id: 64dc5208c38427829de81b16
paper_authors.avatarUrl: /avatars/43a08e46a7a78f1e3d1f6645a9b1d26b.svg
paper_authors.fullname: Dongwon
paper_authors.type: user
paper_authors.user: kdwon
paper_authors._id: 678734168c1e7b6c4a6e5ffa
paper_authors.name: Ju He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:48:25.249Z
paper_authors._id: 661c9059bcd78151e5c06ea1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/661c9059bcd78151e5c06ea1/27bfNo1LZeZQ77vWuAa10.png
paper_authors.fullname: Ju He
paper_authors.type: user
paper_authors.user: turkeyju
paper_authors._id: 678734168c1e7b6c4a6e5ffb
paper_authors.name: Qihang Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:54:50.093Z
paper_authors._id: 677b60e17279b5c57354108b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/677b60e17279b5c57354108b/YOwDhVf9DkeRjOCOLErb6.png
paper_authors.fullname: QihangYu
paper_authors.type: user
paper_authors.user: QihangYu
paper_authors._id: 678734168c1e7b6c4a6e5ffc
paper_authors.name: Chenglin Yang
paper_authors._id: 678734168c1e7b6c4a6e5ffd
paper_authors.name: Xiaohui Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:55:07.851Z
paper_authors._id: 6430aa1b32a732121cd81f98
paper_authors.avatarUrl: /avatars/5419f8d6d4d36fa5ac83e30667b9fd99.svg
paper_authors.fullname: Xiaohui Shen
paper_authors.type: user
paper_authors.user: XiaohuiShen
paper_authors._id: 678734168c1e7b6c4a6e5ffe
paper_authors.name: Suha Kwak
paper_authors._id: 678734168c1e7b6c4a6e5fff
paper_authors.name: Liang-Chieh Chen
paper_title: Democratizing Text-to-Image Masked Generative Models with Compact
    Text-Aware One-Dimensional Tokens
paper_summary: Image tokenizers form the foundation of modern text-to-image generative
  models but are notoriously difficult to train. Furthermore, most existing
  text-to-image models rely on large-scale, high-quality private datasets, making
  them challenging to replicate. In this work, we introduce Text-Aware
  Transformer-based 1-Dimensional Tokenizer (TA-TiTok), an efficient and powerful
  image tokenizer that can utilize either discrete or continuous 1-dimensional
  tokens. TA-TiTok uniquely integrates textual information during the tokenizer
  decoding stage (i.e., de-tokenization), accelerating convergence and enhancing
  performance. TA-TiTok also benefits from a simplified, yet effective, one-stage
  training process, eliminating the need for the complex two-stage distillation
  used in previous 1-dimensional tokenizers. This design allows for seamless
  scalability to large datasets. Building on this, we introduce a family of
  text-to-image Masked Generative Models (MaskGen), trained exclusively on open
  data while achieving comparable performance to models trained on private data.
  We aim to release both the efficient, strong TA-TiTok tokenizers and the
  open-data, open-weight MaskGen models to promote broader access and democratize
  the field of text-to-image masked generative models.
paper_ai_summary: TA-TiTok, a text-aware transformer-based tokenizer, integrates textual information during decoding, improving training efficiency and enabling MaskGen models to achieve comparable performance using open datasets.
paper_ai_keywords: image tokenizers text-to-image generative models TA-TiTok transformer-based 1-dimensional tokens textual information tokenizer decoding de-tokenization convergence performance one-stage training distillation MaskGen open data open-weight