title: QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive
    Multimodal Understanding and Generation
summary: We introduce Quantized Language-Image Pretraining (QLIP), a visual
  tokenization method that combines state-of-the-art reconstruction quality with
  state-of-the-art zero-shot image understanding. QLIP trains a
  binary-spherical-quantization-based autoencoder with reconstruction and
  language-image alignment objectives. We are the first to show that the two
  objectives do not need to be at odds. We balance the two loss terms dynamically
  during training and show that a two-stage training pipeline effectively mixes
  the large-batch requirements of image-language pre-training with the memory
  bottleneck imposed by the reconstruction objective. We validate the
  effectiveness of QLIP for multimodal understanding and text-conditioned image
  generation with a single model. Specifically, QLIP serves as a drop-in
  replacement for the visual encoder for LLaVA and the image tokenizer for
  LlamaGen with comparable or even better performance. Finally, we demonstrate
  that QLIP enables a unified mixed-modality auto-regressive model for
  understanding and generation.
paper_authors._id: 67a99dfe98423dca45d8f659
paper_authors.name: Yue Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:49:43.493Z
paper_authors._id: 638fe91639f7e2a7f9d2a8c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638fe91639f7e2a7f9d2a8c6/hB7DMVODcdAEUdQnXxWA8.jpeg
paper_authors.fullname: Yue Zhao
paper_authors.type: user
paper_authors.user: zhaoyue-zephyrus
paper_authors._id: 67a99dfe98423dca45d8f65a
paper_authors.name: Fuzhao Xue
paper_authors._id: 67a99dfe98423dca45d8f65b
paper_authors.name: Scott Reed
paper_authors._id: 67a99dfe98423dca45d8f65c
paper_authors.name: Linxi Fan
paper_authors._id: 67a99dfe98423dca45d8f65d
paper_authors.name: Yuke Zhu
paper_authors._id: 67a99dfe98423dca45d8f65e
paper_authors.name: Jan Kautz
paper_authors._id: 67a99dfe98423dca45d8f65f
paper_authors.name: Zhiding Yu
paper_authors._id: 67a99dfe98423dca45d8f660
paper_authors.name: Philipp Krähenbühl
paper_authors._id: 67a99dfe98423dca45d8f661
paper_authors.name: De-An Huang
paper_title: QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive
    Multimodal Understanding and Generation
paper_summary: We introduce Quantized Language-Image Pretraining (QLIP), a visual
  tokenization method that combines state-of-the-art reconstruction quality with
  state-of-the-art zero-shot image understanding. QLIP trains a
  binary-spherical-quantization-based autoencoder with reconstruction and
  language-image alignment objectives. We are the first to show that the two
  objectives do not need to be at odds. We balance the two loss terms dynamically
  during training and show that a two-stage training pipeline effectively mixes
  the large-batch requirements of image-language pre-training with the memory
  bottleneck imposed by the reconstruction objective. We validate the
  effectiveness of QLIP for multimodal understanding and text-conditioned image
  generation with a single model. Specifically, QLIP serves as a drop-in
  replacement for the visual encoder for LLaVA and the image tokenizer for
  LlamaGen with comparable or even better performance. Finally, we demonstrate
  that QLIP enables a unified mixed-modality auto-regressive model for
  understanding and generation.
paper_ai_summary: Quantized Language-Image Pretraining (QLIP) integrates binary-spherical-quantization for image reconstructions and language-image alignment, enhancing multimodal understanding and text-conditioned image generation.
paper_ai_keywords: binary-spherical-quantization autoencoder reconstruction language-image alignment multimodal understanding text-conditioned image generation mixed-modality auto-regressive model LLaVA LlamaGen