title: JPEG-LM: LLMs as Image Generators with Canonical Codec Representations
summary: Recent work in image and video generation has been adopting the
  autoregressive LLM architecture due to its generality and potentially easy
  integration into multi-modal systems. The crux of applying autoregressive
  training in language generation to visual generation is discretization --
  representing continuous data like images and videos as discrete tokens. Common
  methods of discretizing images and videos include modeling raw pixel values,
  which are prohibitively lengthy, or vector quantization, which requires
  convoluted pre-hoc training. In this work, we propose to directly model images
  and videos as compressed files saved on computers via canonical codecs (e.g.,
  JPEG, AVC/H.264). Using the default Llama architecture without any
  vision-specific modifications, we pretrain JPEG-LM from scratch to generate
  images (and AVC-LM to generate videos as a proof of concept), by directly
  outputting compressed file bytes in JPEG and AVC formats. Evaluation of image
  generation shows that this simple and straightforward approach is more
  effective than pixel-based modeling and sophisticated vector quantization
  baselines (on which our method yields a 31% reduction in FID). Our analysis
  shows that JPEG-LM has an especial advantage over vector quantization models in
  generating long-tail visual elements. Overall, we show that using canonical
  codec representations can help lower the barriers between language generation
  and visual generation, facilitating future research on multi-modal
  language/image/video LLMs.
paper_authors._id: 66c2a4ae2777c05070a6db3a
paper_authors.name: Xiaochuang Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:13:21.157Z
paper_authors._id: 635ea3969f24f6db0a1e2d0b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635ea3969f24f6db0a1e2d0b/RxaJKAx4MYMShb58yuUxy.png
paper_authors.fullname: Xiaochuang Han
paper_authors.type: user
paper_authors.user: xhan77
paper_authors._id: 66c2a4ae2777c05070a6db3b
paper_authors.name: Marjan Ghazvininejad
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:27:55.990Z
paper_authors._id: 660f0fd377a1e2509aa5a679
paper_authors.avatarUrl: /avatars/e04ef05bed0bf6cefdc7e3e39674e2f9.svg
paper_authors.fullname: Marjan Ghazvininejad
paper_authors.type: user
paper_authors.user: mghazvininejad
paper_authors._id: 66c2a4ae2777c05070a6db3c
paper_authors.name: Pang Wei Koh
paper_authors._id: 66c2a4ae2777c05070a6db3d
paper_authors.name: Yulia Tsvetkov
paper_title: JPEG-LM: LLMs as Image Generators with Canonical Codec Representations
paper_summary: Recent work in image and video generation has been adopting the
  autoregressive LLM architecture due to its generality and potentially easy
  integration into multi-modal systems. The crux of applying autoregressive
  training in language generation to visual generation is discretization --
  representing continuous data like images and videos as discrete tokens. Common
  methods of discretizing images and videos include modeling raw pixel values,
  which are prohibitively lengthy, or vector quantization, which requires
  convoluted pre-hoc training. In this work, we propose to directly model images
  and videos as compressed files saved on computers via canonical codecs (e.g.,
  JPEG, AVC/H.264). Using the default Llama architecture without any
  vision-specific modifications, we pretrain JPEG-LM from scratch to generate
  images (and AVC-LM to generate videos as a proof of concept), by directly
  outputting compressed file bytes in JPEG and AVC formats. Evaluation of image
  generation shows that this simple and straightforward approach is more
  effective than pixel-based modeling and sophisticated vector quantization
  baselines (on which our method yields a 31% reduction in FID). Our analysis
  shows that JPEG-LM has an especial advantage over vector quantization models in
  generating long-tail visual elements. Overall, we show that using canonical
  codec representations can help lower the barriers between language generation
  and visual generation, facilitating future research on multi-modal
  language/image/video LLMs.
paper_ai_summary: Using canonical codec representations (JPEG, AVC) as a direct modeling approach for image and video generation outperforms pixel-based and vector quantization methods, facilitating multi-modal LLMs.
paper_ai_keywords: autoregressive LLM architecture discretization vector quantization compressed file bytes JPEG-LM AVC-LM FID long-tail visual elements multi-modal language/image/video LLMs