title: Chameleon: Mixed-Modal Early-Fusion Foundation Models
summary: We present Chameleon, a family of early-fusion token-based mixed-modal models
  capable of understanding and generating images and text in any arbitrary
  sequence. We outline a stable training approach from inception, an alignment
  recipe, and an architectural parameterization tailored for the early-fusion,
  token-based, mixed-modal setting. The models are evaluated on a comprehensive
  range of tasks, including visual question answering, image captioning, text
  generation, image generation, and long-form mixed modal generation. Chameleon
  demonstrates broad and general capabilities, including state-of-the-art
  performance in image captioning tasks, outperforms Llama-2 in text-only tasks
  while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and
  performs non-trivial image generation, all in a single model. It also matches
  or exceeds the performance of much larger models, including Gemini Pro and
  GPT-4V, according to human judgments on a new long-form mixed-modal generation
  evaluation, where either the prompt or outputs contain mixed sequences of both
  images and text. Chameleon marks a significant step forward in a unified
  modeling of full multimodal documents.
paper_authors._id: 6646b4a06d5f0538ee0176cc
paper_authors.name: Chameleon Team
paper_title: Chameleon: Mixed-Modal Early-Fusion Foundation Models
paper_summary: We present Chameleon, a family of early-fusion token-based mixed-modal models
  capable of understanding and generating images and text in any arbitrary
  sequence. We outline a stable training approach from inception, an alignment
  recipe, and an architectural parameterization tailored for the early-fusion,
  token-based, mixed-modal setting. The models are evaluated on a comprehensive
  range of tasks, including visual question answering, image captioning, text
  generation, image generation, and long-form mixed modal generation. Chameleon
  demonstrates broad and general capabilities, including state-of-the-art
  performance in image captioning tasks, outperforms Llama-2 in text-only tasks
  while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and
  performs non-trivial image generation, all in a single model. It also matches
  or exceeds the performance of much larger models, including Gemini Pro and
  GPT-4V, according to human judgments on a new long-form mixed-modal generation
  evaluation, where either the prompt or outputs contain mixed sequences of both
  images and text. Chameleon marks a significant step forward in a unified
  modeling of full multimodal documents.
paper_ai_keywords: early-fusion token-based mixed-modal models mixed-modal models visual question answering image captioning text generation image generation long-form mixed modal generation state-of-the-art performance Llama-2 Mixtral 8x7B Gemini-Pro Pro GPT-4V long-form mixed generation unified modeling of full multimodal documents