title: Analyzing The Language of Visual Tokens
summary: With the introduction of transformer-based models for vision and language
  tasks, such as LLaVA and Chameleon, there has been renewed interest in the
  discrete tokenized representation of images. These models often treat image
  patches as discrete tokens, analogous to words in natural language, learning
  joint alignments between visual and human languages. However, little is known
  about the statistical behavior of these visual languages - whether they follow
  similar frequency distributions, grammatical structures, or topologies as
  natural languages. In this paper, we take a natural-language-centric approach
  to analyzing discrete visual languages and uncover striking similarities and
  fundamental differences. We demonstrate that, although visual languages adhere
  to Zipfian distributions, higher token innovation drives greater entropy and
  lower compression, with tokens predominantly representing object parts,
  indicating intermediate granularity. We also show that visual languages lack
  cohesive grammatical structures, leading to higher perplexity and weaker
  hierarchical organization compared to natural languages. Finally, we
  demonstrate that, while vision models align more closely with natural languages
  than other models, this alignment remains significantly weaker than the
  cohesion found within natural languages. Through these experiments, we
  demonstrate how understanding the statistical properties of discrete visual
  languages can inform the design of more effective computer vision models.
paper_authors._id: 672e9decf2e997ba83d56e44
paper_authors.name: David M. Chan
paper_authors._id: 672e9decf2e997ba83d56e45
paper_authors.name: Rodolfo Corona
paper_authors._id: 672e9decf2e997ba83d56e46
paper_authors.name: Joonyong Park
paper_authors._id: 672e9decf2e997ba83d56e47
paper_authors.name: Cheol Jun Cho
paper_authors._id: 672e9decf2e997ba83d56e48
paper_authors.name: Yutong Bai
paper_authors._id: 672e9decf2e997ba83d56e49
paper_authors.name: Trevor Darrell
paper_title: Analyzing The Language of Visual Tokens
paper_summary: With the introduction of transformer-based models for vision and language
  tasks, such as LLaVA and Chameleon, there has been renewed interest in the
  discrete tokenized representation of images. These models often treat image
  patches as discrete tokens, analogous to words in natural language, learning
  joint alignments between visual and human languages. However, little is known
  about the statistical behavior of these visual languages - whether they follow
  similar frequency distributions, grammatical structures, or topologies as
  natural languages. In this paper, we take a natural-language-centric approach
  to analyzing discrete visual languages and uncover striking similarities and
  fundamental differences. We demonstrate that, although visual languages adhere
  to Zipfian distributions, higher token innovation drives greater entropy and
  lower compression, with tokens predominantly representing object parts,
  indicating intermediate granularity. We also show that visual languages lack
  cohesive grammatical structures, leading to higher perplexity and weaker
  hierarchical organization compared to natural languages. Finally, we
  demonstrate that, while vision models align more closely with natural languages
  than other models, this alignment remains significantly weaker than the
  cohesion found within natural languages. Through these experiments, we
  demonstrate how understanding the statistical properties of discrete visual
  languages can inform the design of more effective computer vision models.
paper_ai_keywords: transformer-based models LLaVA Chameleon discrete tokenized representation image patches visual and human data joint alignments Visual Question Answering (VQA) Zipfian distributions token innovation entropy compression cohesive grammatical structures perplexity hierarchical organization computer vision models statistical properties discrete visual languages