title: The Geometry of Tokens in Internal Representations of Large Language
    Models
summary: We investigate the relationship between the geometry of token embeddings and
  their role in the next token prediction within transformer models. An important
  aspect of this connection uses the notion of empirical measure, which encodes
  the distribution of token point clouds across transformer layers and drives the
  evolution of token representations in the mean-field interacting picture. We
  use metrics such as intrinsic dimension, neighborhood overlap, and cosine
  similarity to observationally probe these empirical measures across layers. To
  validate our approach, we compare these metrics to a dataset where the tokens
  are shuffled, which disrupts the syntactic and semantic structure. Our findings
  reveal a correlation between the geometric properties of token embeddings and
  the cross-entropy loss of next token predictions, implying that prompts with
  higher loss values have tokens represented in higher-dimensional spaces.
paper_authors._id: 6790ffb0561e5b824100c29a
paper_authors.name: Karthik Viswanathan
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-22T14:24:50.240Z
paper_authors._id: 6632140cf69134729c68e65f
paper_authors.avatarUrl: /avatars/f68426b505752606223bb78fd82a1af9.svg
paper_authors.fullname: Karthik Viswanathan
paper_authors.type: user
paper_authors.user: vkarthik095
paper_authors._id: 6790ffb0561e5b824100c29b
paper_authors.name: Yuri Gardinazzi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-22T15:35:14.198Z
paper_authors._id: 6618fbba78bd2f4adc475834
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6618fbba78bd2f4adc475834/dJBRN4jucKOnMHtq2aq42.jpeg
paper_authors.fullname: YuriGardinazzi
paper_authors.type: user
paper_authors.user: YuriGardinazzi
paper_authors._id: 6790ffb0561e5b824100c29c
paper_authors.name: Giada Panerai
paper_authors._id: 6790ffb0561e5b824100c29d
paper_authors.name: Alberto Cazzaniga
paper_authors._id: 6790ffb0561e5b824100c29e
paper_authors.name: Matteo Biagetti
paper_title: The Geometry of Tokens in Internal Representations of Large Language
    Models
paper_summary: We investigate the relationship between the geometry of token embeddings and
  their role in the next token prediction within transformer models. An important
  aspect of this connection uses the notion of empirical measure, which encodes
  the distribution of token point clouds across transformer layers and drives the
  evolution of token representations in the mean-field interacting picture. We
  use metrics such as intrinsic dimension, neighborhood overlap, and cosine
  similarity to observationally probe these empirical measures across layers. To
  validate our approach, we compare these metrics to a dataset where the tokens
  are shuffled, which disrupts the syntactic and semantic structure. Our findings
  reveal a correlation between the geometric properties of token embeddings and
  the cross-entropy loss of next token predictions, implying that prompts with
  higher loss values have tokens represented in higher-dimensional spaces.
paper_ai_keywords: token embeddings next token prediction transformer models empirical measure token point clouds mean-field interacting picture intrinsic dimension neighborhood cosine similarity cross-entropy loss syntactic semantic structure project embeddings