title: LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context
    Memory of Transformers
summary: We introduce methods to quantify how Large Language Models (LLMs) encode and
  store contextual information, revealing that tokens often seen as minor (e.g.,
  determiners, punctuation) carry surprisingly high context. Notably, removing
  these tokens -- especially stopwords, articles, and commas -- consistently
  degrades performance on MMLU and BABILong-4k, even if removing only irrelevant
  tokens. Our analysis also shows a strong correlation between contextualization
  and linearity, where linearity measures how closely the transformation from one
  layer's embeddings to the next can be approximated by a single linear mapping.
  These findings underscore the hidden importance of filler tokens in maintaining
  context. For further exploration, we present LLM-Microscope, an open-source
  toolkit that assesses token-level nonlinearity, evaluates contextual memory,
  visualizes intermediate layer contributions (via an adapted Logit Lens), and
  measures the intrinsic dimensionality of representations. This toolkit
  illuminates how seemingly trivial tokens can be critical for long-range
  understanding.
paper_authors._id: 67bc1a4a72499ce2ba28cc70
paper_authors.name: Anton Razzhigaev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:11:53.576Z
paper_authors._id: 6172aaeec8e66e2aa84c06b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6172aaeec8e66e2aa84c06b9/ZdRZSp3P1SU6CIDbvQwkv.jpeg
paper_authors.fullname: Anton Razzhigaev
paper_authors.type: user
paper_authors.user: razzant
paper_authors._id: 67bc1a4a72499ce2ba28cc71
paper_authors.name: Matvey Mikhalchuk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:11:58.863Z
paper_authors._id: 64ee45a944f4b3b1bccc02d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ee45a944f4b3b1bccc02d1/SoidO9HQ4mftzbUPtuBBf.png
paper_authors.fullname: Matvey Mikhalchuk
paper_authors.type: user
paper_authors.user: matveymih
paper_authors._id: 67bc1a4a72499ce2ba28cc72
paper_authors.name: Temurbek Rahmatullaev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:40:40.159Z
paper_authors._id: 659c33fe1801bc22227b8ff6
paper_authors.avatarUrl: /avatars/837cdb3351cfd84dc9dcef37bcf18dff.svg
paper_authors.fullname: Temurbek
paper_authors.type: user
paper_authors.user: raxtemur
paper_authors._id: 67bc1a4a72499ce2ba28cc73
paper_authors.name: Elizaveta Goncharova
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:12:27.998Z
paper_authors._id: 6310ff34bc152fa3e810c186
paper_authors.avatarUrl: /avatars/bfd63bcd81548283f5e496e3693bf143.svg
paper_authors.fullname: Elizaveta Goncharova
paper_authors.type: user
paper_authors.user: Elizaveta
paper_authors._id: 67bc1a4a72499ce2ba28cc74
paper_authors.name: Polina Druzhinina
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:12:33.837Z
paper_authors._id: 65d5e094cd05bc1eaa0fafc9
paper_authors.avatarUrl: /avatars/ea3d52def6ef4d9af07728a76a499a9f.svg
paper_authors.fullname: Polina Druzhinina
paper_authors.type: user
paper_authors.user: plina2polina
paper_authors._id: 67bc1a4a72499ce2ba28cc75
paper_authors.name: Ivan Oseledets
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:12:40.295Z
paper_authors._id: 6169a581d05945bfd8718dfa
paper_authors.avatarUrl: /avatars/1892ab06a7ddb557232777de3cbec470.svg
paper_authors.fullname: Ivan Oseledets
paper_authors.type: user
paper_authors.user: oseledets
paper_authors._id: 67bc1a4a72499ce2ba28cc76
paper_authors.name: Andrey Kuznetsov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:40:42.629Z
paper_authors._id: 643984dceb7c5616ef3f5d54
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
paper_authors.fullname: Andrey Kuznetsov
paper_authors.type: user
paper_authors.user: kuznetsoffandrey
paper_title: LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context
    Memory of Transformers
paper_summary: We introduce methods to quantify how Large Language Models (LLMs) encode and
  store contextual information, revealing that tokens often seen as minor (e.g.,
  determiners, punctuation) carry surprisingly high context. Notably, removing
  these tokens -- especially stopwords, articles, and commas -- consistently
  degrades performance on MMLU and BABILong-4k, even if removing only irrelevant
  tokens. Our analysis also shows a strong correlation between contextualization
  and linearity, where linearity measures how closely the transformation from one
  layer's embeddings to the next can be approximated by a single linear mapping.
  These findings underscore the hidden importance of filler tokens in maintaining
  context. For further exploration, we present LLM-Microscope, an open-source
  toolkit that assesses token-level nonlinearity, evaluates contextual memory,
  visualizes intermediate layer contributions (via an adapted Logit Lens), and
  measures the intrinsic dimensionality of representations. This toolkit
  illuminates how seemingly trivial tokens can be critical for long-range
  understanding.
paper_ai_keywords: Large Language Models (LLMs) tokens determiners punctuation stopwords articles commas MMLU BABILong-4k contextualization linearity embeddings LLM-Microscope Logit Lens intrinsic dimensionality long-range understanding