title: Your Context Is Not an Array: Unveiling Random Access Limitations in
    Transformers
summary: Despite their recent successes, Transformer-based large language models show
  surprising failure modes. A well-known example of such failure modes is their
  inability to length-generalize: solving problem instances at inference time
  that are longer than those seen during training. In this work, we further
  explore the root cause of this failure by performing a detailed analysis of
  model behaviors on the simple parity task. Our analysis suggests that length
  generalization failures are intricately related to a model's inability to
  perform random memory accesses within its context window. We present supporting
  evidence for this hypothesis by demonstrating the effectiveness of
  methodologies that circumvent the need for indexing or that enable random token
  access indirectly, through content-based addressing. We further show where and
  how the failure to perform random memory access manifests through attention map
  visualizations.
paper_authors._id: 66bbc1a985f17cea2cf9a8a8
paper_authors.name: MohammadReza Ebrahimi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T08:11:39.279Z
paper_authors._id: 648a1075e8bee5332919e0ab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648a1075e8bee5332919e0ab/KDzi904zGPuy_PL3gbUYE.jpeg
paper_authors.fullname: M.Reza Ebrahimi
paper_authors.type: user
paper_authors.user: mamaj92
paper_authors._id: 66bbc1a985f17cea2cf9a8a9
paper_authors.name: Sunny Panchal
paper_authors._id: 66bbc1a985f17cea2cf9a8aa
paper_authors.name: Roland Memisevic
paper_title: Your Context Is Not an Array: Unveiling Random Access Limitations in
    Transformers
paper_summary: Despite their recent successes, Transformer-based large language models show
  surprising failure modes. A well-known example of such failure modes is their
  inability to length-generalize: solving problem instances at inference time
  that are longer than those seen during training. In this work, we further
  explore the root cause of this failure by performing a detailed analysis of
  model behaviors on the simple parity task. Our analysis suggests that length
  generalization failures are intricately related to a model's inability to
  perform random memory accesses within its context window. We present supporting
  evidence for this hypothesis by demonstrating the effectiveness of
  methodologies that circumvent the need for indexing or that enable random token
  access indirectly, through content-based addressing. We further show where and
  how the failure to perform random memory access manifests through attention map
  visualizations.
paper_ai_summary: Transformer-based models fail to generalize to longer problem instances due to an inability to randomly access memory within their context window, as demonstrated through analysis of attention maps and examination of effective memory access methodologies.
paper_ai_keywords: Transformer-based models length-generalization parity task random memory access context window content-based addressing attention maps