title: TRAMS: Training-free Memory Selection for Long-range Language Modeling
summary: The Transformer architecture is crucial for numerous AI models, but it still
  faces challenges in long-range language modeling. Though several specific
  transformer architectures have been designed to tackle issues of long-range
  dependencies, existing methods like Transformer-XL are plagued by a high
  percentage of ineffective memories. In this study, we present a plug-and-play
  strategy, known as TRAining-free Memory Selection (TRAMS), that selects tokens
  participating in attention calculation based on one simple metric. This
  strategy allows us to keep tokens that are likely to have a high attention
  score with the current queries and ignore the other ones. We have tested our
  approach on the word-level benchmark (WikiText-103) and the character-level
  benchmark (enwik8), and the results indicate an improvement without having
  additional training or adding additional parameters.
paper_authors._id: 6538872d678950f2f184e173
paper_authors.name: Haofei Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:53:14.949Z
paper_authors._id: 636453547cf2c0b4f0a3ee1e
paper_authors.avatarUrl: /avatars/29b4bf0e6abd3b70bb0dbd58188e4ac8.svg
paper_authors.fullname: Haofei Yu
paper_authors.type: user
paper_authors.user: lwaekfjlk
paper_authors._id: 6538872d678950f2f184e174
paper_authors.name: Cunxiang wang
paper_authors._id: 6538872d678950f2f184e175
paper_authors.name: Yue Zhang
paper_authors._id: 6538872d678950f2f184e176
paper_authors.name: Wei Bi
paper_title: TRAMS: Training-free Memory Selection for Long-range Language Modeling
paper_summary: The Transformer architecture is crucial for numerous AI models, but it still
  faces challenges in long-range language modeling. Though several specific
  transformer architectures have been designed to tackle issues of long-range
  dependencies, existing methods like Transformer-XL are plagued by a high
  percentage of ineffective memories. In this study, we present a plug-and-play
  strategy, known as TRAining-free Memory Selection (TRAMS), that selects tokens
  participating in attention calculation based on one simple metric. This
  strategy allows us to keep tokens that are likely to have a high attention
  score with the current queries and ignore the other ones. We have tested our
  approach on the word-level benchmark (WikiText-103) and the character-level
  benchmark (enwik8), and the results indicate an improvement without having
  additional training or adding additional parameters.
paper_ai_keywords: Transformer architecture long-range language modeling long-range dependencies Transformer-XL TRAining-free Memory Selection (TRAMS) attention calculation