TRAMS: Training-free Memory Selection for Long-range Language Modeling
The Transformer architecture is crucial for numerous AI models, but it still
faces challenges in long-range language modeling. Though several specific
transformer architectures have been designed to tackle issues of long-range
dependencies, existing methods like Transformer-XL are plagued by a high
percentage of ineffective memories. In this study, we present a plug-and-play
strategy, known as TRAining-free Memory Selection (TRAMS), that selects tokens
participating in attention calculation based on one simple metric. This
strategy allows us to keep tokens that are likely to have a high attention
score with the current queries and ignore the other ones. We have tested our
approach on the word-level benchmark (WikiText-103) and the character-level
benchmark (enwik8), and the results indicate an improvement without having
additional training or adding additional parameters.
6538872d678950f2f184e173
Haofei Yu
admin_assigned
2023-10-26T11:53:14.949Z
636453547cf2c0b4f0a3ee1e
/avatars/a453340b44d08eec2281ecbe5e993707.svg
Haofei Yu
user
lwaekfjlk
6538872d678950f2f184e174
Cunxiang wang
6538872d678950f2f184e175
Yue Zhang
6538872d678950f2f184e176
Wei Bi
TRAMS: Training-free Memory Selection for Long-range Language Modeling
The Transformer architecture is crucial for numerous AI models, but it still
faces challenges in long-range language modeling. Though several specific
transformer architectures have been designed to tackle issues of long-range
dependencies, existing methods like Transformer-XL are plagued by a high
percentage of ineffective memories. In this study, we present a plug-and-play
strategy, known as TRAining-free Memory Selection (TRAMS), that selects tokens
participating in attention calculation based on one simple metric. This
strategy allows us to keep tokens that are likely to have a high attention
score with the current queries and ignore the other ones. We have tested our
approach on the word-level benchmark (WikiText-103) and the character-level
benchmark (enwik8), and the results indicate an improvement without having
additional training or adding additional parameters.
A new memory selection strategy in the Transformer architecture enhances long-range language modeling by selecting relevant tokens based on a simple metric, improving performance without additional training or parameters.
Transformer long-range language modeling Transformer-XL TRAining-free Memory Selection (TRAMS) attention calculation attention score word-level benchmark character-level benchmark