Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn
  More
Large Language Models (LLMs) are discovered to suffer from accurately
retrieving key information. To address this, we propose Mask-Enhanced
Autoregressive Prediction (MEAP), a simple yet effective training paradigm that
seamlessly integrates Masked Language Modeling (MLM) into Next-Token Prediction
(NTP) to enhance the latter's in-context retrieval capabilities. Specifically,
MEAP first randomly masks a small fraction of input tokens and then directly
performs the standard next-token prediction autoregressive using a decoder-only
Transformer. MEAP eliminates the need for bidirectional attention or
encoder-decoder architectures for MLM, incurring no additional computational
overhead during pre-training or inference. Intensive experiments demonstrate
that MEAP substantially outperforms NTP on key information retrieval and
long-context reasoning tasks, while performing on par or better on commonsense
reasoning tasks. The benefits of MEAP also extend to supervised fine-tuning,
where it shows remarkable advantages in lost-in-the-middle scenarios,
outperforming NTP by 11.77 percentage points. Our analysis indicates that
MEAP's effectiveness arises from its ability to promote more distinguishable
attention scores by concentrating on a reduced set of non-masked tokens. This
mechanism improves the model's focus on task-relevant signals while mitigating
the influence of peripheral context. These findings position MEAP as a
promising training paradigm for large language models.
67ac8bc95cc4f961c9550320
Xialie Zhuang
claimed_verified
2025-02-13T08:25:06.266Z
6575a625b951d40e7a4d8685
https://cdn-avatars.huggingface.co/v1/production/uploads/6575a625b951d40e7a4d8685/mcnyqU--r-vi11aXN02cZ.jpeg
Lie
user
Lie24
67ac8bc95cc4f961c9550321
Zhikai Jia
claimed_verified
2025-06-18T12:18:04.853Z
66ab85b1cd26e168a58333d7
/avatars/0bde87e64700aab178fae99120399a9d.svg
Zhikai Jia
user
jazka
67ac8bc95cc4f961c9550322
Jianjin Li
claimed_verified
2025-07-05T07:54:42.467Z
657fb5abcec775bfe0184a90
https://cdn-avatars.huggingface.co/v1/production/uploads/657fb5abcec775bfe0184a90/vxhORbZzNN_x4YPpGwieb.jpeg
Lil2J
user
Lil2J
67ac8bc95cc4f961c9550323
Zhenyu Zhang
claimed_verified
2025-02-26T08:39:17.521Z
649c888f67fd6c6aa97e5f85
/avatars/9967b729916d1128773102797fed1673.svg
Zhenyu Zhang
user
Kyriection
67ac8bc95cc4f961c9550324
Li Shen
67ac8bc95cc4f961c9550325
Zheng Cao
67ac8bc95cc4f961c9550326
Shiwei Liu
claimed_verified
2025-03-03T11:15:23.571Z
65b04d2291e63920a7898c9e
https://cdn-avatars.huggingface.co/v1/production/uploads/65b04d2291e63920a7898c9e/iUHs235G4bqK-KnH_94ti.jpeg
Liu
user
Shiweiliuiiiiiii
Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn
  More
Large Language Models (LLMs) are discovered to suffer from accurately
retrieving key information. To address this, we propose Mask-Enhanced
Autoregressive Prediction (MEAP), a simple yet effective training paradigm that
seamlessly integrates Masked Language Modeling (MLM) into Next-Token Prediction
(NTP) to enhance the latter's in-context retrieval capabilities. Specifically,
MEAP first randomly masks a small fraction of input tokens and then directly
performs the standard next-token prediction autoregressive using a decoder-only
Transformer. MEAP eliminates the need for bidirectional attention or
encoder-decoder architectures for MLM, incurring no additional computational
overhead during pre-training or inference. Intensive experiments demonstrate
that MEAP substantially outperforms NTP on key information retrieval and
long-context reasoning tasks, while performing on par or better on commonsense
reasoning tasks. The benefits of MEAP also extend to supervised fine-tuning,
where it shows remarkable advantages in lost-in-the-middle scenarios,
outperforming NTP by 11.77 percentage points. Our analysis indicates that
MEAP's effectiveness arises from its ability to promote more distinguishable
attention scores by concentrating on a reduced set of non-masked tokens. This
mechanism improves the model's focus on task-relevant signals while mitigating
the influence of peripheral context. These findings position MEAP as a
promising training paradigm for large language models.
MEAP integrates MLM into NTP for LLMs, enhancing key information retrieval and long-context reasoning without extra computational cost.
Mask-Enhanced Autoregressive Prediction Masked Language Modeling Next-Token Prediction decoder-only Transformer bidirectional attention encoder-decoder architectures key information retrieval long-context reasoning commonsense reasoning supervised fine-tuning lost-in-the-middle scenarios attention scores