title: Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn
    More
summary: Large Language Models (LLMs) are discovered to suffer from accurately
  retrieving key information. To address this, we propose Mask-Enhanced
  Autoregressive Prediction (MEAP), a simple yet effective training paradigm that
  seamlessly integrates Masked Language Modeling (MLM) into Next-Token Prediction
  (NTP) to enhance the latter's in-context retrieval capabilities. Specifically,
  MEAP first randomly masks a small fraction of input tokens and then directly
  performs the standard next-token prediction autoregressive using a decoder-only
  Transformer. MEAP eliminates the need for bidirectional attention or
  encoder-decoder architectures for MLM, incurring no additional computational
  overhead during pre-training or inference. Intensive experiments demonstrate
  that MEAP substantially outperforms NTP on key information retrieval and
  long-context reasoning tasks, while performing on par or better on commonsense
  reasoning tasks. The benefits of MEAP also extend to supervised fine-tuning,
  where it shows remarkable advantages in lost-in-the-middle scenarios,
  outperforming NTP by 11.77 percentage points. Our analysis indicates that
  MEAP's effectiveness arises from its ability to promote more distinguishable
  attention scores by concentrating on a reduced set of non-masked tokens. This
  mechanism improves the model's focus on task-relevant signals while mitigating
  the influence of peripheral context. These findings position MEAP as a
  promising training paradigm for large language models.
paper_authors._id: 67ac8bc95cc4f961c9550320
paper_authors.name: Xialie Zhuang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:25:06.266Z
paper_authors._id: 6575a625b951d40e7a4d8685
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6575a625b951d40e7a4d8685/mcnyqU--r-vi11aXN02cZ.jpeg
paper_authors.fullname: zhuangxialie
paper_authors.type: user
paper_authors.user: ZhuangXialie
paper_authors._id: 67ac8bc95cc4f961c9550321
paper_authors.name: Zhikai Jia
paper_authors._id: 67ac8bc95cc4f961c9550322
paper_authors.name: Jianjin Li
paper_authors._id: 67ac8bc95cc4f961c9550323
paper_authors.name: Zhenyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:39:17.521Z
paper_authors._id: 649c888f67fd6c6aa97e5f85
paper_authors.avatarUrl: /avatars/9967b729916d1128773102797fed1673.svg
paper_authors.fullname: Zhenyu Zhang
paper_authors.type: user
paper_authors.user: Kyriection
paper_authors._id: 67ac8bc95cc4f961c9550324
paper_authors.name: Li Shen
paper_authors._id: 67ac8bc95cc4f961c9550325
paper_authors.name: Zheng Cao
paper_authors._id: 67ac8bc95cc4f961c9550326
paper_authors.name: Shiwei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-03T11:15:23.571Z
paper_authors._id: 65b04d2291e63920a7898c9e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65b04d2291e63920a7898c9e/iUHs235G4bqK-KnH_94ti.jpeg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Shiweiliuiiiiiii
paper_title: Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn
    More
paper_summary: Large Language Models (LLMs) are discovered to suffer from accurately
  retrieving key information. To address this, we propose Mask-Enhanced
  Autoregressive Prediction (MEAP), a simple yet effective training paradigm that
  seamlessly integrates Masked Language Modeling (MLM) into Next-Token Prediction
  (NTP) to enhance the latter's in-context retrieval capabilities. Specifically,
  MEAP first randomly masks a small fraction of input tokens and then directly
  performs the standard next-token prediction autoregressive using a decoder-only
  Transformer. MEAP eliminates the need for bidirectional attention or
  encoder-decoder architectures for MLM, incurring no additional computational
  overhead during pre-training or inference. Intensive experiments demonstrate
  that MEAP substantially outperforms NTP on key information retrieval and
  long-context reasoning tasks, while performing on par or better on commonsense
  reasoning tasks. The benefits of MEAP also extend to supervised fine-tuning,
  where it shows remarkable advantages in lost-in-the-middle scenarios,
  outperforming NTP by 11.77 percentage points. Our analysis indicates that
  MEAP's effectiveness arises from its ability to promote more distinguishable
  attention scores by concentrating on a reduced set of non-masked tokens. This
  mechanism improves the model's focus on task-relevant signals while mitigating
  the influence of peripheral context. These findings position MEAP as a
  promising training paradigm for large language models.
paper_ai_summary: MEAP integrates MLM into NTP for LLMs, enhancing key information retrieval and long-context reasoning without extra computational cost.
paper_ai_keywords: Mask-Enhanced Autoregressive Prediction Masked Language Modeling Next-Token Prediction decoder-only Transformer bidirectional attention encoder-decoder architectures key information retrieval long-context reasoning commonsense reasoning supervised fine-tuning lost-in-the-middle scenarios attention scores