N-gram Prediction and Word Difference Representations for Language Modeling
<|reference_start|>N-gram Prediction and Word Difference Representations for Language Modeling: Causal language modeling (CLM) serves as the foundational framework underpinning remarkable successes of recent large language models (LLMs). Despite its success, the training approach for next word prediction poses a potential risk of causing the model to overly focus on local dependencies within a sentence. While prior studies have been introduced to predict future N words simultaneously, they were primarily applied to tasks such as masked language modeling (MLM) and neural machine translation (NMT). In this study, we introduce a simple N-gram prediction framework for the CLM task. Moreover, we introduce word difference representation (WDR) as a surrogate and contextualized target representation during model training on the basis of N-gram prediction framework. To further enhance the quality of next word prediction, we propose an ensemble method that incorporates the future N words' prediction results. Empirical evaluations across multiple benchmark datasets encompassing CLM and NMT tasks demonstrate the significant advantages of our proposed methods over the conventional CLM.<|reference_end|>
@article{heo2024n-gram,
    title={N-gram Prediction and Word Difference Representations for Language
  Modeling},
    author={DongNyeong Heo, Daniela Noemi Rim, Heeyoul Choi},
    journal={arXiv preprint arXiv:2409.03295},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.03295},
    primaryClass={cs.CL cs.AI}
}