Reconsidering Token Embeddings with the Definitions for Pre-trained Language Models
<|reference_start|>Reconsidering Token Embeddings with the Definitions for Pre-trained Language Models: Learning token embeddings based on token co-occurrence statistics has proven effective for both pre-training and fine-tuning in natural language processing. However, recent studies have pointed out the distribution of learned embeddings degenerates into anisotropy, and even pre-trained language models (PLMs) suffer from a loss of semantics-related information in embeddings for low-frequency tokens. This study first analyzes fine-tuning dynamics of a PLM, BART-large, and demonstrates its robustness against degeneration. On the basis of this finding, we propose DefinitionEMB, a method that utilizes definitions to construct isotropically distributed and semantics-related token embeddings for PLMs while maintaining original robustness during fine-tuning. Our experiments demonstrate the effectiveness of leveraging definitions from Wiktionary to construct such embeddings for RoBERTa-base and BART-large. Furthermore, the constructed embeddings for low-frequency tokens improve the performance of these models across various GLUE and four text summarization datasets.<|reference_end|>
@article{zhang2024reconsidering,
    title={Reconsidering Degeneration of Token Embeddings with Definitions for
  Encoder-based Pre-trained Language Models},
    author={Ying Zhang, Dongyuan Li, and Manabu Okumura},
    journal={arXiv preprint arXiv:2408.01308},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.01308},
    primaryClass={cs.CL}
}