Token-wise Influential Training Data Retrieval for Large Language Models
<|reference_start|>Token-wise Influential Training Data Retrieval for Large Language Models: Given a Large Language Model (LLM) generation, how can we identify which training data led to this generation? In this paper, we proposed RapidIn, a scalable framework adapting to LLMs for estimating the influence of each training data. The proposed framework consists of two stages: caching and retrieval. First, we compress the gradient vectors by over 200,000x, allowing them to be cached on disk or in GPU/CPU memory. Then, given a generation, RapidIn efficiently traverses the cached gradients to estimate the influence within minutes, achieving over a 6,326x speedup. Moreover, RapidIn supports multi-GPU parallelization to substantially accelerate caching and retrieval. Our empirical result confirms the efficiency and effectiveness of RapidIn.<|reference_end|>
@article{lin2024token-wise,
    title={Token-wise Influential Training Data Retrieval for Large Language Models},
    author={Huawei Lin, Jikai Long, Zhaozhuo Xu, Weijie Zhao},
    journal={arXiv preprint arXiv:2405.11724},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.11724},
    primaryClass={cs.CL cs.AI cs.CR cs.IR}
}