In-Context Probing Approximates Influence Function for Data Valuation
<|reference_start|>In-Context Probing Approximates Influence Function for Data Valuation: Data valuation quantifies the value of training data, and is used for data attribution (i.e., determining the contribution of training data towards model predictions), and data selection; both of which are important for curating high-quality datasets to train large language models. In our paper, we show that data valuation through in-context probing (i.e., prompting a LLM) approximates influence functions for selecting training data. We provide a theoretical sketch on this connection based on transformer models performing "implicit" gradient descent on its in-context inputs. Our empirical findings show that in-context probing and gradient-based influence frameworks are similar in how they rank training data. Furthermore, fine-tuning experiments on data selected by either method reveal similar model performance.<|reference_end|>
@article{jiao2024in-context,
    title={In-Context Probing Approximates Influence Function for Data Valuation},
    author={Cathy Jiao and Gary Gao and Chenyan Xiong},
    journal={arXiv preprint arXiv:2407.12259},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12259},
    primaryClass={cs.CL}
}