DE-COP: Detecting Copyrighted Content in Language Models Training Data
<|reference_start|>DE-COP: Detecting Copyrighted Content in Language Models Training Data: How can we detect if copyrighted content was used in the training process of a language model, considering that the training data is typically undisclosed? We are motivated by the premise that a language model is likely to identify verbatim excerpts from its training text. We propose DE-COP, a method to determine whether a piece of copyrighted content was included in training. DE-COP's core approach is to probe an LLM with multiple-choice questions, whose options include both verbatim text and their paraphrases. We construct BookTection, a benchmark with excerpts from 165 books published prior and subsequent to a model's training cutoff, along with their paraphrases. Our experiments show that DE-COP surpasses the prior best method by 9.6% in detection performance (AUC) on models with logits available. Moreover, DE-COP also achieves an average accuracy of 72% for detecting suspect books on fully black-box models where prior methods give approximately 4% accuracy. The code and datasets are available at https://github.com/LeiLiLab/DE-COP.<|reference_end|>
@article{duarte2024de-cop:,
    title={DE-COP: Detecting Copyrighted Content in Language Models Training Data},
    author={Andr'e V. Duarte, Xuandong Zhao, Arlindo L. Oliveira and Lei Li},
    journal={arXiv preprint arXiv:2402.09910},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.09910},
    primaryClass={cs.CL cs.LG}
}