Cuckoo: An IE Free Rider Hatched by Massive Nutrition in LLM's Nest
Massive high-quality data, both pre-training raw texts and post-training
annotations, have been carefully prepared to incubate advanced large language
models (LLMs). In contrast, for information extraction (IE), pre-training data,
such as BIO-tagged sequences, are hard to scale up. We show that IE models can
act as free riders on LLM resources by reframing next-token prediction
into extraction for tokens already present in the context. Specifically,
our proposed next tokens extraction (NTE) paradigm learns a versatile IE model,
Cuckoo, with 102.6M extractive data converted from LLM's pre-training
and post-training data. Under the few-shot setting, Cuckoo adapts effectively
to traditional and complex instruction-following IE with better performance
than existing pre-trained IE models. As a free rider, Cuckoo can naturally
evolve with the ongoing advancements in LLM data preparation, benefiting from
improvements in LLM training pipelines without additional manual effort.
67b3fa2862838a378b21860d
Letian Peng
claimed_verified
2025-02-19T09:04:47.684Z
64323dd503d81fa4d26deaf9
https://cdn-avatars.huggingface.co/v1/production/uploads/64323dd503d81fa4d26deaf9/x3ES8VXEZJljxDWvFWaAf.png
Letian Peng
user
KomeijiForce
67b3fa2862838a378b21860e
Zilong Wang
67b3fa2862838a378b21860f
Feng Yao
67b3fa2862838a378b218610
Jingbo Shang
Cuckoo: An IE Free Rider Hatched by Massive Nutrition in LLM's Nest
Massive high-quality data, both pre-training raw texts and post-training
annotations, have been carefully prepared to incubate advanced large language
models (LLMs). In contrast, for information extraction (IE), pre-training data,
such as BIO-tagged sequences, are hard to scale up. We show that IE models can
act as free riders on LLM resources by reframing next-token prediction
into extraction for tokens already present in the context. Specifically,
our proposed next tokens extraction (NTE) paradigm learns a versatile IE model,
Cuckoo, with 102.6M extractive data converted from LLM's pre-training
and post-training data. Under the few-shot setting, Cuckoo adapts effectively
to traditional and complex instruction-following IE with better performance
than existing pre-trained IE models. As a free rider, Cuckoo can naturally
evolve with the ongoing advancements in LLM data preparation, benefiting from
improvements in LLM training pipelines without additional manual effort.
An IE model named Cuckoo leverages LLM data to perform extraction without requiring additional manual data, showing superior performance in few-shot adaptation to traditional and complex IE tasks.
large language models LLM information extraction IE BIO-tagged sequences next-token prediction extraction NTE paradigm