title: Cuckoo: An IE Free Rider Hatched by Massive Nutrition in LLM's Nest
summary: Massive high-quality data, both pre-training raw texts and post-training
  annotations, have been carefully prepared to incubate advanced large language
  models (LLMs). In contrast, for information extraction (IE), pre-training data,
  such as BIO-tagged sequences, are hard to scale up. We show that IE models can
  act as free riders on LLM resources by reframing next-token prediction
  into extraction for tokens already present in the context. Specifically,
  our proposed next tokens extraction (NTE) paradigm learns a versatile IE model,
  Cuckoo, with 102.6M extractive data converted from LLM's pre-training
  and post-training data. Under the few-shot setting, Cuckoo adapts effectively
  to traditional and complex instruction-following IE with better performance
  than existing pre-trained IE models. As a free rider, Cuckoo can naturally
  evolve with the ongoing advancements in LLM data preparation, benefiting from
  improvements in LLM training pipelines without additional manual effort.
paper_authors._id: 67b3fa2862838a378b21860d
paper_authors.name: Letian Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:04:47.684Z
paper_authors._id: 64323dd503d81fa4d26deaf9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64323dd503d81fa4d26deaf9/x3ES8VXEZJljxDWvFWaAf.png
paper_authors.fullname: Letian Peng
paper_authors.type: user
paper_authors.user: KomeijiForce
paper_authors._id: 67b3fa2862838a378b21860e
paper_authors.name: Zilong Wang
paper_authors._id: 67b3fa2862838a378b21860f
paper_authors.name: Feng Yao
paper_authors._id: 67b3fa2862838a378b218610
paper_authors.name: Jingbo Shang
paper_title: Cuckoo: An IE Free Rider Hatched by Massive Nutrition in LLM's Nest
paper_summary: Massive high-quality data, both pre-training raw texts and post-training
  annotations, have been carefully prepared to incubate advanced large language
  models (LLMs). In contrast, for information extraction (IE), pre-training data,
  such as BIO-tagged sequences, are hard to scale up. We show that IE models can
  act as free riders on LLM resources by reframing next-token prediction
  into extraction for tokens already present in the context. Specifically,
  our proposed next tokens extraction (NTE) paradigm learns a versatile IE model,
  Cuckoo, with 102.6M extractive data converted from LLM's pre-training
  and post-training data. Under the few-shot setting, Cuckoo adapts effectively
  to traditional and complex instruction-following IE with better performance
  than existing pre-trained IE models. As a free rider, Cuckoo can naturally
  evolve with the ongoing advancements in LLM data preparation, benefiting from
  improvements in LLM training pipelines without additional manual effort.
paper_ai_summary: An IE model named Cuckoo leverages LLM data to perform extraction without requiring additional manual data, showing superior performance in few-shot adaptation to traditional and complex IE tasks.
paper_ai_keywords: large language models LLM information extraction IE BIO-tagged sequences next-token prediction extraction NTE paradigm