title: UniversalNER: Targeted Distillation from Large Language Models for Open
    Named Entity Recognition
summary: Large language models (LLMs) have demonstrated remarkable generalizability,
  such as understanding arbitrary entities and relations. Instruction tuning has
  proven effective for distilling LLMs into more cost-efficient models such as
  Alpaca and Vicuna. Yet such student models still trail the original LLMs by
  large margins in downstream applications. In this paper, we explore targeted
  distillation with mission-focused instruction tuning to train student models
  that can excel in a broad application class such as open information
  extraction. Using named entity recognition (NER) for case study, we show how
  ChatGPT can be distilled into much smaller UniversalNER models for open NER.
  For evaluation, we assemble the largest NER benchmark to date, comprising 43
  datasets across 9 diverse domains such as biomedicine, programming, social
  media, law, finance. Without using any direct supervision, UniversalNER attains
  remarkable NER accuracy across tens of thousands of entity types, outperforming
  general instruction-tuned models such as Alpaca and Vicuna by over 30 absolute
  F1 points in average. With a tiny fraction of parameters, UniversalNER not only
  acquires ChatGPT's capability in recognizing arbitrary entity types, but also
  outperforms its NER accuracy by 7-9 absolute F1 points in average. Remarkably,
  UniversalNER even outperforms by a large margin state-of-the-art multi-task
  instruction-tuned systems such as InstructUIE, which uses supervised NER
  examples. We also conduct thorough ablation studies to assess the impact of
  various components in our distillation approach. We will release the
  distillation recipe, data, and UniversalNER models to facilitate future
  research on targeted distillation.
paper_authors._id: 64d1a8e99c245c6ba778dd80
paper_authors.name: Wenxuan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T14:33:52.120Z
paper_authors._id: 619414c7979eb76ec3cc5bab
paper_authors.avatarUrl: /avatars/c1a7055db6150b1bba92e7b13266b489.svg
paper_authors.fullname: Wenxuan Zhou
paper_authors.type: user
paper_authors.user: wzhouad
paper_authors._id: 64d1a8e99c245c6ba778dd81
paper_authors.name: Sheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-08T13:45:14.680Z
paper_authors._id: 6234c11b7d5de9839bc44163
paper_authors.avatarUrl: /avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
paper_authors.fullname: Sheng Zhang
paper_authors.type: user
paper_authors.user: shengz
paper_authors._id: 64d1a8e99c245c6ba778dd82
paper_authors.name: Yu Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-09T00:43:14.384Z
paper_authors._id: 645a02972abf6165a3ba5df8
paper_authors.avatarUrl: /avatars/6166e4f213b9020ba99f853cb04f8db0.svg
paper_authors.fullname: Yu Gu
paper_authors.type: user
paper_authors.user: aidenygu
paper_authors._id: 64d1a8e99c245c6ba778dd83
paper_authors.name: Muhao Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:26:30.917Z
paper_authors._id: 666be8ef81f01fbd60e84f01
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/666be8ef81f01fbd60e84f01/PiCUqr7XT96HGpa_GVRLr.jpeg
paper_authors.fullname: Muhao Chen
paper_authors.type: user
paper_authors.user: Muhao
paper_authors._id: 64d1a8e99c245c6ba778dd84
paper_authors.name: Hoifung Poon
paper_title: UniversalNER: Targeted Distillation from Large Language Models for Open
    Named Entity Recognition
paper_summary: Large language models (LLMs) have demonstrated remarkable generalizability,
  such as understanding arbitrary entities and relations. Instruction tuning has
  proven effective for distilling LLMs into more cost-efficient models such as
  Alpaca and Vicuna. Yet such student models still trail the original LLMs by
  large margins in downstream applications. In this paper, we explore targeted
  distillation with mission-focused instruction tuning to train student models
  that can excel in a broad application class such as open information
  extraction. Using named entity recognition (NER) for case study, we show how
  ChatGPT can be distilled into much smaller UniversalNER models for open NER.
  For evaluation, we assemble the largest NER benchmark to date, comprising 43
  datasets across 9 diverse domains such as biomedicine, programming, social
  media, law, finance. Without using any direct supervision, UniversalNER attains
  remarkable NER accuracy across tens of thousands of entity types, outperforming
  general instruction-tuned models such as Alpaca and Vicuna by over 30 absolute
  F1 points in average. With a tiny fraction of parameters, UniversalNER not only
  acquires ChatGPT's capability in recognizing arbitrary entity types, but also
  outperforms its NER accuracy by 7-9 absolute F1 points in average. Remarkably,
  UniversalNER even outperforms by a large margin state-of-the-art multi-task
  instruction-tuned systems such as InstructUIE, which uses supervised NER
  examples. We also conduct thorough ablation studies to assess the impact of
  various components in our distillation approach. We will release the
  distillation recipe, data, and UniversalNER models to facilitate future
  research on targeted distillation.
paper_ai_summary: Targeted distillation using mission-focused instruction tuning distills ChatGPT into the smaller UniversalNER model, achieving higher NER accuracy across diverse domains than both general and state-of-the-art multi-task tuned models.
paper_ai_keywords: large language models LLMs instruction tuning student models targeted distillation mission-focused instruction tuning open information extraction named entity recognition NER UniversalNER multi-task instruction-tuned systems ablation studies