UniversalNER: Targeted Distillation from Large Language Models for Open
  Named Entity Recognition
Large language models (LLMs) have demonstrated remarkable generalizability,
such as understanding arbitrary entities and relations. Instruction tuning has
proven effective for distilling LLMs into more cost-efficient models such as
Alpaca and Vicuna. Yet such student models still trail the original LLMs by
large margins in downstream applications. In this paper, we explore targeted
distillation with mission-focused instruction tuning to train student models
that can excel in a broad application class such as open information
extraction. Using named entity recognition (NER) for case study, we show how
ChatGPT can be distilled into much smaller UniversalNER models for open NER.
For evaluation, we assemble the largest NER benchmark to date, comprising 43
datasets across 9 diverse domains such as biomedicine, programming, social
media, law, finance. Without using any direct supervision, UniversalNER attains
remarkable NER accuracy across tens of thousands of entity types, outperforming
general instruction-tuned models such as Alpaca and Vicuna by over 30 absolute
F1 points in average. With a tiny fraction of parameters, UniversalNER not only
acquires ChatGPT's capability in recognizing arbitrary entity types, but also
outperforms its NER accuracy by 7-9 absolute F1 points in average. Remarkably,
UniversalNER even outperforms by a large margin state-of-the-art multi-task
instruction-tuned systems such as InstructUIE, which uses supervised NER
examples. We also conduct thorough ablation studies to assess the impact of
various components in our distillation approach. We will release the
distillation recipe, data, and UniversalNER models to facilitate future
research on targeted distillation.
64d1a8e99c245c6ba778dd80
Wenxuan Zhou
admin_assigned
2023-08-08T14:33:52.120Z
619414c7979eb76ec3cc5bab
/avatars/c1a7055db6150b1bba92e7b13266b489.svg
Wenxuan Zhou
user
wzhouad
64d1a8e99c245c6ba778dd81
Sheng Zhang
claimed_verified
2023-08-08T13:45:14.680Z
6234c11b7d5de9839bc44163
/avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
Sheng Zhang
user
shengz
64d1a8e99c245c6ba778dd82
Yu Gu
claimed_verified
2023-08-09T00:43:14.384Z
645a02972abf6165a3ba5df8
/avatars/6166e4f213b9020ba99f853cb04f8db0.svg
Yu Gu
user
aidenygu
64d1a8e99c245c6ba778dd83
Muhao Chen
claimed_verified
2024-06-14T07:26:30.917Z
666be8ef81f01fbd60e84f01
https://cdn-avatars.huggingface.co/v1/production/uploads/666be8ef81f01fbd60e84f01/PiCUqr7XT96HGpa_GVRLr.jpeg
Muhao Chen
user
Muhao
64d1a8e99c245c6ba778dd84
Hoifung Poon
UniversalNER: Targeted Distillation from Large Language Models for Open
  Named Entity Recognition
Large language models (LLMs) have demonstrated remarkable generalizability,
such as understanding arbitrary entities and relations. Instruction tuning has
proven effective for distilling LLMs into more cost-efficient models such as
Alpaca and Vicuna. Yet such student models still trail the original LLMs by
large margins in downstream applications. In this paper, we explore targeted
distillation with mission-focused instruction tuning to train student models
that can excel in a broad application class such as open information
extraction. Using named entity recognition (NER) for case study, we show how
ChatGPT can be distilled into much smaller UniversalNER models for open NER.
For evaluation, we assemble the largest NER benchmark to date, comprising 43
datasets across 9 diverse domains such as biomedicine, programming, social
media, law, finance. Without using any direct supervision, UniversalNER attains
remarkable NER accuracy across tens of thousands of entity types, outperforming
general instruction-tuned models such as Alpaca and Vicuna by over 30 absolute
F1 points in average. With a tiny fraction of parameters, UniversalNER not only
acquires ChatGPT's capability in recognizing arbitrary entity types, but also
outperforms its NER accuracy by 7-9 absolute F1 points in average. Remarkably,
UniversalNER even outperforms by a large margin state-of-the-art multi-task
instruction-tuned systems such as InstructUIE, which uses supervised NER
examples. We also conduct thorough ablation studies to assess the impact of
various components in our distillation approach. We will release the
distillation recipe, data, and UniversalNER models to facilitate future
research on targeted distillation.
Targeted distillation using mission-focused instruction tuning distills ChatGPT into the smaller UniversalNER model, achieving higher NER accuracy across diverse domains than both general and state-of-the-art multi-task tuned models.
large language models LLMs instruction tuning student models targeted distillation mission-focused instruction tuning open information extraction named entity recognition NER UniversalNER multi-task instruction-tuned systems ablation studies