TnT-LLM: Text Mining at Scale with Large Language Models
Transforming unstructured text into structured and meaningful forms,
organized by useful category labels, is a fundamental step in text mining for
downstream analysis and application. However, most existing methods for
producing label taxonomies and building text-based label classifiers still rely
heavily on domain expertise and manual curation, making the process expensive
and time-consuming. This is particularly challenging when the label space is
under-specified and large-scale data annotations are unavailable. In this
paper, we address these challenges with Large Language Models (LLMs), whose
prompt-based interface facilitates the induction and use of large-scale pseudo
labels. We propose TnT-LLM, a two-phase framework that employs LLMs to automate
the process of end-to-end label generation and assignment with minimal human
effort for any given use-case. In the first phase, we introduce a zero-shot,
multi-stage reasoning approach which enables LLMs to produce and refine a label
taxonomy iteratively. In the second phase, LLMs are used as data labelers that
yield training samples so that lightweight supervised classifiers can be
reliably built, deployed, and served at scale. We apply TnT-LLM to the analysis
of user intent and conversational domain for Bing Copilot (formerly Bing Chat),
an open-domain chat-based search engine. Extensive experiments using both human
and automatic evaluation metrics demonstrate that TnT-LLM generates more
accurate and relevant label taxonomies when compared against state-of-the-art
baselines, and achieves a favorable balance between accuracy and efficiency for
classification at scale. We also share our practical experiences and insights
on the challenges and opportunities of using LLMs for large-scale text mining
in real-world applications.
65fa578be12a435e55d6f1a8
Mengting Wan
claimed_verified
2024-03-27T08:46:34.885Z
660328d9738bdf26e72dd454
/avatars/7af51be1cc1dfaaa3a78e0b546d97cf8.svg
Mengting Wan
user
mengtingwan
65fa578be12a435e55d6f1a9
Tara Safavi
admin_assigned
2024-03-21T12:07:32.971Z
63ffc744cafc9d54985fdf4f
/avatars/0820cad83df012182cdf4094a7dcf744.svg
Tara Safavi
user
tararootcake
65fa578be12a435e55d6f1aa
Sujay Kumar Jauhar
65fa578be12a435e55d6f1ab
Yujin Kim
65fa578be12a435e55d6f1ac
Scott Counts
65fa578be12a435e55d6f1ad
Jennifer Neville
65fa578be12a435e55d6f1ae
Siddharth Suri
65fa578be12a435e55d6f1af
Chirag Shah
claimed_verified
2024-04-22T06:55:16.126Z
6624485b0e31d65ecce09748
/avatars/b78085aa72e673e570843b85b116f8bc.svg
Chirag Shah
user
chiragshah
65fa578be12a435e55d6f1b0
Ryen W White
65fa578be12a435e55d6f1b1
Longqi Yang
admin_assigned
2024-03-21T12:08:30.808Z
64d048b197ca59bcf726bdf2
/avatars/6384aa7558382cad60ee94d9e08d6182.svg
Longqi Yang
user
ylongqi
65fa578be12a435e55d6f1b2
Reid Andersen
65fa578be12a435e55d6f1b3
Georg Buscher
65fa578be12a435e55d6f1b4
Dhruv Joshi
65fa578be12a435e55d6f1b5
Nagu Rangan
TnT-LLM: Text Mining at Scale with Large Language Models
Transforming unstructured text into structured and meaningful forms,
organized by useful category labels, is a fundamental step in text mining for
downstream analysis and application. However, most existing methods for
producing label taxonomies and building text-based label classifiers still rely
heavily on domain expertise and manual curation, making the process expensive
and time-consuming. This is particularly challenging when the label space is
under-specified and large-scale data annotations are unavailable. In this
paper, we address these challenges with Large Language Models (LLMs), whose
prompt-based interface facilitates the induction and use of large-scale pseudo
labels. We propose TnT-LLM, a two-phase framework that employs LLMs to automate
the process of end-to-end label generation and assignment with minimal human
effort for any given use-case. In the first phase, we introduce a zero-shot,
multi-stage reasoning approach which enables LLMs to produce and refine a label
taxonomy iteratively. In the second phase, LLMs are used as data labelers that
yield training samples so that lightweight supervised classifiers can be
reliably built, deployed, and served at scale. We apply TnT-LLM to the analysis
of user intent and conversational domain for Bing Copilot (formerly Bing Chat),
an open-domain chat-based search engine. Extensive experiments using both human
and automatic evaluation metrics demonstrate that TnT-LLM generates more
accurate and relevant label taxonomies when compared against state-of-the-art
baselines, and achieves a favorable balance between accuracy and efficiency for
classification at scale. We also share our practical experiences and insights
on the challenges and opportunities of using LLMs for large-scale text mining
in real-world applications.
TnT-LLM is a two-phase framework using Large Language Models to automate label generation and classification with minimal human effort, demonstrating superior accuracy and efficiency in text mining tasks.
Large Language Models prompt-based interface zero-shot reasoning multi-stage reasoning label taxonomy data labelers lightweight supervised classifiers text mining classification real-world applications