title: TnT-LLM: Text Mining at Scale with Large Language Models
summary: Transforming unstructured text into structured and meaningful forms,
  organized by useful category labels, is a fundamental step in text mining for
  downstream analysis and application. However, most existing methods for
  producing label taxonomies and building text-based label classifiers still rely
  heavily on domain expertise and manual curation, making the process expensive
  and time-consuming. This is particularly challenging when the label space is
  under-specified and large-scale data annotations are unavailable. In this
  paper, we address these challenges with Large Language Models (LLMs), whose
  prompt-based interface facilitates the induction and use of large-scale pseudo
  labels. We propose TnT-LLM, a two-phase framework that employs LLMs to automate
  the process of end-to-end label generation and assignment with minimal human
  effort for any given use-case. In the first phase, we introduce a zero-shot,
  multi-stage reasoning approach which enables LLMs to produce and refine a label
  taxonomy iteratively. In the second phase, LLMs are used as data labelers that
  yield training samples so that lightweight supervised classifiers can be
  reliably built, deployed, and served at scale. We apply TnT-LLM to the analysis
  of user intent and conversational domain for Bing Copilot (formerly Bing Chat),
  an open-domain chat-based search engine. Extensive experiments using both human
  and automatic evaluation metrics demonstrate that TnT-LLM generates more
  accurate and relevant label taxonomies when compared against state-of-the-art
  baselines, and achieves a favorable balance between accuracy and efficiency for
  classification at scale. We also share our practical experiences and insights
  on the challenges and opportunities of using LLMs for large-scale text mining
  in real-world applications.
paper_authors._id: 65fa578be12a435e55d6f1a8
paper_authors.name: Mengting Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-27T08:46:34.885Z
paper_authors._id: 660328d9738bdf26e72dd454
paper_authors.avatarUrl: /avatars/7af51be1cc1dfaaa3a78e0b546d97cf8.svg
paper_authors.fullname: Mengting Wan
paper_authors.type: user
paper_authors.user: mengtingwan
paper_authors._id: 65fa578be12a435e55d6f1a9
paper_authors.name: Tara Safavi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:07:32.971Z
paper_authors._id: 63ffc744cafc9d54985fdf4f
paper_authors.avatarUrl: /avatars/0820cad83df012182cdf4094a7dcf744.svg
paper_authors.fullname: Tara Safavi
paper_authors.type: user
paper_authors.user: tararootcake
paper_authors._id: 65fa578be12a435e55d6f1aa
paper_authors.name: Sujay Kumar Jauhar
paper_authors._id: 65fa578be12a435e55d6f1ab
paper_authors.name: Yujin Kim
paper_authors._id: 65fa578be12a435e55d6f1ac
paper_authors.name: Scott Counts
paper_authors._id: 65fa578be12a435e55d6f1ad
paper_authors.name: Jennifer Neville
paper_authors._id: 65fa578be12a435e55d6f1ae
paper_authors.name: Siddharth Suri
paper_authors._id: 65fa578be12a435e55d6f1af
paper_authors.name: Chirag Shah
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-22T06:55:16.126Z
paper_authors._id: 6624485b0e31d65ecce09748
paper_authors.avatarUrl: /avatars/b78085aa72e673e570843b85b116f8bc.svg
paper_authors.fullname: Chirag Shah
paper_authors.type: user
paper_authors.user: chiragshah
paper_authors._id: 65fa578be12a435e55d6f1b0
paper_authors.name: Ryen W White
paper_authors._id: 65fa578be12a435e55d6f1b1
paper_authors.name: Longqi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:08:30.808Z
paper_authors._id: 64d048b197ca59bcf726bdf2
paper_authors.avatarUrl: /avatars/6384aa7558382cad60ee94d9e08d6182.svg
paper_authors.fullname: Longqi Yang
paper_authors.type: user
paper_authors.user: ylongqi
paper_authors._id: 65fa578be12a435e55d6f1b2
paper_authors.name: Reid Andersen
paper_authors._id: 65fa578be12a435e55d6f1b3
paper_authors.name: Georg Buscher
paper_authors._id: 65fa578be12a435e55d6f1b4
paper_authors.name: Dhruv Joshi
paper_authors._id: 65fa578be12a435e55d6f1b5
paper_authors.name: Nagu Rangan
paper_title: TnT-LLM: Text Mining at Scale with Large Language Models
paper_summary: Transforming unstructured text into structured and meaningful forms,
  organized by useful category labels, is a fundamental step in text mining for
  downstream analysis and application. However, most existing methods for
  producing label taxonomies and building text-based label classifiers still rely
  heavily on domain expertise and manual curation, making the process expensive
  and time-consuming. This is particularly challenging when the label space is
  under-specified and large-scale data annotations are unavailable. In this
  paper, we address these challenges with Large Language Models (LLMs), whose
  prompt-based interface facilitates the induction and use of large-scale pseudo
  labels. We propose TnT-LLM, a two-phase framework that employs LLMs to automate
  the process of end-to-end label generation and assignment with minimal human
  effort for any given use-case. In the first phase, we introduce a zero-shot,
  multi-stage reasoning approach which enables LLMs to produce and refine a label
  taxonomy iteratively. In the second phase, LLMs are used as data labelers that
  yield training samples so that lightweight supervised classifiers can be
  reliably built, deployed, and served at scale. We apply TnT-LLM to the analysis
  of user intent and conversational domain for Bing Copilot (formerly Bing Chat),
  an open-domain chat-based search engine. Extensive experiments using both human
  and automatic evaluation metrics demonstrate that TnT-LLM generates more
  accurate and relevant label taxonomies when compared against state-of-the-art
  baselines, and achieves a favorable balance between accuracy and efficiency for
  classification at scale. We also share our practical experiences and insights
  on the challenges and opportunities of using LLMs for large-scale text mining
  in real-world applications.
paper_ai_summary: TnT-LLM is a two-phase framework using Large Language Models to automate label generation and classification with minimal human effort, demonstrating superior accuracy and efficiency in text mining tasks.
paper_ai_keywords: Large Language Models prompt-based interface zero-shot reasoning multi-stage reasoning label taxonomy data labelers lightweight supervised classifiers text mining classification real-world applications