title: Language models are weak learners
summary: A central notion in practical and theoretical machine learning is that of a
  weak learner, classifiers that achieve better-than-random
  performance (on any given distribution over data), even by a small margin. Such
  weak learners form the practical basis for canonical machine learning methods
  such as boosting. In this work, we illustrate that prompt-based large language
  models can operate effectively as said weak learners. Specifically, we
  illustrate the use of a large language model (LLM) as a weak learner in a
  boosting algorithm applied to tabular data. We show that by providing (properly
  sampled according to the distribution of interest) text descriptions of tabular
  data samples, LLMs can produce a summary of the samples that serves as a
  template for classification and achieves the aim of acting as a weak learner on
  this task. We incorporate these models into a boosting approach, which in some
  settings can leverage the knowledge within the LLM to outperform traditional
  tree-based boosting. The model outperforms both few-shot learning and
  occasionally even more involved fine-tuning procedures, particularly for tasks
  involving small numbers of data points. The results illustrate the potential
  for prompt-based LLMs to function not just as few-shot learners themselves, but
  as components of larger machine learning pipelines.
paper_authors._id: 649a48b5478082a13ece75c5
paper_authors.name: Hariharan Manikandan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-29T07:30:55.364Z
paper_authors._id: 63e8013ee02ee67e8e56da42
paper_authors.avatarUrl: /avatars/b4c514433c3c72bd3b09dbf20fb76a09.svg
paper_authors.fullname: Hariharan Manikandan
paper_authors.type: user
paper_authors.user: vshshv3
paper_authors._id: 649a48b5478082a13ece75c6
paper_authors.name: Yiding Jiang
paper_authors._id: 649a48b5478082a13ece75c7
paper_authors.name: J Zico Kolter
paper_title: Language models are weak learners
paper_summary: A central notion in practical and theoretical machine learning is that of a
  weak learner, classifiers that achieve better-than-random
  performance (on any given distribution over data), even by a small margin. Such
  weak learners form the practical basis for canonical machine learning methods
  such as boosting. In this work, we illustrate that prompt-based large language
  models can operate effectively as said weak learners. Specifically, we
  illustrate the use of a large language model (LLM) as a weak learner in a
  boosting algorithm applied to tabular data. We show that by providing (properly
  sampled according to the distribution of interest) text descriptions of tabular
  data samples, LLMs can produce a summary of the samples that serves as a
  template for classification and achieves the aim of acting as a weak learner on
  this task. We incorporate these models into a boosting approach, which in some
  settings can leverage the knowledge within the LLM to outperform traditional
  tree-based boosting. The model outperforms both few-shot learning and
  occasionally even more involved fine-tuning procedures, particularly for tasks
  involving small numbers of data points. The results illustrate the potential
  for prompt-based LLMs to function not just as few-shot learners themselves, but
  as components of larger machine learning pipelines.
paper_ai_summary: Prompt-based large language models can effectively serve as weak learners in boosting algorithms, enhancing performance on tabular data with limited samples compared to traditional methods.
paper_ai_keywords: weak learner boosting large language models LLM tabular data few-shot learning fine-tuning classification