title: Specialized Language Models with Cheap Inference from Limited Domain
    Data
summary: Large language models have emerged as a versatile tool but are challenging to
  apply to tasks lacking large inference budgets and large in-domain training
  sets. This work formalizes these constraints and distinguishes four important
  variables: the pretraining budget (for training before the target domain is
  known), the specialization budget (for training after the target domain is
  known), the inference budget, and the in-domain training set size. Across these
  settings, we compare different approaches from the machine learning literature.
  Limited by inference cost, we find better alternatives to the standard practice
  of training very large vanilla transformer models. In particular, we show that
  hyper-networks and mixture of experts have better perplexity for large
  pretraining budgets, while small models trained on importance sampled datasets
  are attractive for large specialization budgets.
paper_authors._id: 65c04b5fef796a0c7f6b3337
paper_authors.name: David Grangier
paper_authors._id: 65c04b5fef796a0c7f6b3338
paper_authors.name: Angelos Katharopoulos
paper_authors._id: 65c04b5fef796a0c7f6b3339
paper_authors.name: Pierre Ablin
paper_authors._id: 65c04b5fef796a0c7f6b333a
paper_authors.name: Awni Hannun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-05T11:56:06.024Z
paper_authors._id: 623c830997ddced06d78699b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623c830997ddced06d78699b/ucB8joTPONCnc_Gj0mssR.jpeg
paper_authors.fullname: Awni Hannun
paper_authors.type: user
paper_authors.user: awni
paper_title: Specialized Language Models with Cheap Inference from Limited Domain
    Data
paper_summary: Large language models have emerged as a versatile tool but are challenging to
  apply to tasks lacking large inference budgets and large in-domain training
  sets. This work formalizes these constraints and distinguishes four important
  variables: the pretraining budget (for training before the target domain is
  known), the specialization budget (for training after the target domain is
  known), the inference budget, and the in-domain training set size. Across these
  settings, we compare different approaches from the machine learning literature.
  Limited by inference cost, we find better alternatives to the standard practice
  of training very large vanilla transformer models. In particular, we show that
  hyper-networks and mixture of experts have better perplexity for large
  pretraining budgets, while small models trained on importance sampled datasets
  are attractive for large specialization budgets.
paper_ai_keywords: large language models inference budget in-domain training set size training budget specialization budget vanilla transformer models hyper-networks mixture of experts perplexity importance sampled datasets