title: Beyond Scale: the Diversity Coefficient as a Data Quality Metric
    Demonstrates LLMs are Pre-trained on Formally Diverse Data
summary: Current trends to pre-train capable Large Language Models (LLMs) mostly focus
  on scaling of model and dataset size. However, the quality of pre-training data
  is an important factor for training powerful LLMs, yet it is a nebulous concept
  that has not been fully characterized. Therefore, we use the recently proposed
  Task2Vec diversity coefficient to ground and understand formal aspects of data
  quality, to go beyond scale alone. Specifically, we measure the diversity
  coefficient of publicly available pre-training datasets to demonstrate that
  their formal diversity is high when compared to theoretical lower and upper
  bounds. In addition, to build confidence in the diversity coefficient, we
  conduct interpretability experiments and find that the coefficient aligns with
  intuitive properties of diversity, e.g., it increases as the number of latent
  concepts increases. We conclude the diversity coefficient is reliable, show
  it's high for publicly available LLM datasets, and conjecture it can be used to
  build useful diverse datasets for LLMs.
paper_authors._id: 649a3ea6685215f724840d65
paper_authors.name: Alycia Lee
paper_authors._id: 649a3ea6685215f724840d66
paper_authors.name: Brando Miranda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-27T14:14:04.912Z
paper_authors._id: 60ec6453046c81764c7e06f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60ec6453046c81764c7e06f0/gnKlFE5bkpvbDQNb2jrnV.png
paper_authors.fullname: Brando Miranda
paper_authors.type: user
paper_authors.user: brando
paper_authors._id: 649a3ea6685215f724840d67
paper_authors.name: Sanmi Koyejo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-27T14:14:30.111Z
paper_authors._id: 64931e7e2da595588288f161
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
paper_authors.fullname: Sanmi Koyejo
paper_authors.type: user
paper_authors.user: sanmikoyejo
paper_title: Beyond Scale: the Diversity Coefficient as a Data Quality Metric
    Demonstrates LLMs are Pre-trained on Formally Diverse Data
paper_summary: Current trends to pre-train capable Large Language Models (LLMs) mostly focus
  on scaling of model and dataset size. However, the quality of pre-training data
  is an important factor for training powerful LLMs, yet it is a nebulous concept
  that has not been fully characterized. Therefore, we use the recently proposed
  Task2Vec diversity coefficient to ground and understand formal aspects of data
  quality, to go beyond scale alone. Specifically, we measure the diversity
  coefficient of publicly available pre-training datasets to demonstrate that
  their formal diversity is high when compared to theoretical lower and upper
  bounds. In addition, to build confidence in the diversity coefficient, we
  conduct interpretability experiments and find that the coefficient aligns with
  intuitive properties of diversity, e.g., it increases as the number of latent
  concepts increases. We conclude the diversity coefficient is reliable, show
  it's high for publicly available LLM datasets, and conjecture it can be used to
  build useful diverse datasets for LLMs.
paper_ai_keywords: Task2Vec diversity coefficient pre-training datasets diversity coefficient latent concepts