title: Paloma: A Benchmark for Evaluating Language Model Fit
summary: Language models (LMs) commonly report perplexity on monolithic data held out
  from training. Implicitly or explicitly, this data is composed of
  domainsx2013varying distributions of language. Rather than assuming
  perplexity on one distribution extrapolates to others, Perplexity Analysis for
  Language Model Assessment (Paloma), measures LM fit to 585 text domains,
  ranging from nytimes.com to r/depression on Reddit. We invite submissions to
  our benchmark and organize results by comparability based on compliance with
  guidelines such as removal of benchmark contamination from pretraining.
  Submissions can also record parameter and training token count to make
  comparisons of Pareto efficiency for performance as a function of these
  measures of cost. We populate our benchmark with results from 6 baselines
  pretrained on popular corpora. In case studies, we demonstrate analyses that
  are possible with Paloma, such as finding that pretraining without data beyond
  Common Crawl leads to inconsistent fit to many domains.
paper_authors._id: 65812daf1fb6cfa0b48e30e2
paper_authors.name: Ian Magnusson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T09:34:09.700Z
paper_authors._id: 62bddd0b1e22ec8427a0f27e
paper_authors.avatarUrl: /avatars/4528b10d6b94bdbee2fe3c2d28331945.svg
paper_authors.fullname: Ian Magnusson
paper_authors.type: user
paper_authors.user: IanMagnusson
paper_authors._id: 65812daf1fb6cfa0b48e30e3
paper_authors.name: Akshita Bhagia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T09:34:16.985Z
paper_authors._id: 6282937a9624ab5f75c977e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
paper_authors.fullname: Akshita Bhagia
paper_authors.type: user
paper_authors.user: akshitab
paper_authors._id: 65812daf1fb6cfa0b48e30e4
paper_authors.name: Valentin Hofmann
paper_authors._id: 65812daf1fb6cfa0b48e30e5
paper_authors.name: Luca Soldaini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T09:34:39.062Z
paper_authors._id: 5f04d8c45d08220171a0ad32
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
paper_authors.fullname: Luca Soldaini
paper_authors.type: user
paper_authors.user: soldni
paper_authors._id: 65812daf1fb6cfa0b48e30e6
paper_authors.name: Ananya Harsh Jha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-20T10:17:05.811Z
paper_authors._id: 6582bee321aa786b76beba44
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6582bee321aa786b76beba44/1YqJBHmLxMNRaGtU-lbKh.jpeg
paper_authors.fullname: Ananya Harsh Jha
paper_authors.type: user
paper_authors.user: ananyahjha93
paper_authors._id: 65812daf1fb6cfa0b48e30e7
paper_authors.name: Oyvind Tafjord
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T09:35:10.332Z
paper_authors._id: 609b1aeff7c1303cfd1760f3
paper_authors.avatarUrl: /avatars/3cb448514885656466886d7c84e12af1.svg
paper_authors.fullname: Oyvind Tafjord
paper_authors.type: user
paper_authors.user: OyvindTafjord
paper_authors._id: 65812daf1fb6cfa0b48e30e8
paper_authors.name: Dustin Schwenk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T09:35:17.065Z
paper_authors._id: 647e791b7f9ad5e44babb965
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647e791b7f9ad5e44babb965/Vjfy4c3ooDwU24N7mvRdy.jpeg
paper_authors.fullname: Dustin Schwenk
paper_authors.type: user
paper_authors.user: drschwenk
paper_authors._id: 65812daf1fb6cfa0b48e30e9
paper_authors.name: Evan Pete Walsh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T09:35:30.295Z
paper_authors._id: 5eec0b3173936372ea2026d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1623085897467-5eec0b3173936372ea2026d3.jpeg
paper_authors.fullname: Evan Pete Walsh
paper_authors.type: user
paper_authors.user: epwalsh
paper_authors._id: 65812daf1fb6cfa0b48e30ea
paper_authors.name: Yanai Elazar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T09:35:39.162Z
paper_authors._id: 623ca115a795593324c4353f
paper_authors.avatarUrl: /avatars/bf11fe728df2786d52ed4d2de12b48d3.svg
paper_authors.fullname: Yanai Elazar
paper_authors.type: user
paper_authors.user: yanaiela
paper_authors._id: 65812daf1fb6cfa0b48e30eb
paper_authors.name: Kyle Lo
paper_authors._id: 65812daf1fb6cfa0b48e30ec
paper_authors.name: Dirk Groeneveld
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T09:36:01.953Z
paper_authors._id: 60369745413a78f892e7339c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
paper_authors.fullname: Dirk Groeneveld
paper_authors.type: user
paper_authors.user: dirkgr
paper_authors._id: 65812daf1fb6cfa0b48e30ed
paper_authors.name: Iz Beltagy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T09:36:09.411Z
paper_authors._id: 5ec07694ed25d76864d553f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1589816744885-5ec07694ed25d76864d553f4.jpeg
paper_authors.fullname: Iz Beltagy
paper_authors.type: user
paper_authors.user: beltagy
paper_authors._id: 65812daf1fb6cfa0b48e30ee
paper_authors.name: Hannaneh Hajishirzi
paper_authors._id: 65812daf1fb6cfa0b48e30ef
paper_authors.name: Noah A. Smith
paper_authors._id: 65812daf1fb6cfa0b48e30f0
paper_authors.name: Kyle Richardson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-08T21:57:30.483Z
paper_authors._id: 659c6f5e1d398a238152227d
paper_authors.avatarUrl: /avatars/dba97fe8cb825102f1eae97104a71f64.svg
paper_authors.fullname: Kyle Richardson
paper_authors.type: user
paper_authors.user: yakazimir
paper_authors._id: 65812daf1fb6cfa0b48e30f1
paper_authors.name: Jesse Dodge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T09:36:53.685Z
paper_authors._id: 6283f38567d336d3e5d5280e
paper_authors.avatarUrl: /avatars/d0a54aaec74a90b050e671c191b87a80.svg
paper_authors.fullname: Jesse Dodge
paper_authors.type: user
paper_authors.user: JesseDodge
paper_title: Paloma: A Benchmark for Evaluating Language Model Fit
paper_summary: Language models (LMs) commonly report perplexity on monolithic data held out
  from training. Implicitly or explicitly, this data is composed of
  domainsx2013varying distributions of language. Rather than assuming
  perplexity on one distribution extrapolates to others, Perplexity Analysis for
  Language Model Assessment (Paloma), measures LM fit to 585 text domains,
  ranging from nytimes.com to r/depression on Reddit. We invite submissions to
  our benchmark and organize results by comparability based on compliance with
  guidelines such as removal of benchmark contamination from pretraining.
  Submissions can also record parameter and training token count to make
  comparisons of Pareto efficiency for performance as a function of these
  measures of cost. We populate our benchmark with results from 6 baselines
  pretrained on popular corpora. In case studies, we demonstrate analyses that
  are possible with Paloma, such as finding that pretraining without data beyond
  Common Crawl leads to inconsistent fit to many domains.
paper_ai_keywords: perplexity PromptDomains Perplexity Analysis for Language Model Assessment (Paloma) text domains benchmark contamination Pareto efficiency pretraining Common Crawl