title: Catwalk: A Unified Language Model Evaluation Framework for Many Datasets
summary: The success of large language models has shifted the evaluation paradigms in
  natural language processing (NLP). The community's interest has drifted towards
  comparing NLP models across many tasks, domains, and datasets, often at an
  extreme scale. This imposes new engineering challenges: efforts in constructing
  datasets and models have been fragmented, and their formats and interfaces are
  incompatible. As a result, it often takes extensive (re)implementation efforts
  to make fair and controlled comparisons at scale.
    Catwalk aims to address these issues. Catwalk provides a unified interface to
  a broad range of existing NLP datasets and models, ranging from both canonical
  supervised training and fine-tuning, to more modern paradigms like in-context
  learning. Its carefully-designed abstractions allow for easy extensions to many
  others. Catwalk substantially lowers the barriers to conducting controlled
  experiments at scale. For example, we finetuned and evaluated over 64 models on
  over 86 datasets with a single command, without writing any code. Maintained by
  the AllenNLP team at the Allen Institute for Artificial Intelligence (AI2),
  Catwalk is an ongoing open-source effort: https://github.com/allenai/catwalk.
paper_authors._id: 65812cc9365f7676d2748f65
paper_authors.name: Dirk Groeneveld
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:00:14.383Z
paper_authors._id: 60369745413a78f892e7339c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
paper_authors.fullname: Dirk Groeneveld
paper_authors.type: user
paper_authors.user: dirkgr
paper_authors._id: 65812cc9365f7676d2748f66
paper_authors.name: Anas Awadalla
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:00:22.472Z
paper_authors._id: 60f1ab3afe0d78e01037eeb1
paper_authors.avatarUrl: /avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
paper_authors.fullname: Anas Awadalla
paper_authors.type: user
paper_authors.user: anas-awadalla
paper_authors._id: 65812cc9365f7676d2748f67
paper_authors.name: Iz Beltagy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:00:31.935Z
paper_authors._id: 5ec07694ed25d76864d553f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1589816744885-5ec07694ed25d76864d553f4.jpeg
paper_authors.fullname: Iz Beltagy
paper_authors.type: user
paper_authors.user: beltagy
paper_authors._id: 65812cc9365f7676d2748f68
paper_authors.name: Akshita Bhagia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:00:38.863Z
paper_authors._id: 6282937a9624ab5f75c977e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
paper_authors.fullname: Akshita Bhagia
paper_authors.type: user
paper_authors.user: akshitab
paper_authors._id: 65812cc9365f7676d2748f69
paper_authors.name: Ian Magnusson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:00:45.665Z
paper_authors._id: 62bddd0b1e22ec8427a0f27e
paper_authors.avatarUrl: /avatars/4528b10d6b94bdbee2fe3c2d28331945.svg
paper_authors.fullname: Ian Magnusson
paper_authors.type: user
paper_authors.user: IanMagnusson
paper_authors._id: 65812cc9365f7676d2748f6a
paper_authors.name: Hao Peng
paper_authors._id: 65812cc9365f7676d2748f6b
paper_authors.name: Oyvind Tafjord
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:01:40.166Z
paper_authors._id: 609b1aeff7c1303cfd1760f3
paper_authors.avatarUrl: /avatars/3cb448514885656466886d7c84e12af1.svg
paper_authors.fullname: Oyvind Tafjord
paper_authors.type: user
paper_authors.user: OyvindTafjord
paper_authors._id: 65812cc9365f7676d2748f6c
paper_authors.name: Pete Walsh
paper_authors._id: 65812cc9365f7676d2748f6d
paper_authors.name: Kyle Richardson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-08T21:57:32.099Z
paper_authors._id: 659c6f5e1d398a238152227d
paper_authors.avatarUrl: /avatars/dba97fe8cb825102f1eae97104a71f64.svg
paper_authors.fullname: Kyle Richardson
paper_authors.type: user
paper_authors.user: yakazimir
paper_authors._id: 65812cc9365f7676d2748f6e
paper_authors.name: Jesse Dodge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:01:56.972Z
paper_authors._id: 6283f38567d336d3e5d5280e
paper_authors.avatarUrl: /avatars/d0a54aaec74a90b050e671c191b87a80.svg
paper_authors.fullname: Jesse Dodge
paper_authors.type: user
paper_authors.user: JesseDodge
paper_title: Catwalk: A Unified Language Model Evaluation Framework for Many Datasets
paper_summary: The success of large language models has shifted the evaluation paradigms in
  natural language processing (NLP). The community's interest has drifted towards
  comparing NLP models across many tasks, domains, and datasets, often at an
  extreme scale. This imposes new engineering challenges: efforts in constructing
  datasets and models have been fragmented, and their formats and interfaces are
  incompatible. As a result, it often takes extensive (re)implementation efforts
  to make fair and controlled comparisons at scale.
    Catwalk aims to address these issues. Catwalk provides a unified interface to
  a broad range of existing NLP datasets and models, ranging from both canonical
  supervised training and fine-tuning, to more modern paradigms like in-context
  learning. Its carefully-designed abstractions allow for easy extensions to many
  others. Catwalk substantially lowers the barriers to conducting controlled
  experiments at scale. For example, we finetuned and evaluated over 64 models on
  over 86 datasets with a single command, without writing any code. Maintained by
  the AllenNLP team at the Allen Institute for Artificial Intelligence (AI2),
  Catwalk is an ongoing open-source effort: https://github.com/allenai/catwalk.
paper_ai_keywords: in-context learning