Catwalk: A Unified Language Model Evaluation Framework for Many Datasets
The success of large language models has shifted the evaluation paradigms in
natural language processing (NLP). The community's interest has drifted towards
comparing NLP models across many tasks, domains, and datasets, often at an
extreme scale. This imposes new engineering challenges: efforts in constructing
datasets and models have been fragmented, and their formats and interfaces are
incompatible. As a result, it often takes extensive (re)implementation efforts
to make fair and controlled comparisons at scale.
  Catwalk aims to address these issues. Catwalk provides a unified interface to
a broad range of existing NLP datasets and models, ranging from both canonical
supervised training and fine-tuning, to more modern paradigms like in-context
learning. Its carefully-designed abstractions allow for easy extensions to many
others. Catwalk substantially lowers the barriers to conducting controlled
experiments at scale. For example, we finetuned and evaluated over 64 models on
over 86 datasets with a single command, without writing any code. Maintained by
the AllenNLP team at the Allen Institute for Artificial Intelligence (AI2),
Catwalk is an ongoing open-source effort: https://github.com/allenai/catwalk.
65812cc9365f7676d2748f65
Dirk Groeneveld
admin_assigned
2023-12-19T10:00:14.383Z
60369745413a78f892e7339c
https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
Dirk Groeneveld
user
dirkgr
65812cc9365f7676d2748f66
Anas Awadalla
admin_assigned
2023-12-19T10:00:22.472Z
60f1ab3afe0d78e01037eeb1
/avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
Anas Awadalla
user
anas-awadalla
65812cc9365f7676d2748f67
Iz Beltagy
admin_assigned
2023-12-19T10:00:31.935Z
5ec07694ed25d76864d553f4
https://cdn-avatars.huggingface.co/v1/production/uploads/1589816744885-5ec07694ed25d76864d553f4.jpeg
Iz Beltagy
user
beltagy
65812cc9365f7676d2748f68
Akshita Bhagia
admin_assigned
2023-12-19T10:00:38.863Z
6282937a9624ab5f75c977e1
https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
Akshita Bhagia
user
akshitab
65812cc9365f7676d2748f69
Ian Magnusson
admin_assigned
2023-12-19T10:00:45.665Z
62bddd0b1e22ec8427a0f27e
/avatars/c433d0fb2945c70d1998f1e0e98d265f.svg
Ian Magnusson
user
IanMagnusson
65812cc9365f7676d2748f6a
Hao Peng
65812cc9365f7676d2748f6b
Oyvind Tafjord
admin_assigned
2023-12-19T10:01:40.166Z
609b1aeff7c1303cfd1760f3
/avatars/3cb448514885656466886d7c84e12af1.svg
Oyvind Tafjord
user
OyvindTafjord
65812cc9365f7676d2748f6c
Pete Walsh
65812cc9365f7676d2748f6d
Kyle Richardson
claimed_verified
2024-01-08T21:57:32.099Z
659c6f5e1d398a238152227d
/avatars/dba97fe8cb825102f1eae97104a71f64.svg
Kyle Richardson
user
yakazimir
65812cc9365f7676d2748f6e
Jesse Dodge
admin_assigned
2023-12-19T10:01:56.972Z
6283f38567d336d3e5d5280e
/avatars/d0a54aaec74a90b050e671c191b87a80.svg
Jesse Dodge
user
JesseDodge
Catwalk: A Unified Language Model Evaluation Framework for Many Datasets
The success of large language models has shifted the evaluation paradigms in
natural language processing (NLP). The community's interest has drifted towards
comparing NLP models across many tasks, domains, and datasets, often at an
extreme scale. This imposes new engineering challenges: efforts in constructing
datasets and models have been fragmented, and their formats and interfaces are
incompatible. As a result, it often takes extensive (re)implementation efforts
to make fair and controlled comparisons at scale.
  Catwalk aims to address these issues. Catwalk provides a unified interface to
a broad range of existing NLP datasets and models, ranging from both canonical
supervised training and fine-tuning, to more modern paradigms like in-context
learning. Its carefully-designed abstractions allow for easy extensions to many
others. Catwalk substantially lowers the barriers to conducting controlled
experiments at scale. For example, we finetuned and evaluated over 64 models on
over 86 datasets with a single command, without writing any code. Maintained by
the AllenNLP team at the Allen Institute for Artificial Intelligence (AI2),
Catwalk is an ongoing open-source effort: https://github.com/allenai/catwalk.
Catwalk provides a unified interface for evaluating large language models across various tasks and datasets, reducing the need for extensive code reimplementation.
large language models natural language processing NLP datasets models canonical supervised training fine-tuning in-context learning abstractions controlled experiments