2.2 Task Domains and Evaluation

Properly assessing the capabilities of LLMs remains an active research area, due to the diversity and
breadth of evaluations to perform. Question-answering datasets (Joshi et al., 2017; Kwiatkowski
et al., 2019; Lin et al., 2022) are very popular evaluation benchmarks, but new benchmarks tailored

5https://commoncrawl.org

4

for LLM assessments have also emerged recently (Dubois et al., 2023; Beeching et al., 2023; Zheng
et al., 2023). In the following section, we explore LLMs capacities across 6 main dimensions:
general capabilities, agent capabilities, logical reasoning (including maths and coding capacities),
long-context modelling, specific applications such as QA or summarization, and trustworthiness.

3 Open-Source LLMs vs. ChatGPT

3.1 General Capabilities
ChatGPT’sOne-yearAnniversary-AreOpen-Source LargeLanguageModelsCatchingup