  Systematic literature reviews (SLRs) play an essential role in summarising,
synthesising and validating scientific evidence. In recent years, there has
been a growing interest in using machine learning techniques to automate the
identification of relevant studies for SLRs. However, the lack of standardised
evaluation datasets makes comparing the performance of such automated
literature screening systems difficult. In this paper, we analyse the citation
screening evaluation datasets, revealing that many of the available datasets
are either too small, suffer from data leakage or have limited applicability to
systems treating automated literature screening as a classification task, as
opposed to, for example, a retrieval or question-answering task. To address
these challenges, we introduce CSMeD, a meta-dataset consolidating nine
publicly released collections, providing unified access to 325 SLRs from the
fields of medicine and computer science. CSMeD serves as a comprehensive
resource for training and evaluating the performance of automated citation
screening models. Additionally, we introduce CSMeD-FT, a new dataset designed
explicitly for evaluating the full text publication screening task. To
demonstrate the utility of CSMeD, we conduct experiments and establish
baselines on new datasets.

Computation and Language;Information Retrieval