  We describe an annotation initiative to capture the scholarly contributions
in natural language processing (NLP) articles, particularly, for the articles
that discuss machine learning (ML) approaches for various information
extraction tasks. We develop the annotation task based on a pilot annotation
exercise on 50 NLP-ML scholarly articles presenting contributions to five
information extraction tasks 1. machine translation, 2. named entity
recognition, 3. question answering, 4. relation classification, and 5. text
classification. In this article, we describe the outcomes of this pilot
annotation phase. Through the exercise we have obtained an annotation
methodology; and found ten core information units that reflect the contribution
of the NLP-ML scholarly investigations. The resulting annotation scheme we
developed based on these information units is called NLPContributions.
  The overarching goal of our endeavor is four-fold: 1) to find a systematic
set of patterns of subject-predicate-object statements for the semantic
structuring of scholarly contributions that are more or less generically
applicable for NLP-ML research articles; 2) to apply the discovered patterns in
the creation of a larger annotated dataset for training machine readers of
research contributions; 3) to ingest the dataset into the Open Research
Knowledge Graph (ORKG) infrastructure as a showcase for creating user-friendly
state-of-the-art overviews; 4) to integrate the machine readers into the ORKG
to assist users in the manual curation of their respective article
contributions. We envision that the NLPContributions methodology engenders a
wider discussion on the topic toward its further refinement and development.
Our pilot annotated dataset of 50 NLP-ML scholarly articles according to the
NLPContributions scheme is openly available to the research community at
https://doi.org/10.25835/0019761.

Computation and Language;Digital Libraries;Information Retrieval