domain, some classification tasks, including MQP (McCreery et al., 2020), RCT (Dernoncourt &
Lee, 2017), and ChemProt (Kringelum et al., 2016), are too challenging for the model, thus we
conduct few-shot prompting and maintain the number of demonstrations the same in each class.
Fine-tuning. In fine-tuning, we utilize a fixed prompt template (the one displayed in Table 11) for
each task to convert input-output into question-answering pairs. The model is then trained on these
pairs for one epoch with the warm-up step as 0, and we compute the loss only on the tokens of the
output answer of each training example (Mukherjee et al., 2023). All other training settings are the
same with domain-adaptive pre-training. Fine-tuning evaluation is similar to prompting evaluation,
but with two differences to align with the fine-tuning training stage: no demonstration is presented
before the prompt input, and the prompt template is the same with the one used in the training stage.
ADAPTINGLARGELANGUAGEMODELSVIA READINGCOMPREHENSION