subjects, and the average score across all subjects is reported.
Law. For the Law domain knowledge probing, we employ the LEDGAR dataset (Tuggener et al.,
2020). This dataset is designed for contract provision classification and encompasses a wide spec-
trum of 100 distinct law topics. Each label represents the principal topic of the given contract
provision. Originally structured as a 100-classification task, we adapt it for knowledge probing by
simplifying it into a four-choice question format. For each data sample, we preserve the label class
and randomly select three additional classes to create the four candidate options.
Similar to biomedicine knowledge probing, we feed into the model the data input using the template
“{CONTRACT} The topic is”, and then compare per-token-likelihood of each option to get the
model prediction. The evaluation is performed individually for each of the 100 law topics, and the
average score across all topics is reported.
ADAPTINGLARGELANGUAGEMODELSVIA READINGCOMPREHENSION