With the development of biomedical language understanding benchmarks, Artificial Intelligence applications are widely used in the medical field. However, most benchmarks are limited to English, which makes it challenging to replicate many of the successes in English for other languages. To facilitate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform far worse than the human ceiling 1 . Our benchmark is released at https://tianchi.aliyun. com/dataset/dataDetail?dataId=95414&lang=en-us . Artificial intelligence is gradually changing the landscape of healthcare, and biomedical research (Yu et al., 2018). With the fast advancement of biomedical datasets, biomedical natural language processing (BioNLP) has facilitated a broad range  Equal contribution and shared co-first authorship. of applications such as biomedical text mining, which leverages textual data in Electronic Health Records (EHRs). A key driving force behind such improvements and rapid iterations of models is the use of general evaluation datasets and benchmarks (Gijsbers et al., 2019). Pioneer benchmarks, such as BLURB (Gu et al., 2020), PubMedQA (Jin et al., 2019), and others, have provided us with the opportunity to conduct research on biomedical language understanding and developing real-world applications. Unfortunately, most of these benchmarks are developed in English, which makes the development of the associated machine intelligence Anglo-centric. Meanwhile, other languages, such as Chinese, have unique linguistic characteristics and categories that need to be considered. Even though Chinese speakers account for a quarter of the world population, there have been no existing Chinese biomedical language understanding evaluation benchmarks. To address this issue and facilitate natural language processing studies in Chinese, we take the first step in introducing a comprehensive C hinese B iomedical L anguage U nderstanding E valuation ( CBLUE ) benchmark with eight biomedical language understanding tasks. These tasks include named entity recognition, information extraction, clinical diagnosis normalization, short text classification, question answering (in transfer learning setting), intent classification, semantic similarity, and so on. We evaluate several pre-trained Chinese language models on CBLUE and report their performance. The current models still perform by far worse than the standard of single-human perfor-7888 mance, leaving room for future improvements. We also conduct a comprehensive analysis using case studies to indicate the challenges and linguistic differences in Chinese biomedical language understanding. We intend to develop a universal GLUElike open platform for the Chinese BioNLP community, and this work helps accelerate research in that direction. Overall, the main contributions of this study are as follows:  We propose the first Chinese biomedical language understanding benchmark, an open-ended, community-driven project with diverse tasks. The proposed benchmark serves as a platform for the Chinese BioNLP community and encourages new dataset contributions. We report a systematic evaluation of 11 Chinese pre-trained language models to understand the challenges derived by these tasks. We release the source code of the baselines as a toolkit for future research purposes. Several benchmarks have been developed to evaluate general language understanding over the past few years. GLUE (Wang et al., 2019b) is one of the first frameworks developed as a formal challenge affording straightforward comparison between task-agnostic transfer learning techniques. SuperGLUE (Wang et al., 2019a), styled after GLUE, introduce a new set of more difficult language understanding datasets. Other similarly motivated benchmarks include DecaNLP (McCann et al., 2018), which recast a set of target tasks into a general question-answering format and prohibit task-specific parameters, and SentEval (Conneau and Kiela, 2018), which evaluate explicitly fixed-size sentence em-beddings. Non-English benchmarks include Rus-sianSuperGLUE (Shavrina et al., 2020) and CLUE (Xu et al., 2020), which is a community-driven benchmark with nine Chinese natural language understanding tasks. These benchmarks in the general domain provide a north star goal for researchers and are part of the reason we can confidently say we have made great strides in our field. For BioNLP, many datasets and benchmarks have been proposed (Wang et al., 2020; Li et al., 2016; Wu et al., 2019) which promote the biomedical language understanding (Beltagy et al., 2019; Lewis et al., 2020; Lee et al., 2020). Tsatsaronis et al. (2015) propose biomedical language understanding datasets as well as a competition on large-scale biomedical semantic indexing and question answering. Jin et al. (2019) propose PubMedQA, a novel biomedical question answering dataset collected from PubMed abstracts. Pappas et al. (2018) propose BioRead, which is a publicly available cloze-style biomedical machine reading comprehension (MRC) dataset. Gu et al. (2020) create a leaderboard featuring the Biomedical Language Understanding & Reasoning Benchmark (BLURB). Unlike a general domain corpus, the annotation of a biomedical corpus needs expert intervention and is labor-intensive and time-consuming. Moreover, most of the benchmarks are based on English; ignoring other languages means that potentially valuable information may be lost, which can be helpful for generalization. In this study, we focus on Chinese to fill the gap and aim to develop the first Chinese biomedical language understanding benchmark . Note that Chinese biomedical text is linguistically different from English and has its domain characteristics, necessitating an evaluation BioNLP benchmark designed explicitly for Chinese. CBLUE consists of 8 biomedical language understanding tasks.The task descriptions and statistics of CBLUE are shown Table 1. Unlike CLUE (Xu et al., 2020) as shown in Table 2, CBLUE has a diverse data source (the annotation is expen-sive), richer task setting, thus, more challenging for NLP models. We introduce the design principle of CBLUE as follows: 1) Diverse tasks : CBLUE contain widespread token-level, sequence-level, sequence-pair tasks. 2) Variety of differently distributed data : CBLUE collect data from various sources, including clinical trials, EHRs, medical forum, textbooks, and search engine logs with a real-world distribution. 3) Quality control in long-term maintenance : We asked domain experts (doctors from Class A tertiary hospitals) to annotate datasets and carefully review data to ensure data quality. CMeEE For this task, the dataset is first released in CHIP2020 2 (Hongying et al., 2020). Given a pre-defined schema, the task is to identify and extract entities from the given sentence and classify them into nine categories: disease, clinical manifestations, drugs, medical equipment, medical procedures, body, medical examinations, microorganisms, and department. CMeIE For this task, the dataset is also released in CHIP2020 (Guan et al., 2020). The goal of the task is to identify both entities and relations in a sentence following the schema constraints. There are 53 relations defined in the dataset, including 10 synonymous sub-relationships and 43 other sub-relationships. CHIP-CDN For this task, the dataset is to standardize the terms from the final diagnoses of Chinese electronic medical records. Given the original phrase, the task is to normalize it to standard terminology based on the International Classification of Diseases (ICD-10) standard for Beijing Clinical Edition v601. CHIP-CTC For this task, the dataset is to classify clinical trials eligibility criteria, which are fundamental guidelines of clinical trials defined to identify whether a subject meets a clinical trial or not (Zong et al., 2021). All text data are collected from the website of the Chinese Clinical Trial Registry (ChiCTR) 3 , and a total of 44 categories are defined. The task is like text classification; although it is not a new task, studies and corpora for the Chinese clinical trial criterion are still limited , and we hope to promote future research for social benefits. CHIP-STS For this task, the dataset is for sentence similarity in the non-i.i.d. (non-independent and identically distributed) setting. Specifically, the task aims to evaluate the generalization ability between disease types on Chinese disease questions and answer data. Given question pairs related to 5 different diseases (The disease types in the training and testing set are different), the task is to determine whether the semantics of the two sentences are similar. KUAKE-QIC For this task, the dataset is for intent classification. Given search engine queries, 3 http://chictr.org.cn/ 7890 the task is to classify each of them into one of 11 medical intent categories defined in KUAKE-QIC. Those include diagnosis, etiology analysis, treatment plan, medical advice, test result analysis and others. KUAKE-QTR For this task, the dataset is used to estimate the relevance of the title of a query document. Given a query (e.g., Symptoms of vitamin B deficiency), the task aims to find the relevant title (e.g., The main manifestations of vitamin B deficiency). KUAKE-QQR For this task, the dataset is used to evaluate the relevance of the content expressed in two queries. Similar to KUAKE-QTR, the task aims to estimate query-query relevance, which is an essential and challenging task in real-world search engines. Since machine learning models are mostly data-driven, data plays a critical role, and it is pretty often in the form of a static dataset (Gebru et al., 2018). We collect data for different tasks from diverse sources, including clinical trials, EHRs, medical books, and search logs from real-world search engines. As biomedical data may contain private information such as the patient's name, age, and gender, all collected datasets are anonymized and reviewed by the IRB committee of each data provider to preserve privacy. We introduce the data collection details followingly. Clinical trial eligibility criteria text is collected from ChiCTR, a non-profit organization that provides information about clinical trial registration for public research use. In each trial registry file, eligibility criteria text is organized as a paragraph in the inclusion criteria and exclusion criteria. Some meaningless texts are excluded, and the remaining texts are annotated to generate the CHIP-CTC dataset. We obtain the final diagnoses of the medical records from several Class A tertiary hospitals and sample a few diagnosis items from different medical departments to construct the CHIP-CDN dataset for research purposes. The diagnosis items are randomly sampled from the items which are not covered by the common medical synonyms dict. Due to the COVID-19 pandemic, online consultation has become more and more popular via the Internet. To promote data diversity, we select the online questions by patients to build the CHIP-STS dataset. Note that most of the questions are chief complaints. To ensure the authority and practicability of the corpus, we also select medical textbooks of Pediatrics (Wang et al., 2018), Clinical Pediatrics (Shen and Gui, 2013) and Clinical Practice 4 . We collect data from these sources to construct the CMeIE and CMeEE datasets. We also collect search logs from real-world search engines like the Alibaba QUARK Search Engine 5 . First, we filter the search queries in the raw search logs by the medical tag to obtain candidate medical texts. Then, we sample the documents for each query with non-zero relevance scores (i.e., to determine if the document is relevant to the query). Specifically, we divide all the documents into three categories, namely high, middle, and tail documents, and then uniformly sample the data to guarantee diversity. We leverage the data from search logs to construct KUAKE-QTC, KUAKE-QTR, and KUAKE-QQR datasets. Each sample is annotated by three to five domain experts , and the annotation with the majority of votes is taken to estimate human performance. During the annotation phase, we add control questions to prevent dishonest behaviors by the domain experts. Consequently, we reject any annotations made by domain experts who fail in the training phase and do not adopt the results of those who achieved low performance on the control tasks. We maintain strict and high criteria for approval and review at least 10 random samples from each worker to decide whether to approve or reject all their HITs. We also calculate the average inter-rater agreement between annotators using Fleiss' Kappa scores (Fleiss, 1971), finding that five out of six annotations show almost perfect agreement (  = 0 . 9 ). Utility-preserving Anonymization Biomedical data may be considered as a breach in the privacy of individuals because they usually contain sensitive information. Thus, we conduct utility-preserving anonymization following (Lee et al., 2017) to anonymize the data before releasing the benchmark. Real-world Distribution To promote the generalization of models, all the data in our CBLUE benchmark follow real-world distribution without up/downsampling. As shown in Figure 1(a), our dataset follows long-tail distribution following Zipf's law and will inevitably be long-tailed. However, long-tail distribution has no significant effect on performance. Further, some datasets, such as CMedIE, have label hierarchy with both coarse-grained and fine-grained relation labels, as shown in Figure 1(b). Diverse Tasks Setting Our CBLUE benchmark includes eight diverse tasks, including named entity recognition, relation extraction, and single-sentence/sentence-pair classification. Besides the independent and i.i.d. scenarios, our CBLUE benchmark also contains a specific transfer learning scenario supported by the CHIP-STS dataset, in which the testing set has a different distribution from the training set. We provide a leaderboard for users to submit their own results on CBLUE. The evaluation system will give final scores for each task when users submit their prediction results. The platform offers 60 free GPU hours from Aliyun 6 to help researchers develop and train their models. Our CBLUE benchmark was released online on April 1, 2021. Up to now, more than 900 researchers have applied the dataset, and over 300 teams have submitted their model predictions to our platform, including medical institutions (Peking Union Medical College Hospital, etc.), universities (Tsinghua University, Zhejiang University, etc.), and AI companies (Baidu, Huawei, etc.). We will continue to maintain the benchmark by adding new tasks. To make it easier to use the CBLUE benchmark, we also offer a toolkit implemented in PyTorch (Paszke et al., 2019) for reproducibility. Our toolkit supports mainstream pre-trained models and a wide range of target tasks. 6 https://tianchi.aliyun.com/ notebook-ai/ 7892 Model CMeEE CMeIE CDN CTC STS QIC QTR QQR Avg. Baselines We conduct experiments with baselines based on different Chinese pre-trained language models. We add an additional output layer (e.g., MLP) for each CBLUE task and fine-tune the pre-trained models. Models We evaluate CBLUE on the following public available Chinese pre-trained models:  BERT-base (Devlin et al., 2018). We use the base model with 12 layers, 768 hidden layers, 12 heads, and 110 million parameters. BERT-wwm-ext-base (Cui et al., 2019). A Chinese pre-trained BERT model with whole word masking. RoBERTa-large (Liu et al., 2019). Compared with BERT, RoBERTa removes the next sentence prediction objective and dynamically changes the masking pattern applied to the training data. RoBERTa-wwm-ext-base/large. RoBERTa-wwm-ext is an efficient pre-trained model which integrates the advantages of RoBERTa and BERT-wwm. ALBERT-tiny/xxlarge (Lan et al., 2019). AL-BERT is a pre-trained model with two objectives: Masked Language Modeling (MLM) and Sentence Ordering Prediction (SOP). ZEN (Diao et al., 2019). A BERT-based Chinese text encoder enhanced by N-gram representations, where different combinations of characters are considered during training. Mac-BERT-base/large (Cui et al., 2020). Mac-BERT is an improved BERT with novel MLM as a correction pre-training task. PCL-MedBERT 7 . A pre-trained medical language model proposed by the Peng Cheng Laboratory. We implement all baselines with PyTorch (Paszke et al., 2019). All the training details can be found in the appendix. We report the results of our baseline models on the CBLUE benchmark in Table 3. We notice that larger pre-trained models obtain better performance. Since Chinese text is composed of terminologies, carefully designed masking strategies may be helpful for representation learning. However, we observe that models which use whole word masking do not always yield better performance than others in some tasks, such as CTC, QIC, QTR, and QQR, indicating that tasks in our benchmark are challenging and more sophisticated technologies should be developed. Further, we find that ALBERT-tiny achieves comparable performance to base models in CDN, STS, QTR, and QQR tasks, illustrating that smaller models may also perform well in specific tasks. We think this is caused by the different distribution between pretraining corpus and Chinese medical text; thus, large PTLMs may not obtain satisfactory performance. Finally, we notice that PCL-MedBERT, which tends to be state-of-the-art in Chinese biomedical text processing tasks, and does not perform as well as we expected. This further demonstrates the difficulty 7 https://code.ihub.org.cn/projects/ 1775 7893 CMeEE CMeIE CDN CTC STS QIC QTR QQR Trainedannotation annotator 1 69.0 62.0 60.0 73.0 94.0 87.0 75.0 80.0 annotator 2 62.0 65.0 69.0 75.0 93.0 91.0 62.0 88.0 annotator 3 69.0 67.0 62.0 80.0 88.0 83.0 71.0 90.0 avg 66.7 64.7 63.7 76.0 91.7 87.0 69.3 86.0 majority 67.0 66.0 65.0 78.0 93.0 88.0 71.0 89.0 best model 62.4 55.9 59.3 70.9 85.6 85.5 62.9 84.7 Table 4: Human performance of two-stage evaluation scores with the best-performed model. avg refers to the mean score from the three annotators. majority indicates the performance taken from the majority vote of amateur humans. Bold text denotes the best result among human and model prediction. For all of the tasks in CBLUE, we ask human amateur annotators with no medical experience to label instances from the testing set and compute the annotators' majority vote against the gold label annotated by specialists. Similar to SuperGLUE (Wang et al., 2019a), we first need to train the annotators before they work on the testing data. Annotators are asked to annotate some data from the development set; then, their annotations are validated against the gold standard. Annotators need to correct their annotation mistakes repeatedly so that they can master the specific tasks. Finally, they annotate instances from the testing data, and these annotations are used to compute the final human scores. The results are shown in Table 4 and the last row of Table 3. In all tasks, humans have better performance. We choose two datasets: CMeEE and KUAKE-QIC, a sequence labeling and classification task, respectively, to conduct case studies. As shown in Figure 2, we report the statistics of the proportion of various types of error cases 8 . For CMeEE, we notice that entity overlap 9 , ambiguity 10 , need domain knowledge 11 , annotation error 12 are major reasons that result in the prediction failure. Furthermore, there exist many instances with entity overlap , which may lead to confusion for the named entity recognition task. While in the analysis for KUAKE-QIC, almost half of bad cases are due to multiple triggers 13 and colloquialism . Colloquialism 14 is natural in search queries, which means that some descriptions of the Chinese medical text are too simplified, colloquial, or inaccurate. We show some cases on CMeEE in Table 5. In the second row, we notice that given the instance of                  8 See definitions of errors in the appendix. 9 There exist multiple overlapping entities in the instance. 10 The instance has a similar context but different meaning, which mislead the prediction. 11 There exist biomedical terminologies in the instance which require domain knowledge to understand. 12 The annotated label is wrong. 13 There exist multiple indicative words which mislead the prediction. 14 The instance is quite different from written language (e.g., with many abbreviations) 7894 Sentence Word Label RO MB               B      12%  19%        Ite Pro Pro The results of blood biochemical analysis show that vitamin B lack rate is about 12% to 19%. blood biochemical analysis Ite Pro Pro                          Bod O Bod The rash can be reduced by the host producing specific anti-toxin antibodies. anti-toxin antibodies Bod O Bod                            1.                                     ,   ,   Sym, Sym, Sym O Sym, Sym, Sym According to the structure and function of genetic material, genetic diseases are divided into five categories: 1. Chromosomal diseases refer to abnormal chromosome number or chromosome structure abnormalities, including deletions, translocations, inversions... deletions, translocations, inversions Sym, Sym, Sym O Sym, Sym, Sym Table 5: Case studies in CMeEE. We evaluate roberta-wwm-ext and PCL-MedBERT on 3 sampled sentences, with their gold labels and model predictions. Ite (medical examination items), Pro (medical procedure), Bod (body), and Sym (clinical symptoms) are labeled for medical named words. O means that the model fails to extract the entity from sentences. RO=roberta-wwm-ext, MB=PCL-MedBERT. Query Model Gold BERT BERT-ext MedBERT                                        Does it matter if the ratio of lymphocytes is high and the ratio of neutrophils is low? Diagnosis Diagnosis Test results analysis Test results analysis                          Consultation: When do children usually get chickenpox? Other Other Other Diseasedescription      160      40                             The systolic blood pressure of the elderly is 160, and the diastolic blood pressure is only more than 40. What is the reason? How to treat? Diagnosis Diagnosis Diagnosis Treatment Table 6: Case studies in KUAKE-QIC. We evaluate the performance of baselines with 3 sampled instances. The correlation between Query and Title is divided into 3 levels (0-2), which means  poorly related or unrelated ',  related ' and  strongly related '. BERT = BERT-base, BERT-ext = BERT-wwm-ext-base, MedBERT = PCL-MedBERT.    ( Rash can be reduced by the host producing specificanti-toxin antibodies. ), ROBERTA and PCL-MedBERT obtain different predictions. The reason is that there exist medical terms such as       ( anti-toxin antibodies ). ROBERTA can not identify those tokens correctly, but PCL-MedBERT, pre-trained on the medical corpus, can successfully make it. Moreover, PCL-MedBERT can extract entities    ,   ,   ( eletions, translocations, inversions ) from the long sentences, which is challenging for other models. We further show some cases on KUAKE-QIC in Table 6. In the first case, we notice that both BERT and BERT-ext fail to obtain the intent label of the query                        ? ( Does it matter if the ratio of lymphocytes is high and the ratio of neutrophils is low? ), while MedBERT can obtain the correct prediction. Since        ( ratio of lymphocytes ) and        ( ratio of neutrophils ) are biomedical terms, and the general pre-trained language model has to leverage domain knowledge to understand those phrases. As shown in Table 5 and Table 6, compared with other languages, the Chinese language is very colloquial even in medical texts. Furthermore, polysemy is prevalent in chinese language. The meaning of a word changes according to its tone, which usually causes confusion and difficulties for machine reading. In summary, we conclude that tasks in CBLUE are not easy to solve since the Chinese language has unique characteristics , and more robust models should be developed. In this paper, we present a Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark. We evaluate 11 current language representation models on CBLUE and analyzed their results. The results illustrate the limited ability of state-of-the-art models to handle some of the more challenging tasks. In contrast to English benchmarks such as GLUE/SuperGLUE and BLURB, whose model performance already matches human performance, we observe that this is far from the truth for Chinese biomedical language understanding. We want to express gratitude to the anonymous reviewers for their hard work and kind comments. This work is funded by Special Project of New Generation Artificial Intelligence of the Ministry of Science and Technology of China (2021ZD0113402), National Natural Science Foundations of China (61876052 and U1813215), National Natural Science Foundation of Guangdong, China (2019A1515011158), Strategic Emerging Industry Development Special Fund of Shen-zhen (20200821174109001), Pilot Project in 5G + Health Application of Ministry of Industry and Information Technology & National Health Commission (5G + Luohu Hospital Group: an Attempt to New Health Management Styles of Resi-dents), Zhengzhou collaborative innovation major special project (20XTZX11020), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Ningbo Natural Science Foundation (2021J190), and Yongjiang Talent Introduction Programme (2021A-156-G). We collected all the data with authorization from the organization that owned the data and signed the agreement. We release the benchmark following the CC BY-NC 4.0 license. All collected datasets are anonymized and reviewed by the IRB committee of each data provider to preserve privacy. Since we collect data following real-world distribution, there may exist popularity bias that cannot be ignored.
abstain abstain objective result other abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain method objective objective objective method objective other other other other other objective other other other other other other other objective other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain method method method abstain method abstain method abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain method result abstain objective other other method method abstain method