Providing a reliable explanation for clinical diagnosis based on the Electronic Medical Record (EMR) is fundamental to the application of Artificial Intelligence in the medical field. Current methods mostly treat the EMR as a text sequence and provide explanations based on a precise medical knowledge base, which is disease-specific and difficult to obtain for experts in reality. Therefore, we propose a counterfactual multi-granularity graph supporting facts extraction (CMGE) method to extract supporting facts from the irregular EMR itself without external knowledge bases in this paper. Specifically, we first structure the sequence of the EMR into a hierarchical graph network and then obtain the causal relationship between multi-granularity features and diagnosis results through counterfactual intervention on the graph. Features having the strongest causal connection with the results provide interpretive support for the diagnosis. Experimental results on real Chinese EMRs of the lymphedema demonstrate that our method can diagnose four types of EMRs correctly, and can provide accurate supporting facts for the results. More importantly, the results on different diseases demonstrate the robustness of our approach, which represents the potential application in the medical field 1 . Electronic Medical Record (EMR) based diagnosis has attracted extensive attention due to its comprehensive historical information and clinical descriptions with the development of natural language processing and medical informatics (Yang et al., 2018; Choi et al., 2018; Liu et al., 2019; Dong et al., 2020; Ma et al., 2020b). The application of deep learning in medicine requires adequate medical explanations for the result. Specific to the diagnosis of EMR, the model needs to provide the text description supporting the diagnosis results. As shown in Figure 1, an irregular EMR is a document of disease-related information, including symptoms, history of the disease, preliminary examination results, and so on, which is disordered and sparse with meaningless noisy text. Existing methods provide explanation through medical entities (Yuan et al., 2020), text spans (Mullenbach et al., 2018) and the weights of external knowledge (Ma et al., 2018). The entity is critical to the diagnosis (Sha and Wang, 2017; Girardi et al., 2018), but for the medical explanation, it cannot provide specific information of symptoms (such as positive or negative). And the form of the span is too fragmented and lacks readability. Therefore, the clause as a more informative and readable representation is needed to be combined above the level of entities. Most of the previous methods provide reliable explanations for diagnosis by calculating the similarity with an external medical knowledge base (ICD 2 and CCS 3 ) (Xu et al., 2019, 2020). KAME 2 https://www.cdc.gov/nchs/icd/icd10cm.htm 3 https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp (Ma et al., 2018) uses the weights of the nodes in the introduced knowledge graph to provide explanations. Depending on the hierarchical relations in the database, GMAN (Yuan et al., 2020) builds a disease hierarchy graph and a causal graph to find critical entities. However, a trusted medical knowledge base requires a mass of expertise in different fields to build, and it may be incomplete or erroneous in practical clinical applications. So far, how to extract supporting facts from the EMR itself without an external medical knowledge base is still a problem. Counterfactual reasoning provides a link between what could have happened when inputs had been changed (Verma et al., 2020). Doctors usually make a judgment based on several related symptoms during diagnosing a disease. In this regard we can consider a question: will a doctor make a misdiagnosis without one of the critical symptoms? The result is clear. In a counterfactual way, if we gradually weaken the features until the diagnosis changes dramatically, then this feature can be considered as a supporting fact. Based on this consensus, we propose a counterfactual multi-granularity graph supporting facts extraction (CMGE) method for the irregular EMR in this paper. First, we model the EMR as a hierarchical graph structure, which contains sentences, clauses, and entities. Specifically, sentences are used to model the temporal relationship, clauses provide a complete descriptive explanation, and entities provide symptom support as others. On this basis, we use a graph attention network to aggregate all information from different granularities. Then, we can do a counterfactual intervention to obtain the causal relation between feature and diagnosis. Specifically, we train a learnable soft-mask matrix to mask the feature of nodes or edges in the graph while keeping the diagnosis unchanged, and the remaining features are the supporting facts of the diagnosis. Counterfactual reasoning on the graph requires enhancing the medical features contained in the text of different granularity, so we use clustering labels 4 to cluster clauses and entities. The experimental results demonstrate the effectiveness of our method. The contributions of this paper are summarized as follows:  We propose a multi-granularity structured 4 Notice that this label is disease-free and can be initially labeled without expert knowledge by crowdsourcing annotation. modeling method based on the hierarchical graph network that decomposes the EMR into sentences, clauses, and entities, and use clustering labels to enhance the expression of medical features. We adapt counterfactual intervention to extract critical supporting facts from the EMR during diagnosis. Importantly, our method is disease-independent and does not require a precise external medical knowledge base, so that it is suitable for a wide range of applications. The evaluation conducted on the real EMR dataset shows that our method can correctly diagnose the types of lymphedema. Keyword coverage and human evaluation show that the counterfactual reasoning method has better extraction accuracy and robustness compared to two existing methods reimplemented by ourselves. Given an irregular EMR in the form of free text X = [ x 1 , x 2 ,    , x L ] with L words, the task for us is to extract supporting facts that can be used to explain the diagnosis result without relying on external knowledge while performing diagnosis. The supporting facts can be entities or clauses of text. The medical features in the EMR are sparse and medical entities are insufficient to provide suffi-cient explanation for diagnosis. Therefore, we do multi-granularity segmentation for EMRs, which Figure 3: An overview of counterfactual multi-granularity graph supporting facts extraction network. enhances the symptom features of entities and explanation of diagnosis, while maintaining the integrity of the text. An EMR can be divided by periods into sentences, which can be further divided into clauses by commas or semicolons as a more granular segmentation. In order to keep the symptom features of entities, we do Named Entity Recognition 5 and number extraction for each clause 6 . In addition, we add two general nodes representing the gender and age of the patient respectively. After segmentation, as shown in Figure 2, we can build a hierarchical tree structure. The nodes at each level represent the text of sentences, clauses, and entities respectively. Specifically, for each EMR, we connect the two general nodes, sentence nodes sequentially. Then, we connect the clause node to the sentence node to which it belongs and the entity nodes disassembled from it. In particular, a fully-connected relationship is established between all the clause nodes, which overcomes the defect that Graph Attention Network (GAT) can only aggregate the information from adjacent nodes when the network is shallow and expands the receptive field of each sub-sentence node to the whole EMR. Then, all clause nodes are connected to an aggregate node which is used to do the diagnosis. All the edges in the graph are bidirectional to make the information between nodes flow better. In the original EMR, all tokens have the same weight, so noisy text will degrade the performance of diagnosis and explanation. To improve the accuracy of symptom presentation, clustering-labels are used to cluster clauses and entities into corresponding medical classifications. Specifically, the clause is divided into 33 classes and the entity is divide into 10 classes, which is a scientific classification method in medicine derived from the textbook "Di-agnostics" (Xuehong Wan, 2013). These labels are disease-free and can be labeled without expert knowledge by crowdsourcing annotation. We manually annotated the corresponding labels for the entire dataset on our own platform. And we have trained a BERT (Devlin et al., 2019) based text classifier on 30% of the data, which can achieve the annotation accuracy of 80.76% on clauses and 97.13% on entities on the remaining data. This shows that our method can easily annotate large-scale data. With these labels, we can gather the same types of features together in the feature space, thereby enhancing the model's overall attention to important types of features. Please refer to Appendix B.2 for more details. After building the multi-granularity graph for a medical record, each node in the graph contains a sequence X node = [ x 1 , x 2 ,    , x n ] with n words, which is tokenized by the tokenizer of BERT (De-vlin et al., 2019). In order to maintain the consistency of the results of different granularity encoding, we use one bi-directional RNN (Schuster and Paliwal, 1997) with GRU (Cho et al., 2014) to cover the sequence of sentences, clauses, entities and general information into hidden state sequence respectively H m = ( h 1 , h 2 ,    , h n ) : h t = BiGRU ( h t  1 , e ( x t )) (1) where h t is the hidden state of the t-th token and e ( x t ) is the embedding vectors with random initialization of x i . Finally, we use the last hidden state of i-th text sequence as the feature H i of node i . Once we get the feature of the node, we use the Graph Attention Network (GAT) (Velickovic et al., 2018) to aggregate the information between different granularity. GAT can obtain the correlation score between nodes based on the attention mechanism, which is the key to the interpretability of our model. Specifically, GAT takes all the node features as input and calculates the attention coeffi-cients  ij by e ij = LeakyReLU ( a T ([ W H i ; W H j ])) (2)  ij = exp( e ij ) (cid:80) k N i exp( e ik ) (3) where H i is the feature of node i , W  R d  d is a learnable weight matrix for the linear projection, a  R 2 d is a learnable weight vector used to transform the adjacent node feature representations to the edge score e ij between the i -th and j -th nodes. Equation (4) means to do a softmax normalization between all the edge attention scores on the edges connected to node i . Then, we update the feature of each node by H (cid:48) i = LeakyReLU ( (cid:88) j N i  ij W H j ) (4) After graph reasoning, the representation H of each node has been updated with the granular information aggregated from adjacent nodes and can be used for subsequent tasks. After obtaining the updated node features, we use them in three subtasks: (i) graph classification for automatic diagnosis; (ii) sub-sentence classification for clustering; and (iii) entity classification for clustering. Taking entity node classification as an example, for each entity node, we use a two-layer MLP with the ReLU activation function to calculate the probability. For an entity node i , we can get P entity,i = MLP entity ( E i ) (5) By the same way, we can obtain the probability P graph , P clause , P entity . The same as the common multi-task learning, we joint all the losses together as: L joint =  1 L graph +  2 L clause +  3 L entity (6) where  1 ,  2 and  3 are hyper-parameters, and all the loss are calculated by cross-entropy loss. Providing supporting information while making the diagnosis is the key to applying Artificial Intelligence into the medical field. Inspired by (Ying et al., 2019), we add node-mask or edge-mask into GAT to obtain the counterfactual result after the training and eliminate the noise nodes while keeping the diagnostic results unchanged. For edge-mask, we introduce a learnable matrix M with the same form as the adjacency matrix of the medical record graph. Each element m ij in the matrix represents the degree of mask for message aggregation from node i to node j in the graph. With this method, the calculation of attention coef-ficients in the GAT has been changed to  ij = exp( e ij m ij ) (cid:80) k N i exp( e ik m ik ) (7) And for node-mask, similarly, we introduce a learnable parameter  i for each node i in the graph. The parameter represents the degree of mask for the feature in the node. After node-mask, the calculation of e ij and H (cid:48) i has been changed to e ij = LeakyReLU ( a T ([  i W H i ;  j W H j ])) (8) H (cid:48) i = LeakyReLU ( (cid:88) j N i  ij  j W H j ) (9) In the training of counterfactual reasoning, we jointly optimize three loss functions to obtain accurate counterfactual results. To ensure that the model can make a correct diagnosis after the counterfactual intervention, we use the original model type train test Secondary Lymphedema 448 36 Primary Lymphedema 185 22 Chylous Reflux Lymphedema 19 21 Others 248 21 All 900 100 Table 1: The statistics of the datasets. to obtain the fact result D i and maximizes the probability of selecting the correct diagnosis in counterfactual reasoning. Besides, we minimize the sum of all elements in the mask matrix to ensure that all noise nodes are filtered as much as possible. Since there is an exponential level possibility of counterfactual intervention on the model through the node-mask or edge-mask, we minimize the information entropy of the mask matrix regarding which node to select to reduce the uncertainty of the result. Finally, the loss of counterfactual reasoning is as follows: L c =   4 logP ( D = D i ) +  5 sum ( M )   6 1 N (cid:88) m i  M m i log ( m i )   6 1 N (cid:88) m i  M (1  m i ) log (1  m i ) (10) where  4 ,  5 and  6 are hyper-parameters, N is the number of elements in the mask matrix M , and all the elements in M are mapped to the [0 , 1] by sigmoid function. For node-mask, the training is similar. After counterfactual reasoning, we extract the nodes or edges (each edge represents the two nodes connected) represented by the top-k elements in the mask matrix as supporting facts. Based on the cooperation with the hospitals, we conducted experiments with real EMR data. We selected the EMRs from the department of lymphedema and diagnose the disease of primary lymphedema (       ), secondary lymphedema (        ), chylous reflux lymphedema (          ) and others (   ). The reasons for us to choose this department are as follows: (I) Lymphedema is a sub-discipline in medicine, so the researches on it, whether in Medicine or Artificial Intelligence, is still limited. For example, ICD10 can not provide full medical supporting. (II) The pathogenesis and treatment methods of different types of lymphedema vary greatly, but their outward manifestations are similar. Therefore, there is an urgent need for a simple method of earlier diagnosis system of lymphedema. (III) Specialist doctors pay more attention to the diagnosis in sub-discipline disease and do not concern with the large-scale rough diagnosis. Formally, there are 1000 EMRs used in our experiment, of which 900 are used for training and 100 are used for testing. The statistics of four types of diseases are shown in Table 1. The average length of all EMRs is 345 words in Chinese. And our model is implemented based on PyTorch (Paszke et al., 2019), and use Adam (Kingma and Ba, 2015) optimizer for training. Please refer to Appendix B.1 for datasets details and Appendix A.1 for implementation details. We designed two representative models to compare the ability to extract medical support facts under similar task conditions based on attention and variational inference: Self-Attention This method represents most of the existing approaches and provides explanations through attention similarity. We use BiGRU to encode the EMR. With the sequence embedding, following (Choi et al., 2016), we use average pooling to obtain the overall representation for automatic diagnosis. For supporting fact extraction, following (Mullenbach et al., 2018), we calculate the self-attention weight of each token, and design a sliding window method to obtain the average attention scores of fixed length spans, among which having high scores are taken as the supporting facts. PostKS This is another method based on variational inference we've designed in addition to attention. Inspired by the dialogue knowledge selection model PostKS (Lian et al., 2019), we convert the pivotal information extraction into a clause selection problem. This method uses the text result of the diagnosis(as shown in Figure 1) to calculate the correlation with the clause as posterior distribution through the attention mechanism, and then uses self-attention and average pooling between clauses to obtain correlation score as the prior distribution. During training, based on variational inference, the model uses posterior information to guide prior selection, so that makes the prior distribution and the Model Diagnosis Clause Entity P/% R/% F1/% P/% R/% F1/% P/% R/% F1/% Self-Attention 94.95 95.00 94.97 ---PostKS 95.13 97.00 96.06 ---CMGE  c  e 96.17 96.00 96.08 1.91 3.72 2.52 14.36 9.05 11.10 CMGE  e 97.40 96.00 96.69 81.26 81.80 81.53 15.22 32.18 20.67 CMGE  c 97.19 97.11 97.15 25.75 1.55 2.92 95.33 95.12 95.22 CMGE 99.04 99.00 99.02 82.49 82.53 82.51 96.43 96.38 96.40 Table 2: The first two lines are the diagnostic performance of the compared model and the last line is ours. posterior distribution consistent. Finally, during inference, we select the clauses with high prior attention scores among clauses as supporting facts. Please refer to Appendix A.2 for more details. To measure the performance of our pivotal information extraction module, we built a simple diagnostic criterion from (Levine, 2017), which is a complete diagnosis and treatment guide for lymphedema written by medical experts. Based on this diagnosis criteria, we used a combination of automatic evaluation and human evaluation. Automatic Evaluation The precision, recall, and F1 are used as the metrics to measure the diagnostic accuracy of the model, which is the basis for the practical application. Specifically, several key-phrases for the three types of lymphedema are manually identified respectively to represent diagnostic features, and they are the re-descriptions of diagnostic criteria in the guide using phrases from EMRs. We use hit@1/3/5 (Bordes et al., 2013) to measure the coverage rate of the extracted results to the key-phrases. These metrics represent whether one of the diagnostic features is included in the top-1/3/5 extracted results. Please refer to Appendix B.3 for more details. Human Evaluation Since some of the implicit medical features cannot be covered by key-phrases, human evaluation is necessary. We used each model to extract the top 3 supporting facts respectively for all 100 EMR samples in the testset, and randomly shuffled the order of the results. Then we invited 3 evaluators with medical backgrounds and having read the guide to determine whether the results conform to medical knowledge. We focus on the comprehensiveness and trustworthiness of each model. Comprehensiveness is used to measure whether the model can provide more medical features, and trustworthiness is used to measure whether the extraction results are helpful for diagnosis. For each item, the evaluator is asked to score in 0  2 . The final indicator is the average of the three evaluators. 4.1 Diagnostic Result The diagnostic results are shown in Table 2. From the results, we can see our model performs better than all the compared models and can achieve about 99% accuracy in the diagnosis of lymphedema, which exceeds the comparison models by 3%-5% in precision, recall, and F1. Based on our model, the categories of clauses and entities can be distinguished correctly, which demonstrates that the clustering information contained in the pseudo-labels is correctly learned by our multi-granularity model. This result indicates that the accuracy of our method in the diagnosis of lymphedema is in line with clinical requirements. Since our goal is to make the model really help doctors in clinical practice with reliable medical explanations, we will focus on the performance of the counterfactual extraction of the supporting facts for the diagnosis that follows. Please refer to Appendix A.4 for the effectiveness of our model in diagnosis on the benchmark data. Automatic Result Table 3 shows the automatic evaluation results of the supporting facts extraction. Since the identified keywords are difficult to accurately cover the features for diagnosis and models have different adaptability to various diseases, the performance is distinguishing on different diseases. Compared with other models, the counterfactual-Model Secondary Lymphedema Primary Lymphedema Chylous Reflux Lymphedema hit@1 hit@3 hit@5 hit@1 hit@3 hit@5 hit@1 hit@3 hit@5 Self-Attention 5.45% 36.36% 50.91% 13.64% 45.45% 54.55% 4.76% 9.52% 33.33% PostKS 9.09% 43.64% 60.00% 9.09% 40.91% 54.55% 0.00% 14.29% 19.05% Node-Mask 25.45% 52.73% 69.09% 22.73 % 31.82% 54.55% 9.52% 19.05% 23.81% Edge-Mask 36.36% 61.82% 70.91% 22.73% 40.91% 50.00% 61.90% 66.67% 76.19% Table 3: The automatic evaluation for the extraction of diagnostic supporting facts for three types of lymphedema. based methods, especially the Edge-Mask method, has an advantage in accuracy and robustness on the whole. Hit@1 shows that the Edge-Mask can locate key facts more quickly than the comparison methods and hit@5 shows that it achieves over 70% accuracy on secondary lymphedema and chylous reflux lymphedema. In the comparison of different lymphedema, other methods have a greater performance degradation, and only the Edge-Mask maintains high accuracy in various diseases, indicating that the Edge-Mask method is highly robust to different diseases. Human Result Table 4 shows the results of the human evaluation of the four categories of diagnosis. Compared with other methods, the counterfactual-based methods have great advantages in comprehensiveness, which indicates that our method can focus more on useful medical information and eliminate invalid noise in the EMR. The fourth category requires focus. This category includes all non-lymphedema medical records, and its diseases are diverse and complex. It can be seen that the method of counterfactual reasoning has strong performance in this type in terms of comprehensiveness and credibility, indicating that our method is truly independent of the type of disease and suitable for large-scale promotion. Table 2 shows the ablation experiment results for the clustering labels. For the experiment without corresponding labels, we used a classifier with random initialization parameters for classification, which can reflect the expectation of the ability to encode medical features of the model. The results show that both the clause label and entity label can improve the accuracy of diagnosis by about 1% on the basis of over 96% accuracy. Since we use the same encoder to encode the three granular texts of the sentence, clause and entity, the addition of clause labels also improves the accuracy of entity classification and vice versa. The result indicates that the introduction of cluster tags enhances the expression of medical information in the model and enables the model to better extract and utilize relevant medical knowledge from irregular text. Results in Primary Lymphedema Since the diagnosis of primary lymphedema is mainly diagnosed by excluding other types of lymphedema, the keywords we established are not standardized in the EMR, the performance of all models in Table 3 has a significant decline and only be used for comparison. And the performance in human evaluation is consistent with other diseases in Table 4. Results in Chylous Reflux Lymphedema Except for Edge-Mask, the performance of the other methods on chylous reflux lymphedema has dropped significantly. Since this type of EMR only accounts for 4% of the dataset, the models based on frequency statistics are difficult to capture key features. And Edge-Mask, using counterfactual intervention to obtain causal relation, is disease-independent and can adapt to few data. Node-Mask and Edge-Mask Edge-Mask is included in Node-Mask. Masking the feature of a node will inevitably reduce the flow of information on all connected edges. So compared to Node-Mask, Edge-Mask is a fine-grained counterfactual intervention. For Node-Mask, the flow of multi-granularity information between nodes will be truncated. For example, when a clause node is masked, the entity features belonging to it are truncated together. Therefore, Node-Mask has a weaker performance than Edge-Mask. Figure 4 is an example randomly obtained from the test set. In this graph, each node represents a clause that contains the entities used to describe the symptoms of the disease and the edges represent the connection between them. All the aforementioned features constitute a hierarchical supporting graph to provide effective help for doctors' diagnosis. As we can see, our model successfully extracted the patient's history of cancer, surgery and chemotherapy, which can clearly indicate that the patient is suffering from secondary lymphedema. This shows that the supporting facts we extracted are effective. We provide a comparison of the extraction results of different models in Appendix A.3. Figure 5 shows an example of the visualization of the Edge-Mask matrix. It can be seen that most of the edges have been masked, and only the edges from two key feature nodes have been preserved. This proves that our method can effectively filter noisy features and extract supporting facts. Explainable Diagnosis with EMR It is necessary to provide explainability for automatic diagnosis systems. CAML (Mullenbach et al., 2018) provides explanations with the spans having the high attention weights in the text sequence and (Feng et al., 2020) calculates a threshold for attention selection. AdaCare (Ma et al., 2020a) calculate the average importance weights in the overall dataset to obtain symptoms strongly associated with the diseases. These works focus on correlations based on attention and ignore causality between features and diagnosis. Document Modeling with Graph Network Document modeling with graph network has been widely used in text classification (Yao et al., 2019), multi-hop reading comprehension (Cao et al., 2019) and abstract extraction (Wang et al., 2020). An EMR can also be considered as a document. There are two main ways to structure a document into a graph, based on the entity (Qiu et al., 2019) or based on the structure of the document (Zheng et al., 2020). (Tu et al., 2019) considers the integration of documents and entities as heterogeneous nodes in the graph network, and (Fang et al., 2019) propose a hierarchical model that combines document structure and entity structure. We used a multi-granularity hierarchical graph network to model the EMR documents. history (Lewis, 1973; Woodward, 2005). In recent years, (Oberst and Sontag, 2019) introduces a kind of structural causal model to genera counterfactual trajectories in a synthetic environment of sepsis management. (Lin et al., 2020) presents a patient simulator to generate informative counterfactual response in the disease diagnosis. (Lenis et al., 2020) identifies salient regions of a medical image by measuring the effect of local counterfactual image-perturbations. We use counterfactual reasoning in EMRs to provide explanations for diagnosis. In this paper, we propose a counterfactual multi-granularity graph supporting facts extraction (CMGE) method for the irregular EMR without an external medical knowledge base. Based on this model, we can correctly diagnose lymphedema. The proposed counterfactual-based approach can discover the causal relationship between symptoms and diagnosis. The results of supporting fact extraction show that our method has strong robustness and can maintain accuracy in various diseases and even in categories with few data resources. In the future, we will introduce multi-modal into the model such as radiology images to discover more medical knowledge from EMRs. This work is supported by the National Key Research and Development Program of China (No.2018YFB1005104) and the Key Research Program of the Chinese Academy of Sciences (ZDBS-SSW-JSC006).
abstain abstain objective objective abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain method objective objective abstain method result method method objective objective objective method abstain result objective method abstain abstain method abstain abstain method method result abstain method method abstain abstain abstain abstain abstain abstain abstain method method method method abstain abstain method method method method other abstain method method abstain abstain abstain method method method abstain method method abstain method abstain method method abstain method method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain abstain other other other other other other other other method abstain abstain abstain abstain method objective abstain abstain result abstain other