Causal inference is the process of capturing cause-effect relationship among variables. Most existing works focus on dealing with structured data, while mining causal relationship among factors from unstructured data, like text, has been less examined, but is of great importance, especially in the legal do-main. In this paper, we propose a novel G raph-based C ausal I nference ( GCI ) framework, which builds causal graphs from fact descriptions without much human involvement and enables causal inference to facilitate legal practitioners to make proper decisions. We evaluate the framework on a challenging similar charge disambiguation task. Experimental results show that GCI can capture the nuance from fact descriptions among multiple confusing charges and provide explainable discrimination, especially in few-shot settings. We also observe that the causal knowledge contained in GCI can be effectively injected into powerful neural networks for better performance and interpretability. Code and data are available at https://github.com/xxxiaol/GCI/ . Causal inference is the process of exploring how changes on variable T affect another variable Y . Here we call T and Y as treatment and outcome , respectively, and the changes on T are called intervention . In other words, the process of drawing a conclusion about whether and how Y changes when intervening on T is called causal inference . Most research in causal inference is devoted to analyzing structured data. Take the research question how smoking causes lung cancer (Pearl and Mackenzie, 2018) as an example. Smoking , lung cancer , together with distractors like age are extracted from structured data, like electronic health records, and considered as factors. Usually, such * Equal contribution. studies properly organize those factors into human-designed structures, e.g., a causal directed acyclic graph (Wright, 1921) with factors { smoking , age , lung cancer } as nodes and causal relations { smoking  lung cancer , age  lung cancer } as edges, and perform inference on such structures. Recent works attempt to integrate text information into causal inference (Egami et al., 2018; Veitch et al., 2019; Yao et al., 2019; Keith et al., 2020), but they mainly treat the text as a single node in the causal graph, which is relatively coarse-grained. For instance, Yao et al. (2019) investigate how the complaints from consumers affect the company's responses (admission or denial). They regard the entire text of complaint as a treatment, without looking into different aspects of the text, like the events that consumers complained about and the compensation that consumers requested. Actually, discovering causal relationship inside unstructured and high-dimensional data, like text, is also beneficial or even crucial for scenarios involving reading comprehensive text and making decisions accordingly. For instance, when a legal AI system assists judges to deal with complicated cases that involve multiple parties and complex events, causal inference could help to figure out the exact distinguishable elements that are crucial for fair and impartial judgements. As shown in Figure 1, if the system can automatically spot two 1929 essential key points, 1) the deceitful acts of the defendant, and 2) obtaining properties from the victim, from the unstructured fact descriptions, then the prediction fraud can be more convincing and helpful, rather than a label from a black box. More details of GFCI algorithm and its advantage is provided in Appendix A. The output of GFCI is a graphical object called Partial Ancestral Graph (PAG). PAG is a mixed graph containing the features common to all Di-1930 Edge Meaning A  B A causes B. A  B There is an unobserved confounder of A and B. A  B Either A causes B, or unobserved confounder. In practice, we would expect a legal AI system to provide human-readable and sound explanations to help the court make the right decisions. It is worthwhile especially for underdeveloped areas, where such techniques could help the judges of rural areas with more trustworthy references from previous judgements. This would further help to maintain the principle of treating like cases alike in many continental law systems. A main challenge in judicial practice is to distinguish between similar charges, we thus propose a new task similar charge disambiguation as a testbed. Cases of similar charges often share similar context, and a system is expected to mine the nuance of reasoning process from the text. However, performing causal inference on fact descriptions of criminal cases is not trivial at all. It poses the following challenges. 1) Without expert involvement, it is not easy to extract factors that are key to prediction, and organize them in a reasonable form that can both facilitate the inference process and tolerate noise. For example, automatically extracted elements may not cover all the key points in the textual descriptions, and automatically built graphs may contain unreliable edges. 2) It is not easy to benefit from both traditional causal inference models and modern neural architectures. In this paper, we propose a novel G raph-based C ausal I nference ( GCI ) framework, which could effectively apply causal inference to legal text analysis. GCI first recognizes key factors by extracting keywords from the fact descriptions and clustering similar ones into groups as individual nodes. Then we build causal graphs on these nodes with a causal discovery algorithm that can tolerate unobserved variables, reducing the impact of missing elements. We further estimate the causal strength of each edge, to weaken unreliable edges as much as possible, and apply the refined graph to help decision making. Experimental results show that our GCI framework can induce reasonable causal graphs and capture the nuance from plain text for legal applications, especially with few training data. neural network (NN) models. We propose two approaches: 1) imposing the causal strength constraints to the NN's attention weights; 2) applying recurrent neural networks on the causal chains extracted from our causal graphs. Experiments indicate that our methods can successfully inject the extracted causal knowledge and thus enhance the NN models. We also show that integrating GCI helps to mitigate the underlying bias in data towards the final prediction. Our main contributions are as follows: 1) We propose a novel graph-based causal inference ( GCI ) framework to apply causal inference to unstructured text, without much human involvement. 2) We explore to equip popular neural network models with our GCI , by encouraging neural models to learn from causal knowledge derived from GCI . 3) We evaluate our methods on a legal text analysis task, similar charge disambiguation, and experimental results show that our GCI can capture the nuance from plain fact descriptions, and further help improve neural models with interpretability. Two types of questions are typically related to causality. The first is whether there is causal relationship between a set of variables, and the second question is when two variables T and Y are causally related, how much would Y change if we change the value of T . Both of them are discussed in our GCI framework, and we first briefly introduce the key concepts. Causal discovery corresponds to the first type of questions. From the view of graph, causal discovery requires models to infer causal graphs from observational data. In our GCI framework, we leverage Greedy Fast Causal Inference (GFCI) algorithm (Ogarrio et al., 2016) to implement causal discovery. GFCI combines score-based and constraint-based algorithms. It merges the best of both worlds, performing well like score-based methods, and not making too unrealistic assumptions like constraint-based ones. Specifically, GFCI does not rely on the assumption of no latent confounders, thus is suitable in our situation. rected Acyclic Graphs (DAGs) that represent the same conditional independence relationship over the measured variables. In other words, PAG entails all the possibilities of valid DAGs concerning the original data. In causal inference settings (Zhang, 2008), four types of edges are provided in PAG, as listed in Table 1. With PAG, we are able to consider unobserved confounders and the uncertainty in causal inference. Causal strength estimation deals with the second type of questions. It is the very task to quantify the causal strength of each learned relation, i.e., whether the relation is strong or weak. To precisely estimate causal strength, confounders need to be kept the same. Confounder is a variable causally influencing both treatment T and outcome Y . Take the example of smoking, lung cancer and age in Section 1. Here we study if there is causal relationship between smoking ( T ) and lung cancer ( Y ), and age is a confounder C . It is straightforward to compare the proportion of lung cancer among smokers and non-smokers. However, age influences both smoking and lung cancer . Older people are more likely to smoke. They also have a much higher risk of suffering from cancer. If we do not consider the value of age , its influence to lung cancer will be regarded as smoking 's influence, thus wrongly amplify the causal effect of smoking to lung cancer . In our GCI framework, we apply Average Treatment Effect (ATE) (Holland, 1986) as a measure of causal strength. All variables are binary in our work. So given an edge T  Y , we quantify how the outcome Y is expected to change if we modify the treatment T from 0 to 1:  T,Y = E [ Y | do ( T = 1)]  E [ Y | do ( T = 0)] , (1) where E means expectation, and the do-calculus do ( T = 1) indicates intervention on T , setting its value to 1 . pairs of comparable samples with the most similar propensity scores, where each pair consists of one sample in the treated group and one in the untreated. Given the great similarity between the two samples, we could make a direct comparison between them. Specifically, propensity score L ( z ) = P ( T = 1 | Z = z ) is the probability of treatment being assigned to 1 given a set of observed confounders z . As T is binary, we have T  Z | L (  means independence). So matching on propensity scores equals matching on the full set of confounders. Our graph-based causal inference ( GCI ) framework consists of three parts, constructing the causal graph, estimating causal strength on it, and making decisions. Figure 2 shows the overall architecture. We first define the similar charge disambiguation task. Given the fact descriptions of criminal cases D = { d 1 , d 2 , . . . , d N } , a system is expected to classify each case into one charge from the similar charge set C = { c 1 , c 2 , . . . , c M } . Extracting Factors. To prepare nodes for the causal graph, we calculate the importance of word w j for charge c i using YAKE (Campos et al., 2020). We enhance YAKE with inverse document frequency (Jones, 1972) to extract more discriminative words of each charge. To discriminate the similar charges, we select p words with the highest importance scores for each charge, cluster them into q classes to merge similar keywords. The q classes together with the M charges form the nodes of the causal graph. All these factors are binary. When the graph is applied to a case, each factor is of value 1 if it exists in this case, and 0 if not. Unlike factors extracted by experts, automatically extracted keywords may be incomplete, resulting in unobserved confounders in causal discovery. Learning Causal Relationship. The next step is to build edges for the graph, in other words, discover the causal relationship between different factors. To learn causal relations and tackle the unobserved confounder problem, we use GFCI (Ogarrio et al., 2016), which does not rely on the assumption 1931 Extract Factors LabeledCasesLabeledCases A B C D Y1 Y2 A B C D Y1 Y2 A B C D A B C D A B C D A B C DA B C D 0.9 0.1 Y1 Y2 Y1 Y2 Y1 ABCD Y1Y1 ABC D Y1Y1 A BCD Y1Y1 Y2Y2 ABCD Y1Y1 A B CD Y1 Y2 ... ... We further introduce constraints to filter noisy edges. First, as the judgement is made based on the fact description, we do not allow edges from charge nodes to other ones, e.g., an edge from fraud to lie is prohibited. Second, given that causes usually appear before effects in time (Black, 1956), and fact descriptions in legal text are often written in the temporal order of events, we thus consider the chronological order of descriptions as temporal constraints to filter noisy edges. If factor A appears after B in most cases, we will not allow the edge from A to B . Note that this constraint does not imply there is an edge from B to A , as chronological order is not a sufficient condition of causality. Sampling Causal Graphs. PAG contains uncertain relations shown in Table 1, which leaves challenges for quantification and further application. So we sample Q causal graphs from PAG. Among the four edge types,  and  are clear: in each sampled graph,  edges are retained and  edges are removed (because they do not indicate causal relations between the two nodes). For  edges, they have two possible choices: being kept (cause) and being removed (unobserved confounder). In the absence of true possibility, we simply keep an edge with 1 / 2 probability, and remove it with another 1 / 2 . And for  edges, we give 1 / 3 probability for  ,  , and no edge, respectively. The quality of each sampled graph G q is measured by its fitness with data X , where we use the Bayesian information criterion BIC( G q , X ) to estimate (Schwarz et al., 1978). As the resulting graphs are noisy in nature, we estimate the strength of the learned causal relations to refine a sampled causal graph. We assign high strength to edges with strong causal effect, and near-zero strength to edges that do not indicate causal relations or with weak effect. We regard the Average Treatment Effect  GT,Y (ATE, Section 2.2) as the strength of T  Y in graph G , and utilize the Propensity Score Matching (PSM, Section 2.2) to measure it:   GT,Y = [ (cid:2) i : t i =1 ( y i  y j ) + (cid:2) i : t i =0 ( y j  y i )] /N, (2) where j = argmin k : t k (cid:3) = t i | L ( z i )  L ( z k ) | means the most similar instance in the opposite group of i , and t i , y i , z i are the value of treatment, outcome and confounders of instance i , respectively. When applying the sampled causal graphs to the similar charge disambiguation task, we simply extract factors and map the case description with the graph accordingly, and decide which charge in C is more appropriate to this case. Firstly, we compute the overall causal strength of each factor T j to Y i among the Q sampled causal graphs, where Y i represents whether charge c i is committed:   T j ,Y i = Q (cid:2) q =1 BIC( G q , X )    G q T j ,Y i , (3) 1932 where   G q T j ,Y i is the measured causal strength in G q , and is 0 if edge T j  Y i does not exist in G q . For each case, we then map the text with the graphs, and calculate scores for each charge: S ( Y i ) = (cid:2) T j  Tr ( Y i )   T j ,Y i   ( T j ) , i  { 1 , . . . , M } , (4) where  ( T j ) is a dummy variable indicating the presence of T j in this case, and T r ( Y i ) is the set of treatments of Y i (from the view of graph, the nodes pointing to Y i ). The calculated scores are fed into a random forest classifier (Ho, 1995) to learn thresholds between the charges. More advanced classifiers can also be used. Neural networks (NN) are considered to be good at exploring large volumes of textual data. This motivates us to integrate the causal framework with NN, to benefit each other. Here we propose two integration methods as shown in Figure 3. First, we inject the estimated causal strength to constrain the attention weights of a Bi-LSTM with attention model (Zhou et al., 2016). A Bi-LSTM layer is first applied to the fact descriptions to obtain contextual embeddings H = { h 1 , h 2 , . . . , h n } , h i  R b 0 , where b 0 is the dimension of embeddings. Then, an attention layer assigns different weights { a 1 , a 2 , . . . , a n } to each word, and sums the words up according to the weights to build a text embedding v : a i = exp( q T  h i ) (cid:3) nk =1 exp( q T  h k ) , v = n (cid:2) i =1 a i  h i , (5) where q  R b 0 is a learnable query vector. Finally, we apply two fully connected layers to the text embedding v , and form the prediction vector r cons . Besides a cross-entropy loss L cross on r cons , we introduce an auxiliary loss L cons to guide the attention module with the causal strength learned from GCI . Given the golden label c j , for each word w i which belongs to the factor f ,   T f ,Y j is the corresponding causal strength, and g i is the normalized strength over the whole sequence. L cons is set to make the attention weights close to the normalized LSTMA. Note that in the validation and testing stages, the inputs do not contain any strength constraint and golden charge information. Therefore, we select the epoch with the least cross-entropy loss in the validation stage to evaluate on the test set. Causal chains are another type of knowledge that can be captured from causal graphs. In the legal scenario, causal chains depict the process of committing crimes. They can also be treated as the summarization of cases or behavioural patterns for each charge. Therefore, the second approach is to leverage the causal chains directly, as the chains may contain valuable information for judgement. For a given text, we extract factors and traverse all causal chains composed by the factors from the sampled causal graphs, Chains = { chain 1 , chain 2 , . . . , chain m } . In this task, we only consider chains ending up with treatments of charges, as they are more relevant with the judgement. An LSTM layer is applied to each chain, and all the chains are pooled to build case representation c  R b 0 : ch i = l i (cid:2) j =1 (LSTM( chain i ) j ) , c = MaxPooling(BIC( G q , X )  ch i ) , 1  i  m, chain i  G q , (7) where l i indicates the length of chain i . The case representation c is then fed to two fully connected layers to make the prediction r chain , and a cross-entropy loss is used to optimize the model. 1933 Charge Sets Charges #Cases Personal Injury Intentional Injury & Murder & 6377 / 2282 / Involuntary Manslaughter 1989 Violent Acquisition Robbery & Seizure & 5020 / 2113 / Kidnapping 622 F&E Fraud & Extortion 3536 / 2149 E&MPF Embezzlement & 2391 / 1998 Misappropriation of Public Funds AP&DD Abuse of Power & 1950 / 1938 Dereliction of Duty Table 2: Summary of the similar charge sets. Causal Chains vs. Whole Text. Both CausalChain and LSTM are a straightforward application of unidirectional LSTM, but over different texts, one for our extracted causal chains and the other for the whole fact description. We find CausalChain outperforms LSTM by 8.2% on average Acc and 11.7% on average F1. The difference shows that causal chains contain condensed key information that contributes to the judgement, while the whole description may contain far more irrelevant information that may disturb the prediction. We also conduct experiments on combining causal chains and the 1934 Models Personal Violent F&E E&MPF AP&DD Average Injury Acquisition LSTM 1% 60.94 / 37.91 58.48 / 29.33 63.91 / 47.00 53.56 / 39.84 52.08 / 46.13 57.79 / 40.04 5% 61.97 / 44.88 67.09 / 35.86 71.60 / 68.68 59.89 / 56.88 54.12 / 48.53 62.93 / 50.97 10% 76.45 / 67.81 65.64 / 47.62 82.14 / 80.74 70.21 / 70.00 55.46 / 51.29 69.98 / 63.49 30% 85.37 / 81.27 74.43 / 66.05 88.10 / 87.33 71.60 / 70.82 65.61 / 65.19 77.02 / 74.13 50% 85.67 / 83.02 80.10 / 72.27 90.04 / 89.06 75.59 / 75.46 69.65 / 69.62 80.21 / 77.89 Bi-LSTM 1% 62.29 / 40.81 53.86 / 33.25 62.95 / 43.27 54.54 / 41.91 48.98 / 37.84 56.52 / 39.42 5% 74.00 / 69.52 65.18 / 38.99 60.34 / 56.96 61.88 / 61.63 51.77 / 46.23 62.63 / 54.66 10% 76.66 / 71.86 67.10 / 46.07 85.31 / 84.37 60.08 / 53.34 60.20 / 57.95 69.87 / 62.72 30% 85.46 / 82.53 75.30 / 64.12 87.57 / 86.58 70.45 / 69.64 65.45 / 65.12 76.85 / 73.60 50% 87.19 / 85.01 78.43 / 69.94 90.43 / 89.83 76.08 / 75.78 71.12 / 70.50 80.65 / 78.21 GCI 1% 69.54 / 49.77 57.08 / 42.55 82.81  / 82.56  74.65  / 70.22  62.47 / 61.72 69.31  / 61.36  5% 81.19 / 75.58 69.70 / 60.39  88.25 / 87.24  83.27  / 83.06  78.09  / 77.95  80.10  / 76.84  10% 80.33 / 74.50 74.06 / 67.31  87.97 / 87.51 85.23  / 84.62  78.36  / 78.31  81.19  / 78.45  30% 84.83 / 80.10 75.99 / 70.64 89.31 / 88.39 88.55  / 88.21  80.82 / 80.56  83.90  / 81.58  50% 85.72 / 81.62 76.31 / 71.45 90.41 / 89.14 89.01 (cid:2) / 88.63 (cid:2) 81.01 (cid:2) / 80.90 (cid:2) 84.49 / 82.35 GCI-co 1% 67.49 / 44.43 63.70  / 34.64 75.72 / 67.60 69.08 / 67.20 64.93  / 64.41  68.19 / 55.66 5% 76.70 / 63.94 67.65 / 34.35 86.63 / 85.81 82.23 / 81.86 73.94 / 73.77 77.43 / 67.95 10% 68.05 / 45.37 69.26 / 46.39 85.62 / 84.41 81.23 / 79.64 74.21 / 74.05 75.67 / 65.97 30% 77.31 / 63.45 70.42 / 50.94 81.44 / 80.54 85.71 / 85.20 74.43 / 74.28 77.86 / 70.88 50% 79.21 / 69.37 70.38 / 50.78 79.30 / 77.58 84.39 / 83.72 74.16 / 73.99 77.49 / 71.09 CausalChain 1% 73.20  / 60.31  63.60 / 44.02  68.01 / 52.93 66.97 / 56.66 63.13 / 62.30 66.98 / 55.24 5% 81.99  / 76.03  70.57 / 59.85 88.64  / 87.21 75.13 / 74.74 71.75 / 70.38 77.62 / 73.64 10% 81.21 / 74.71 73.50 / 66.66 87.59 / 86.36 79.75 / 79.45 74.43 / 74.11 79.30 / 76.26 30% 85.61 / 81.00 74.93 / 67.30 89.10 / 88.19 81.63 / 81.25 80.90  / 80.50 82.43 / 79.65 50% 86.41 / 83.11 75.66 / 68.47 90.45 / 89.21 81.25 / 80.09 80.03 / 79.89 82.76 / 80.16 Bi-LSTM+Att 1% 62.16 / 41.70 58.21 / 32.97 67.99 / 62.80 57.90 / 50.67 53.20 / 41.78 59.89 / 45.99 5% 78.29 / 72.81 67.50 / 50.68 85.30 / 84.28 61.86 / 55.38 58.76 / 53.03 70.34 / 63.23 10% 81.51 / 78.36 67.97 / 58.26 88.07 / 87.33 75.38 / 74.86 58.82 / 55.82 74.35 / 70.93 30% 86.07 / 83.49 80.47 / 72.55 88.97 / 88.41 81.53 / 81.14 72.84 / 72.65 81.98 / 79.65 50% 87.25 / 85.38 82.27 / 74.15 91.56 / 91.05 82.29 / 82.11 73.70 / 73.65 83.41 / 81.27 Bi-LSTM+Att+Cons 1% 70.12 / 59.46 54.29 / 40.34 78.25 / 76.80 61.03 / 60.62 53.84 / 44.93 63.51 / 56.43 5% 79.07 / 75.89 73.09  / 56.84 86.80 / 86.35 66.86 / 59.89 72.27 / 72.18 75.62 / 70.23 10% 83.33  / 79.70  76.26  / 64.62 88.76  / 88.02  80.03 / 79.64 73.53 / 73.48 80.38 / 77.09 30% 86.55  / 83.85  81.48  / 73.15  89.80  / 89.35  81.82 / 81.31 79.46 / 79.35 83.82 / 81.40 50% 88.31 (cid:2) / 86.18 (cid:2) 82.72 (cid:2) / 76.03 (cid:2) 92.05 (cid:2) / 91.55 (cid:2) 83.02 / 82.69 80.72 / 80.64 85.36 (cid:2) / 83.42 (cid:2) Table 3: Performance on similar charge disambiguation. Dataset. For the similar charge disambiguation task, we pick five similar charge sets from the Criminal Law of the People's Republic of China (Congress, 2017), which are hard to discriminate in practice (Ouyang et al., 1999), and select the corresponding fact descriptions from the Chinese AI and Law Challenge (CAIL2018) (Xiao et al., 2018). Detailed statistics of the charge sets are given in Table 2. Note we filter out the cases whose judgements include multiple charges from one charge set. The fact descriptions in our dataset are in Chinese. Our Models. We evaluate our graph-based causal inference ( GCI ) framework as described in Section 3, and two models integrating GCI with NN ( Bi-LSTM+Att+Cons and CausalChain ) as described in Section 4. Comparison Models. To study the effect of causal relationship captured by GCI , we implement a variant called GCI-co , which is built upon a correlation-based graph rather than our discovered causal graph. In detail, we compute the Pearson correlation coefficient  for every two factors, and draw an edge if  > 0 . 5 . The direction of the edge is from the factor that appears earlier in the text more often, to the other. Then we compare GCI and two integration methods with NN baselines, including LSTM , Bi-LSTM and Bi-LSTM+Att . Bi-LSTM+Att is a common backbone of legal judgement prediction models, while we do not add multitask learning (Luo et al., 2017) and expert knowledge (Xu et al., 2020) for simplicity. Since the prior knowledge learned from pre-trained models may result in unfair comparison, we do not choose the models such as BERT (Devlin et al., 2018) as baselines and backbones to eliminate the influence. Previous works integrating text into causal inference are not able to find causal relationships inside text, so we do not take them into comparison. We select a set of training ratios, 1%, 5%, 10%, 30%, and 50%, to study how the performance gap changes along with different training data available. For each setting, we run the experiments on three random seeds and report the average accuracy (Acc) and macro-F1 (F1). More details about baselines, parameter selection, and training process are in Appendix B. 5.2 Main Results Table 3 reports the charge disambiguation performance of our models and comparison models. Causal Graph vs. Correlation-based Graph. GCI outperforms GCI-co by 4.5% on average Acc, and 9.8% on average F1, indicating the graph constructed by mining causal relations better captures the relationship between charges and factors. Causal Inference vs. Neural Networks. Comparing GCI with NN baselines LSTM , Bi-LSTM and Bi-LSTM+Att , we observe in few-shot settings (1%, 5%), GCI outperforms NNs by about 10% on average, since NNs tend to underfit in few-shot settings. However, with the increase of training data, the performance gap becomes narrower and consequently, NNs outperform GCI in several cases. Compared with GCI , NNs have the advantage of learning from large amounts of unstructured data. Adding Strength Constraints. We can see that Bi-LSTM+Att+Cons outperforms Bi-LSTM+Att by around 1-5%. The performance gap is much larger in few-shot settings. This suggests that our estimated causal strength is helpful for attention-based models to capture the key information in the text. whole plain text, but simply concatenating them does not work well since the whole text may introduce noise, and better integration methods are needed, which we leave for future work. To analyze the robustness of the causal discovery process, we apply sensitivity analysis on the causal graphs. In detail, we make disturbance to the original causal relations, and examine the sensitivity of causal effect towards the violations. Following Kiciman and Sharma (2018), we use three refuters for examination: 1) Random Confounder , a new confounder with random value is added to the graph, and ideally, the causal strength should remain the same as before. 2) Placebo Treatment , the value of a treatment is replaced by a random value, so the treatment becomes a placebo, and the strength should be zero. 3) Subset of Data , we use a subset of cases to recalculate the strength. Ideally, the strength estimation will not vary significantly. We take a sampled causal graph of F&E (Fraud & Extortion) as an example, and exhibit the refuters on the treatments of charge extortion in Figure 4. Causal strength is almost the same as before after Random Confounder and Subset of Data refutation; and turns to nearly zero after Placebo Treatment . The results show that our graph construction method is robust against disturbance. Causal chains manifest how effective GCI is in terms of inducing common patterns of suspect's 1935 6.3 Effect of Integrating Causal Strength with Attention Following Lei et al. (2017), we conduct human evaluation on words accorded with high attention Charge Sets Bi-LSTM+Att Bi-LSTM+Att+Cons Personal Injury 3.03 3.17 Violent Acquisition 3.18 3.74 F&E 3.34 3.65 E&MPF 3.13 3.27 AP&DD 3.08 3.13 Table 4: Results of human evaluation. behaviours. It is helpful for people to better understand the core part of legal cases. Here we select the causal graph of Personal Injury (Intentional Injury & Murder & Involuntary Manslaughter) and showcase several causal chains underlying the text. As shown in Figure 5, the chains depict common patterns of these charges, from the initial causes, to the final criminal behaviour. More examples are provided in Appendix D. Now the question is how the graph structures help to discriminate similar charges. Here we analyze the nuance between the causal chains of two similar charges E&MPF (Embezzlement & Misappropriation of Public Funds). For both charges, the cases often first give the background that someone held a certain position of authority, thus had power. Then both kinds of cases describe that the person illegally obtained a large amount of money by utilizing his power. While the cases in E&MPF share very similar context, there exists nuance between them: embezzlement emphasizes that someone privately and illegally possesses the bulk of money, while misappropriation of public funds emphasizes that someone would temporally use the money for a certain purpose. By observing the causal chains of two charges, GCI could capture the slight difference well: for embezzlement, the causal chains tend to be work / take charge of  take advantage of (position/power) ; for misappropriation of public funds, the causal chains tend to be take charge of  take advantage of (position/power)  misappropriate  profit . The difference between the former and latter chains is whether the person had subsequent behaviour (e.g., using the money for purposes like making profits). We could observe that the difference in causal chains accords with the definitions of the two charges. weights and compare the evaluation results of standard ( Bi-LSTM+Att ) and constraint-based attention models ( Bi-LSTM+Att+Cons ). For each set of charges, we train both models with 10% data, and randomly select 30 cases that both models predict correctly. For the total 150 cases, we showcase content, charge names, attention weights above 0.05, and corresponding words. Each participant is asked to score from 1 to 5 for the extent of how beneficial the extracted keywords are to disambiguation. A higher score means that the attention weights indeed capture the keywords. Each case is assigned to at least four participants. Results are shown in Table 4. We observe that the constraint-based model is better at explanation than normal attention-based models on all five charge groups. Take Violent Acquisition as an example. Although the cases are predicted correctly by Bi-LSTM+Att , the model tends to attend to words bag , RMB and value , which frequently occur but cannot be treated as clues for judgement. Instead, Bi-LSTM+Att+Cons values factors such as grab , rob and hold , which are more helpful for judgement. Causal Inference with Text. Recently, a few works try to take text into account when performing causal inference. Landeiro and Culotta (2016) introduce causal inference to text classification, and manage to remove bias from certain out-of-text confounders. Wood-Doughty et al. (2018) use text as a supplement of missing data and measurement error for the causal graphs constructed by structured data. Egami et al. (2018) focus on mapping text to a low-dimensional representation of the treatment or outcome. Veitch et al. (2019) and Yao et al. (2019) treat text as confounder and covariate, which help to make causal estimation more accurate. These works all build causal graphs manually, and regard text as a whole to be one of the factors. In contrast, we set our sights on text containing rich causal information in itself. Paul (2017) looks into text by computing propensity score for each word, but only 1936 focuses on causal relationship between words and sentiment. We instead take a causal graph perspective, discover and utilize causal relationship inside text to perform reasoning. Neural Networks for Causal Discovery. Recently, researchers attempt to apply neural networks to causal discovery (Ponti and Korhonen, 2017; Alvarez-Melis and Jaakkola, 2017; Ning et al., 2018; Gao et al., 2019; Weber et al., 2020). However, Alvarez-Melis and Jaakkola (2017) model causal relationship by correlation, which may introduce bias into causal inference; Ning et al. (2018) and Gao et al. (2019) merely focus on capturing causality by explicit textual features or supervision from labeled causal pairs. There are also a line of works focusing on how to use neural networks to summarize confounders and estimate treatment effects (Louizos et al., 2017; Yao et al., 2018; Knzel et al., 2018), which are parts of the whole causal inference process. Weber et al. (2020) show how to formalize causal relationships in script learning, but it is limited to pairwise learning of events and cannot be generalized to sequential and compositional events. Legal Judgement Prediction. Previous works in legal text analysis focus on the task of legal judgement prediction (LJP). Luo et al. (2017) and Zhong et al. (2018) exploit neural networks to solve LJP tasks. Zhong et al. (2020) provide interpretable judgements by iteratively questioning and answering. Another line pays attention to confusing charges: Hu et al. (2018) manually design discriminative attributes, and Xu et al. (2020) use attention mechanisms to highlight differences between similar charges. Using knowledge derived from causal graphs, GCI exhibits a different and interpretable discrimination process. Although GCI is effective on its own and when working with powerful neural network models, there is still room for further improvement. More Precise Causal Inference Models. The causal estimation results of GCI is based on the constructed causal graphs in former stages, and the automated construction process may bring imprecise factors and even omissions. As clustering algorithms try to summarize the general characteristics of text, descriptions with subtle differences may be clustered into one factor, but the differences matter in legal judgement. For example, in Personal Injury's graph, different ways of killing are summarized as the factor kill , therefore lose valuable information. Specifically, beaten to death might occur in cases of involuntary manslaughter, while shooting cases are more likely to be associated with murder. Also, factors with low frequency may be omitted in clustering, but are actually useful for discrimination. Overall, under the circumstance without much expert effort, it is worthwhile to explore how to construct a more reliable causal graph. Deep understanding on legal documents. Although GCI to some extent tackles the challenges of incapability of causal inference methods on unstructured text, it may make mistakes when facing complex fact descriptions. Negation semantics is a typical example. It is occasional to see negation word usage in fact descriptions, which usually indicates that someone did not have a certain behaviour. However, GCI has not considered this aspect, and may be awry in cases containing negation usage. Besides, pronoun resolution is also an important aspect that may confuse models. For example, certain behaviour is done by the victim and the subject of the behaviour is a pronoun. If the model was unaware of the subject of the behaviour, it would be counted as criminal's behaviour and introduces noise to later inference stage. Moreover, intent could be a decisive factor when discriminating charges, for example, murder and involuntary manslaughter. But it may not be mentioned explicitly in the fact descriptions. It should be better to recover the complete course of fact and recognize the implicit intents between the lines with deep understanding of the context and relevant common sense knowledge. We propose GCI , a graph-based causal inference framework to discover causal information in text, and design approaches to integrate causal models and neural networks. In the similar charge disambiguation task, we show our approaches capture important evidence for judgement and nuance among charges. Further analysis demonstrates the quality of causal graphs, value of causal chains, and interpretability of computed causal strength. 1937 10 Ethical Considerations 10.1 Intended Use We aim to facilitate legal service with our proposed GCI , providing valuable evidence instead of directly making judgements. Acknowledgments This work is supported in part by National Hi-Tech R&D Program of China (2018YFC0831900). We would like to thank the anonymous reviewers for the helpful discussions and suggestions. Also, we would thank Yuxuan Lai, Liunian Harold Li, Jieyu Zhao, Chen Wu and Xingchen Lan for advice about 1938 experiments and writing. We hope that legal AI models could assist legal workers in underdeveloped areas, helping them to explore key points in cases, discriminate from similar crimes, and make better decisions. By treating like cases alike in different regions of the country, influences of judges' intellectual flaws and arbitrariness are weakened, and rights of people, both the defendants and the victims, are protected. Failure Mode. The model may give wrong evidence in some cases, but this will not cause signifi-cantly bad impact. The process of GCI is transparent. Extracted factors, the causal relations between them, and causal chains that lead to the final decision are all shown to users. By checking these rationales of model reasoning, users can clearly find where goes wrong, and not adopt the outputs, or intervene and correct the model. Misuse Potential. We emphasize that such a model cannot be used individually, as the trial process is seriously conducted and regulated by the judicial system. In the actual judicial process, the prosecutors, judges, and lawyers are under strict supervision. We do not think there is a possibility for them to misuse computer models. Criminal behaviour is very unbalanced in gender. Take the three charges in our Personal Injury charge set as an example. Lin and Zou (2020) counted gender ratio in criminal cases from 2013 to 2017 in China, and the ratios of male defendant are 94.70% (Intentional Injury), 87.72% (Murder), and 93.97% (Involuntary Manslaughter). The disparity in defendants of male and female leads to the small proportion of female cases in training corpus. Therefore, female cases may be inadequately trained. If this results in more incorrect predictions for female cases, women's rights are violated. Following Dixon et al. (2018) and Park et al. (2018), we use False Positive Equality Difference (FPED) and False Negative Equality Difference (FNED) to examine the performance difference in Metrics Bi-LSTM+Att Bi-LSTM+Att+Cons FPED 0.048 0.032 FNED 0.065 0.049 Table 5: Results of equality difference. two genders. They are defined as: FPED = (cid:2) t  T | FPR  FPR t | , FNED = (cid:2) t  T | FNR  FNR t | , (8) where FPR is false positive rate of classification, FNR is false negative rate, and T = { male, female } . The two metrics quantify the extent of variation between the performances of two genders. Applying them to Bi-LSTM+Att and Bi-LSTM+Att+Cons models in Personal Injury charge set, the results are shown in Table 5. The model with causal constraints achieves smaller variance measured by both metrics, which reduce between 1 / 4 to 1 / 3 of the unfairness in performance of Bi-LSTM+Att . This shows the superiority of our model with causal knowledge. Compared with normal neural networks, our constraint-based model utilizes causal relations, which are more stable to the number of occurrences. Though adding causal knowledge narrows the equality difference, it still exists in Bi-LSTM+Att+Cons (the metrics are greater than zero). Other types of bias may also exist in our model, given that the training corpora contain decisions of humans and systemic bias of humans may be preserved. Further debiasing method is needed if the model is put into real use. In general, we believe that adding causal knowledge to decision making will help debiasing, and the transparent exhibition of causal graphs and chains will enable people to find biases in time and correct them. For any correspondence, please contact Yansong Feng. 1986. Statistics and causal inference. Journal of the American statistical Association , 81(396):945960. Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation . Katherine A Keith, David Jensen, and Brendan O'Connor. 2020. Text and causal inference: A review of using text to remove confounding from causal estimates. arXiv preprint arXiv:2005.00649 . Emre Kiciman and Amit Sharma. 2018. Tutorial on causal inference and counterfactual reasoning. In ACM KDD International Conference on Knowledge Discovery and Data Mining . Virgile Landeiro and Aron Culotta. 2016. Robust text classification in the presence of confounding bias. In Thirtieth AAAI Conference on Artificial Intelligence . Tao Lei et al. 2017. Interpretable neural models for natural language processing . Ph.D. thesis, Massachusetts Institute of Technology. Wei Lin and Shaokun Zou. 2020. The Bluebook on the Big Data of Criminal Justice . Peking University Press.
abstain abstain objective method abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other method abstain abstain objective abstain abstain abstain abstain abstain abstain objective abstain method method result abstain objective method result objective objective result other objective objective other other method other other abstain other other other method other other other other other method other other other other method method abstain method other abstain other abstain other method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method objective method abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain method method method other abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain method abstain abstain abstain abstain method method other abstain abstain abstain method method abstain method abstain method abstain method method method method abstain method method abstain abstain method method method method method method method abstain abstain abstain method abstain abstain abstain method abstain method method method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other objective other method other other other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result abstain objective abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain