We consider the problem of using observational data to estimate the causal effects of linguistic properties. For example, does writing a complaint politely lead to a faster response time? How much will a positive product review increase sales? This paper addresses two technical challenges related to the problem before developing a practical method. First, we formalize the causal quantity of interest as the effect of a writer's intent , and establish the assumptions necessary to identify this from observational data. Second, in practice, we only have access to noisy proxies for the linguistic properties of intereste.g., predictions from classifiers and lexicons. We propose an estimator for this setting and prove that its bias is bounded when we perform an adjustment for the text. Based on these results, we introduce TEXTCAUSE , an algorithm for estimating causal effects of linguistic properties. The method leverages (1) distant supervision to improve the quality of noisy proxies, and (2) a pre-trained language model (BERT) to adjust for the text. We show that the proposed method outperforms related approaches when estimating the effect of Amazon review sentiment on semi-simulated sales figures. Finally, we present an applied case study investigating the effects of complaint politeness on bureaucratic response times. Social scientists have long been interested in the causal effects of language, studying questions like: How should political candidates describe their personal history to appeal to voters (Fong and Grimmer, 2016)? How can business owners write product descriptions to increase sales on e-commerce platforms (Pryzant et al., 2017, 2018a)? How can consumers word their complaints to receive faster responses (Egami et al., 2018)? What conversational strategies can mental health counselors use to have more successful counseling sessions (Zhang et al., 2020)? To study the causal effects of linguistic properties, we must reason about interventions: what would the response time for a complaint be if we could make that complaint polite while keeping all other properties (topic, pragmatics, etc.) fixed? Although it is sometimes feasible to run such experiments where text is manipulated and outcomes are recorded (Grimmer and Fong, 2020), analysts typically have observational data consisting of texts and outcomes obtained without intervention. This paper formalizes the estimation of causal effects of linguistic properties in observational settings. Estimating causal effects from observational data requires addressing two challenges. First, we need to formalize the causal effect of interest by specifying the hypothetical intervention to which it corresponds. The first contribution of this paper is articulating the causal effects of linguistic properties; we imagine intervening on the writer of a text document and telling them to use different linguistic properties. The second challenge of causal inference is identification: we need to express causal quantities in terms of variables we can observe. Often, instead of the true linguistic property of interest we have access to a noisy measurement called the proxy label . Analysts typically infer these values from text with classifiers, lexicons, or topic models (Grimmer and Stewart, 2013; Lucas et al., 2015; Prabhakaran et al., 2016; Voigt et al., 2017; Luo et al., 2019; Lucy et al., 2020). The second contribution of this paper is establishing the assumptions we need to recover the true effects of a latent linguistic property from these noisy proxy labels. In particular, we propose an adjustment for the confounding information in a text document and prove that this bounds the bias of the resulting estimates. an algorithm for estimating the causal effects of linguistic properties. The algorithm uses distantly supervised label propagation to improve the proxy label (Zhur and Ghahramani, 2002; Mintz et al., 2009; Hamilton et al., 2016), then BERT to adjust for the bias due to text (Devlin et al., 2018; Veitch et al., 2020). We demonstrate the method's accuracy with partially-simulated Amazon reviews and sales data, perform a sensitivity analysis in situations where assumptions are violated, and show an application to consumer finance complaints. Data and a package for performing text-based causal inferences is available at https:// github.com/rpryzant/causal-text . Causal inference from observational data is well-studied (Pearl, 2009; Rosenbaum and Rubin, 1983, 1984; Shalizi, 2013). In this setting, analysts are interested in the effect of a treatment T (e.g., a drug) on an outcome Y (e.g., disease progression). For ease, we consider binary treatments. The average treatment effect (ATE) on the outcome Y is,  = E [ Y ; do( T = 1)]  E [ Y ; do( T = 0)] , (1) where the operation do( T = t ) means that we hypothetically intervene and set the treatment T to some value (Pearl, 2009). Typically, the ATE  is not the simple difference in average conditional outcomes, E [ Y | T = 1]  E [ Y | T = 0] . This is because confounding variables C are associated with both the treatment and outcome, inducing non-causal associations between them, referred to as open backdoor paths (Pearl, 2009). When all the confounding variables are observed, we can write the ATE in terms of observed variables using the backdoor-adjustment formula (Pearl, 2009),  = EC (cid:104) E [ Y | T = 1 , C ]  E [ Y | T = 0 , C ] (cid:105) . For example, if the confounding variable C is discrete, we group the data into values of C , calculate the average difference in outcomes between the treated and untreated samples of each group, and take the average over groups. We are interested in the causal effects of linguistic properties. To formalize this as a treatment, we Figure 1: The proposed causal model of text and outcomes. imagine intervening on the writer of a text, e.g., telling people to write with a property (or not). We show that to estimate the effect of using a linguistic property, we must consider how a reader of the text perceives the property. These dual perspectives of the reader and writer are well studied in linguistics and NLP; 1 we adapt the idea for causal inference. Figure 1 illustrates a causal model of the setting. Let W be a text document and let T (binary) be whether or not a writer uses a particular linguistic property of interest. 2 For example, in consumer complaints, the variable T can indicate whether the writer intends to be polite or not. The outcome is a variable Y , e.g., how long it took for this complaint to be serviced. Let Z be other linguistic properties that the writer communicated (consciously or unconsciously) via the text W , e.g. topic, brevity or eloquence. The linguistic properties T and Z are typically correlated, and both variables affect the outcome Y . 1 Literary theory argues that language is subject to two perspectives: the artistic pole  the text as intended by the author  and the aesthetic pole  the text as interpreted by the reader (Iser, 1974, 1979). The noisy channel model (Yuret and Yatbaz, 2010; Gibson et al., 2013) connects these poles by supposing that the reader perceives a noisy version of the author's intent. This duality has also been modeled in linguistic pragmatics as the difference between speaker meaning and literal or utterance meaning (Potts, 2009; Levinson, 1995, 2000). Gricean pragmatic models like RSA (Goodman and Frank, 2016) similarly formalize this as the reader using the literal meaning to help make inferences about the speaker's intent. where we imagine intervening on writers and telling them to use the linguistic property of interest (setting T = 1 ,  write politely ) or not ( T = 0 ). This causal effect is appealing because the hypothetical intervention is well-defined  it corresponds to an intervention we could perform in theory. However, without further assumptions,  wri. is not identified from the observational data. The reason is that we would need to adjust for the unobserved linguistic properties Z , which create open backdoor paths because they are correlated with both the treatment T and outcome Y (Fig-ure 1). To solve this problem, we observe that the reader is the one who produces outcomes. Readers use the text W to perceive a value for the property of interest (captured by the variable  T ) as well as other properties (captured by  Z ) then produce the outcome Y based on these perceived values. For example, a customer service representative reads a consumer complaint, judges whether (among other things) the complaint is polite or not, and chooses how quickly to respond based on this. Consider the average treatment effect,  rea. where we imagine intervening on the reader's perception of a linguistic property  T . The following result shows that we can identify the causal effect of interest,  wri. , by exploiting this ATE  rea. . Theorem 1. Let  Z = f ( W ) be a function of the words W such that E [ Y | W ] = E (cid:104) Y |  T ,  Z (cid:105) . Suppose that the following assumptions hold: 1. (no unobserved confounding) W blocks backdoor paths between  T and Y , 2. (agreement of intent and perception) T =  T . 3. (overlap) For some constant (cid:15) > 0 , (cid:15) < P (  T = 1 |  Z ) < 1  (cid:15) with probability 1. 3 3 Informally, it must be possible to perceive a property (  T =1) for all settings of  Z , and  Z cannot perfectly predict  T . Then the ATE  rea. is identified as,  rea. = EW (cid:20) E (cid:104) Y |  T = 1 ,  Z = f ( W ) (cid:105)  (5) E (cid:104) Y |  T = 0 ,  Z = f ( W ) (cid:105) (cid:21) . (6) Moreover, the ATE  rea. is equal to  wri. . The proof is in Appendix A. Intuitively, the result says that the information in the text W that the reader uses to determine the outcome Y splits into two parts: the information the reader uses to perceive the linguistic property of interest (  T ), and the information used to perceive other properties (  Z = f ( W ) ). The information captured by the variable  Z is confounding; it affects the outcome and is also correlated with the treatment  T . Under certain assumptions, adjusting for the function of text  Z that captures confounding suf-fices to identify the  rea. ; in Figure 1, the backdoor path  T  W   Z  Y is blocked. 4 Moreover, if we assume that readers correctly perceive the writer's intent, the effect  rea. , which can expressed in terms of observed variables, is equivalent to the effect that we want,  wri. . If we observed  T , the reader's perception of the linguistic property of interest, then we could proceed by estimating the effect  rea. (equivalently,  wri. ). However, in most settings, one does not observe the linguistic properties that a writer intends to use ( T and Z ) or that a reader perceives (  T and the information in  Z ). Instead, one uses a classifier or lexicon to predict values for this property from the text, producing a proxy label  T (e.g. predicted politeness). For this setting, where we only have access to proxy labels, we introduce the estimand  proxy which substitutes the proxy  T for the unobserved treatment  T in the effect  rea. :  proxy = EW (cid:20) E (cid:104) Y |  T = 1 ,  Z = f ( W ) (cid:105) (7)  E (cid:104) Y |  T = 0 ,  Z = f ( W ) (cid:105) (cid:21) . 4 Grimmer and Fong (2020) studied a closely related setting where text documents are randomly assigned to readers who produce outcomes. From this experiment, they discover text properties that cause the outcome. Their causal identification result requires an exclusion restriction assumption, which is related to the no unobserved confounding assumption that we make. This estimand only requires an adjustment for the confounding information  Z . We show how to extract this information using pretrained language models in Section 5. Prior work on causal inference with proxy treatments (Wood-Doughty et al., 2018) requires an adjustment using the measurement model P (  T |  T ) , i.e. the true relationship between the proxy label  T and its target  T , which is typically unobserved. In contrast, the estimand  proxy does not require the measurement model. The following result shows that the estimand  proxy only attenuates the ATE that we want,  rea. . That is, the bias due to proxy treatments is benign; it can only decrease the magnitude of the effect but it does not change the sign. Theorem 2. Let (cid:15) 0 = Pr(  T = 0 |  T = 1 ,  Z ) and let (cid:15) 1 = Pr(  T = 1 |  T = 0 ,  Z ) . Then,  proxy =  rea. The proof is in Appendix E. This result shows that the proposed estimand  proxy , which we can estimate, is equal to the ATE  rea. that we want, minus a bias term related to measurement error. In particular, if the classifier is better than chance and the treatment effect sign is homogeneous across possible texts  i.e., it always helps or always hurts, an assumption the analyst must carefully assess  then the bias term is positive with the degree of attenuation dependent on the error rate of the proxy label  T . The result tells us to construct the most accurate proxy treatment  T possible, so long as we adjust for the confounding part of the text. 5 This is a novel result for causal inference with proxy treatments and sidesteps the need for the measurement model. We introduce a practical algorithm for estimating the causal effects of linguistic properties. Motivated by Theorem 2, we first describe an approach for improving the accuracy of proxy labels. We then use the improved proxy labels, text and outcomes to fit a model that extracts and adjusts for the confounding information in the text (  Z ). In practice, one may observe additional covariates 5 We prove in Appendix F that without the adjustment for confounding information  Z , estimates of the ATE  rea. C that capture confounding properties, e.g., the product that a review is about or complaint type. We will include these covariates in the estimation algorithm. The first stage of TEXTCAUSE is motivated by Theorem 2, which said that a more accurate proxy can yield lower estimation bias. Accordingly, this stage uses distant supervision to improve the fidelity of lexicon-based proxy labels  T . In particular, we exploit an inductive bias of frequently used lexicon-based proxy treatments: the words in a lexicon correctly capture the linguistic property of interest (i.e., high precision, Tausczik and Pennebaker, 2010), but can omit words and discourse-level elements that also map to the desired property (i.e., low recall, Kim and Hovy, 2006; Rao and Ravichandran, 2009). Motivated by work on lexicon induction and label propagation (Hamilton et al., 2016; An et al., 2018), we improve the recall of proxy labels, training a classifier P  to predict the proxy label  T , then using that classifier to relabel examples which were labeled  T = 0 but look like  T = 1 . Formally, given a dataset of tuples { ( Y i , W i , C i ,  T i ) } ni =1 the algorithm is: 1. Train a classifier to predict P  (  T | W ) , e.g., logistic regression trained with bag-of-words features and  T labels. 2. Relabel some  T = 0 examples (we experiment with ablating this in Appendix C):  T  i = (cid:40) 1 if  T i = 1 1 [ P  (  T i = 1 | W i ) > 0 . 5] otherwise The second stage of TEXTCAUSE estimates the effect  proxy using the text W , improved proxy labels  T  , and outcomes Y . This stage is motivated by Theorem 1, which described how to adjust for the confounding parts of the text. We approximate this confounding information in the text,  Z = f ( W ) , with a learned representation b ( W ) that predicts the expected outcomes E [ Y |  T  = t, b ( W ) , C ] for t = 0 , 1 (Eq. 7). We use DistilBERT (Sanh et al., 2019) to produce a representation of the text b ( W ) by embedding the text then selecting the vector corresponding to a prepended [CLS] token. We proceed Figure 2: The second stage of TEXTCAUSE adapts word embeddings to predict both of Y 's potential outcomes. to optimize the model so that the representation b ( W ) directly approximates the confounding information  Z = f ( w ) . In particular, we train an estimator for the expected conditional outcome Q ( t, b ( W ) , C ) = E [ Y |  T  = t, b ( W ) , C ] :  Q ( t, b ( W ) , C ) =  ( M bt b ( W ) + M ct c + b )) , where the vector c is a one-hot encoding of the covariates C , the vectors M bt  R 768 and M ct  R | C | are learned, one for each value t of the treatment, and the scalar b is a bias term. Letting  be all parameters of the model, our training objective is to minimize, min  n (cid:88) i =1 L ( Y i ,  Q  (  T  i , b ( W i ) , C i ) +   R ( W i ) , where L (  ) is the cross-entropy loss and R (  ) is the original BERT masked language modeling objective, which we include following Veitch et al. (2020). The hyperparameter  is a penalty for the masked language modeling objective. The parameters M t are updated on examples where  T  i = t . Once  Q (  ) is fitted, an estimator   proxy for the effect  proxy (Eq. 7) is,   proxy = 1 n (cid:88) i (cid:104)  Q (1 , b ( W i ) , C i )   Q (0 , b ( W i ) , C i ) (cid:105) , (9) where we approximate the outer expectation over the text W with a sample average. Intuitively, this procedure works because the representation b ( W ) extracts the confounding information  Z = f ( W ) ; it explains the outcome Y as well as possible given the proxy label  T  . We evaluate the proposed algorithm's ability to recover causal effects of linguistic properties. Since ground-truth causal effects are unavailable without randomized controlled trials, we produce a semisynthetic dataset based on Amazon reviews where only the outcomes are simulated. We also conduct a real-world study using real-world complaints and bureaucratic response times. Our key findings are  More accurate proxies combined with text adjustment leads to more accurate ATE estimates. Naive proxy-based procedures significantly underestimate true causal effects. ATE estimates can lose fidelity when the proxy is less than 80% accurate. Dataset. Here we use real world and publicly available Amazon review data to answer the question, how much does a positive product review affect sales? We create a scenario where positive reviews increase sales, but this effect is confounded by the type of product. Specifically:  The text W is a publicly available corpus of Amazon reviews for digital music products (Ni et al., 2019). For simplicity, we only include reviews for mp3, CD, or Vinyl. We also exclude reviews for products worth more than $100 or fewer than 5 words. The observed covariate C is a binary indicator for whether the associated review is a CD or not, and we use this to simulate a confounded outcome. The treatment T =  T is whether that review is positive (5 stars) or not (1 or 2 stars). Hence, we omit reviews with 3 or 4 stars. Note that here it is reasonable to assume writer's intention ( T ) equals the reader's perception (  T ), as the author is deliberately communicating their sentiment (or a very close proxy) with the stars. We use this variable to (1) simulate outcomes and (2) calculate ground truth causal effects for evaluation. The proxy treatment  T is computed via two strategies: (1) a randomly noised version of T fixed to 93% accuracy (to resemble a reasonable classifier's output, later called  proxy-noised ), and (2) a binary indicator for whether any words in W overlap with a positive sentiment lexicon (Liu et al., 2010). The outcome Y  Bernoulli(  (  c (  ( C )   o ) +  t  T + N (0 ,  ))) represents whether a product received a click or not. The parameter  c controls confound strength,  t controls treatment strength,  o is an offset and the propensity  ( C ) = P ( T = 1 | C ) is estimated from data. The final data set consists of 17,000 examples. Protocol. All nonlinear models were implemented using PyTorch (Paszke et al., 2019). We use the transformers 6 implementation of Distill-BERT and the distilbert-base-uncased model, which has 66M parameters. To this we added 3,080 parameters for text adjustment (the M bt and M ct vectors). Models were trained in a cross-validated fashion, with the data being split into 12,000, 2,000, and 4,000-example train, validation, and test sets. 7 BERT was optimized for 3 epochs on each fold using Adam (Kingma and Ba, 2014), a learning rate of 2 e  5 , and a batch size of 32. The weighting on the potential outcome and masked language modeling heads was 0.1 and 1.0, respectively. Linear models were implemented with sklearn . For T-boosting, we used a vocab size of 2,000 and L2 regularization with a strength of c = 1 e  4 . Each experiment was replicated using 100 different random seeds for robustness. Each trial took an average of 32 minutes with three 1.2 GHz CPU cores and one TITAN X GPU. Baselines. The unadjusted baseline is   naive =  E [ Y |  T = 1]   E [ Y |  T = 0] , the expected difference in outcomes conditioned on  T . 8 The 6 https://huggingface.co/transformers 7 See Egami et al. (2018) for an investigation into train/test splits for text-based causal inference. the observed covariate C and are based on Sridhar and Getoor (2019):   naive+C = 1 | C | (cid:80) c ( E [ Y |  T = 1 , C = c ]   E [ Y |  T = 0 , C = c ]) , using randomly drawn and lexicon-based  T proxies. We also compare against semi-oracle,   matrix , an estimator which assumes additional access to the ground truth measurement model P (  T | T ) (Wood-Doughty et al., 2018); see Appendix G for derivation. Note that for clarity, we henceforth refer to the treatment-boosting and text-adjusting stages of TEXTCAUSE as T-boost and W-Adjust . Our primary results are summarized in Table 1. Individually, T-boost and W-Adjust perform well, generating estimates which are closer to the oracle than the naive unadjusted and proxy-lex' baselines. However, these components fail to outperform the highly accurate proxy-noised baseline unless they are combined (i.e., the TEXTCAUSE algorithm). Only the full T extCause algorithm consistently outperformed (i.e. produced higher quality ATE estimates) than the baselines. This result is robust to varying levels of noise and treat-ment/confound strength. Indeed TEXTCAUSE 's estimates were on average within 2% of the semi-oracle. Furthermore, these results support Theorem 2: methods which adjusted for the text always attenuated the true ATE. Adjusting for the confounding parts of text is crucial: the results show that estimators that adjust for the covariates C but not the text perform poorly, sometimes even worse than the unadjusted estimator   naive . Does it always help to adjust for the text? We consider the case where confounding information in the text causes a naive estimator which does not adjust for this information (  naive ) to have the opposite sign of the true effect  . Does our proposed text adjustment help in this situation? Theorem 2 says it should, because  proxy estimates are bounded in [0,  ]. This ensures that the most important of bits, the bit of directional information, is preserved. Table 2 shows results from such a scenario. We see that the true ATE of T ,  , has a strong negative effect, while the naive estimator  naive+C produces a positive effect. Adding an adjustment for the confounding parts of the text with TEXTCAUSE Noise: Low High Treatment: Low High Low High Mean delta Confounding: Low High Low High Low High Low High from oracle oracle (  ) 9.92 10.03 18.98 19.30 8.28 8.28 16.04 16.19 0.0 semi-oracle (   matrix ) 9.73 9.82 18.77 19.08 8.25 8.28 16.02 16.21 0.13 unadjusted (   naive ) 6.84 7.66 13.53 14.50 5.79 6.42 11.51 12.26 3.58 proxy-lex (   naive+C ) 6.67 6.73 12.88 13.09 5.65 5.67 10.98 11.12 4.43 proxy-noised (   naive+C ) 8.25 8.27 15.90 16.12 6.69 6.72 13.22 13.33 2.35 + T-boost (   naive+C ) 8.11 8.16 15.53 15.73 6.78 6.80 13.19 13.32 2.51 + W-Adjust (   proxy ) 7.82 8.57 14.96 16.13 6.62 7.22 12.95 13.76 2.39 + T-boost + W-Adjust 9.42 10.27 18.20 19.32 7.85 8.53 15.45 16.30 0.11 (TEXTCAUSE ,   proxy ) Table 1: ATE estimates: expected change in click probabilities if one were to manipulate the sentiment of a review from negative to positive. successfully brings the proxy-based estimate to 0, which is indicative of the bounded behavior that Theorem 2 suggests. Sensitivity analysis. In Figure 3 we synthetically vary the accuracy of a proxy  T by dropping random subsets of the data. This is to evaluate the robustness of various estimation procedures. We would expect (1) methods that do not adjust for the text to behave unpredictably, and (2) methods that do adjust for the text to be more robust. These results support our first hypothesis: boosting treatment labels without text adjustment can behave unpredictably, as proxy-lex and T-boost both overestimate the true ATE. In other words, the predictions of both estimators grow further from the oracle as  T 's accuracy increases. the text ( W-Adjust and TEXTCAUSE ) consistently attenuate the true ATE, which is in line with Theorem 2. However, we find that TEXTCAUSE , which makes use of T-boost and W-Adjust , may not always provide the highest quality ATE estimates in finite data regimes. Notably, when  T is less than 90% accurate, both proxy-lex and T-boost can produce higher-quality estimates than the proposed TEXTCAUSE algorithm. Note that all estimates quickly lose fidelity as the proxy  T becomes noisier. It rapidly becomes difficult for any method to recover the true ATE when the proxy  T is less than 80% accurate. We proceed to offer an applied pilot study which seeks to answer, how does the perceived politeness of a complaint affect the time it takes for that complaint to be addressed? We consider complaints filed with the Consumer Financial Protection Bureau (CFPB). 9 This is a government agency which solicits and handles complaints about finan-cial products. When they receive a complaint it is forwarded to the relevant company. The time it takes for that company to process the complaint is recorded. Some submissions are handled quickly ( < 15 days) while others languish. This 15-day threshold is our outcome Y . We additionally adjust for an observed covariate C that captures what product and company the complaint is about (mort-gage or bank account). To reduce other potentially confounding effects, we pair each Y = 1 complaint with the most similar Y = 0 complaint according to cosine similarity of TF-IDF vectors (Mozer et al., 2020). From this we select the 4,000 most similar pairs for a total of 8,000 complaints. For our treatment (politeness), we use a state-of-the-art politeness detection package geared towards social scientists (Yeomans et al., 2018). This package reports a score from a trained classifier using expert features of politeness and a hand-labeled dataset. We take examples in the top and bottom 25% of the scoring distribution to be our  T = 1 and  T = 0 examples and throw out all others. The final dataset consists of 4,000 complaints, topics, and outcomes. We use the same training procedure and hyper-parameters as Section 6.1, except now W-Adjust is trained for 9 epochs and each cross validation fold is of size 2,000. Results are given in Figure 3 and suggest that perceived politeness may have an effect on reducing response time. We find that the effect size increases as we adjust for increasing amounts of information. The unadjusted approach which does not perform any adjustment produces the smallest ATE. proxy-lex, which only adjusts for covariates, indicated the second-smallest ATE. The W-Adjust and TEXTCAUSE methods, which adjust for covariates and text, produced the largest ATE 9 https://www.consumer-action.org/ downloads/english/cfpb_full_dbase_report.pdf/ Estimator ATE SE unadjusted (   naive ) 3.01  0.3 proxy-lex (   naive+C ) 4.03  0.4 + T-boost (   naive+C ) 9.64  0.5 + W-Adjust (   proxy ) 6.30  1.6 + T-boost + W-Adjust 10.30  2.1 TEXTCAUSE , (   proxy ) Table 3: Effect size can vary across estimation methods, with methods that adjust for more information producing larger ATEs. estimates. This suggests that there is a significant amount of confounding in real world studies, and the choice of estimator can yield highly varying conclusions. Our focus fits into a body of work on text-based causal inference that includes text as treatments (Egami et al., 2018; Fong and Grimmer, 2016; Grimmer and Fong, 2020; Wood-Doughty et al., 2018), text as outcomes (Egami et al., 2018), and text as confounders (Roberts et al. (2020); Veitch et al. (2020); see Keith et al. (2020) for a review of that space). We build on Veitch et al. (2020), which proposed a BERT-based text adjustment method similar to our W-Adjust algorithm. This paper is related to work by Grimmer and Fong (2020), which discusses assumptions needed to estimate causal effects of text-based treatments in randomized controlled trials. There is also work on discovering causal structure in text, as topics with latent variable models (Fong and Grimmer, 2016) and as words and n-grams with adversarial learning (Pryzant et al., 2018b) and residualization (Pryzant et al., 2018a). There is also a growing body of applications in the social sciences (Hall, 2017; Olteanu et al., 2017; Saha et al., 2019; Mozer et al., 2020; Karell and Freedman, 2019; Sobolev, 2019; Zhang et al., 2020). This paper also fits into a long-standing body of work on measurement error and causal inference (Pearl, 2012; Kuroki and Pearl, 2014; Buonaccorsi, 2010; Carroll et al., 2006; Shu and Yi, 2019; Oktay et al., 2019; Wood-Doughty et al., 2018). Most of this work deals with proxies for confounding variables. The present paper is most closely related to Wood-Doughty et al. (2018), which also deals with proxy treatments, but instead proposes an adjustment using the measurement model. This paper addressed a setting of interest to NLP and social science researchers: estimating the causal effects of latent linguistic properties from observational data. We clarified critical ambiguities in the problem, showed how causal effects can be interpreted, presented a method, and demonstrated how it offers practical and theoretical advantages over the existing practice. We also release a package for performing text-based causal inferences. 10 This work opens new avenues for further conceptual, methodological, and theoretical refinement. This includes improving non-lexicon based treatments, heterogeneous effects, overlap violations, counterfactual inference, ethical considerations, extensions to higher-dimensional outcomes and covariates, and benchmark datasets based on paired randomized controlled trials and observational studies. This project recieved partial funding from the Stanford Data Science Institute, NSF Award IIS-1514268 and a Google Faculty Research Award. We thank Justin Grimmer, Stefan Wager, Percy Liang, Tatsunori Hashimoto, Zach Wood-Doughty, Katherine Keith, the Stanford NLP Group, and our anonymous reviewers for their thoughtful comments and suggestions.
method abstain abstain objective objective abstain objective result abstain objective objective abstain objective objective objective objective method abstain objective abstain objective objective result abstain abstain objective objective abstain abstain objective other other other method abstain other other abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method method method abstain method other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method method abstain method abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain other abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain other abstain method other other abstain abstain other abstain objective method objective abstain other other