extfieldsofstudy:
Computer Science
Biology
Mathematics

text:
Causal inference in drug discovery and development

To discover new drugs is to seek and to prove causality. As an emerging approach leveraging human knowledge and creativity, data, and machine intelligence, causal inference holds the promise of reducing cognitive bias and improving decision making in drug discovery. While it has been applied across the value chain, the concepts and practice of causal inference remain obscure to many practitioners. This article offers a non-technical introduction to causal inference, reviews its recent applications, and discusses opportunities and challenges of adopting the causal language in drug discovery and development.

Introduction
Causal inference is the process of identifying causal effects based on prior knowledge, hypothesis, and correlations observed in data. This article aims at equipping practitioners of drug discovery and development, especially those without formal training in mathematics and statistics, with necessary knowledge to start working with causal inference at three levels: (1) to recognize situations where causal inference is advantageous and to understand why a causal model is needed (Section 2), (2) to perform causal inference step by step with software (Section 3), and (3) to learn and get inspiration from recent applications (Section 4). Finally, in Section 5, we discuss opportunities and challenges of adopting the causal language in drug discovery and development.
Causal inference identifies causations from correlations. In statistics, correlation means the relationship between two random variables, whether causal or not. For instance, high temperatures lead to higher ice-cream sales, therefore the variables temperature and ice-cream sales are correlated. Similarly, high temperatures make wildfire more likely to happen, therefore the variables temperature and wildfire frequency are correlated. Apparently, ice-cream sales and wildfire frequency are correlated but do not cause each other. The task of causal inference in this context is to infer the causal relationship between temperature, ice-cream sales, and wildfire frequency based on observations.
There is neither a consensus definition of causality nor a single way to identify it (Box 1). It is partially because the nature of causation is still under debate 1 , and partially because even simple causality like higher temperature causes higher ice-cream sales can be broken down into an infinite chain of causal relationships that can involve entities across both physical (molecules, receptors, cells, organs and systems, society, etc.) and time scales.
In this article, we introduce the statistical causal inference approach, and define causality as a probabilistic relationship that satisfies four conditions: regular probabilistic update, manipulation, counterfactual condition, and mechanism of action.
Take the example of testing whether a drug is causal for halting deadly cancer progression in a clinical trial. Regular probabilistic update means that taking the drug modifies the conditional probability of dying from the disease within a defined time window, irrespective of where and when the trial happens. Manipulation means that drug treatment shows additional benefit even if we consider all other factors affecting patients' survival, for instance age and comorbidities. Counterfactual condition means that the death of a patient would not have been postponed had the drug not been taken. Finally, mechanism of action means that we understand why the drug prolongs patients' survival, for instance by activating tumour-infiltrating immune cells. Taken together, the four conditions ensure both statistical correlation and mechanistic understanding. They put the causality criteria by Austin Bradford Hill 2 in a causal context, and implement a practical test for the philosophical reasoning of establishing causality in healthcare. 3 2. What is causal inference and why do we need it 2.1 Why causality matters in drug discovery and development Causality is indispensable for predicting outcomes of intervention, for answering counterfactual, what-if questions, and for human understanding. Although it takes time and effort to identify causes from correlations, the investment is rewarding. The history of drug discovery is bound with cases where causality inspires repurposing of a drug, developing new classes of chemical matters, or refuting the use of a drug.
One prominent example is the Shakespearean history of thalidomide 4 . Chemically, thalidomide is a mixture of two enantiomers: the (R)-enantiomer is sedative while the (S)-enantiomer is teratogenic. While its use was first correlated with an unusually high incidence of birth deficiency 5 , it took much time and work to identify the cause for the teratogenicity, i.e. the degradation of SALL4 protein via E3 ubiquitin ligase complex 6 . In the meantime, thalidomide was found potentially effective against drug-resistant multiple myeloma. Follow-up study based on the first causal finding offered a causal explanation for this correlation, too: thalidomide induces protein degradation of key transcription factors Ikaros (IKZF1) and Aiolos (IKZF3) with the same ubiquitin apparatus. Last but not least, the causal link between a small molecule and protein degradation inspired a flourishing search for new modalities including proteolysis-targeting chimera (PROTAC) 7 and other multispecific drugs 8 .
Repurposing drugs to treat COVID-19 offers another recent example demonstrating the importance of causality. Much literature reported a correlation between treatment with drugs targeting one or both human sigma receptors (σ 1 and σ 2 ), among others hydroxychloroquine, and negative modulation of SARS-Cov-2 infection. Researchers, however, noticed the intriguing discrepancy that while most compounds have nanomolar affinity against sigma receptors, the cellular antiviral activities showed large variation. It raised the question whether pharmacological modulation of sigma receptors is causal for SARS-Cov-2 inhibition. The question led Tummino et al. to find that phospholipidosis, instead of target-mediated mechanisms, underlies the antiviral activity of many drugs 9 . Phospholipidosis is caused by physicochemical properties of the drugs, can be reliably predicted experimentally and computationally 10 , and results in dose-limiting toxicity. The example, opposite to the example of thalidomide, highlights the importance of dissecting causal mechanisms from correlations, too.
While correlations suffice for predictions, causality enables us to understand why a drug works or causes harm. What is the relationship between correlation and causation? And how can we systematically identify causations from correlations?

Distinguishing causation from correlation
Causality manifests itself by correlations, though correlations are not always caused by causality. There are four reasons why we observe a correlation between two events: causation, confounding, coincidence, and conspiracy.
We assume that two variables, x and y, depict expression levels of two proteins, X and Y, in a population of cells. If x and y is correlated, four scenarios are possible: 1. Causation: expression of X causes expression of Y, or expression of Y causes expression of X . We may use additional knowledge to favour one direction over the other. For instance, if X is a known transcription factor while Y is not, we may favour the model X→Y (→ reads causes) over Y→X.

2.
Confounding: a third, potentially unobserved, protein U causes expression of both X and Y, i.e. X←U→Y. 3. Coincidence: the correlation is solely by chance. If so, we shall observe diminishing correlations as we collect more data. Given the data, we can perform statistical inferences, for instance permutation test and bootstrapping, to test how likely we observe the correlation by chance. 4. Conspiracy: the correlation is due to deliberate manipulation of data or the sampling process. We may, for instance, create a good correlation by removing data from all cells where two proteins are not correlated. Figure 1 illustrates the point that both causation and confounding can generate correlation. While statistical tools are available to assess the probability of a coincidence, and conspiracy may be discovered by science, statistical models alone cannot tell causation and confounding apart. For that we need causal inference, which requires a causal model of how data is generated, and operates the model with statistical modelling.
For some applications, correlation is all we need. In the toy example above, if our goal is to predict the expression of Y given the expression of X in a similar yet unobserved cell from the same population, correlation suffices and there is little need to distinguish between causation and confounding. Off-the-shelf statistical and machine-learning models are good enough, because they exploit correlations between input variables and the target variable.
Delineating causation from correlation is essential for many other applications. If our task is to predict the expression of Y in a cell where we inhibit the expression of X ( Figure 1G), for instance by gene silencing, it is essential to know which of the models below is the correct one: X→Y, or Y→X, or X←U→Y. Because if X→Y is true, expression of Y shall become residual; given the other two models, expression of Y shall not be affected. If we do not have a model, collecting more data does not help us with the task.
The simple example illustrates three critical points: (1) causality is required to predict the outcome of intervention, (2) data alone is not able to tell causality, and (3) we need both scientific models, either prior knowledge or hypothesis, and data to infer causality.
The take-away message is that while statistical and machine-learning models are useful if we ask for correlation, we need additional tools to recover causality from correlations if our goal is to intervene in the system and predict the outcome. While correlation helps us to predict or even to explain 11,12 , causality helps us to understand.

Causal modelling with Directed Acyclic Graphs (DAGs)
Directed Acyclic Graphs, or DAGs, are computational models of causal inference. DAGs contain nodes, which represent variables, and edges, which represent causal relationships. Like the two-variable cases described above, DAGs represent either knowledge or hypotheses about causal relationships between the variables. The edge, i.e. the causal relationship, can have any discrete, linear, or nonlinear functional form. A lack of edge between any pair of nodes means that we exclude the possibility that they have direct causality on each other. An indirect causal relationship may still exist in such cases due to the propagation of causality through the DAG, as we shall see below.
Three 3-node structures are prevalent in DAGs. Understanding these common structures, known as motifs, allows us to interpret more complex causal models.
1. The pipe (Figure 2A) describes the simplest chain of causality: X→Z→Y. 2. The fork ( Figure 2B) describes a variable Z causing both X and Y: X←Z→Y. 3. The collider ( Figure 2C) describes the situation where a variable Z is caused by both X and Y: X→Z←Y.
A pipe transduces causality in a chain, for instance Vemurafenib binding to V600E-mutated BRAF→reduced BRAF signalling→reduced tumour size. If the intermediate cause is manipulated, for instance by mutations enhancing BRAF signalling, the causal effect from drug on tumour size shall diminish or even reverse. Statistically speaking, X and Y are marginally correlated, and they are independent conditional on Z.
A fork, like the temperature example raised before, generates correlations between variables that do not cause each other. Example: PI3K phosphorylation←RAS signalling→BRAF proliferation. If the common cause is present, its both effects are correlated; if the common cause is absent, or if we focus on the subset of the data where the common cause takes the same value, the effects become independent. Similar to the situation of pipes, X and Y are marginally correlated, and they become independent conditional on Z.
Colliders behave very differently from pipes and forks. For instance, cells that lose the expression of the BOP1 (Block of Proliferation 1) gene become resistant against Vemurafenib 13 . The mechanism can be described by a collider structure: BRAF V600E→increased phosphorylation of ERK1/2←loss of BOP1, namely both BRAF V600E mutation (V600E for short) and loss of BOP1 (BOP1↓) increases ERK1/2 phosphorylation (↑pERK1/2). As long as either condition is satisfied, we can observe ↑pERK1/2. Assuming that they are the only two factors influencing pERK1/2, the collider creates a dependency between V600E and BOP1↓. When ↑pERK1/2 is true, either V600E or BOP1↓ or both must exist; otherwise, neither V600E nor BOP1↓ is possible. Even if V600E and BOP1↓ are statistically independent from each other, i.e. knowing V600E does not give us any information about whether BOP1 is lost or not, the two events become correlated once we observe ↑pERK1. In contrast to pipes and forks, X and Y are marginally independent but become correlated conditional on Z.
Knowledge of causal structure allows us to better recognize cognitive bias. We demonstrate this point with the example of colliders. With simulated data generated by the three 3-node structures in Figure 1, we show that we can remove or create correlation between x and y if we stratify by the third variable z, depending on the graph structure. The conclusion holds if z is a continuous variable and if we regress it out in a statistical model. Imagine now we face a dataset with two variables, x and y, we observe a good linear correlation, and our goal is to predict the value of y given a new set of values x. Does a linear model suffice? Not necessarily, because if there is a variable z forming a collider with x and y, and its value changes for the new set of values x, then our linear model is likely to fail. Even the simplest causal graphs show us it is important to know which factors affect the generation of data.
Besides the pipes, forks, and colliders, we highlight two 4-node motifs: 1. The descendant describes a variant of the collider, where the variable caused by two variables X and Y is not directly observed, but its descendant, namely another variable that is caused by the unobserved variable, is observed. Since we often use the variable U to indicate an unobserved variable, the DAG can be represented as: X→U←Y and U→Z. The structure, for instance, is useful to model the effect of known risk factors (X) and treatment (Y) on disease status (U). Though U may not be directly observed, a biomarker (Z), which is partially affected by U, can be measured. 2. The paw or 3-pan describes a class of motifs, including the one shown in Figure 5A.
The tail variable is known as instrumental variable. Models with instrumental variables are applicable in a wide variety of contexts, including Mendelian Randomization, a technique for target identification to be described in Section 4, and the analysis of clinical-trial data with non-compliance, i.e. when some patients in the treatment arm do not take the medicine as prescribed 14 .
DAGs consisting of these and other motifs 15 can represent causal relationships between any number of variables. We refer readers interested in reading and understanding DAGs better to several outstanding tutorials [16][17][18] .
Since DAG structure specifies marginal and conditional correlations, it is possible to rank models by their plausibility given data, and to learn the strength of causal relationships. The two tasks are known as causal identification, or causal discovery, in which the truth value of a claim of the form 'C causes E' is determined; and quantitative causal estimation, in which a numerical value s (strength) is estimated for a claim of the form 'C has an effect on E'. Causal discovery is more challenging than causal estimation, and discovered causal effects usually require causal estimation before they can be used for high-stake applications 19,20 . Nevertheless, it plays an important role in reconstructing causal networks for disease understanding and target identification, among others because we lack much biological knowledge in order to hypothesize how the observed data are generated. See Section 4 for details.

Relationship between causal inference and established modelling techniques
Causal inference completes statistical modelling and mechanistic modelling, two commonly used techniques in drug discovery and development (Table 1). Statistical modelling, including pattern-recognition-based machine-learning models, aim at identifying prediction-relevant features, and functions that transform the features to approximate the target variable and thereby exploit correlations. The models may or may not have a causal interpretation. On the other hand, mechanistic modelling uses mathematical models of biological processes to describe and predict how components, information, or energy of a system evolve with time. Examples include pharmaco-kinetics and -dynamics (PK/PD) and Quantitative System Pharmacology (QSP) models based on ordinary or partial differential equations (ODEs/PDEs), agent-based models, and Hidden Markov Models. While mechanistic models usually have a causal interpretation and are powerful for predicting outcome of intervention, we often miss information of important factors that affect the system's behaviour.
The three modelling approaches complement and benefit from each other. Reciprocal benefits between statistical and causal models are apparent. By discovering and quantifying causal relationships between observed variables, we can render statistical models causal interpretations. By using advanced machine-learning approaches, we may learn highly complex functional forms describing the causal effect 21 , or discover causality from high-dimensional data 22 . Practically, since we are often limited by the volume of high-quality data, causal inference may likely profit from parsimonious statistical models and Bayesian approaches 23 .
Results of causal inference help refine and improve existing mechanistic models by including causal factors and removing confounding factors. In turn, outputs from mechanistic models complement the current causal inference regime by predicting time-dependent outcomes of intervention, in particular those of components that regulate each other.

Causal inference for experimental data and observational data
The power of causal inference to connect and enhance existing modelling approaches is particularly prominent in the analysis of observational studies. Classically, we distinguish observational studies from controlled experimental studies. In a controlled experiment, we assign test objects, for instance animals in preclinical experiments, to groups. One group receives the treatment while the other group does not. If we go one step further to require that the grouping is randomized with regard to any relevant attributes of the test objects (passage, sex, body weight, etc.), then we have a randomised controlled experiment, a gold standard to establish causality. In an observational study, in contrast, we measure or survey members of a sample without trying to affect them. Such data can come from epidemiological studies, electronic health records (EHRs), insurance claims, and other data that come from natural (instead of controlled) experiments, for instance omics and behavioural data of healthy individuals and patients.
Several reasons make it imperative to use causal models to analyse data generated by observational studies. First, it allows us to integrate knowledge and hypotheses about inevitable biases in the data generation process. Second, we can investigate the causal effect of independent variables of interest while considering other variables that affect the outcome, known as covariates. Third, we can potentially resolve the effects of variables that influence both the independent variable and the outcome, known as confounding variables.
If we analyse observational data as if we were handling randomised experimental data, without considering the causal structure underlying the data generation and collection, we may derive false, sometimes ridiculous, conclusions 24 .
Even if a study is set up as a classical randomised experiment, various reasons may break the randomization and demand causal inference to identify the treatment effect. For instance, in clinical trials, noncompliance is a common issue, where some patients do not take the treatment as prescribed. To identify the real treatment effect while acknowledging the noncompliance, it is necessary to employ a causal model, with the instrumental variable model illustrated above as the simplest example. Other issues that have been successfully addressed by causal inference include missing data 25,26 , for instance when patients drop out of the trial, and intercurrent events, i.e. events occurring after randomization which can either preclude observation of the outcome of interest or affect its interpretation (ICH E9(E1)), for instance development of anti-drug antibodies (ADAs) [27][28][29][30] . The book by Guido Imbens and Donald Rubin provides an outstanding introduction to the analysis of these and other aberrations from classical randomised experiments with causal inference 14 .
Causal inference can leverage both experimental and observational data to generate insight. Eichler et al. proposed a new framework, known as threshold-crossing, that generates evidence by synthesising historical randomised clinical trials and real-world data resources 31 . A recent landscape assessment and elicited comments have also shown that causal inference penetrates and contributes significantly to clinical-trial study design and analysis by judging the suitability and integrating real-world data from observational studies 32,33 .
Finally, we emphasise the importance of study design for both experimental and observational studies. It is widely acknowledged that well-designed randomised controlled experiments, both in preclinical drug discovery 34 and in clinical drug development 35 , are the gold standard of verifying causal relationships. The quality and strength of evidence provided by an observational study and causal inference is also determined largely by the study design 36 . Study design should be scrutinised prior to causal inference in order to gauge whether we can answer the question of interest.

A step-by-step guide of causal inference
We introduce causal inference as an iterative process of six steps: modelling → identification → estimation → refutation → refinement → application. They are both illustrated in Figure 3 and detailed below. To assist practitioners acquiring hand-on experience with causal inference, we offer complementary interactive tutorials with case studies as Supplementary Information. The tutorials, implemented in programming languages Python (with Jupyter Notebooks) and R (with Rmarkdown), are available at https://github.com/Accio/causal_drug_discovery.
Step 1: Modelling To start, we construct a DAG to model causal mechanisms that generated the observed data by synthesising common sense, scientific knowledge, or explicit assumptions. If multiple hypotheses exist, we may build several DAGs and subject them to analysis and comparison.
In case there is little knowledge and hypothesis available, we may apply causal discovery techniques to hypothesize about how data may be generated. Commonly used methods include Bayesian networks 37 , factor analysis 38 , encoder-decoder and other deep generative models 39 . Limitations of such methods include (1) they are often technically challenging (NP-hard problems), (2) the results often include many alternative hypotheses that explain the data equally well, and (3) the proposed models may not be causal, i.e. fail to be validated by interventional studies. Nevertheless, the discovered causal graphs, combined with expert's review and curation, may serve as reasonable starting points.
Step 2: Identification Once a causal model is set up, we test whether we can answer the question of interest quantitatively. The quantity that addresses the causal question, for instance how strong is the effect of the drug on disease progression, is known as the estimand. In this step our goal is to assess whether we can estimate the estimand from the data at all, because some graph structures prohibit us from doing so.
The concept of estimand is closely associated with estimator and estimate, all of which are core to statistical inference from data. To illustrate them with an example, assume that we give six randomly chosen patients drug treatment and make the following observations on disease progression within a predefined period: 1,1,0,1,1,0 (1=slower progression than placebo, 0= similar progression as placebo). Our goal is to estimate the probability p that the treatment slows down disease progression.
1. The estimand is the quantity of interest, in this case the probability p. Apparently the estimand is defined by the objective of the experiment and data analysis. 2. The estimator is the rule for estimating the estimand. Multiple estimators may exist for the same estimand. In the given example, the maximum-likelihood estimator of p is , and a Bayesian estimator with a non-informative prior is .
3. The estimate is the numeric value that we get by analysing the data by following the rule of the estimator. In this case, =0.67 and =0. 71. ' In a causal model, the estimand is usually the causal effect pointing from a treatment (cause) to its target (effect) variable, which can be a regression coefficient or other numeric values that quantify the strength of the causal relationship. The causal graph model structure determines which estimators are available, and the numerical values of estimates are gauged by the users to interpret the results and challenged by refutation analysis.
An interesting and important result from theoretical studies is that we can assert whether an estimand can be identified using graphical models alone, independent of the choice of the estimator and without access to the data. This means that we can tell whether the question can be answered at all by causal inference before we collect or generate any data, which is usually the most time-and labour-consuming step in knowledge acquisition.
In this step, we need to identify the target estimand by specifying the quantity of interest in the DAG model. Usually we use software to identify whether the estimand can be estimated based on the graph structure. If the estimand is not identifiable, we have little choice than modifying our model, redefining our question, or admitting that the problem cannot be solved with causal inference. Otherwise, the model is said to be identifiable, and we can continue with the estimation step.
A causal model is intrinsically a generative model, namely it can be used to generate simulated data. Once the identifiability is established, a common practice is to simulate data with the model with specified parameters, to run the inference with simulated data, and to examine both whether the simulated data reassemble observed data, and whether the estimate is consistent with the input parameters. Inference with simulated data offers us experience with the model, opportunities to identify bugs and flaws of the model before the real run, and confidence in the model and the choice of estimator.
Step 3: Estimation Next we collect existing or generate new data according to the DAG. The importance of quality control of data cannot be overestimated: garbage in, garbage out. Data may be filtered so that only the subset that satisfies the DAG structure is used for the estimation.
Once we have decided on the estimand and the data, we can estimate the causal effect by identifying an appropriate estimator and deriving the estimate from the data, which is usually done by software (see Box 2 for popular open-source software packages for causal inference).
The choice of estimator depends on the DAG structure and by the nature of data. The popular DoWhy package, for instance, offers diverse estimators depending on the graph structure, including (1) (generalised) linear regression, (2) stratification, matching, or weighting by propensity score, namely the probability of each unit being assigned to a treatment group given a set of observed covariates, (3) using instrumental variables, and (4) machine-learning based estimators implemented in the EconML package. While the choice of appropriate estimators is context-and question-specific, several measures can help us make better choices, especially (1) using simulated data, (2) learning from case studies and past experience, and (3) applying refutation techniques discussed in the next step.
Step 4: Refutation While estimation is commonly deemed as the most important step in causal inference, in high-stakes applications such as drug discovery and development, we need to test the robustness of our estimates. This is achieved by refutation, a collective name of many modelling techniques to test the strength, validity, and our confidence of the estimated causal effect.

Commonly used refutation techniques include
• Sufficiency test with unobserved random common cause: Does the estimate change when we add an independent random variable as a common cause (confounder) to both the treatment and the outcome? The estimate should not be too sensitive. • Placebo treatment: What happens to the estimate when we replace the true treatment variable with an independent random variable? The effect should go to zero. • Dummy outcome: What happens to the estimate when we replace the true outcome variable with an independent random variable? The effect, again, should go to zero.
In addition, one can perform data partition or bootstrapping to estimate the variability of the estimates. Furthermore, bespoke refutation techniques can be used to address application-specific questions, for instance non-compliance in randomised clinical trials 40 . We refer interested readers to a recent review about these and other techniques known as sensitivity analysis 41 .
Statistical causal inference is not the only way to identify causality (Box 1). Alternative evidence from other models, for instance, results from mechanistic models and results from randomised controlled experiments, should be used to strengthen or challenge our belief in the output of the causal model.
Step 5: Refinement Even when the estimated causal effects withstand refutation analysis, they are seldom the sole goal of our investigation. Given that we lack a comprehensive understanding for virtually all biological problems, we often need to refine the model, which includes selection from multiple models, addition of new variables, interpretation of latent variables and of estimates, and answering what-if questions. The refined model can be used on one hand to guide design of new experiments to further refute or update the model, and on the other hand to guide interventions.
Step 6: Application One of the ultimate goals of causal inference is to perform interventions in the real world. The outcomes can be further analysed in the causal inference framework, therefore closing the loop.
In contrast to a 'hypothesis-free' paradigm, where one expects to learn mechanisms generating the data from data alone, the cycle of modelling, identification, estimation, refutation, refinement, and application integrates knowledge, hypothesis, and data to address scientific questions. Interdisciplinary teamwork is required to construct explicit models of causality, to collect estimand-specific data, and to identify causal relationships.

Literature review and case studies 4.1 Literature review
To gain an overview of applications of causal inference in drug discovery and development, we compiled a list of publications by querying the MEDLINE/PubMed database. We found more than 800 scientific publications (Supplementary Table 1). As comparison, we also queried publications on machine learning and artificial intelligence and stratified the number of publications by years ( Figure 4A). We note two interesting patterns: (1) publications on causal inference in drug discovery have increased almost steadily since 1990, and (2) the topic receives much less coverage compared with machine learning and artificial intelligence. Despite that the scope of the latter two concepts is admittedly much broader, the patterns underscore the importance of populating the causal mindset and language among practitioners.
We classified the literature by applications, and observed that causal inference has been applied throughout the value chain of drug discovery and development ( Figure 4B and 4C). The majority of publications was about analysis of observational studies, especially drug safety, pharmacovigilance, and real-world data. Methodological papers and publication reporting applications for target identification and assessment, clinical trial design and analysis, and biomarker studies followed with distance. Applications in other areas, in particular preclinical and translational research, are still scarce.
Given the broad application of causal inference, and given that applications for clinical data, real-world data, and observational studies have been extensively reviewed elsewhere 32,33,42,43 , we focus below on case studies of causal inference for disease understanding and target identification in translational research.
Translational research includes activities that establish the causal relationship between drug candidates and disease progression. It has two interlinked components: forward and reverse translation. Forward translation predicts drug's dosing and risk-benefit ratio in patients with in silico models and data from in vitro, ex vivo, in vivo animal models, and microdosing human studies. Reverse translation informs and improves decision making in forward translation by analysing data collected in clinical trials, real-world data, as well as data generated with clinical-stage or marketed molecules in preclinical models. Forward and reverse translation form a closed loop and complement each other to validate or refute the proposed causal relationship between treatment and patient's health status.
While translational research has many facets, we choose disease understanding and target identification to highlight the impact of causal inference for two main reasons. First, leveraging the causal engine in the target identification and selection phase may have the strongest potential in reducing costs of disease and of drug discovery to society [44][45][46] . Second, applications in this area involve heterogeneous and high-dimensional data, for instance EHRs and genomics and other omics data. Though the data type varies in other tasks of translational research, for instance risk-benefit assessment, biomarker selection, and patient stratification and enrichment, the concepts, software tools, and practice can be transferred.

Learning causal associations from natural experiments and observational studies
While target identification was driven by druggability and in vitro or animal disease models for a long time, reverse translation is upending this pattern. Causal inference contributes evidence to support or disaffirm drug targets by analysing data from natural experiments and observational studies. The need for causal inference is particularly strong in emerging fields such as microbiome medicine where high-quality data are relatively scarce and associative analyses are still prevalent 47 . Similarly, causal inference is needed when the study of disease aetiology is challenging, for instance in neuroscience. Path analysis, a precursor to and variant of causal inference, has been applied to prioritise or refute targets in neurogenesis, Parkinson's Disease, and autism spectrum disorder 48-50 . Recently, researchers used path analysis to dissect the quantitative contribution of Alzheimer's biomarkers and risk factors to cognitive impairment and decline by analysing longitudinal biomarker data 51 . The study reported that about 16% of variance in both cross-sectional and longitudinal cognitive impairment was accounted for by amyloid-beta and about 46% by tau. The findings suggest that both removing Aβ and completely shutting down Tau downstream events may only be partially effective in slowing down cognitive decline or reversing cognitive impairment. The results provide predictions of the theoretical boundary of efficacy of existing and new molecules modulating these targets. By integrating insights from such studies and from other modelling approaches, for instance the Q-ATN model developed by Norman Mazer and colleagues 52 , we may be able to better understand disease aetiology and to propose better targets based on real-world evidence.
Application of causal inference on multi-omics (genomics, transcriptomics, proteomics, metabolomics) data leads us to identify causal genes, pathways and gene regulatory networks that have a direct effect on disease states. [53][54][55][56] Genome-wide association studies (GWAS) have mapped the genetic architecture of common diseases in humans 57 , and have identified genetic loci that affect drug response and susceptibility to adverse drug reactions 58 . Drugs targeting genetically supported targets tend to be twice as likely to be successful as other drugs. 59,60 Exploiting GWAS data in drug discovery is, however, challenging because there are generally hundreds to thousands of genetic risk variants, mostly lying in non-coding genomic regions and each typically contributing only a small amount of risk.
"Mendelian randomization" (MR) 61,62 is a statistical approach that uses genetic variants as randomised instruments ("natural experiments") to identify causal associations between heritable traits. It is based on the fact that genotypes are independently assorted and randomly distributed in a population by Mendel's laws, or in a less strict manner, independent of each other outside haplotype blocks, 63 and not affected by environmental or genetic confounders that affect both the exposure (target manipulation) and the outcome (disease state). If it can be assumed that a genetic locus affects the outcome only through the exposure, then the causal effect of the exposure on the outcome can be derived from their relative associations to the genetic locus, which acts as an instrumental variable 64 ( Figure 5A).
By integrating GWAS with studies that map genetic effects on the transcriptome or proteome (expression or protein quantitative trait loci; eQTLs or pQTLs, respectively), MR can identify causal associations between molecular traits and disease. When a molecular QTL (mQTL) and GWAS locus overlap, MR suggests causal variants that affect phenotype via regulating gene expression, and can distinguish them from pleiotropic causal variants that affect gene expression and phenotype simultaneously, or multiple causal variants that are in linkage and regulate expression and phenotypes independently 65 . When applied to proteomic data, inferred causal associations between proteins and disease can suggest drug repurposing opportunities 66 or new candidate drug targets 67,68 . An advantage of MR is that it can be performed using summary statistics alone 69 . A limitation is that MR tests effects of molecular traits on diseases one-by-one and that no molecular pathways are reconstructed. This is important because the candidate causal factors may act indirectly or redundantly with other factors, or the candidate factors may not be druggable themselves but affect intermediate druggable targets.
Statistical model selection can orient the direction of causality among correlated genes or proteins by merging genetics and functional genomics data from individuals in a segregating population. This insight was first put forward in a visionary paper by Jansen and Nap 70 , and then formalised in a Bayesian model selection procedure 71 . This procedure compares the conditional independence relations implied by an independent, causative or reactive DAG among QTLs and pairs of genes or complex traits to infer the direction of causality from the model that best fits the available data ( Figure 5B).
Further theoretical developments have extended this model selection procedure by quantifying the statistical significance of the best fitting model to allow multiple testing correction [72][73][74] , introducing additional DAGs with hidden confounders in the model comparison 75,76 , and providing an efficient software implementation that can handle the size of modern genome-wide studies 75 . Statistical significance is estimated by expressing the conditional independence implications of the tested DAGs as combinations of likelihood ratio tests. Either their maximum P-value is treated as an omnibus hypothesis test 73,74,76 , or false discovery estimation techniques are used to express the results of the individual tests as probabilities of their null or alternative hypothesis being true, which can then be combined by the usual rules of probability theory 72,75 . Accounting for hidden confounders (e.g., coregulation by an unknown upstream factor) requires additional assumptions on the genetic instrument, usually satisfied if the instrument is a cis-acting eQTL or pQTL for the "exposure" variable, and involves a trade-off between a high false negative rate or an increased false positive rate, when hidden confounders are ignored or included in the model comparison, respectively 75,77 .
Bayesian networks combine the results of pairwise causal inferences into a comprehensive causal network model of the underlying biological system. Bayesian networks are probabilistic machine learning models for expressing both prior knowledge and inferred conditional independence and causal relationships among variables [78][79][80] . A Bayesian gene network consists of a directed graph without cycles, which connects regulatory genes to their targets 81,82 ( Figure 5C). The structure of a Bayesian network is inferred from the data using score-based or constraint-based methods 79 . Score-based methods maximise the likelihood of the model, or sample from the posterior distribution using Markov chain Monte Carlo (MCMC), using edge additions, deletions or inversions to search the space of DAG structures. Score-based methods have been shown to perform well for reconstructing Bayesian gene networks from genetics and genomics data using simulated and model organism data [83][84][85][86] . Constraint-based methods first learn the undirected skeleton of the network using repeated conditional independence tests, and then assign edge directions by resolving directional constraints (v-structures and acyclicity) on the skeleton. They have been used for instance in the joint genetic mapping of multiple complex traits 87 .
When individual-level genotype and gene or protein expression data are available from the same individuals, the results of causal inference tests are used to constrain the DAG search space [81][82][83][84][85][86][87] . This is done most elegantly by expressing the joint likelihood of observing both data types as a standard likelihood term for observing the expression data given the DAG and an additional likelihood term for observing the genotype data given the expression data and causal interaction implied by the DAG 81,82,85-87 ( Figure 5C). For large systems, the latter term can be expressed as an independent product over pairwise causal edge probabilities inferred from the model selection procedure ( Figure 5B) 86 , whereas for sufficiently small systems, genetic association mapping and Bayesian network reconstruction can be performed simultaneously 81,82,85,87 .
Using causal inference and Bayesian networks to identify causal genes, pathways and gene regulatory networks from multi-omics data has resulted in a system-level understanding of complex diseases and proposed novel target genes. Bayesian networks were used to map the genetic architecture of human liver gene expression and propose causal genes at loci associated with risk for type 1 diabetes, risk for coronary artery disease, and plasma low-density lipoprotein cholesterol levels 88 . Using genomic and transcriptomic data from post-mortem samples from four brain regions of late-onset Alzheimer disease cases and nondemented individuals, the gene TYROBP was identified as a key regulator of a causal immune and microglia gene network 89 . A role for TYROBP in the morphology of amyloid deposits and level of TAU phosphorylation was later confirmed experimentally. [90][91][92] Another analysis of genomic, transcriptomic, and proteomic data from four brain regions from 315 AD cases identified VGF as a key regulator in multiple AD causal networks, and overexpression of VGF in an Alzheimer mouse model was shown to reduce neuropathology. 93 Using genomic and transcriptomic data from seven vascular and metabolic tissues from coronary artery disease (CAD) cases undergoing surgical intervention, CAD-causal Bayesian gene networks were identified that replicated in data from matching tissues in the Hybrid Mouse Diversity Panel 94 . It was later found that one of these causal networks is susceptible to treatment by antiretroviral therapy drugs, and that targeting it may help reduce CAD side effects of ART drugs 95 . Further computational analysis of multi-omics causal networks in the context of protein interaction and pharmacological databases can identify established and novel druggable targets and target tissues 96 .
To bridge from the molecular scale to disease states, clinical phenotypes are not usually modelled as nodes in the causal network. Instead genome-wide omics data is first partitioned into co-expression modules associated with clinical traits 89,93,94,97 , causal networks are learned for each module separately, and connected in a higher-level Bayesian or coexpression network where each node is a module "eigengene", a representation of the module's expression profile by its first principal component 98 ( Figure 5E). This higher-level network can model tissue-specific and inter-tissue communication processes that are the result of the "collective" states of molecular networks and that are more proximal to the physiological disease processes 55,94,97,99 .
As in almost all other areas of genomics, single-cell technologies are creating new opportunities for learning causal disease mechanisms from population-based data 100 . Cost-effective strategies for generating single-cell RNA-seq data across individuals have been developed 101,102 and the first large-scale population genetics studies have been published recently [103][104][105][106][107] . A particularly attractive application will be to exploit the fact that allele-specific expression can be mapped in single cells 108 . This will allow us to quantify eQTL effect sizes 109 at the level of an individual instead of population, which will lead to a better understanding of the variation of causal effects across individuals. Moreover, since allele-specific expression can also be linked to transcriptional bursting 108 , an intrinsically random event, it can potentially be used as another type of randomised instrument or natural experiment to infer causal associations from observational single-cell RNA-seq data.

Learning causal associations from controlled perturbation experiments
Having considered the impact of causal inference on reverse translation, now we turn to its impact on forward translation by analysing controlled perturbation experiments ( Figure 5D).
For target-based discovery programs, canonical gene knockout or perturbation experiments in model systems establish causality in a manner similar to randomised controlled trials.
When these experiments are combined with genome-wide readouts, they provide insights into the direct and indirect causal targets of the perturbed gene. Systematic small molecule and perturbation screens using RNAi technology followed by reduced representation transcriptome sequencing have been performed in cancer cell lines 110 . CRISPR-based technology combined with single-cell RNA sequencing is now rapidly expanding the scope of conducting genome-wide perturbation screens in human cells [111][112][113][114][115] .
Causal inference from perturbation experiments allows us to disentangle direct from indirect effects and predict the outcome of a new intervention or perturbation. Nested effect models are probabilistic graphical models to infer a genetic hierarchy from the nested structure of observed perturbation effects 116,117 . A more formal approach uses causal DAGs and do-calculus to model causal effects from perturbation screens 118,119 .
Pioneers and experts in phenotypic drug discovery have long realised the importance of establishing causal relationships. Moffat et al. 120 proposed a model of chain of translatability, namely from the chemical matter to the assay phenotype, to the preclinical disease model, and finally to the human disease. We note the multiscale nature of this chain of translatability and the resemblance of challenges faced by target-based and those by phenotypic programs except for the target identification and assessment question. A key open question is whether causal inference, and in particular causal discovery techniques, can help phenotypic programs identify molecular targets and mode of action.
A fundamental issue with forward translation is that cellular or animal models differ from human physiology. Therefore a critical additional request is that the identified link must be persistent in humans, in the form of in vitro-in vivo translation, governed by physiological constraints, and between-species translation, governed by evolutionary forces.
In order to decide whether experimental findings can be extrapolated across domains that differ both in their distributions and in their inherent causal characteristics, we need prior scientific knowledge about the invariance of certain mechanisms is available and represented in "selection diagrams", DAGs in which the causal mechanisms are explicitly encoded and in which differences in populations are represented as local modifications of those mechanisms 121,122 . Switching between populations corresponds to conditioning on different values of the variables that locate the mechanisms where structural discrepancies between the two populations are suspected to take place. The transportability problem can then be formulated and decided using do-calculus and graphical criteria on the selection diagrams.
Translational study is undergoing a profound change. While animal models are traditionally used as a bridge between in vitro experiments and studies with human, the field is witnessing strong interest and investment in human model systems such as stem-cell-derived cell lines, primary cells, organoids, organ-on-a-chip, or other microphysiological systems. They are currently being tested for many purposes, especially efficacy and safety assessment, prediction and modelling of pharmacokinetic and pharmacodynamic parameters, biomarker identification, and disease modelling 123 .
From the point of view of causal inference, the key question is how much the causal model relevant for the disease and the drug candidate in human systems is conserved in model systems, let it be in vitro, in vivo, or human model systems. To answer this question, it is necessary to establish causal models with each model system, and to compare the models between the systems. So far, the translational value of most human model systems is judged by looking alike, i.e. to assess similarity in morphology and/or omics profiles with regard to primary human material. In order to mimic the causal relationship, more work is required to assess these systems by functioning alike, i.e. to assess their response upon perturbation and conditions that mimic disease aetiology and development 124,125 .

Discussions and conclusions
This review addresses three questions: when do we need a causal model, how to perform causal inference, and which activities in drug discovery have been or will be empowered by causal inference. We argue that causal models are essential for predictive modelling of outcome of intervention, for answering what-if questions, and for understanding disease biology and why drugs work (or not). We introduced causal inference as a six-step iterative process with hand-on examples in the complementary tutorial. And we performed literature review and discussed recent application of causal inference in translational research.
As discussed in the introduction, causality is multiscale. This renders causal inference a suited tool to investigate interactions between drugs and human biological systems, which are per se multiscale 126 . By leveraging advances in statistical modelling and mechanism-based multiscale modelling, it has the potential to transform heterogeneous data into unified knowledge, which helps us understand the impact of genetic, epigenetic, and environmental factors on drug action 127 .
An emerging consensus is that the quality of in silico model predictions outweighs the speed of prediction, and qualitative statements that enable scientists to focus on promising targets and regions of chemical space are often more impactful than quantitative predictions for decision making 128,129 . In line this, we foresee three key opportunities for causal inference: (1) high-quality prediction for predictive modelling of interventional outcomes, including out-of-distribution predictions 128,129 , (2) individualised causal inference and estimation of heterogeneous treatment effects 130 , and (3) causal-and counterfactual-based decision making 16,131 .
Besides the opportunities, we foresee three major challenges to be overcome by researchers and practitioners of causal inference in the coming decade: (1) the methodology of causal inference needs to be further developed to embrace the reality and complexity of biology, (2) the community needs to share and have open access to high-quality data and models, and (3) we need a change in both language and mindset.
The methodology of causal inference warrants further research. DAGs, though very powerful, have intrinsic limitations when being used to model biological systems. Edges in DAGs must be acyclic, i.e. we are never trapped in a closed loop by following the directions of edges. DAGs are therefore not suited for analysing reciprocal causal relationships, though they are prevalent in biology, across scales from molecular interactions 132 to consciousness 133 . To model systems with such relationships, one can either take advantage of longitudinal data to identify causal relationships 134 , or model the net output of nodes involved in reciprocal relationships as a variable instead of modelling individual nodes, or employ computational models other than DAGs, for instance graph neural networks and other representation learning techniques 135 . Both the theory and the software required for performing such analyses need further research and development. Despite recent progress discussed above 22 , it is still challenging to perform causal inference with high-dimensional data, for systems with multiple complex traits, i.e. many factors with small effect sizes [136][137][138] , and for complex adaptive systems where feedback loops are abound [139][140][141] .
Open contribution and access to improve biological models is essential for the community to refine and leverage causal models. Resources of causal models are emerging quickly 142 , and community-wide efforts like DREAM challenges have been hosted to identify causal relationships from omics datasets 143 . Nevertheless, the causal language remains foreign to many researchers, we do not always distinguish correlation from causation explicitly, and in many cases scientific findings are not represented as causal models. While it remains a complex and unsolved problem of how to populate the causal language and how to encourage researchers to share data and causal models, we are cautiously optimistic about further development because of the potential gain. The Linus's Law formulated by Eric S. Raymond 144 , Given enough eyeballs, all bugs are shallow, is a motivation for all of us to adopt the causal language and share the causal models: to share and to refine causal models in a community is to understand how biology and drugs work collectively.
Several reasons may make practitioners of drug discovery and development, including many colleagues we have interviewed, wary of causal inference. We need the front-up payment of bringing up knowledge and hypothesis, often exposing our naivety and lack of knowledge. In a real-world setting, setting up a model is particularly challenging because we often have incomplete information and knowledge about human biology, pharmacology and toxicology of drugs, not to mention the abundance of invisible or unquantifiable quantities and events that affect efficacy and safety of drugs. We have to give up the hope about 'learning from data alone' and 'hypothesis-free approaches'. We know that machine-learning models such as deep neural networks and graphical neural networks can learn patterns if they are trained with enough data. Isn't it better to accumulate more data and let the machine find out the causality? Why do we stick to and propose causal inference?
We welcome these questions and doubts. They lead us to believe that we need a change in both language and mindset to think and talk about causality. We argue that (1) causality exists beyond data and it is not possible to learn causality from data alone without prior knowledge and hypotheses, and (2) the central task of drug discovery and development is to discover and validate multiscale causal relationships. The true causal network, in contrast to causal DAG models, can contain feedback loops and therefore cannot be subject to the analysis introduced here as a whole. However, the network can be separated into smaller sub-networks, each verified by the six-step cycle of causal inference, and the causality propagates between the scales. The network connects the drug molecule with disease outcome via its interactions with biological molecules. The interactions emerge as pharmacology and toxicology on cellular, organ-and-system, and population levels. Furthermore, the network can be enriched with individual traits such as genetics, medical history, lifestyle, and environmental factors to allow population analysis.
In conclusion, causal inference offers a principled approach to model-and data-driven predictive modelling and decision making. It complements statistical and mechanistic models to empower an understandable synthesis and integration of knowledge and data across scales. We foresee that further development of its methodology, open-accessible causal models and high-quality data, and a broader adoption of the causal language and mindset will fuel both forward and reverse translation to seek and to prove causality with new drugs.
Supplementary           Mendelian Randomization (MR) estimates the causal effect of an exposure ("x", e.g. a gene or protein expression level) on an outcome ("y", e.g. disease status or clinical trait) using a genetic instrument for the exposure ("z", an eQTL or pQTL for a gene or protein exposure). MR is applicable if the instrument is independent of unobserved confounders ("u") and if the outcome is associated with the instrument only through the exposure (no alternative paths) (indicated by red crosses). (B) Simulated scatter plot of two coexpressed genes X and Y, with samples colored according to the genotype of a genetic marker Z for X (e.g. a cis-eQTL for X). Model selection determines the causal model underlying the data by testing the conditional independencies implied by each causal DAG. Representative causal models without (top) and with (bottom) unobserved confounders are shown. Model selection assumes that the edge Z→X must be included in the model and that Z is independent of any unobserved confounders. (C) When a larger set of coexpressed genes is considered (left), the data is modelled by a causal Bayesian network (right). Black nodes represent genes (x) and white nodes genetic instruments (z) that are used to orient the causal directions of gene-gene edges, and the Bayesian network represents the probability distribution of jointly observing x and z given a set of model parameters Θ learned from the training data. (D) Controlled perturbation experiments give direct information about the variables causally downstream of the perturbed node (yellow nodes) and can be used to refute or refine networks reconstructed from observational data. (E) Models inferred at one scale (e.g. cell-type or tissue-specific gene or protein networks) can be integrated into higher-level models. In the figure, nodes represent tissue-specific causal Bayesian networks (node colour, tissue; node size, network size) and edges represent "eigengene" similarities. Panel E obtained from Talukdar  • Directed acyclic graph (DAG): A directed graph without directed cycles. A graph consists of vertices and edges (also called arcs) connecting vertices. If the edges are directed from one vertex to another, the graph is called directed. If by following the edge directions, we never form a closed loop (also called a cycle), the directed graph is called acyclic. See https://en.wikipedia.org/wiki/Directed_acyclic_graph • Potential outcomes: The outcomes of an individual in both the factual and counterfactual worlds. For instance, for an individual participating in the treatment arm of a randomised clinical trials (RCTs), the potential outcomes are the outcomes to both treatment (factual) and control (counterfactual). See https://en.wikipedia.org/wiki/Causal_model#Potential_outcome • Counterfactual: A hypothetical scenario of how the world would have been under circumstances that are counter to the facts. For instance, for an individual participating in the treatment arm of a RCT, the counterfactual is the hypothetical world in which that individual would not have received treatment. See https://en.wikipedia.org/wiki/Causal_model#Counterfactuals • Backdoor criterion: A sufficient condition to find a set of variables Z to deconfound the analysis of the causal effect of a variable X on a variable Y using observational data. A set of confounder variables Z satisfies the backdoor criterion if (1) no confounder variable Z is a descendent of X and (2) all backdoor paths between X and Y are blocked by the set of confounders. A backdoor path from X to Y is any path from X to Y that starts with an arrow pointing to X. See https://en.wikipedia.org/wiki/Causal_model#Confounder/deconfounder • Frontdoor criterion: A sufficient condition to find a set of variables Z to measure the causal effect of a variable X on a variable Y using observational data. A set of variables Z satisfies the frontdoor criterion if (1) Z intercepts all directed paths from X to Y, (2) there are no unblocked backdoor paths from X to Z, and (3) all backdoor paths from Z to Y are closed by X. • Instrument variable: A variable Z that can be used to determine the causal effect of a variable X on a variable Y using observational data. A variable Z is a valid instrument variable if (1) it induces changes in X, (2) it has no effects on Y other than through the mediation by X, and (3) it is independent of any confounders that affect both X and Y. • Estimation trinity: estimator (the rule), estimate (the numeric value), and the estimand (the quantity to be estimated) • Sensitivity analysis (model refutation): The study of how the uncertainty in the output of a mathematical model can be divided and allocated to different sources of uncertainty in its inputs. • Mendelian randomization: An instrument variable method to determine the causal effect of an exposure X on an outcome Y where the instrument is a genetic variant. See Section 4.

Box 1: An anarchy of approaches to causality
Despite a long history of causality research in multiple disciplines, especially statistics, computer science, epidemiology, healthcare, and philosophy of science, there is no consensus way of defining and inferring causality 24 . Impactful approaches for causal inference include the counterfactual outcome framework, the Campbell's framework, Bayesian decision theory, and the Directed Acyclic Graph (DAG) approach 145 . We note that they are not mutually exclusive. In this review we particularly focused on the counterfactual framework and the DAG approach.
A variety of philosophical and scientific approaches have been developed to investigate causality. Below is a list of approaches that mostly impacted the authors: • Causal machine learning aiming at individualised causal inference using high-dimensional features to subdivide population, with emerging work in medicine and healthcare 43,151 . • Learning causal relationships represents the next-level challenge that poses a more substantial challenge than causal effect learning 20 . • Challenges of estimation from finite sample, see Shipley (2016) 152 We close this box with a quote from Richard McElreath: There is no method for making causal models other than science. There is no method to science other than honest anarchy.


XDGAN: Multi-Modal 3D Shape Generation in 2D Space

Generative models for 2D images has recently seen tremendous progress in quality, resolution and speed as a result of the efficiency of 2D convolutional architectures. However it is difficult to extend this progress into the 3D domain since most current 3D representations rely on custom network components. This paper addresses a central question: Is it possible to directly leverage 2D image generative models to generate 3D shapes instead? To answer this, we propose XDGAN, an effective and fast method for applying 2D image GAN architectures to the generation of 3D object geometry combined with additional surface attributes, like color textures and normals. Specifically, we propose a novel method to convert 3D shapes into compact 1-channel geometry images and leverage StyleGAN3 and image-to-image translation networks to generate 3D objects in 2D space. The generated geometry images are quick to convert to 3D meshes, enabling real-time 3D object synthesis, visualization and interactive editing. Moreover, the use of standard 2D architectures can help bring more 2D advances into the 3D realm. We show both quantitatively and qualitatively that our method is highly effective at various tasks such as 3D shape generation, single view reconstruction and shape manipulation, while being significantly faster and more flexible compared to recent 3D generative models.

Introduction
Generative Adversarial Networks [15] have achieved remarkable progress in generating high-resolution realistic images, typically using convolutional architectures such as [5,23]. Extending these advances to the 3D domain remains a challenge and an active area of research, with newest methods introducing implicit or explicit 3D awareness into the GAN generative process [7,8,17,34,55]. In this work, we show that it is possible to directly leverage a 2D GAN architecture designed for images to generate high-quality 3D shapes. The key to our approach is parameterization of 3D shapes as 2D planar geometry images [18], which we use as the training dataset for an image-based generator. Perhaps one of the greatest challenges to the development of generative models for 3D content is converging on the right representation. Direct extension of 2D pixel grids to 3D voxel grids [50] suffers from high memory demands of 3D convolutions, limiting resolution. Point cloud samples of surface geometry, while popular for generative tasks [2,6,28], are limited in their ability to model sharp features or high-resolution textures. Recently implicit 3D representations, such as NeRF [33] and DeepSDF [35], as well as hybrids, have shown great promise for generative tasks [7,11,52]. However, these approaches typically are too slow for interactive applications and their output does not interface with existing 3D tools, calling for costly or lossy conversion of models before use. To this date, 3D meshes, augmented with normals and textures, remain the most widely adopted 3D representation in 3D software, movies and games. Our method uses 2D generative architectures to produce fixed-topology textured meshes with a single forward pass, and is thus immediately practical.
We present XDGAN (X-Dimensional GAN), a method for using standard fast 2D GAN architectures for generating high-resolution 3D meshes with additional surface properties like textures and normals. XDGAN is trained on a collection of 3D shapes, which are first unwrapped into planar geometry images. Each pixel of a geometry image represents a vertex position of a fixed topology mesh, and is thus a direct representation of surface geometry. To our knowledge, ours is the first work to show that 2D convolutional generators can produce high-fidelity 3D shapes by operating on planar parameterization of 3D geometry, an idea that can help bring more advances in 2D architectures to the 3D realm. Specifically, we experiment with StyleGAN [23,24] modified to train on higher-precision geometry images that represent 3D shapes. In order to augment generated geometry with additional per-vertex properties, we show that it is possible to directly apply an image-to-image translation framework [36] to predict these properties for each geometry image. We demonstrate that our approach can generate diverse, high-quality textured 3D meshes, beating existing generators in quality of surface detail, presence of color or practical usefulness of output representation. Further, we demonstrate that powerful properties of the StyleGAN latent space can also be exploited for 3D generation, reconstruction, as well as supervised and unsupervised manipulation, just as it has been shown for 2D images.

Related Work
Generative Adversarial Networks (GANs) have become a popular technique for image generation since their introduction by Goodfellow et al. [15]. Striking progress in the quality and resolution of GAN-generated images has been achieved in recent years [5,22,23,24], including in conditional settings [36]. While some degree of view control of such models has been exploited for downstream 3D tasks [56], these architectures have remained primarily confined to generation in the 2D domain. Most recently, approaches that leverage an intermediate 3D representation to improve 3D consistency and view control of GANs have also been proposed [4,7,8,17,34]. Our method is orthogonal to this line of work as we show that it is possible to leverage unmodified 2D GAN architectures, to learn 3D geometry directly. Beyond unconditional generation, latent spaces of large-scale GANs have been successfully used for both coarse and fine-grained manipulations of images. While some techniques focus on editing real images [1,3,26,30,40], others devise meaningful exploration of the GAN latent space. Supervised approaches typically use pre-trained attribute classifiers to optimize edit directions [14,43,44]. Other works show that it is possible to find meaningful directions in latent space in an unsupervised way [20,21,49]. In our work, we use a modified version of InterFaceGAN [44] to demonstrate how latent space manipulation techniques, originally proposed for 2D image manipulation, can be leveraged for 3D generation as well.
While there have been remarkable advances in 3D shape generation in the recent years, deployment of 3D generative models poses challenges due to generation quality and speed, as well as compatibility of the output format with downstream tasks (see Tb.1 for summery).
The most direct 3D analog of the 2D pixel array is the 3D voxel grid, and early generative approaches adapted convolutional architectures to generate objects by operating on this 3D grid [50,51]. Due to fixed resolution of the voxel grid and high memory demands of 3D convolutions these approaches are limited to low quality outputs. Later methods mitigate high memory requirements, for example, by using octrees to represent the 3D space more efficiently [47] or through local generation [54], but are still constrained in their ability to represent high-resolution smoothly varying geometry and texture (Tb.1, A).
Due to their simplicity, point clouds, or unordered (x, y, z) samples of surface geometry, are a popular representation for 3D generation [2, 6, 28], reconstruction [29] and segmentation [13,38,39]. Despite continued progress in generative modeling  Table 1: Comparison of 3D generative methods by the representation used. We include quantitative and qualitative comparisons with the starred methods in our experiments.
of point clouds [53,57], they cannot overcome the inherent limitations of the representation, which cannot model sharp surface details, or be rendered as high-fidelity textured meshes. We quantitatively compare our method against the recently published Diffusion Point Cloud (DPC) [31] (See also Tb.1, B). DPC proposes a probabilistic diffusion model and uses point cloud representation of 3D shapes for shape generation, auto-encoding and shape completion.
Recently there has also been a surge in using implicit 3D representations, such as learned signed distance functions (SDFs) [19,35] or occupancy [11,32]. Like point clouds and voxel grids, these representations allow modeling varying object topology, but can also represent high-resolution surface detail. Most related to ours among implicit methods is IM-GAN [11], which proposes to use an implicit decoder IM-NET in conjunction with features learned by a latent-GAN model [2] to yield a general 3D generative model. Because the outputs of this and related methods are implicit in the network weights, high-resolution results typically cannot be visualized or exported in real-time, requiring several seconds on state-of-the-art GPU, and precluding interactive applications (See Tb.1, C). In addition, a number of hybrid approaches are emerging. For example, ShapeFormer [52] employs transformers in the space of discrete implicit shape elements to support shape completion, and DMTet [42] produces detailed tetrahedral meshes using sparse SDF samples. While DMTet generates a mesh-based representation quickly, like our method, it is only designed to work when conditioned on a rough input shape, not as a general generator.

Methodology
Our approach leverages the power of 2D convolutional generator networks in order to generate 3D geometry. Given a training set of (optionally textured) 3D objects, we first convert them into geometry images and corresponding normal maps, textures or other attribute maps ( §3.1). The one-channel geometry images represent vertex positions of a fixed topology mesh, and are used to train an image-based GAN model to generate such plausible geometry images ( §3.2). In order to augment output with per-vertex attributes like textures and normals, we similarly train an image-to-image translation network on these aligned attribute maps ( §3.2). After training (Fig.2a), these two image-based generators produce detailed textured 3D meshes in real-time ( §2b). Fast generation makes interactive exploration of the GAN latent space espe-  , we first convert a training dataset of textured 3D meshes into 2D geometry images with corresponding textures, normals or any other surface attributes. Next, we train a GAN model on geometry images to generate geometry, and an image-to-image translation network to generate Xchannel attribute images for an input geometry image. At test-time (b), feed-forward evaluation of these networks followed by a trivial meshing step produces textured 3D output meshes in real-time.
cially attractive. Although our method generates 3D output, we reap the benefits of latent space exploration techniques developed for the 2D domain to explore and edit 3D shapes ( §3.3).

Conversion to Geometry Images
Geometry images [18] are a representation that captures 3D surface as a 2D array of (x, y, z) values via an implicit surface parameterization, which is a bi-directional mapping from an object's surface to a 2D plane. Such parameterizations are often used in computer graphics to map 2D textures and other attributes onto a 3D surface mesh. Once the parameterization is determined, the plane is sampled to create a 3-channel n × n image where each pixel represents an (x, y, z) vertex location of a new geometry image mesh, which is independent from the original object's mesh topology or representation. The order and connectivity of each vertex within this geometry image mesh is encoded by its pixel index, where faces connect neighboring pixel-vertices using a regular structure. Geometry images can contain additional channels encoding surface properties like color texture, normals, object segmentation and similar. Because the same surface parameterization is used when sampling these additional properties, all channels of the geometry image are aligned, with each pixel representing location and other properties of the same 3D vertex. In our work, we generate normals and texture geometry images and propose applying 2D convolutional generators to these aligned X-channel images.
In order to map a closed 3D surface onto a 2D plane, we need to determine seams, make cuts across the surface to conform to a chosen topology (e.g. disk, sphere, cylinder), and unwrap it onto a flat square surface. Creating appropriate parameterizations that are well-suited to individual 3D object topology and shape is an active research area [18,27,37,41,46]. Since our goal is to train a generative model, we want to reduce the representation variability as much as possible to allow the model to learn the true data distribution instead of the variability in parameterizations. For this reason, we choose to work with a specific family of spherical parameterizations, and a fixed seam structure to convert 3D shapes to geometry images and create consistent input for training the 2D generator. We start with two realistic assumptions about the geometry of our shapes. First, we assume that most of our 3D shapes are symmetric, which means we only need to represent half of the shape in the geometry image. Cutting the closed surface of the shape in half also defines an edge of the surface that is needed for the parameterization. Second, we assume our shapes are mostly genus-zero or close to it. This means that the surface can be mapped onto a surface of a sphere. While the spherical projection method imposes limitations on output geometry, our aim is to explore generation in the most consistent setting first. Given above assumptions, we define a three step process to obtain a geometry image from a collection of 3D shapes (Refer to Fig.3). We first scale and align all input shapes to ensure consistent orientation and scale and place them within the projection sphere. We then project the surface of the shape onto a spherical domain using the normal direction of each point on the sphere. This creates a unique mapping between each point on the sphere's surface and the shape's surface. Finally, we cut the sphere in half along the symmetry axis of the shape and map the surface of the hemisphere onto a 2D plane using a fixed, predefined areapreserving parameterization. Because projection from the sphere onto a 3D surface has only one degree of freedom (distance along the normal direction of the sphere, see Fig.3), we represent the 3D location of a surface point as a single value measuring the distance (which we call "Rad") from the origin along the projection ray, effectively reducing the degrees of freedom of the representation from 3 channels per pixel to only 1. To compute the original (x, y, z) coordinates of all the vertices, one only needs to multiply the Rad image by the image representing the normals of the projection sphere, fixed for the whole dataset. In addition to positions, we produce aligned images of normals and color textures.

Generating Geometry Images and Other Attributes
Once our training dataset is converted to Rad geometry images ( §3.1), we train standard 2D GAN architectures to produce "realistic" Rad images. The only modification required is to adapt them to high-precision geometry images instead of the traditional 8-bit RGB images. Specifically, we experiment using StyleGAN3 [25] for geometry generation, but nothing suggests that our approach will not generalize to other GAN architectures (see Supplemental for experiments with StyleGAN2 [24]). We found training over single-channel Rad images to differ little in its stability from standard image-based GAN training.
During the generation of Rad geometry images ( §3.1), we also produce aligned normal and texture maps using the same surface parameterization. Normal maps describe the local curvature of a 3D surface and are often used in computer graphics to add fine details to a 3D shape without adding actual geometry. To generate smooth normal maps even for shapes with low resolution, we use the smooth shading technique which interpolates the normals along a flat mesh triangle to give the illusion of smooth shading. The computed smooth normals are then mapped onto a 2D image using the same parameterization. The texture maps are generated by using an environment map around the object which can generate both realistic lighting and reflection for the object. The result of the rendering is then baked as a texture and mapped into a 2D image using the same procedure as with normal maps.
We use these aligned image channels to train an image-to-image translation network to produce X attribute channel images given an input Rad image of the same resolution. Specifically, we experiment with SPADE image-to-image translation network [36], but nothing in our setup is specific to this particular network choice. We note that SPADE is designed to be able to produce diverse outputs given a single input.

Exploiting the Latent Space
A major advantage of our 3D generative model is that it is based on standard GAN architectures, which allows us to directly apply various techniques developed for image-based GANs. To demonstrate this, we integrate the existing technique of In-terFaceGAN [44] in order to find editing directions within our trained latent space. To this end, we first project our training shapes into the latent space W to create a set of of projected latent vectors w with lables (e.g. sports-car, SUV, etc.). We then train one linear SVMs for each label using these labeled examples, and use the unit normal vector to the separation hyperplane of the SVM n T as an editing direction in latent space. The target attribute of a real or generated 3D shape can then be enhanced or negated by simply steering its latent vector p along this editing direction n T by t steps with a step size of α to produce a new latent vector p = p + (α × t × n T )

Experiments
We evaluate our model's performance on shape projection, editing and single-view reconstruction tasks. We compare our method to three state-of-the-art 3D shape generation methods that use triangular meshs (AtlasNet [16]), point-clouds (DPC [31]) and implicit representation (IM-Net [11]). The experiments show that our method achieves significantly better results on some object classes and competitive results on others, all while performing inference several orders of magnitude faster than the closest competitor IM-NET [11].

Datasets and Experimental Setup
For the quantitative comparisons, we use the ShapeNetV2 [9] car and airplane categories consisting of 3509 and 4045 shapes respectively and use 10% of each set for testing. We choose these two categories as they are sufficiently distinct structurally and demonstrate the range of our method. Additionally, we use a proprietary new dataset consisting of 3050 high-quality car models which we call HQCars. In addition to the higher quality of meshes, this dataset provides high quality texture maps and semantic labels, like model, maker, year and type, which are not available in ShapeNet.
To create our training set, we first converted the set of 3D meshes to 1-channel Rad geometry images of size 512 x 512 and the corresponding normals and texture maps. (See Supplementary Material for comparisons with geometry images using 3 (x, y, z) channels) We use the Rad geometry images to train a modified version of the official StyleGAN3 implementation [25] to process high-precision values instead of the 8bit RGB images. We train this StyleGAN3-r variant without any data augmentation with batch size 8 and gamma value of 1 for 500 epochs. To generate the additional texture and normal maps, we train an image-to-image translation model separately for each using the official implementation of SPADE [36], similarly modified to deal with the range and precision of geometry images. Each model is trained on aligned pairs of ground-truth geometry images and texture or normals maps, for 300 epochs with batch size of 32.
We train our model and all baselines from scratch on the car and airplane training data subsets separately and evaluate on the respective test subsets. We use the official Pytorch implementations of AtlasNet, DPC and IM-Net with the default parameters.
For all experiments, we use reconstruction-based tasks and compare the generated shapes to ground truth shapes using Chamfer distance (CD) and Earth Mover's Distance (EMD) metrics. CD and EMD metrics are computed by sampling 5000 points from the generated and ground truth shapes, matching the nearest points across the sets of points and computing their sum of Euclidean and Manhattan distances respectively. In addition to CD and EMD, we use the Light Field Distance (LFD) [10] as a visual similarity metric. To compute LFD, the generated and ground truth shapes are rendered from various camera angles. The rendered images are then encoded using Zernike moments and Fourier descriptors to compute similarity. Finally, we report the average inference time of ours and competing methods as real-time generation is critical for interactive applications.

Quantitative Evaluation
For evaluating on the task of 3D shape reconstruction, we use an iterative latent projection approach to obtain a reconstruction of ground-truth test model. We start from a randomly sampled latent vector and iteratively compute the loss between the generated geometry image from this vector and the ground-truth geometry image, then back-propagate the loss through the generator to update the latent vector. We repeat this procedure for 500 iterations for each test model. For the single-view reconstruction task, we train a ResNet-based encoder on the rendered RGB images [12] of a ShapeNet model from a random viewpoint, and predicts a latent vector which is then used to generate a geometry image. For both tasks, the final evaluation metrics are computed between the triangle mesh created from the generated geometry image and the original ShapeNet test mesh.  Results in Table 2 show that our method achieves by far the best CD and EMD on the car category and ranks second on the airplane category after IM-Net, while being faster by factor of 500. This finding is not surprising, as we have made some simplifying assumptions during conversion to geometry images ( §3.1), and it is harder to map airplanes with many sharp and spiky features onto the spherical domain than it is for relatively compact cars. These results also demonstrate that when the 2D parameterization is appropriate (as it is for cars), 2D generative models using our technique outperform even implicit 3D representation models, which is an unexpected outcome. We further note that our method is the fastest by at least a factor of 3 over any baseline. For Single-View reconstruction experiment (Table 3), the results are similar where our methods achieves very good results on the car category while falling short from IM-Net on the airplane category.   Table 4 show 3D meshes projected with XDGAN using the method in §4.2, visually compared to other competitors. Our methods generates water-tight meshes with high-fidelity that preserve even small details like car mirrors and wheel patterns. The inclusion of generated surface normals also add another level of detail and smoothness where needed, making the results from XDGAN visually outstanding compared to other methods.

IM-Net
Ours (no normals) Ours  Furthermore, Fig.4 shows the the powerful properties of XDGAN's latent space. Editing directions optimized using InterFaceGAN [44] ( §3.3) allow semantic editing of the 3D object in real time, for example by making a car appear sportier (Fig.4, top). The latent space additionally allows for generating intermediates between two input models by linearly interpolating their projected latent space vectors.

Conclusion
We presented XDGAN, the first method to show that 2D convolutional architectures can be directly used for generative modeling of 3D objects with high fidelity. Our approach is based on a novel combination of planar image parameterization with an image-based GAN. We show that our method generates high-quality textured 3D meshes, allows unconditional sampling, projection, and can be used in several conditional settings. In addition, XDGAN's speed and convenient mesh output allow for a number of applications that can be directly employed in practice. The shortcoming of our approach is its reliance on a fixed topology mesh, associated with the geometry image. We believe that there are many ways future work could mitigate this limitation, for example through part-based generation. Despite this limitation, we have shown impressive generation results from our method, and hope that this work may help close the gap between advances in 2D generative architectures and comparable approaches in 3D.

acknowledgments
We thank Stefan Liske and Timotheus Gmeiner from PCH Innovations GmbH for the insightful discussions and making this collaboration possible. Part of this research work has been done while Hassan Abu Alhaija was at PCH Innovations GmbH.