text:
\section{Preliminary}
\label{sec:background}

We introduce performance debugging in \S~\ref{subsec:perf-debug} and formulate
causality analysis in \S~\ref{subsec:causality}. We introduce CE, an emerging
success paradigm in testing complex (distributed) software in
\S~\ref{subsec:chaos}.

\subsection{Database Performance Diagnosis}
\label{subsec:perf-debug}

Database developers and users, when observing unexpected performance behaviors,
may often wish to identify relevant information to debug the anomalies.
Depending on the accessible information, we classify performance debugging into
two categories:

\parh{Blackbox Debugging: Localizing KPIs.}~From a holistic view, today's
database performance downgrades are primarily due to abnormal system components,
such as kernel, network, or pod failure in container clusters~\cite{pod}.
Blackbox debugging is therefore becoming the mainstream in performance
debugging, where users do not need to access database internals. For instance, a
developer may wonder what causes an intermittently slow SQL
query~\cite{ma2020diagnosing}, and the end goal is to find KPIs like high CPU
usage or low disk throughput, that are the root causes of the slow query.

Blackbox debugging of database performance defects is a demanding, yet unsolved
problem. Runtime environments often comprise hundreds of KPIs (e.g., our
evaluated container clusters have at most 254 KPIs, see
\S~\ref{sec:implementation}), making it challenging to flag root cause KPIs.
Statistical debugging
(SD)~\cite{jones2005empirical,liblit2005scalable,liu2006statistical} can
identify KPIs correlated to anomalies. Nevertheless, as frequently quoted,
correlation does \textit{not} imply causation and SD is fundamentally limited in
its applicability and accuracy. A promising trend is to launch \textit{causality
analysis}~\cite{gan2021sage,chen2014causeinfer} to construct causal relations
between KPIs. Identifying root cause KPIs is thus recast to predict KPIs that
manifest a major causation with anomalies based on the causality graph.

\parh{Whitebox Debugging: Localizing Program Components.}~In general, software
bugs can also result in performance anomalies. This ``whitebox'' scenario
assumes that programmers can monitor software internals, which often need to
\textit{instrument} the software to monitor execution behaviors. Typical debug
approaches include software profiling~\cite{attariyan2012x,zhao2016non},
visualization~\cite{bezemer2015understanding}, and program analysis techniques
like program slicing~\cite{soremekun2021locating}, delta debugging, and
statistical
debugging~\cite{jones2005empirical,liblit2005scalable,liu2006statistical,jin2010instrumentation,zuo2016low}.
Often, the end goal is to isolate buggy program components that represent
performance bottlenecks.

\parh{Whitebox vs. Blackbox.}~We clarify that whereas whitebox performance
debugging is widespread in conventional software debugging, its use in database
contexts is \textit{uncommon}, if even possible. This may be primarily due to
the high complexity of database systems, and the lack of access to the
underlying software internals for commercial products. Accordingly, this paper
presents \textsc{PerfCE}\xspace, a blackbox performance debugging tool toward real-world
(distributed) databases where traditional program analysis or tracing techniques
are hardly possible. Understanding performance anomalies is important for
production
databases~\cite{aguilera2003performance,ma2020diagnosing,gan2021sage}. \textsc{PerfCE}\xspace\ is
designed based on the observation that existing causality-based approaches are
of limited accuracy and comprehensiveness in real-world usage (see
\S~\ref{sec:motivation}). The next section formulates causality analysis in
performance debugging.

\subsection{Causality Analysis}
\label{subsec:causality}

\parh{Qualitative Csaulity Analysis.}~A desiderata in performance debugging is
to \textit{automatically} identify the root cause and also provide actionable
suggestions for repair. In Fig.~\ref{fig:motivating-example}, developers may
anticipate to know that abnormal CPU usage causes the processing time spike.
This denotes a qualitative view of performance debugging enabled by causality
analysis, which flags one or several KPIs $X_i$ deemed as the root causes of the
performance defects. With causality graph defined below, qualitative analysis is
straightforward.

\vspace{-2pt}
\begin{definition}[Causal Graph]
A causal graph (a.k.a., Bayesian network) is a directed acyclic graph (DAG)
$G=(V,E)$. Each node $X$ represents a random variable and each edge $X\to Y$
encodes their cause-effect relationships, which denotes that $X$ is a direct
cause of $Y$. We use $Pa_G(X)$ to denote the parent nodes of $X$ in $G$.
\end{definition}
\vspace{-2pt}

With causal graph, identifying root causes of anomalies is recast to identify
the cause-effect relationships from graph ancestors and descendants.
Particularly, given an abnormal KPI $Y$, a common approach is to backtrack the
ancestors of $Y$ and identify the most ancestral abnormal KPI $X_i$ of $Y$ as
the root cause~\cite{chen2014causeinfer}.

\parh{Quantitative Causality Analysis.}~Developers may also demand quantitative
performance debugging, which is often referred to as \textit{counterfactual
analysis}. Let $X_1,X_2,Y$ denote filesystem I/O, CPU usage and processing time.
When observing $X_1=x_1, X_2=x_2$ in a particular time point, developers would
wish to know $\mathbb{E}(Y\mid X_1=x_1, do(X_2=x_2^*))$, where the $do(\cdot)$
operator, a counterfactual query, represents an intervention over value
of a variable like $X_2$. The formulation estimates the processing time in a
counterfactual world which can be interpreted as ``\textit{we have observed
$X_1=x_1, X_2=x_2$ in the real world, if $X_2$ were $x_2^*$, then what would $Y$
be?}''

Counterfactual analysis in causality
analysis~\cite{pearl1991theory,balke2022probabilistic} can address how a
quantitative counterfactual change of a KPI impacts outcomes of interest, such
as processing time. In addition to causal graphs, counterfactual analysis
entails building the structural equation model (SEM) of the underlying
distribution~\cite{pearl2012calculus}.

\vspace{-2pt}
\begin{definition}[SEM]
    A SEM $M$ is defined as follows:
    \begin{enumerate}
        \item A set of exogenous variables $U$, representing the factors outside
        the model;
        \item A set of observed endogenous variables $V$; each variable $X$ is
        functionally dependent on $U_X\cup Pa_G(X)$ where $U_X\subseteq U$.
        \item A set of deterministic functions $f_X \in F$ such that each
        $f_X:Pa_G(X)\times U_X\to X$ computes the value of $X$.
    \end{enumerate}
\end{definition}
\vspace{-2pt}

A counterfactual query $do(X=x)$ simulates interventions, by deleting the edges
between $X$ and its parent nodes from the causal graph in SEM. The target
variable $X$ is replaced with a constant such that $X=x$, while the rest of the
model are unchanged~\cite{pearl1991theory,balke2022probabilistic}.

To obtain SEM, we need to: 1) obtain the causal graph and then 2) estimate each
function $f_X \in F$ from data. Overall, these two steps are often referred to
as structure learning and parameter learning in causality
analysis~\cite{meek1997structure}. Given observational data, structure learning
recovers causal graphs, where underlying cause-effect relations are inferred
from graph ancestors/descendants. Then, $f_X$ of every non-root variable $X$ can
be learned by training a regression model from historical data. For instance,
$f_Y$ in Fig.~\ref{fig:motivating-example} can be estimated by regressing $Y$ on
$X_1,X_2$ using collected historical data.

\begin{figure}[!ht]
    \centering
    \vspace*{-10pt}
    \includegraphics[width=0.90\linewidth]{fig/chaos.pdf}
    \vspace*{-10pt}
    \caption{Using CE to aid software testing.}
    \label{fig:chaos}
  \end{figure}

\subsection{Chaos Engineering (CE)}
\label{subsec:chaos}

As large-scale, distributed systems evolve, traditional software testing
methods, which primarily mutate program inputs, become less powerful. CE is the
discipline of testing a system to ensure it can sustain turbulent conditions in
production~\cite{chaos}. In a cloud/container scenario, typical turbulent
conditions may include infrastructure, pod, network, and application failures.
CE typically takes three steps to uncover software bugs:

\parh{\ding{172}~Defining Steady State.}~CE often begins by defining testing
oracles, often known as ``steady states,'' as easily measurable outputs (rather
than internal states) of a system that indicate normal behaviors. Accordingly,
CE would make the important hypothesis such that \textit{a steady state will
persist in both a control group and the experimental group subjected to CE
stress.}

Steady states may subsume throughput and query outputs, by considering domain
specific demands. For instance, SQLSmith~\cite{sqlsmith} compares the outputs of
one database (when it is being stressed by CE) and MySQL. The steady state in
this scenario is defined such that the SQL execution outputs are
\textit{consistent} between two databases. Knowledgeable audiences may notice
that this setup expands the standard ``differential testing''
procedure~\cite{rigger2020testing,kallas2020diffstream,sotiropoulos2021data}, in
which the experimental group must act consistently with the reference
even while the former group is under the stress of CE.

\parh{\ding{173}~Picking Chaos Variables.}~Typically, CE comprises a collection
of chaos variables, each of which denotes a critical, low-level factor that may
induce various failures in infrastructure, network, and systems. For instance,
modern CE frameworks like Chaos Mesh~\cite{cm} are deeply coupled with containerization
environments like Kubernetes~\cite{k8s}, and its offered chaos variables subsume
a comprehensive set of failures that may occur in Kubernetes clusters (e.g.,
container-kill, pod-kill).
To clarify a potential confusion, chaos variables are \textit{not} the same as
KPIs; usually there are considerably more KPIs than chaos variables. Mutating
each chaos variable (e.g., an IO-concerned chaos variable) may influence many
KPIs (e.g., average I/O time), and several chaos variables may simultaneously
influence the same KPI as well. See \S~\ref{sec:implementation} about our
mutated chaos variables.

\parh{\ding{174}~Launching CE and Testing.}~Once properly defining \ding{172}
and \ding{173}, the next step is to launch CE and test the target system,
looking for the steady state's inconsistency between the reference group and the
target system (experimental group) under CE. Findings at this step can be
wrapped up for debugging and error fix.
\section{Conclusion}

We have presented \textsc{PerfCE}\xspace, a database performance anomaly diagnosis framework.
\textsc{PerfCE}\xspace\ adopts chaos engineering, an emerging engineering practice for stressing
complex software systems. \textsc{PerfCE}\xspace\ novelly employs Chaos Mesh\ for augmenting
causality-based performance debugging. \textsc{PerfCE}\xspace\ features high-quality qualitative
root cause identification and quantitative counterfactual analysis. \textsc{PerfCE}\xspace\
addresses observable and latent confounders in causality analysis. Our
evaluation over MySQL, TiDB, and synthetic datasets shows that \textsc{PerfCE}\xspace\ offers
accurate performance diagnosis, and it outperforms existing works across nearly
all settings.
\subsection{Offline CE-Enhanced SEM Learning Phase}
\label{subsec:offline}

\parh{CE Usage.}~It is a long-standing problem in causal inference to learn
high-quality SEM from observational data. The validity of output SEMs is
dependent on a set of assumptions. In performance diagnosis, these assumptions
are frequently violated, posing considerable hurdles to subsequent causality
analysis. This work employs CE in SEM learning. In \ding{172} and \ding{173}, we
explain how CE enables accurate and comprehensive SEM learning. In \ding{174},
we detail the operations for generating training data using CE.

\sparh{\ding{172} Improving Observability on Normal Data.}~One famous assumption
is \textit{faithfulness}, which guarantees d-separation on causal graphs when
conditional independence on the joint probability distribution (detected in
observational data) exists~\cite{spirtes2000causation}. This assumption helps us
refute causal relations in the absence of \textit{correlations}. However, the
lack of correlations may be simply due to insufficient data, which is frequent
in this research. Since many database KPIs remain unchanged or vary only little
in daily usage, we can hardly establish meaningful causal relations on them.
Existing research has to either ignore these KPIs or collect abnormal behaviors
manually from logs to explore causal relations at the best effort. Manual data
collection is costly, domain-specific, and less scalable. \textsc{PerfCE}\xspace\ uses Chaos Mesh\ to
form an active intervention during causal learning, as it systematically mutates
chaos variables to achieve extensive observability on possible value ranges and
causality among KPIs. For instance, mutating a chaos variable, network loss
probability (which ranges from 0\% to 100\%), can affect many network-related
KPIs, such as ``network duration of request.'' Mutating chaos variables in de
facto CE frameworks can influence hundreds of KPIs. Thus, CE improves
observability on the normal states of KPIs, leading to more comprehensive and
accurate causal learning.

\sparh{\ding{173} Improving Observability on Anomalies.}~Furthermore, CE allows
for triggering anomalies offline by heavily mutating certain chaos variables. By
inspecting real-world performance anomalies, we find that many basic performance
anomalies (e.g., ``I/O Saturation'') are outcomes of the I/O-concerned chaos
variable. Moreover, some complex performance anomalies (e.g., ``Workload
Spike'') can be simulated by combining chaos variables. This way, we not only
augment the comprehensiveness of observed normal data (as in \ding{172}), but
also largely enrich the knowledge of anomaly data. This would mitigate OOD
(out-of-distribution) faced by previous works, since online anomaly data was
rarely seen in offline training data collected by previous research.

\sparh{\ding{174} Collecting Training Data.}~To collect offline training data,
in addition to constant queries emitted by workload simulators (such as the
TPC-C benchmark~\cite{tpcc}), we also use a set of chaos variables to stress the
target databases. Among the chaos variables included in our experiments (see
details in \S~\ref{sec:implementation}), some, when enabled, directly inject
faults to the target databases. When generating training data, we enable each of
them and record how KPIs change. Other chaos variables are
\textit{configurable}, allowing us to mutate their values to determine the
extent of their impact on databases. When generating training data, for each of
them, we divide its valid input range equally into three to five thresholds
(depending on the sensitivity of this chaos variable; less sensitive variables
are mutated for five times and vice versa). We mutate each chaos variable using
all of its thresholds, and we collect the KPI changes for each iteration. KPIs are
initially used to learn the structure of the causal graph, and then chaos
variables and KPIs are used together to learn the SEM's parameters (see details
in the following paragraphs).

\parh{Causal Graph Structure Learning.}~Once gathered offline training data, we
first learn causal graphs. Causal graph structure learning generates a DAG,
where each node denotes one KPI, and each edge represents causality relations of
a KPI pair. This enables qualitative causal analysis, such that given a
performance anomaly monitored during the online phase (denoting a node on the
DAG), we collect its ancestors to form the potential root causes (see details in
\S~\ref{subsec:online}). Previous work like CauseInfer~\cite{chen2014causeinfer}
has relied heavily on constraint-based algorithms (e.g., the PC
algorithm~\cite{spirtes2000causation}), which returns an incomplete causal graph
with undirected edges (CPDAG) and arbitrarily assign directions to undirected
edges. In \textsc{PerfCE}\xspace, we aim to harness the power of score-based algorithms to learn
a causal graph that maximizes a predefined score over observational data, with
all edges in the resulting graph being directed. In particular, we employ a
two-stage technique based on BLIP~\cite{scanagatta2015learning} to learn causal
graphs. To begin, for each variable, we identify its possible parent sets with
local scores; then, we use a global structure optimization algorithm to identify
the causal graph that maximizes the global score, which is computed using cached
local scores in the first stage. According to the
literatures~\cite{ding2020reliable}, BLIP produces empirically much better
results than the PC algorithm~\cite{spirtes2000causation} employed in previous
works. Nevertheless, \textsc{PerfCE}\xspace\ is not bounded with BLIP; we use it as it is one
recent work that offers off-the-shelf implementation with good engineering
quality. Users can replace BLIP with other algorithms whenever
needed~\cite{spirtes2000causation,ding2020reliable}.

\parh{Causal Graph Parameter Learning.}~Given the DAG representing causal graph
structures, we further conduct parameter learning to assign relationships toward
each edge of the DAG. This step enables quantitative counterfactual analysis.
Considering the sample case in Fig.~\ref{fig:motivating-example}, given filesystem
I/O $X_1=x_1$ and CPU usage $X_2=x_2$, quantitative causality analysis infers
the consequent processing time $Y$. For this simple case, we can directly train
a predictive model that predicts $Y$ given $X_1$ and $X_2$. However, such
methods are \textit{not} suitable when $X_1$ and $X_2$ has a causal relationship
($X_1$ forms a confounder). For example, heavy filesystem I/O itself may result
in poor CPU usage, as processes may be blocked while waiting for I/O. In this
case, $X_1$ is an observable confounder of $X_2$. We start to analyze a
potential solution by assuming linear causal relations and will extend
to general cases later. Consider the following relations:
\begin{align}
    X_2 &= \theta_1 X_1 + b_1 + \epsilon_1 \\
    Y   &= \theta_2 X_1 + \theta_3 X_2 + b_2 + \epsilon_2 \label{eq:linear-sem}
\end{align}

\noindent where $\theta_{1,2,3}$ denote linear coefficients describing the
effect of $X_1$ on $X_2$, $X_1$ on $Y$ and $X_2$ on $Y$, respectively; $b_1,b_2$
are the constant terms of the linear model and $\epsilon_1,\epsilon_2$ denote the
zero-mean noise (or disturbance) introduced by exogenous variables. A naive
solution is to construct a sophisticated machine learning estimator
$\hat{\theta_2} X_1 + \hat{\theta_3}X_2+\hat{b_2}$ for learning the regression
function $\theta_2 X_1 + \theta_3 X_2$. However, as pointed out
in~\cite{chernozhukov2016double}, \textit{such naive estimator does not converge
to the true regression function due to regularization bias} and the bias also
exists in general settings. In other words, since our observations on $X_2$ are
actually dependent on $X_1$, $\hat{\theta_2} X_1 + \hat{\theta_3}X_2+\hat{b_2}$
learned from data cannot precisely predict $Y$ given arbitrary (counterfactual)
$X_1$ and $X_2$, where $X_1,X_2$ are independent. We now introduce how \textsc{PerfCE}\xspace\
handles observable and latent confounders.

\parh{Observable Confounders.}~To handle observable confounders, we adopt double
machine learning~\cite{chernozhukov2016double}, a popular strategy in causality
analysis. In short, double machine learning first estimates an unbiased
$\hat{\theta_1}X_1+\hat{b_1}$ from data and obtains the first residual
$\hat{\epsilon_1}=X_2-\hat{X_2}$. Then, it uses $X_1$ to regress $Y$ as
$\hat{Y}$ and obtains the second residual
$\hat{\epsilon_2}+\hat{\theta_3}\hat{\epsilon_1}=Y-\hat{Y}$. Recall that
$\epsilon_2$ is zero-mean, and we eventually use the first residual
$\hat{\epsilon_1}$ to regress the second residual
$\hat{\epsilon_2}+\hat{\theta_3}\hat{\epsilon_1}$ to derive the unbiased
estimation $\hat{\theta_3}$. With $\hat{\theta_3}$, $\hat{\theta_2}$ can be
estimated accordingly by using $X$ to regress $\theta_2 X_1+\hat{\theta_3} X_2
+b_2$. Note that this pipeline (for augmenting linear regression) is extensible
to arbitrary functions. Particularly, deep learning models can be used to
perform non-linear regression and model more flexible causal relations.

\parh{Latent Confounders.}~The above technique is still based on the premise
that all confounders (i.e., $X_1$ in our example) are observable. It cannot be
extended to a more complex setting with latent confounders, which pervasively
exist in real-world settings. Overall, the presence of latent confounders makes
counterfactual predictions technically challenging, if not impossible, as we
have no method of disentangling the effect of $X_1$ from the causal relationship
between $X_2$ and $Y$. We note that none of existing works reviewed in
Table~\ref{tab:comparison} ever consider latent confounders, or they simply assume
the absence of latent confounders.

\begin{figure}[!ht]
\centering
\includegraphics[width=0.32\linewidth]{fig/iv.pdf}
\vspace*{-10pt}
\caption{Counterfactual predictions with an IV.}
\label{fig:iv}
\end{figure}

A commonly-used tactic, namely instrumental variable (IV), can be introduced to
address such difficult scenarios, in which we need to learn a regression
function under the influence of an ``interventional
distribution''~\cite{pearl2009causality}. Consider Fig.~\ref{fig:iv}, where IV is
introduced to serve the cause of treatment (i.e., $X_2$), whose effect on the
outcome (i.e., $Y$), if any, are propagated with $X_2$.

It is unclear how prior works can leverage IV, given they primarily collect
system logs in a \textit{passive} manner. In contrast, \textsc{PerfCE}\xspace\ \textit{novelly}
uses CE to form the IV, which has a direct influence on the CPU usage $X_2$ and
an indirect effect on processing time $Y$ via $X_2$. We clarify that CE is
proper to form IV: CE typically mutates low-level chaos variables (e.g., CPU,
network) by injecting failures. Thus, it is reasonable to assume the causal
relationships between CE (through the mutated chaos variables) and the
influenced ``high-level'' KPIs are indirect, allowing for the use of IV. To
illustrate, we will revisit the linear case and show how it is generalized into
arbitrary function forms. Considering the following linear form:
\begin{align}
    X_2 &= f(X_1) + \theta_1 IV + b_1\\
    Y   &= h(X_1) + \theta_2 X_2 + b_2
\end{align}

\noindent where $f(X_1)$ and $h(X_1)$ denote two unknown arbitrary (zero-mean)
functions of $X_1$ on $X_2$ and $Y$, respectively. Note that here $X_1$ denotes
an unseen exogenous variable (latent confounder). $\theta_{1,2}$ denote linear
coefficients describing the effect of $IV$ on $X_2$ and $X_2$ on $Y$,
respectively. $b_1,b_2$ are the constant terms of the linear model. The
procedure of using $IV$ also comprises a two-stage
regression~\cite{angrist1996identification}. First, we use $IV$ to regress $X_2$
and obtain $\hat{X_2}=\hat{\theta_1} IV + \hat{b_1}$. Here, we obtain the
residual as the effect of latent confounders on $X_2$:
$\hat{f(X_1)}=X_2-\hat{X_2}$. Then, in the second stage, we use the estimated
$\hat{X_2}$ to regress $Y=h(X_1) + \theta_2 \hat{X_2} + b_2 +
\theta_2\hat{f(X_1)}$. Recall that $f(X_1),h(X_1)$ is zero-mean. We obtain
unbiased estimation of $\hat{\theta_2},\hat{b_2}$. By doing this, we recover the
cause-effect relation as $Y=\hat{\theta_2}X_2+\hat{b_2}$, which allows us to
infer counterfactual queries.

Note that for generalizability, \textsc{PerfCE}\xspace\ uses a popular IV framework,
DeepIV~\cite{hartford2017deep}, to enable counterfactual analysis with deep
learning models. This enables to approximate arbitrary non-linear function
forms. We refer readers to~\cite{hartford2017deep} for full details.
\subsection{Online Root Cause Analysis}
\label{subsec:online}

\begin{algorithm}[!htbp]
\scriptsize
\caption{Root Cause Analysis (RCA)}
\label{alg:rca}
\KwIn{Observed KPIs $\bm{x}=(\bm{x}_1,\cdots,\bm{x}_n)$,
KPI of Interest $Y$, SEM $M$}
\KwOut{Ordered Potential Root Causes $\bm{X}=\{X_i,\cdots\}$}
$y\leftarrow \bm{x}_Y$; // observed KPI of interest\\
$\bm{X}_C=\{X_a\mid X_a \text{~is an ancestor of }Y \text{~on } M\}$; //initialize candidates\\
\ForEach{$X_a\in \bm{X}_C$}{
  $\hat{y}\leftarrow\mathbb{E}[ Y\mid \overbrace{X_j=\bm{x}_j}^{\forall j\neq a},do(X_a=\mathbb{E}\left[X_a\right])]$\;
  $s_a\leftarrow \texttt{PDF}_{Y}(\hat{y}) - \texttt{PDF}_{Y}(y)$\;
  assign $s_a$ as $X_a$'s blame\;
}
$\bm{X}\leftarrow\{X_a\mid X_a \in \bm{X}_C, s_c>0\}$\;
rank $\bm{X}$ by blames\;
\Return{$X$}
\end{algorithm}

\parh{Workflow.}~Alg.~\ref{alg:rca} outlines the workflow of conducting root cause
analysis for performance anomalies encountered during the online phase. $\bm{x}$
stands for a vector of KPIs currently being observed by the database monitoring
utility, and $Y$ denotes a KPI of interest specified by users (e.g., $Y$ can be
the query processing time). $M$ denotes the SEM learned in the offline phase.

Suppose $Y$ be the performance anomaly KPI, denoting that the database is under
high load. Alg.~\ref{alg:rca} first extracts the current observation of $Y=y$ from
the vector $\bm{x}$ (line 1). Then, it backwardly traverses the causal graph $M$
to find all ancestors of $Y$. These ancestors are candidates of root causes to
the abnormal $Y$ (line 2). For each candidate, Alg.~\ref{alg:rca} computes the
blame of the candidate (lines 3--7). Here, a counterfactual expectation is
predicted under an intervention to the current candidate $X_a$ (line 4; see
details shortly). Overall, it estimates \textit{``how would $Y$ become if $X_a$
were back to normal?''}  where ``back to normal'' is defined as
$X_a=\mathbb{E}\left[X_a\right]$ ($\mathbb{E}\left[X_a\right]$ is the average
value of $X_a$), and we retain all other KPIs in the vector $\bm{x}$ (line 4).
Let the estimated $Y$ be $\hat{y}$,
we diff the probability density of the current KPI value $y$ and the estimated
counterfactual KPI value $\hat{y}$ to compute the blame score of $X_a$ (lines
5--6), where $\texttt{PDF}_{Y}$ represents the probability density function of
random variable $Y$. By ranking causes with their blames, Alg.~\ref{alg:rca}
prioritizes users' attention to the most influential root cause KPIs. We
elaborate on computing counterfactual predictions and probability densities as
follows.

\begin{algorithm}[!htbp]
\scriptsize
\caption{Counterfactual Prediction}
\label{alg:cp}
\KwIn{Observed KPIs $\bm{x}=(\bm{x}_1,\cdots,\bm{x}_n)$,
KPI of Interest $Y$, Candidate Cause $X_a$, SEM $M$}
\KwOut{Counterfactual Prediction $\hat{y}$}
$\hat{\bm{x}}\leftarrow \bm{x}$\;
$\hat{\bm{x}_a}\leftarrow \mathbb{E}[X_a]$\;
$\bm{X}_D=\{X_d\mid X_d \text{~is a descendant of }X_a \text{~on } M\}$\;
sort $\bm{X}_D$ in the topological order on $M$\;
\ForEach{$X_d\in \bm{X}_D$}{
  Total\_TE $\leftarrow \sum_{X_p\in Pa(X_d)} \left(\mathbb{E}[X_d\mid do(X_p=\hat{\bm{x}_p})] - \mathbb{E}[(]X_d\mid do(X_p=\bm{x}_p)]\right)$\;
  $\hat{\bm{x}_d} \leftarrow \bm{x}_d + $Total\_TE\;
  \lIf{$X_d$ is $Y$}{break}
}
\Return{$\hat{\bm{x}_Y}$}
\end{algorithm}

\parh{Counterfacutal Prediction.}~Performing counterfactual predictions is
difficult, given the complexity of causal structures. As noted on line 4 of
Alg.~\ref{alg:cp}, we proceed the counterfactual predictions as follows:
\begin{equation}
  \label{eq:counterfactual}
  \hat{y}=\mathbb{E}[ Y\mid \overbrace{X_j=\bm{x}_j}^{\forall j\neq a},do(X_a=\mathbb{E}\left[X_a\right])]
\end{equation}

\noindent which estimates the conditional expectation of $Y$ given the currently
observed KPIs $\bm{x}$ and an intervention on one KPI, $X_a$ while keeping the
remaining KPIs ($j\neq a$) unchanged. When $X_a$ is a simple direct cause of
$Y$, estimating Eqn.~\ref{eq:counterfactual} is straightforward: we can simply add
the average treatment effect (ATE) of $X_a$ on $Y$ to the currently observed
$y$. Formally,
\begin{equation}
  \hat{y}=y + (\mathbb{E}[Y \mid do(X_a=\mathbb{E}[X_a])] - \mathbb{E}[Y \mid do(X_a=\bm{x}_a)])
\end{equation}

\noindent where the second operand (in the parentheses) of the addition
represents the ATE of $X_a$ on $Y$, which can be further computed using the SEM
learned in the offline phase. For instance, suppose we have
obtained the unbiased estimation of $\theta_2$ in Eqn.~\ref{eq:linear-sem} as
$\hat{\theta_2}$. Then, the ATE of $X_1$ on $Y$ can be computed as
$\hat{\theta_2}\times (\mathbb{E}[X_1]-\bm{x}_1)$.

However, in general cases where $X_a$ is an ancestor of $Y$, counterfactual
changes on $X_a$ may influence other KPIs, whose changes may propagate to $Y$ as
well. To systematically model the effect of $X_a$ on $Y$, we recursively update
all $X_a$'s descendants on the causal graph with respect to the counterfactual
change and estimate $\hat{y}$.
Alg.~\ref{alg:cp} outlines the procedure. We use $\hat{\bm{x}}$ to maintain the
counterfactual values of KPIs (line 1) and update $\hat{\bm{x}_a}$ to its mean
value (line 2). With $do(X_a=\mathbb{E}[X_a])$, only the descendants of $X_a$
will be modified. Therefore, we estimate each of its descendants in topological
order (lines 3--9) such that a descendant is updated when only all its ancestors
have been updated to the counterfactual values. For each descendant $X_d$, we
compute the total treatment effect (Total\_TE) as the sum of its parents'
treatment effects on $X_d$ (line 6) and derive the counterfactual $X_d$
accordingly (line 7). When $Y$ is updated, we terminate the procedure and return
$\hat{\bm{x}_Y}$ as the counterfactual prediction of $Y$ given the intervention
of $do(X_a=\mathbb{E}[X_a])$.

\begin{figure}[!ht]
\centering
\vspace{-10pt}
\includegraphics[width=1.00\linewidth]{fig/distribution.pdf}
\vspace{-10pt}
\caption{Modeling KPI with different techniques.}
\label{fig:kde}
\end{figure}

\parh{Probablistic Modeling.}~In general, the distribution of $X_k$ is unknown
and computing $\texttt{PDF}_{Y}(y)$ is technically infeasible. It is common to
assume a prior distribution family for $X_k$, and therefore, estimating the
parameters from observations and computing $\texttt{PDF}_{Y}(y)$ with closed
form PDF becomes feasible. For instance, when normal distribution is assumed, we
can estimate its parameters (i.e., mean and variance) via maximum likelihood
estimations, and with estimated parameters, the corresponding PDF is attainable.
Having that said, given the diversity of real-world KPIs, our preliminary study
shows that neither Gaussian distribution nor a more general beta distribution
can plausibly estimate the underlying distribution. For instance, when the
underlying distribution is of multiple modals (i.e., multiple peaks), simply
assuming normal or beta distributions would apparently undermine the estimation.
We employ Gaussian KDE (kernel density
estimation)~\cite{rosenblatt1956remarks,parzen1962estimation} to perform
non-parametric approximation of the underlying distribution. We present a sample
comparison in Fig.~\ref{fig:kde} where we apply both Gaussian KDE and normal
distribution on a typical KPI (i.e., ``Current QPS'') collected from MySQL. It
can be observed that our approach achieves a reasonably good approximation to
underlying distributions of KPIs, while normal distribution fails to capture the
statistical multi-modality in the empirical data. We note that we have observed
consistently good performance when using Gaussian KDE across other KPIs.
\section{Design of \textsc{PerfCE}\xspace}
\label{sec:overview}

Fig.~\ref{fig:overview} depicts \textsc{PerfCE}\xspace's workflow. \textsc{PerfCE}\xspace\ composes offline learning
and online diagnose phases. During the offline phase, \textsc{PerfCE}\xspace\ adopts
Chaos Mesh~\cite{cm}, an industry-standard CE framework integrated in cloud container
environments. We use Chaos Mesh\ to launch CE toward the target databases to collect
training data for constructing SEM (\S~\ref{subsec:offline}). CE enables active
intervention, resulting in high-quality SEM. \textsc{PerfCE}\xspace\ considers both observable
and latent confounders (see Fig.~\ref{fig:local-structure}). We present
configuration details of Chaos Mesh\ in \S~\ref{sec:implementation}.
With the learned SEM and a performance anomaly occurred during the online
database execution, \textsc{PerfCE}\xspace\ can identify the root cause KPIs. Furthermore, \textsc{PerfCE}\xspace\
allows the user to issue counterfactual queries toward the localized root causes
and obtain quantitative fix, such as reducing CPU usage below 45\%. We discuss
the details at this step in \S~\ref{subsec:online}.

\parh{Application Scope.}~\textsc{PerfCE}\xspace\ debugs real-world database performance
anomalies. Modern databases are complex and prone to performance issues. In
evaluation, we assess \textsc{PerfCE}\xspace\ using synthetic test suites and real-world
databases (MySQL and TiDB). These test suites/databases were also evaluated by
existing works. We deploy MySQL and TiDB in Kubernetes~\cite{k8s} clusters;
Kubernetes is the de facto container management system. Other databases can also
be integrated into Kubernetes and analyzed by \textsc{PerfCE}\xspace.

We also assume the databases have KPI monitoring utilities. Designing
performance monitoring is \textit{orthogonal} to this research, and our focus is
to analyze the monitored anomalies. There are several performance monitoring
tools integrated in Kubernetes~\cite{grafana,prometheus}.
See details about monitoring in \S~\ref{sec:implementation}.

\input{design-offline}
\input{design-online}
\section{Threats to Validity}

\section{Evaluation}
\label{sec:evaluation}

This section evaluates \textsc{PerfCE}\xspace\ on MySQL in \S~\ref{subsec:mysql}, and studies the
scalability of \textsc{PerfCE}\xspace\ on a large distributed DBMS, TiDB in \S~\ref{subsec:tidb}.
We also compare \textsc{PerfCE}\xspace's counterfactual analysis with standard machine learning
models on synthetic data in \S~\ref{subsec:synthetic}.

\parh{Processing Time.}~For the MySQL evaluation, it takes about 50 minutes for
causal graph structure learning, and takes about 8 minutes for causal graph
parameter learning. After that, to analyze one performance anomaly of MySQL
(including both quantitative and qualitative), \textsc{PerfCE}\xspace\ takes less than one
minute. As for TiDB, \textsc{PerfCE}\xspace\ takes about 1.3 hour for causal graph learning, and
about 40 minutes for parameter learning. Then, each anomaly analysis takes 3 to
5 minutes (depends on the number of involved KPIs). Evaluating each synthetic
dataset takes several seconds to finish. We clarify that parameter inference
uses one Nvidia GeForce RTX 3090 GPU and the causal graph construction (using
BLIP) is performed on a Intel Xeon CPU E5-2683 with 256 GB RAM; we use 64 cores
at this step. Thus, while \textsc{PerfCE}\xspace's exact processing time may vary across
machines, we find its overall speed as encouraging, especially with GPUs and CPU
parallels.

\subsection{Evaluation of MySQL}
\label{subsec:mysql}

\begin{table*}[!htbp]
  \centering
  \caption{Taxonomy of performance anomalies used in the MySQL experiment.}
  \vspace{-10pt}
   \resizebox{0.65\linewidth}{!}{
  \begin{tabular}{c|c|l}
    \hline
    \textbf{Component} & \textbf{Anomaly} & \textbf{Description}\\
    \hline
    Client & Workload Spike & invoke BenchBase to launch additional queries. \\\hline
    \multirow{4}{*}{Database} & Database Backup & invoke \textit{mysqldump}. \\\cline{2-3}
    & Database Restore & dump the pre-dumped history table back into database. \\\cline{2-3}
    & Flush Log & invoke \textit{mysqladmin} to flush all logs. \\\cline{2-3}
    & Flush Table & invoke \textit{mysqladmin} to flush all tables.
    \\\hline
    \multirow{3}{*}{I/O} & I/O Saturation & invoke \textit{stress-ng} to increase disk usage.\\\cline{2-3}
    & I/O Latency & invoke Chaos Mesh\ to launch I/O delays. \\\cline{2-3}
    & I/O Fault & invoke Chaos Mesh\ to inject I/O faults
    \\\hline
    \multirow{2}{*}{Network} & Network Delay & invoke Chaos Mesh\ to increase network latency. \\\cline{2-3}
    & Network Partition & invoke Chaos Mesh\ to inject network partition faults. \\\hline
    Memory & Memory Stress & invoke Chaos Mesh\ to increase memory usage.\\\hline
    CPU & CPU Stress & invoke Chaos Mesh\ to increase CPU usage. \\\hline
  \end{tabular}
   }
\label{tab:mysql-anomaly}
\end{table*}

\parh{Environment and Data Collection.}~We setup a MySQL instance on the
Kubernetes framework, execute queries in TPC-C using
BenchBase~\cite{DifallahPCC13} for 12 hours, in which there are 40 minutes under
chaos experiments. We use mysql-exporter~\cite{mysql-exporter} to collect
relevant KPIs. Then, in
the online phase, we follow the method in DBSherlock~\cite{yoon2016dbsherlock}
to simulate performance anomalies, as reported in Table~\ref{tab:mysql-anomaly}.
These anomalies are simulated in accordance with different key components in
MySQL. We create a total of 12 anomaly instances that affect the KPI of
interest (i.e., Query\_Duration). We use a total of 79 KPIs. We have already reported the KPIs
in Table~\ref{tab:mysql-kpi}.

\parh{Baselines.}~We re-implement CauseInfer~\cite{chen2014causeinfer},
FluxInfer~\cite{liu2020fluxinfer}, and ExplainIt~\cite{jeyakumar2019explainit}
as three baseline methods. They are all reviewed in Table~\ref{tab:comparison}. For
CauseInfer~\cite{chen2014causeinfer}, we also implement a variant of CauseInfer
(CauseInfer+CE), by augmenting the causal discovery process of CauseInfer with
our CE module. Regarding other relevant research works reviewed in
Table~\ref{tab:comparison}, we exclude from comparing with methods that do not
feature general KPI supports. We also omit comparison with
DBSherlock~\cite{yoon2016dbsherlock} and iSQUAD~\cite{ma2020diagnosing}, which
require labeling normal and abnormal regions for data collected during the
offline phase.

\begin{figure}[!ht]
\centering
\vspace{-5pt}
\includegraphics[width=0.95\linewidth]{fig/question.pdf}
\vspace{-5pt}
\caption{Example of human evaluation questions.}
\label{fig:human-eval}
\end{figure}

\parh{Evaluation.}~\textsc{PerfCE}\xspace\ and the three baseline methods yield a list of KPIs
ranked by their likelihood of being the performance anomaly's root causes. For
each anomaly experiment, we collect the top-5 KPIs. For the methods that provide
fewer than five KPIs, we collect all returned KPIs.
At this step, we invite seven experts, including two database engineers and five
academy researchers with expertise in databases, to pick a subset of KPIs from
the KPIs that are recommended by at least one method. We illustrate one question
posed to our participants in Fig.~\ref{fig:human-eval}. Experts are first educated
with the application scenario of database performance debugging (root cause
analysis) and the meaning of KPIs. Then, they are asked to comprehend the
detailed implementation of the root cause (Fig.~\ref{fig:human-eval}~(b)) before
selecting KPIs that are believed to be closely related to each anomaly's root
cause (Fig.~\ref{fig:human-eval}~(c)). Lastly, each KPI is assigned a score as
$\frac{\text{\# vote}}{\text{\# participant}}$ for a given anomaly; the higher
score indicates the higher consensus for this KPI being the true root cause,
from the human expert's perspective. The average completion time is 45 minutes.
The details of the human evaluation is presented in Supplementary Material~\cite{snapshot}.

\parh{Metric.}~With the scores collected from human evaluation, we consider the
following metrics for comparison. First, we compute the average score (AS) of
top-1/3/5 KPIs suggested by each method. In addition, we also employ two metrics
over the ranked KPIs provided by each method, MAP@R (Mean Average Precision) and
NDCG (Normalized Discounted Cumulative Gain), that are widely-used in
recommendation systems to evaluate the accuracy of the ranked list given user
preferences.

\begin{table}[t]

  \setlength{\tabcolsep}{3.0pt}
  \caption{Comparison of different methods. \#Anomaly denotes number of
  anomalies in this component. We highlight the \colorbox{best}{best} and
  \colorbox{second}{second best} methods in each component.}
  \vspace{-10pt}
  \label{tab:full-comparison}
  \begin{center}
  \resizebox{1.00\linewidth}{!}{\begin{tabular}{
    @{\hspace{1pt}}c@{\hspace{2.0pt}}
    @{\hspace{1.5pt}}c@{\hspace{1.5pt}}
    @{\hspace{1.5pt}}c@{\hspace{1.5pt}}
    @{\hspace{1.5pt}}c@{\hspace{1.5pt}}
    @{\hspace{1.5pt}}c@{\hspace{1.5pt}}
    @{\hspace{1.5pt}}c@{\hspace{1.5pt}}
    @{\hspace{1.5pt}}c@{\hspace{1pt}}
    }
  \toprule
  \textbf{Component} & \multirow{2}{*}{\textbf{Metric}} & \multirow{2}{*}{\textbf{\textsc{PerfCE}\xspace}} & \multirow{2}{*}{\textbf{CauseInfer}} & \multirow{2}{*}{\textbf{CauseInfer+CE}} & \multirow{2}{*}{\textbf{FluxInfer}} & \multirow{2}{*}{\textbf{ExplainIt}} \\
  \textbf{(\#Anomaly)} & & & & & & \\
  \midrule
  \multirow{5}{*}{\makecell{Client\\(1)}}
  & Top-1 AS & \colorbox{second}{0.71} & 0.00 & 0.00 & \colorbox{best}{0.86} & 0.00 \\
  & Top-3 AS & \colorbox{second}{0.81} & 0.00 & 0.00 & \colorbox{best}{0.86} & 0.29 \\
  & Top-5 AS & \colorbox{best}{0.83} & 0.00 & 0.00 & \colorbox{second}{0.69} & 0.17 \\
  & MAP@R    & \colorbox{best}{0.43} & 0.00 & 0.00 & \colorbox{second}{0.36} & 0.03 \\
  & NDCG     & \colorbox{best}{0.89} & 0.00 & 0.00 & \colorbox{second}{0.81} & 0.16 \\
  \cmidrule(r){1-7}
  \multirow{5}{*}{\makecell{Database\\(4)}}
  & Top-1 AS & \colorbox{best}{0.43} & 0.00 & \colorbox{second}{0.18} & \colorbox{best}{0.43} & 0.00 \\
  & Top-3 AS & \colorbox{best}{0.36} & 0.15 & \colorbox{second}{0.30} & 0.29 & 0.05 \\
  & Top-5 AS & \colorbox{best}{0.44} & 0.10 & 0.27 & \colorbox{second}{0.29} & 0.09 \\
  & MAP@R    & \colorbox{best}{0.13} & 0.02 & 0.09 & \colorbox{second}{0.11} & 0.01 \\
  & NDCG     & \colorbox{best}{0.51} & 0.12 & 0.37 & \colorbox{second}{0.38} & 0.09 \\
  \cmidrule(r){1-7}
  \multirow{5}{*}{\makecell{I/O\\(4)}}
  & Top-1 AS & \colorbox{best}{0.89} & \colorbox{second}{0.39} & 0.21 & 0.18 & 0.00 \\
  & Top-3 AS & \colorbox{best}{0.83} & 0.27 & \colorbox{second}{0.42} & 0.14 & 0.06 \\
  & Top-5 AS & \colorbox{best}{0.77} & 0.16 & \colorbox{second}{0.36} & 0.26 & 0.16 \\
  & MAP@R    & \colorbox{best}{0.48} & 0.10 & \colorbox{second}{0.18} & 0.10 & 0.03 \\
  & NDCG     & \colorbox{best}{0.73} & 0.23 & \colorbox{second}{0.35} & 0.27 & 0.14 \\
  \cmidrule(r){1-7}
  \multirow{5}{*}{\makecell{Network\\(2)}}
  & Top-1 AS & \colorbox{best}{0.85} & \colorbox{second}{0.43} & 0.14 & 0.14 & 0.00 \\
  & Top-3 AS & \colorbox{best}{0.88} & 0.24 & \colorbox{second}{0.33} & 0.07 & 0.07 \\
  & Top-5 AS & \colorbox{best}{0.89} & 0.14 & \colorbox{second}{0.29} & 0.21 & 0.14 \\
  & MAP@R    & \colorbox{best}{0.33} & 0.08 & \colorbox{second}{0.09} & 0.06 & 0.03 \\
  & NDCG     & \colorbox{best}{0.98} & 0.22 & \colorbox{second}{0.29} & 0.21 & 0.12 \\
  \cmidrule(r){1-7}
  \multirow{5}{*}{\makecell{Memory\\(1)}}
  & Top-1 AS & \colorbox{best}{1.00} & \colorbox{best}{1.00} & \colorbox{second}{0.86} & \colorbox{second}{0.86} & 0.00 \\
  & Top-3 AS & \colorbox{best}{1.00} & 0.33 & \colorbox{second}{0.90} & 0.29 & 0.00 \\
  & Top-5 AS & \colorbox{best}{1.00} & 0.33 & \colorbox{second}{0.77} & 0.40 & 0.20 \\
  & MAP@R    & 0.17 & 0.08 & \colorbox{best}{0.72} & \colorbox{second}{0.29} & 0.03 \\
  & NDCG     & \colorbox{best}{1.00} & 0.31 & \colorbox{second}{0.96} & 0.54 & 0.16 \\
  \cmidrule(r){1-7}
  \multirow{5}{*}{\makecell{CPU\\(1)}}
  & Top-1 AS & \colorbox{best}{0.57} & 0.00 & 0.00 & \colorbox{second}{0.29} & 0.00 \\
  & Top-3 AS & \colorbox{second}{0.57} & 0.00 & 0.00 & \colorbox{best}{0.76} & 0.00 \\
  & Top-5 AS & \colorbox{second}{0.57} & 0.00 & 0.00 & \colorbox{best}{0.66} & 0.11 \\
  & MAP@R    & \colorbox{second}{0.11} & 0.00 & 0.00 & \colorbox{best}{0.50} & 0.03 \\
  & NDCG     & \colorbox{second}{0.57} & 0.00 & 0.00 & \colorbox{best}{0.74} & 0.09 \\
  \cmidrule(r){1-7}
  \multirow{5}{*}{\makecell{Overall\\(13)}}
  & Top-1 AS & \colorbox{best}{0.71} & 0.26 & 0.21 & \colorbox{second}{0.36} & 0.00 \\
  & Top-3 AS & \colorbox{best}{0.68} & 0.19 & 0.34 & \colorbox{second}{0.29} & 0.06 \\
  & Top-5 AS & \colorbox{best}{0.70} & 0.13 & 0.30 & \colorbox{second}{0.34} & 0.14 \\
  & MAP@R    & \colorbox{best}{0.29} & 0.06 & 0.15 & \colorbox{second}{0.16} & 0.02 \\
  & NDCG     & \colorbox{best}{0.72} & 0.17 & 0.34 & \colorbox{second}{0.39} & 0.12 \\
  \cmidrule(r){1-7}

  \end{tabular}}
    \end{center}
    \vspace{-2em}
\end{table}

\parh{Result Overview.}~We report the results of Top-1/3/5 Average Score (AS),
MAP@R and NDCG in Table~\ref{tab:full-comparison}. We observe that \textsc{PerfCE}\xspace\
substantially outperforms existing methods for nearly all settings. For
instance, its overall NDCG is 84.6\% higher than the second best method,
FluxInfer. It provides the best root causes for five out of six anomaly
components compared to baseline methods. In short, we consider \textsc{PerfCE}\xspace\ as
sufficient to produce reasonable KPIs to assist developers in identifying the
root cause. Empirical results also show that CE offers a general augmentation
toward causality-based approaches. As in Table~\ref{tab:full-comparison}, after
being augmented with CE, CauseInfer manifests much better performance in most
settings (6.7\%/14.7\% improvement on the overall NDCG/MAP@R scores,
respectively) and provides better root causes compared to its original version.

\parh{Error Analysis.}~\textsc{PerfCE}\xspace\ exhibits sub-optimal performance in a few
settings. We find that \textsc{PerfCE}\xspace\ only recommends one KPIs for the anomaly triggered
by memory stress (5th component in Table~\ref{tab:full-comparison}) and CPU stress
(6th component). Therefore, AS is computed using the only KPI. For memory
stress, \textsc{PerfCE}\xspace\ recommends ``Node.Memory\_Distribution\_Free'', which is voted by
all human experts and thus assigned a score of 1. Thus, its Top-1/3/5 AS and
NDCG are computed as 1.00. However, experts also annotate some other useful
albeit less important KPIs with lower scores (e.g.,
``MySQL.Query\_Cache\_Memory\_Free\_Memory''). As a result, when a method (such
as CauseInfer) recommends more KPIs besides ``Node.Swap\_Activity\_Swap\_In'',
it will gain a higher MAP score. We inspect the cause of this inaccuracy and
find that it is primarily due to inaccurate counterfactual analysis over a
lengthy causal chain (with 11 hops on causal graphs), where regression errors
are accumulated and propagated, resulting in inaccurate estimations.

In another case where an anomaly was triggered by CPU stress (6th component in
Table~\ref{tab:full-comparison}), \textsc{PerfCE}\xspace\ also fails to suggest several highly
relevant KPIs. We find that it is due to the inaccurate causal graph, where the
causal relationships between the KPI of interest (i.e., ``Query\_Duration'') and
many CPU-related KPIs are incorrectly missed. In turn, the performance of
baseline methods (e.g., CauseInfer and ExplainIt) downgrades as well. Given the
inherent difficulty of causal structure learning, we anticipate that \textsc{PerfCE}\xspace\
users will incorporate their domain knowledge to revise certain spurious edges
and improve the quality of causal graphs. In particular, we find that when four
edges are revised, \textsc{PerfCE}\xspace's performance is improved from 0.11 to 0.27 on the
MAP@R score and from 0.57 to 0.61 on the NDCG score, respectively.

\begin{table}[!htbp]
  \centering
  \caption{Statistics of generated causal graphs. \#Node and \#Edge denotes the number of
  non-isolated nodes/edges in the causal graphs. BIC denotes Bayesian Information
  Criterion.}
  \vspace{-10pt}
   \resizebox{0.85\linewidth}{!}{
  \begin{tabular}{l|c|c|c|c}
    \hline
    \textbf{Method} & \textbf{\#Node} & \textbf{\#Edge} & \textbf{BIC}& \textbf{Accuracy}\\
    \hline
    \textsc{PerfCE}\xspace\ w/o CE & 64 & 63 & -614055.5 & 0.73 \\\hline
    \textsc{PerfCE}\xspace\  & 78 & 89 & -584910.3 & 0.87\\\hline
    Improvement  & +22\% & +42\% & +6\% & +16\%\\\hline
  \end{tabular}
   }
\label{tab:causal-graph-stat}
\end{table}

\parh{Plausibility of Causal Graph.}~We report the statistics of causal graphs
generated by \textsc{PerfCE}\xspace\ and its ablated versions (\textsc{PerfCE}\xspace\ w/o CE) that excludes the
CE module in Table~\ref{tab:causal-graph-stat}. Here, the \textbf{BIC} score is a
standard metric for assessing the ``fitness'' of causal graphs on observational
data. We also measure \textbf{Accuracy}: a metric examining if the pairwise
correlations are properly represented in the causal graph in terms of
d-separations and d-connections. We refer readers to~\cite{pgmpy} for the full
details of these evaluation metrics. The two metrics are computed using the
unseen data collected during the online phase. For both metrics, a greater value
indicates a better fitness with online data. We interpret the overall results as
encouraging: 78 out of 79 KPIs (after excluding constantly unchanged KPIs) are
not isolated in the causal graph, and 26 additional edges are identified
compared to the ablated version. Furthermore, \textsc{PerfCE}\xspace\ manifests a high degree of
agreement with the unseen online data, with an accuracy of 0.87. The
improvements in the BIC score further demonstrates the effectiveness of CE. We
supply the full graph in Supplementary Material~\cite{snapshot}. Overall, we consider that the \textsc{PerfCE}\xspace's causal
discovery can learn a plausible causal graph, and CE facilitates learning causal
graph with much better quality.

\subsection{Scalability Evaluation on TiDB}
\label{subsec:tidb}

\parh{Environment.}~We also evaluate \textsc{PerfCE}\xspace\ on a distributed database, TiDB. As
noted in \S~\ref{sec:implementation}, we host TiDB on a Kubernetes cluster,
which has three \texttt{TiKV} pods (\texttt{tikv-0,1,2}), one \texttt{TiDB} pod
(\texttt{tidb}), and one \texttt{PD} pod (\texttt{pd}). As mentioned in
\S~\ref{sec:implementation}, 254 KPIs are involved in the TiDB scenario, which
is much larger than that of MySQL. Therefore, we view this evaluation as
appropriate to benchmark the scalability of \textsc{PerfCE}\xspace\ on real-world distributed
databases. We present two case studies in particular in this section: we setup
two anomalies to TiDB, namely \texttt{tidb-pod-failure} and
\texttt{network-loss}. \texttt{tidb-pod-failure} injects a failure directly to
the \texttt{TiDB} pod and makes it temporally unavailable. \texttt{tikv-0-network-loss}
randomly drops 80\% packets that are sent to a specific KV pod (i.e.,
\texttt{tikv-0}).

\begin{table}[!htbp]
  \centering
  \caption{Top-10 KPIs suggested by \textsc{PerfCE}\xspace\ for answering \texttt{tidb-pod-failure}.}
  \vspace{-10pt}
   \resizebox{\linewidth}{!}{
  \begin{tabular}{c|c|l}
    \hline
    \textbf{Rank} & \textbf{KPI Family} & \textbf{Interpretation}\\
    \hline
    \multirow{3}{*}{1-5} & \multirow{3}{*}{\makecell{Lock\\Resolve OPS}} &
       The number of \texttt{TiDB} operations that resolve \\
    && locks. When a \texttt{TiDB} pod fails, \texttt{TiDB}-related \\
    && operations would decrease. \\\hline
    \multirow{4}{*}{6, 9} & \multirow{4}{*}{\makecell{Statement\\OPS}} &
       The number of different SQL statements \\
    && executed per second by \texttt{TiDB} pod. When a \\
    && \texttt{TiDB} pod fails, it would be unable to \\
    && execute statements. \\\hline
    \multirow{3}{*}{7, 8} & \multirow{3}{*}{\makecell{CPS By\\Instance (OK)}} &
       The succeed command statistics on each \texttt{TiDB}\\
    && instance. When a \texttt{TiDB} pod fails, the number\\
    &&  of succeed commands would decrease.\\\hline
    \multirow{3}{*}{10} & \multirow{3}{*}{\makecell{KV Cmd\\OPS}} &
       The number of executed KV commands emitted\\
    && by \texttt{TiDB} pod. When a \texttt{TiDB} pod fails, it  \\
    && would be unable to execute statements. \\\hline
  \end{tabular}
   }

\label{tab:tide-case1}
\end{table}

\parh{Case 1:}~In \texttt{tidb-pod-failure}, we inject a failure to the \texttt{TiDB}
pod. Then, we observe that the duration of the request launched by \texttt{PD} pod (i.e.,
``Handle Requests Duration'') becomes abnormal and ask \textsc{PerfCE}\xspace\ to diagnose this
anomaly. In short, we group the top-10 KPIs suggested by \textsc{PerfCE}\xspace\ and use TiDB's
official documentation to comprehend each of them. As shown in
Table~\ref{tab:tide-case1}, while the target KPI (PD's request duration) is not
directly relevant to \texttt{TiDB} pod failure, we find that all top-10 KPIs suggested by
\textsc{PerfCE}\xspace\ are highly relevant to the root cause. All of these KPIs indicate exactly
different activities performed by the failed \texttt{TiDB} pod. Therefore, we interpret
the outputs of \textsc{PerfCE}\xspace\ as reasonable and highly informative: when users are
provided with these KPIs, it should be accurate to assume that they can easily
identify the anomaly's root cause.

\begin{table}[!htbp]
  \centering
  \caption{Top-10 KPIs suggested by \textsc{PerfCE}\xspace\ for answering \texttt{tikv-0-network-loss}.}
  \vspace{-10pt}
   \resizebox{\linewidth}{!}{
  \begin{tabular}{c|c|l}
    \hline
    \textbf{Rank} & \textbf{KPI Family} & \textbf{Interpretation}\\
    \hline
    \multirow{5}{*}{1, 3, 7} & \multirow{5}{*}{\makecell{\texttt{TiKV}\\Write\\Leader}} &
       The number of leaders that are writing on each \\
    && \texttt{TiKV} instance. When network packets to a \texttt{TiKV} \\
    && pod are constantly lost, writes on this \texttt{TiKV} pod\\
    && would decrease and writes on other pods would  \\
    && increase. \\\hline
    \multirow{6}{*}{\makecell{2, 4-6,\\10}} & \multirow{6}{*}{\makecell{\texttt{TiKV}\\Resource}} &
       The usage of resources (e.g., CPU and memory) \\
    && on each \texttt{TiKV} instance. When network packets \\
    && to a \texttt{TiKV} pod is constantly lost, it would process \\
    && less requests thus uses less resources, while  \\
    && others are responsible for more requests and  \\
    && use more resources. \\\hline
    \multirow{4}{*}{8, 9} & \multirow{4}{*}{\makecell{Duration}} &
       The duration for processing different activities.\\
    && When network packets to a \texttt{TiKV} pod is constantly \\
    && lost, the duration of completing different\\
    && activities would increase.\\\hline
  \end{tabular}
   }
\label{tab:tide-case2}
\end{table}

\parh{Case 2:}~In \texttt{tikv-0-network-loss}, a notable proportion of network
packets are dropped, reducing the overall performance of the database. In
particular, we observe that the duration of command completion, ``Completed
Commands Duration (seconds)'', is abnormal, and we apply \textsc{PerfCE}\xspace\ for root cause
analysis. We group and present the results of \textsc{PerfCE}\xspace\ in Table~\ref{tab:tide-case2}.
We find the outputs of \textsc{PerfCE}\xspace\ as valuable. Note that the affected \texttt{TiKV} pod
(\texttt{tikv-0}) can only serve limited functionality, while other pods
(\texttt{tikv-1,2}) would serve extra responsibility instead. Therefore, the
decrease of workload on \texttt{tikv-0} and the surge of workload on
\texttt{tikv-1,2} are reasonable, and it should provide enough hints for users
to investigate the detailed status of \texttt{tikv-0}.

\subsection{Evaluation on Counterfactual Analysis}
\label{subsec:synthetic}

In addition to the evaluation conducted on real-world DBMSs, we also seek to
determine if \textsc{PerfCE}\xspace\ yields plausible counterfactual analysis results under
complex causal graphs. However, collecting or labeling the ground-truth outcomes
of quantitative counterfactual changes in real-world DBMSs is highly difficult,
if not impossible (therefore we use human evaluation to assess the accuracy of
\textsc{PerfCE}\xspace\ in \S~\ref{subsec:mysql}). Hence, we use synthetic data to compare \textsc{PerfCE}\xspace\
and other baselines, which is common setup when evaluating causal inference
algorithms.

\begin{table}[!htbp]
  \centering
  \caption{DGP models for generating synthetic data. LS stands for local structure.}
  \vspace{-10pt}
   \resizebox{\linewidth}{!}{
  \begin{tabular}{c|l}
    \hline
    \textbf{LS (see Fig.~\ref{fig:local-structure})} & \textbf{Formulation}\\
    \hline
    \multirow{3}{*}{\makecell{(a)\\no confounder}}  & $X_1\sim \text{Uniform}(-1,1)$ \\
     & $X_2\sim \text{Uniform}(-1,1)$ \\
     & $Y=\theta_1(X_1) + \theta_2(X_2) + \epsilon, \epsilon\sim \text{Uniform}(-1,1)$ \\\hline
     \multirow{3}{*}{\makecell{(b)\\observable confounder}}  & $X_1\sim \text{Uniform}(-1,1)$ \\
     & $X_2=\theta_1(X_1)+\eta,\eta\sim \text{Uniform}(-1,1)$ \\
     & $Y=\theta_2(X_1) + \theta_3(X_2) + \epsilon, \epsilon\sim \text{Uniform}(-1,1)$ \\\hline
     \multirow{4}{*}{\makecell{(c)\\latent confounder}}  & $IV\sim \text{Uniform}(-1,1)$ \\
     & $X_1\sim \text{Uniform}(-1,1)$ (\underline{unobserved})\\
     & $X_2=\theta_1(X_1)+\theta_2(IV)+\eta,\eta\sim \text{Uniform}(-1,1)$ \\
     & $Y=\theta_3(X_1) + \theta_4(X_2) + \epsilon, \epsilon\sim \text{Uniform}(-1,1)$ \\\hline
  \end{tabular}
   }
\label{tab:local-structure}

\end{table}

\parh{DGP Model.}~As a common setup, we formulate a set of DGP (Data Generating
Process) models~\cite{econml} used in this evaluation in
Table~\ref{tab:local-structure}. Note that these models correspond to the graphical
forms (in which we illustrate confounders) in Fig.~\ref{fig:local-structure}. We
use the linear form of $\theta$ and employ different random states to generate
$100$ datasets for each DGP model. In sum, we have a total of 300 ($3\times
100$) DGP models for evaluation. Then, for each DGP model, we train the model
with 5,000 data samples and craft 1,000 counterfactual queries for evaluation.
Each counterfactual query answers the treatment effects of changing $X_2=x_2$ to
$X_2=x_2'$ when $X_1=x_1$, where $x_1,x_2,x_2'$ are generated randomly and the
ground-truth treatment effects are generated by the specification of $Y$ in DGP
models.

\parh{Baseline \& Metric.}~As two baseline methods, we employ standard Multi
Layer Perception (MLP) and Random Forest (RF) models, which treat counterfactual
analysis as a regression problem. The MLP model consists of three hidden layers
and uses the Adam optimizer~\cite{kingma2014adam} with default parameters. The
RF model consists of 300 estimators with the maximal depth of seven. We report
MSE (mean squared error; lower is better) and $R^2$ (coefficient of
determination; higher is better) of each method on the dataset generated by each
DGP model.

\begin{table}[!htbp]
  \centering
  \vspace{-5pt}
  \caption{MSE and $R^2$ on synthetic data. LS stands for local structure. Best
  metrics are \colorbox{green}{highlighted}.}
  \vspace{-10pt}
   \resizebox{0.75\linewidth}{!}{
  \begin{tabular}{c|c|c|c|c|c}
    \hline
    \textbf{LS} & \textbf{$\theta$} & \textbf{Metric} & \textbf{\textsc{PerfCE}\xspace} & \textbf{MLP} & \textbf{RF}\\\hline
    \multirow{4}{*}{(a)} & \multirow{2}{*}{MSE} & mean & \bestcell{0.0003} & 0.0020 & 0.0109 \\\cline{3-6}
     & & std & \bestcell 0.0003 & 0.0011 & 0.0020 \\\cline{2-6}
     & \multirow{2}{*}{$R^2$} & mean & \bestcell 0.9988 & 0.9914 & 0.9522 \\\cline{3-6}
     & & std &\bestcell 0.0018 & 0.0063 & 0.0297 \\\hline
     \multirow{4}{*}{(b)} & \multirow{2}{*}{MSE} & mean &\bestcell 0.0002 & 0.0030 & 0.0176 \\\cline{3-6}
     & & std &\bestcell 0.0003 & 0.0022 & 0.0072 \\\cline{2-6}
     & \multirow{2}{*}{$R^2$} & mean &\bestcell 0.9988 & 0.9875 & 0.9282 \\\cline{3-6}
     & & std &\bestcell 0.0019 & 0.0093 & 0.0463 \\\hline
     \multirow{4}{*}{(c)} & \multirow{2}{*}{MSE} & mean &\bestcell 0.0174 & 0.3758 & 0.1029 \\\cline{3-6}
     & & std &\bestcell 0.0302 & 0.2098 & 0.0960 \\\cline{2-6}
     & \multirow{2}{*}{$R^2$} & mean &\bestcell 0.9382 & -0.8258 & 0.1540 \\\cline{3-6}
     & & std &\bestcell 0.0936 & 1.9738 & 1.2979 \\\hline
    \end{tabular}
   }

\label{tab:synthetic-result}
\end{table}

Table~\ref{tab:synthetic-result} reports the results. \textsc{PerfCE}\xspace\ excels in all
settings. In particular, on LS (c), the most challenging setting, the MLP model
largely overfits the data, resulting in unsatisfactory and unstable performance.
We interpret such downgrade as reasonable; the MLP model cannot distinguish the
effects of $X_1,X_2$ on $Y$ from the data generated by the DGP models under LS
(c). The same reason leads to the similar performance downgrade on the RF
models. However, in contrast to the MLP and RF models, we observe that \textsc{PerfCE}\xspace\
correctly estimates the effects of $X_2$ on $Y$ across all settings, even in the
presence of latent confounders, as in LS (c).

\section{Discussion}

\parh{Enhance Structure Learning with Human Knowledge.}~As shown in
\S~\ref{subsec:mysql}, flaws in causal graphs may undermine the accuracy of
\textsc{PerfCE}\xspace. In \textbf{Error Analysis} of \S~\ref{subsec:mysql}, we further
illustrate the applicability of incorporating expert knowledge to improve the
accuracy of \textsc{PerfCE}\xspace: we enhance the CPU case's MAP@R score from 0.11 to 0.27 and
the NDCG score from 0.57 to 0.61. As a practical future work, we anticipate that
\textsc{PerfCE}\xspace\ can be improved by encoding prior knowledge~\cite{meek1995casual} from
users to assist in the causal structure learning phase. We leave it for future
exploration.

\parh{Limitation on Achieved Observability.}~While we have primarily improved
the observability of causality analysis with chaos engineering, there are still
gaps between the achieved observability (using chaos experiments) and real-world
anomalies. We assume that it is often impossible to avoid encountering
real-world performance anomalies that do not exist in the collected offline
training data. This would result in the infamous OOD problems and inaccuracies
in the counterfactual analysis. In particular, we find that when the
counterfactual analysis is performed on a lengthy causal chain, errors propagate
and lead to incorrect outcomes of \textsc{PerfCE}\xspace. Currently, we collect KPIs when
mutating a limited collection of chaos variables; note that these chaos
variables are offered by to test general-purpose distributed systems. We see it
as demanding for CE frameworks tailored for databases, allowing for more
comprehensive, database-specific, and anomaly-oriented data collection.

\parh{Quantative Evaluation of Counterfactual Analysis.}~For real-world database
scenarios, it is generally impossible to prepare the ground-truth results of
quantitative counterfactual analysis. Evaluating real-world counterfactual
analysis is a long-lasting problem in the field of causality analysis. In this
work, we employ human evaluations to collect the ground-truth in the form of
\textit{order relationships} (as we assign each KPI a score based on its votes
of human experts). This makes end-to-end quantitative comparison of different
methods practically feasible. Looking ahead, we plan to explore developing
testbeds for counterfactual analysis in database KPIs, where controlled
experiments (with which we can collect the ground-truth of counterfactual
analysis) become possible.

\parh{Capturing Temporal Characteristics of KPIs.}~As is (often implicitly)
assumed by most works in this
field~\cite{chen2014causeinfer,lin2018microscope,jeyakumar2019explainit}, every
observation of KPIs is independent and identically distributed (iid), making
causality analysis practical feasible. This research shares the same underlying
assumption about KPI with prior works. In reality, KPIs may however manifest
temporal characteristics. Therefore, current assumptions may result in missing
edges during causal structure learning and inaccurate estimations in
counterfactual analysis. Nevertheless, given that the theory and tools of causal
discovery and causal inference on time series are mainly understudied, we leave
it for future research the consideration of the temporal properties of KPIs when
conducting counterfactual analysis.

\section{Related Work}

\parh{Theoretical Causality Analysis.}~Causality analysis is the foundation of
scientific discovery with applications in the field of economics and clinical
trails~\cite{pearl2009causality}. Identifying qualitative (i.e., SEM structure
learning) and quantitative (i.e., SEM parameter learning) causal relationships
is challenging. Typically, causal discovery algorithms are applied to learn a
causal graph that encodes causal relationships among a set of variables. The
methods for causal discovery include constraint-based
methods~\cite{spirtes2000causation,ding2020reliable},
score-based~\cite{scanagatta2015learning,tsamardinos2006max},
gradient-based~\cite{zheng2018dags} and supervision-based~\cite{ml4s}. Without
clear qualitative causal relationships, causal inference (i.e., identifying
quantitative causal relationships) is infeasible or inaccurate. The methods for
causal inference is diverse. For instance, Halpern and Pearl proposed
\textit{actual causality} that quantifies the blame of an event to the
outcome~\cite{halpern2020causes}. Rubin proposed potential outcomes
framework~\cite{rubin2005causal,rubin1974estimating}. In essence, it computes
the difference on an individual when a hypothetical intervention is enforced
versus an individual without the intervention. \textsc{PerfCE}\xspace\ uses Rubin's framework;
because it is more suitable when KPIs are modeled as random variables and ATE
(average treatment effect) naturally allows for propagating influence over
causal paths.

\parh{SE Applications of Causality Analysis.}~Recently, it has been witnessed a
number of researches that applied causality analysis to address problems in
software
engineering~\cite{fariha2020causality,dubslaff2022causality,johnson2020causal,krishna2020cadet,hsiao2017asyncclock}.
These tools usually focus on analyzing how program inputs, configurations or
behaviors impacts the software (e.g., execution time or crash) where causal
relationships are known and clear. Then, causal inference (such as actual
causality techniques) can be smoothly applied to diagnose software faults or
failures. However, in distributed systems (e.g., database, microservices),
causal relationships among numerical KPIs are unknown and presumably complicated
(e.g., latent confounders). Such scenarios necessitate causal discovery before
performing causal inference. Nevertheless, limited observability on such systems
further hinders the ability of standard off-the-shelf causal discovery
algorithms to comprehensively identify causal relationships. As a result, root
cause analysis for such scenarios is inherently difficult and challenging and
cannot be handled by existing tools. To address this challenge, \textsc{PerfCE}\xspace\ harnesses
the power of chaos engineering to improve the quality of causality analysis.
\section{Implementation}
\label{sec:implementation}

\begin{table}[!htbp]
  \centering
  \caption{Chaos variables used in our implementation.}
  \label{tab:chaos}
  \vspace{-10pt}
   \resizebox{1.00\linewidth}{!}{
  \begin{tabular}{c|c|l}
    \hline
    \textbf{Category} & \textbf{Variable} & \textbf{Description}\\
    \hline
    \multirow{2}{*}{Stress (2)}  & CPU & increase the load of CPU. \\\cline{2-3}
     & Memory & increase the load of memory. \\\hline
    \multirow{2}{*}{I/O (2)}  & Latency & delay file system calls. \\\cline{2-3}
     & Error & returns an error for filesystem calls. \\\hline
    \multirow{5}{*}{Network (5)}  & Loss & randomly drop network packets.\\\cline{2-3}
     & Latency & increase the latency of network. \\\cline{2-3}
     & Bandwidth & set bandwidth limit and buffer. \\\cline{2-3}
     & Corrupt & randomly corrupt network packets. \\\cline{2-3}
     & Duplicate & randomly duplicate network packets. \\\hline
    Pod (1) & Fault & specify which type of pod fault is injected on pods. \\\hline
    Time (1) & TimeOffset & specify the length of time offset. \\\hline
    Location (1) & Location & specify which pods the anomaly is triggered on. \\\hline
  \end{tabular}
   }
\label{tab:chaos-var}
\end{table}

We implement \textsc{PerfCE}\xspace\ with about 2K lines of Python code. During the offline
phase, \textsc{PerfCE}\xspace\ employs an industry-strength CE framework, Chaos Mesh\ (ver. 2.1.3). Chaos Mesh\
is commonly deployed on Kubernetes clusters, enabling CE on cloud
containerization environments. Here, we configure Chaos Mesh\ with about 3000 lines of
yaml. Therefore, (distributed) databases that are currently deployed in
Kubernetes clusters can be analyzed. Moreover, by being integrated with
Kubernetes, \textsc{PerfCE}\xspace\ is agnostic to the underlying DBMSs (e.g., MySQL, PostgreSQL
and TiDB). Also, we clarify that the technical solution of \textsc{PerfCE}\xspace\ is
\textit{not} specifically designed for Kubernetes clusters. \textsc{PerfCE}\xspace\ is general to
conduct performance debugging of other scenarios (e.g., databases deployed on a
single machine). We adopt Kubernetes given its good integration with Chaos Mesh\ and
the popularity (and technical challenge) of analyzing cloud, distributed
databases. We list all employed chaos variables in Table~\ref{tab:chaos}. For the
MySQL experiments, we use the first three categories (as MySQL is not a
distributed setup). We use all categories for the TiDB experiments.

\begin{table}[!htbp]
  \centering
  \caption{MySQL KPIs used in our implementation.}
  \vspace{-10pt}
   \resizebox{\linewidth}{!}{
  \begin{tabular}{c|l}
    \hline
    \textbf{Type} & \textbf{KPI}\\
    \hline
    \multirow{5}{*}{Workload (43)}
    & Current\_QPS, MySQL.Connections\_\{Conn/Max\_Used\_Conn/Max\_Conn\}, \\
     & MySQL.Questions, MySQL.Client\_Thread\_Activity\_\{Peak\_Threads\_-\\
     & Connected/Peak\_Threads\_Running/Avg\_Threads Running\}, MySQL.Abor-\\
     & ted\_Connections\_\{attempts/timeouts\}, MySQL.Slow\_Queries, MySQL.\\
     & Handlers, MySQL.Transaction\_Handlers, Top\_Command\_Counters.\\\hline
     \multirow{8}{*}{\makecell{MySQL\\Memory (23)}}
     & MySQL.Thread\_\{Thread\_Cache\_Size/Threads\_Cached/Threads\_Created\},\\
     & MySQL.Internal\_Mem\_\{InnoDB\_Buffer\_Pool\_Data/InnoDB\_Log\_Buffer-\\
     & \_Size/Additional\_Mem\_Pool\_Size/InnoDB\_Dictionary\_Size/Key\_Buffer-\\
     & \_Size/Query\_Cache\_Size/Ada\_Hash\_Index\_Size/TokuDB\_Cache\_Size\},\\
     & MySQL.Query\_Cache\_\{Free\_Mem/Query\_Cache\_Size/Hits/Inserts/Not-\\
     & \_Cached/Prunes/Queries\_in\_Cache/\}, MySQL.Table\_Open\_Cache\_Status\_-\\
     & \{Openings/Hits/Misses/Misses\_Due\_to\_Overflow/Table\_Open\_Cache\_-\\
     & Hit\_Ratio\}.\\\hline
     \multirow{9}{*}{\makecell{MySQL\\Internals (28)}}
     & MySQL.File\_Openings, MySQL.Open\_Files\_\{Open\_Files/Open\_Files\_Limit/\\
     & InnoDB\_Open\_Files\}, MySQL.Open\_Tables\_\{Open\_Tables/Table\_Open\_-\\
     & Cache\}, MySQL.Temporary\_Objects\_Created\_Tmp\_\{Tables/DIsk\_Tables/\\
     & Files\}, MySQL.Select\_Types\_\{Full\_Join/Full\_Range\_Join/Range/Scan/\\
     & Range\_Check\}, MySQL.Sorts\_\{Rows/Range/Merge\_Passes/Scan\}, \\
     & MySQL.Table\_Definition\_Cache\_\{Open\_Table\_Definitions/Table\_-\\
     & Definitions\_Cache\_Size/Opened\_Table\_Definitions\}, MySQL.Table\_-\\
     & Locks\_\{immediate/waited\}, MySQL.Network\_Traffic\_\{Inbound/Outbound\},\\
     & MySQL.Network\_Usage\_Hourly\_\{Received/Sent\}, MySQL.Query\_Duration.\\\hline
     \multirow{4}{*}{\makecell{Node\\Resources (18)}}
     & Node.IO\_Activity\_\{Page\_In/Page\_Our\}, Node.Memory\_Distribution\_-\\
     & \{Free/Total\}, Node.CPU\_Usage\_\{\_Load/\_Load\_Max\_Core\_Utilization\},\\
     & Node.Network\_Traffic\_\{Inbound/Outbound\}, Node.Swap\_Activity\_\{In\_-\\
     & Reads/Out\_Writes\}, Node.Disk\_Latency.\\\hline

    \end{tabular}
   }
\label{tab:mysql-kpi}
\end{table}

\parh{Database Selection.}~We instantiate \textsc{PerfCE}\xspace\ for performance diagnosis of
MySQL and TiDB. We deploy MySQL on Kubernetes, where we create MySQL with single
instance. Note that this setup is consistently adopted in prior
works~\cite{yoon2016dbsherlock}. We also deploy TiDB on Kubernetes. TiDB is a
popular distributed database, which is generally more complex than the MySQL
experiments due to much larger KPI size. To deploy TiDB, we follow its
recommended test configuration~\cite{tidbConfig}, deploying a basic test TiDB
cluster with one \texttt{PD} pod, one \texttt{TiDB} pod and three \texttt{TiKV}
pods on a 3-node Kubernetes cluster. Here, \texttt{PD}, \texttt{TiDB}, and
\texttt{TiKV} serve as manager, interface, and storage components of the
cluster, respectively. In Kubernetes, a pod denotes the minimal unit of an
application. For the 3-node Kubernetes cluster, one node works as the
control-plane, and other two nodes work as worker.

\parh{KPI Monitoring.}~\textsc{PerfCE}\xspace\ uses Grafana~\cite{grafana} with the
Prometheus~\cite{prometheus} backend to monitor KPIs. This is a popular solution
for event monitoring in Kubernetes. Table~\ref{tab:mysql-kpi} characterizes all 112
monitored KPIs in our MySQL experiments. As for the TiDB experiments, 254 KPIs
are involved. Audiences can refer to~\cite{tidbkpi} for information about these
KPIs (note that there are multiple KPIs under each ``Panel Name''
at~\cite{tidbkpi}). \textsc{PerfCE}\xspace\ records each KPI's value every one second.

\parh{Offline Training.}~\S~\ref{subsec:offline} explains that \textsc{PerfCE}\xspace\ extends
the codebase of BLIP~\cite{scanagatta2015learning} to learn causal graphs. We
adopt BLIP due to its SOTA performance and high engineering quality. We extend
its codebase with 112 extra lines of Python code. \textsc{PerfCE}\xspace\ is orthogonal to
particular causal graph learning algorithms. We leave it as one future work to
explore leveraging other SOTA algorithms like REAL~\cite{ding2020reliable} and
ML4C~\cite{dai2021ml4c}. We employ EconML~\cite{econml} to perform double
machine learning and DeepIV~\cite{hartford2017deep} to estimate the parameters
of causal relationships, where random forest models and neural networks (Multi
Layer Perception) are jointly employed for regression. We present model details
in Supplementary Material~\cite{snapshot}.

\section{Introduction}\label{sec:introduction}}
\label{sec:intro}

Performance anomalies of databases is severe, since databases are critical
infrastructures that support daily operations and businesses. Service outages or
performance defects can result in a negative user experience, a decline in
sales, and even brand damage. Google, for instance, assesses page speed for
ranking websites~\cite{wire19}. According to reports, every 100ms of latency
would cost Amazon 1\% in revenue~\cite{gigaspace}, and every 0.5s of additional
load delay for Google search results would result in a 20\% loss in
traffic~\cite{mayer}.

Database performance diagnosis is difficult. Modern databases often entail
complex resource management, and dependencies between modules of a (distributed)
database may introduce subtle performance bottlenecks, degrading system
performance. Diagnosing such performance issues empirically is cumbersome and
error-prone, especially as typical (distributed) databases on the cloud or in
containerization environments may be exposed to hundreds of potentially
influencing key performance indicators (KPIs).

\begin{figure}[t]
\centering
\includegraphics[width=1.00\linewidth]{fig/motivating-example.pdf}
\vspace{-15pt}
\caption{Motivating example of performance debugging.}
\label{fig:motivating-example}
\end{figure}

\parh{Usage Scenario.}~We show an illustrative example in
Fig.~\ref{fig:motivating-example}, which contains three KPIs. Filesystem IO $X_1$
and CPU usage $X_2$, as two causes, influence the database query processing time
$Y$. A developer, Bob, observes a processing time spike (5.1s) and wonders the
root cause of this spike. He manually checks all performance metrics and
identifies that the spike is due to high CPU usage.

\parh{Ideal Solution.}~The given scenario is usually human-intensive. As
disclosed by database vendors~\cite{ma2020diagnosing}, a burst of performance
anomalies may last only a few minutes, whereas human-intensive diagnoses can
take tens of minutes, which is time consuming and error-prone. Ideally, Bob
wishes to obtain a causal graph and a counterfactual oracle, derived from KPI
causal relations (see Fig.~\ref{fig:motivating-example}(a)) for performance
debugging. Bob would first backwardly traverse the causal graph and scope all
ancestors of the node denoting processing time $Y$. These ancestors, as in
Fig.~\ref{fig:motivating-example}(a), represent (in)direct causes of performance
downgrades. Then, Bob would inquire the oracle (the inquiry is often dubbed as a
``counterfactual query'') whether the performance downgrade can be resolved if a
particular cause were changed to a given extent: a counterfactual change that
can fix performance downgrade forms the root cause. Considering the table in
Fig.~\ref{fig:motivating-example}(d), where Bob submits two counterfactual queries
(yellow cells) to the oracle and receives $\hat{Y}$ after such counterfactual
changes. Bob observes that when $X_1$ (filesystem IO) resumes normal (i.e., its
mean value, $22.83\%$ in our case), processing time $\hat{Y}$ remains elevated
(first row in Fig.~\ref{fig:motivating-example}(d)). When $X_2$ (CPU usage) drops
to its mean value $29.69\%$, $\hat{Y}$ returns to normal (second row in
Fig.~\ref{fig:motivating-example}(d)). Thus, Bob attributes this spike to high CPU
usage.

\parh{Practical Challenge.}~Establishing an accurate causal graph and a
counterfactual oracle are long-standing challenges in causality analysis.
Existing works primarily employ off-the-shelf causal discovery algorithms or
hand-coded rules to identify causal graphs from observational data (e.g., events
related to infrastructure, storage, networks). Then, a predictive model is
trained on causal relations to support counterfactual analysis. However, we deem
that existing works are insufficient for systematic, in-depth performance
diagnosis due to \textit{limited observability}. Data samples of real-life
performance downgrades are rare. Therefore, the causal graph learned from mostly
normal data (the ``steady status'' in Fig.~\ref{fig:motivating-example}(b)) may be
biased and unsuitable for analyzing performance downgrades. Moreover, the
predictive model rarely considers bias due to confounders, leading to erroneous
counterfactual predictions~\cite{rosenbaum1983central,hartford2017deep}.

\parh{Our Solution.}~We adopt chaos engineering (CE), an emerging engineering
practice that aids testing complex software systems. We expand the common usage
of CE to create sufficient and authentic abnormal data, denoting
system events causing performance anomalies (see ``chaos exp.'' in
Fig.~\ref{fig:motivating-example}(b)). This enables learning high-quality causal
graphs and oracles.
In the \textit{offline phase}, we would launch CE to learn the causal graph and
the associated structural equation model (SEM) of the database. In addition to
data passively collected in mundane steady status, we proactively collect
abnormal data incurred by CE (``chaos exp.'' in
Fig.~\ref{fig:motivating-example}(b)) to learn an accurate and comprehensive
causal graph. Moreover, we use CE for active intervention, facilitating
estimation of SEM under (latent) confounders. In the \textit{online phase},
when an anomaly is observed (Fig.~\ref{fig:motivating-example}(c)), we collect
the ancestors of the abnormal processing time $Y$ on the causal graph and use
the learned SEM to diagnose the root cause of the anomaly by answering
counterfactual queries.

\parh{Main Results.}~This paper presents an automated framework, \textsc{PerfCE}\xspace, that
delivers the aforementioned performance diagnosis using CE-enhanced causality
analysis. \textsc{PerfCE}\xspace\ uses an industrial-strength chaos framework, Chaos Mesh~\cite{cm}, to
establish causal graphs and SEMs with high quality. Evaluation using synthetic
datasets shows that \textsc{PerfCE}\xspace\ offers highly accurate causality analysis. Moreover,
we evaluate \textsc{PerfCE}\xspace\ by setting up MySQL and a distributed database
TiDB~\cite{tidb} on the Kubernetes~\cite{k8s} container environments. For this
setup, human evaluations (involving two database engineers and five academy
researchers) show that \textsc{PerfCE}\xspace\ can reliably diagnose performance defects incurred
by various system resources, outperforming existing works in nearly all
settings. In sum, we make the following contributions:

\vspace{-5pt}
\begin{itemize}[leftmargin=3mm]
    \item This work, for the first time, introduces the usage of CE in the
    context of causality analysis. Combining passive observations  and proactive
    CE achieves comprehensive observability, enabling qualitative root cause
    identification and quantitative counterfactual analysis with high accuracy.

    \item We design \textsc{PerfCE}\xspace\ to conduct automated performance anomaly diagnosis
    for complex databases. \textsc{PerfCE}\xspace\ incorporates a set of design principles and
    optimizations to deliver an effective diagnosis with moderate cost.

    \item Our evaluation of synthetic datasets and real-world (distributed)
    databases hosted on containerization environments subsumes different
    practical and complex scenarios. Under all scenarios, \textsc{PerfCE}\xspace\ enables
    accurate debugging of performance anomalies.
\end{itemize}

\subsection{Subsection Heading Here}

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}

\section{Motivation \& Literature Review}
\label{sec:motivation}

This section discusses key technical challenge of causality analysis and review
existing works. We then illustrate the synergistic effect of integrating
causality analysis with CE.

\begin{figure}[!ht]
\centering
\vspace{-10pt}
\includegraphics[width=0.9\linewidth]{fig/local-structure.pdf}
\vspace{-10pt}
\caption{Three typical local structures in a causal graph.}
\label{fig:local-structure}
\end{figure}

\parh{Challenges in Rule-Based Causality Analysis.}~Existing causal-based
performance debugging can be categorized into \textit{rule-based construction}
and \textit{learning-based construction}. However, the accuracy of estimated
causal graphs by both methods remain questionable. First, rule-based
construction is highly dependent on expert knowledge and human-intensive, which
may not be always correct with respect to rigorous mathematical properties of
causal relationships~\cite{spirtes2000causation}. In fact, rule-based methods
usually aim at a specific application with limited types of metrics, e.g.,
Sage~\cite{gan2021sage}, one state-of-the-art work, only supports
latency-related KPIs.

\parh{Challenges in Learning-Based Causality Analysis.}~Learning-based causality
construction is generally superior than rule-based methods. Recent
works~\cite{jeyakumar2019explainit,chen2014causeinfer,lin2018microscope} use
learning-based approaches to creating causal graphs for performance debugging.
This allows more flexible causality inference with broader applications.
Nevertheless, in performance debugging, it is challenging to construct SEM for
qualitative/quantitative causality analysis. The key issue is the
\textit{limited observability}. Overall, data collected during normal database
execution suffer from selection bias, where abnormal data, denoting performance
anomalies, are rare or absent. According to our observation, a considerable
proportion of KPIs (e.g., a KPI denoting failed queries, known as \texttt{Failed
Query OPM}) are unchanged or change negligibly during normal database execution.
This hinders learning plausible causal relations. With this regard, existing
works often process a huge amount of logs~\cite{ma2020diagnosing}, thereby
subsuming possible (anomaly) data at the best effort.

\parh{Challenges Owing to Confounders.}~Despite above challenges of forming
causal graphs, we find that existing rule-/learning-based approaches neglect one
key influential factor---\textit{confounders}---in causality analysis due to
limited observability. Confounders, either observable or non-observable (called
``latent confounders''), are ubiquitous and hinder causality analysis. For
instance, software input size $Z$ may simultaneously increase filesystem I/O
$X_1$ and the overall processing time $Y$. The resulting causality model that
regresses $Y$ on $X_1$ is biased and inaccurate. When $Z$ is observed (i.e.,
Fig.~\ref{fig:local-structure}(b)), we may employ a doubly robust
estimator~\cite{rosenbaum1983central} to debias. For realistic settings where
$Z$ is not observed ($Z$ is a latent confounder, as in
Fig.~\ref{fig:local-structure}(c)), it is impossible to estimate an unbiased model
from data~\cite{hartford2017deep}.
The state-of-the-art (SOTA) works either assume the absence of
confounders~\cite{gan2021sage} (as in Fig.~\ref{fig:local-structure}(a)) or assume
that all confounders are observable~\cite{jeyakumar2019explainit}, as in
Fig.~\ref{fig:local-structure}(b). As our evaluation shows
(\S~\ref{subsec:synthetic}), neglecting observable/latent confounders impede
causality analysis accuracy.

\parh{Existing Works.}~Table~\ref{tab:comparison} compares \textsc{PerfCE}\xspace\ with existing
works conceptually and technically. We categorize each work in terms of either
rule-based or learning-based causality analysis. As aforementioned, rule-based
methods are often limited to specific domains and metrics;
Sage~\cite{gan2021sage} and Groot~\cite{wang2021groot} manually defined several
rules to constitute causal graphs. As shown in the last column of
Table~\ref{tab:comparison}, they only support limited KPIs (e.g., network
latency-related KPIs) specified in the manual rules.
Sieve~\cite{gan2019seer} uses Granger causality tests to discover causal
relations between time series. It evaluates if one time series can forecast
another, which is not necessarily the true causality. Most learning-based
methods train causal graphs using offline data collected during normal
execution; such passively collected observations (e.g., system logs) are often
biased.

Moreover, given a causal graph, most works rate the contribution of identified
causes using heuristics pertaining to graphical structures, e.g., PageRank-based
solutions add heuristically-designed ``weights'' to graph edges. We clarify that
such heuristics-based methods can hardly provide meaningful quantitative
relations between the root cause and performance anomalies, preventing
developers from comprehending how the root cause leads to an anomaly. Methods
based on graph traversal begin with abnormal KPI nodes and backtrack via their
ancestors to identify the root cause. Sage~\cite{gan2021sage} applies predictive
models to quantify the influence of a possible cause. Sage is the only attempt
for quantitative counterfactual analysis. Sage, however, only supports
quantitative analysis of network latency-related KPIs, as it primarily focuses
on cloud microservices. Due to limited observability, Sage may be biased in
regards to the prevalence of confounders.

\parh{\textsc{PerfCE}\xspace.}~CE is currently used to stress software systems and aid in-house
(differential) testing. This paper views CE as a nearly ``out-of-the-box''
option for enhancing causality analysis. CE readily introduces negative impacts
on software performance. Its chaos variables often influence hundreds of KPIs,
forming a wealth of training data for learning-based causality analysis. \textsc{PerfCE}\xspace\
exhibits general KPI support, not limited to a few domain-specific instances.
Also, instead of passively collecting system logs for training, \textsc{PerfCE}\xspace\ benefits
from \textit{active intervention} (by mutating chaos variables) to achieve
comprehensive observability and develop more accurate SEMs. In addition, \textsc{PerfCE}\xspace,
for the first time, delivers quantitative counterfactual analysis in the
presence of (latent) confounders with arbitrary KPIs. \textsc{PerfCE}\xspace\ uses double machine
learning (DML)~\cite{chernozhukov2016double} to address observable confounders,
while CE allows the use of instrumental variable (IV)~\cite{pearl2009causality}
to overcome latent confounders.

Note that this research uses the \textit{domain-general} CE framework (as shown
in Fig.~\ref{fig:chaos}). We do not require those domain-specific ``steady
states'' (Fig.~\ref{fig:chaos}). We show that CE is not limited to assist in-house
testing, and together with causality analysis, it can largely boost performance
debugging of (distributed) databases.

redpajama_set_name: RedPajamaArXiv


text:
Senior housing takes another step forward
BY PATRICIA ROY, PROY@HOLDENLANDMARK.COM
PAXTON – The shovel is not yet ground, but members of the Paxton Housing Partnership have passed some major milestones in their goal of creating affordable senior housing, after almost a decade of effort.
Recently, the Zoning Board of Appeals granted two variances for the project. Plans for the building call for 50 apartments – 45 one bedroom and 5 two bedroom units - while the town's zoning bylaws allow for no more than four residential units in a single building. A second variance was granted because the building is six inches higher than allowed by current zoning laws.
On Jan. 9, representatives of JKS Paxton LLC of Falmouth, project developer, went before the Planning Board to give a detailed presentation of the proposed development, PHP chair Martha Akstin said.
Jim Regis, vice president of S-C Management Corp. who will manage the building, was at the meeting, along with Gary Shepard, of David E. Ross Associates and engineer for the project. An attorney and architect were also present, Akstin said.
"It was a very detailed presentation," Akstin said. "The Planning Board asked that the meeting be continued until Feb. 3 so they had time to review it."
The senior housing will be built on 10 acres of the 46 acres left to the town by the late Arnold Klingele. An additional 20 acres was set aside as part of the project to meet zoning requirements, but will not be developed. JKS Paxton LLC has a 99-year lease in place for the land that is located off Grove Street and Holden Road. The lease, a 50-page document, was signed by the select board on Jan. 10.
The project is estimated to come in at $11 million to $12 million. The bulk of it will be financed through low income housing tax credits, that JKS Paxton LLC is in the process of applying for now, Akstin said.
The application process for these tax credits is very competitive, she said.
The federal government developed the tax credit program to provide the private market with an incentive to invest in affordable rental housing. The tax credits are awarded to developers of qualified projects. The developer can then sell the credits to investors to raise capital for building the housing. The credits work to reduce the developer's borrowing, so the housing is able to be offered for an affordable rate. The money is granted by the federal government, but is overseen by the state's Department of Housing and Community Development.
"One thing in our favor is this project is shovel-ready, and they favor those," Akstin said.
Only seniors age 62 or older will be eligible for the housing and their income must fall within guidelines set by HUD for the Worcester area. A single person may not earn more than $38,000 a year, while income for couples is limited to $42,000.
Rent will be assessed at 30 percent of yearly income. None of the apartments will be market rate.
"It's strictly affordable senior housing," Akstin said.
Thirty-five of the units will be set aside for residents or people who have a history of living in town.
"Say you sold your house and moved out, you could be eligible to come back. S-C Management will be doing an outreach program on this," Akstin said.
She is pleased with the amount of experience that S-C Management has in the senior housing field. They have been in the business for 20 years and manage 3,000 senior apartments throughout the state.
"They have a track record. This is not their first time at the rodeo," Akstin said.
The affordable senior housing units will also boost the town's inventory of Chapter 40B, or affordable housing. According to state formula, 10 percent of the town's 1,500 residences should be set aside for affordable housing. Currently, two group homes make up the town's entire affordable housing stock.
The senior housing will bring the town's inventory affordable housing up to 30 percent, Akstin said. Still, the housing will be a taxproducing entity for the town.
JKS Paxton Inc. should hear in July if it has been approved for the tax credits. The project could start within a month of that and should take 10 months to complete, she said.
"They have spent a lot of money on engineers, architects and the project manager's salary, so they want to get this started as soon as possible," she said.

redpajama_set_name: RedPajamaCommonCrawl


text:
KANYE'S TRIBUTE
What does Donda mean?
Published: 12:19 ET, Aug 29 2021
Updated: 12:19 ET, Aug 29 2021
Published: Invalid Date,
THE name Donda is on everyone's lips right now.
But not everyone knows what it means and why it is important to Kanye West.
Kanye West is releasing his new album called DondaCredit: AP
What is Donda?
Donda is Yeezy's new album that fans have been waiting patiently to be released.
His latest offering finally dropped TODAY (29 August, 2021).
Donda was due to be released earlier this month, but Kanye missed the deadline, despite saying he was "focusing on music".
The album also marks West's first full-length album of new material since 2019's Jesus Is King, which debuted at No. 1 on the Billboard 200 albums chart.
Kanye West and his late mother Donda WestCredit: Getty Images - Getty
Who is Donda named after?
This is Kanye West's 10th studio album to date and the title is a tribute to his late mother, Donda West, who passed away 14 years ago.
Donda was born on July 12, 1949, in Oklahoma City, Oklahoma.
She was married to Ray West, father of Kanye, but after their divorce, she moved with her son to Chicago in 1980.
Read more on Kanye West's new album release here...
How did Donda West pass away?
On November 10, 2007, Donda died of coronary artery disease and multiple post-operative factors due to or as a consequence of liposuction and mammoplasty.
She underwent liposuction, a tummy tuck, and a breast reduction the day before she died.
She was 58 years old at the time of her death.
Kanye was extremely close to his mother, and her death seemed to impact him greatly.
Kanye told Q magazine in 2015: "My mother was my everything.
"If I had never moved to LA, she'd be alive. I don't want to go far into it because it will bring me to tears."
more on kanye
MOVING ON?
Kardashian fans think Kim dropped clue she's dating A-list rapper in post
PETE WHO?
Kim asking friends 'to set her up on dates' after ex Pete moves on with Chase
TIKTOK TO BIG SCREEN
Kim Kardashian and Kanye West's daughter North, 9, scores major role
HI, MOM
North showed 'loyalty' to Kim at dinner with Kanye and his new wife, expert reveals
When was Kanye West's Donda listening event?
Kanye hosted another listening party for his long-awaited 10th studio album Donda yesterday, (August 29, 2021)
It was held at the 40,000-seater Soldier Field stadium and featured a replica set of the rapper's childhood home in South Shore.
Making a surprise appearance was Kayne's estranged wife Kim Kardashian West who came onto the stage dressed in a wedding gown.
This was despite Kim filing for a divorce in February.
Other guests at the event also included controversial artists Marlyn Manson and DaBaby at the event.
The pair appeared on stage next to West, standing at the front of the fake house as West's 'jail' track was played.
Elsewhere, The Weeknd and Travis Scott both appeared on the album, while a sample from Lauryn Hill could be heard in one of the tracks.
At one point, West even appeared to set himself on fire inside the stadium.
The whole event was streamed live on Apple Music.
Kanye West sets himself on FIRE and Kim Kardashian wears a wedding dress at Donda listening party


If energy and water are subsidized to boost crop production, could it lead to more and cheaper food but a shrinking, degraded water supply? Growing crops for biofuels might promise more abundant, cleaner energy, but what happens to food security when cultivable land is diverted?
The Water-Energy-Food Security Nexus Approach, an increasingly influential concept pioneered by ICIMOD and applied in 2015 to climate change adaptation and sustainable development, is a tool to highlight precisely those questions and linkages.
The approach steers the focus away from individual sectors, which can result in competing and counterproductive strategies, and stresses an integrated perspective that identifies trade-offs and maximizes positive impact and policy coherence.
An ICIMOD article applying the approach to climate adaptation, published by Climate Policy, quickly logged over 5,200 views and became the second most-read article on the site. Another influential article applies the concept to the United Nations Sustainable Development Goals in South Asia, while previous work has highlighted Himalayan upstreamdownstream linkages.
ICIMOD research is helping to lead the way towards balanced, effective strategies to improve the future of mountain people and ecosystems.


Cacharel is a French ready-to-wear brand particularly known for his very special universe. The visual identity of the House is very specific. It has been created and developed thanks to different people.
First of all, the stylist Corinne Sarrut is one of the first stylists who give to Ca charel his own design style such as prints, flowers prints.
In 1968, Jean Bousquet, Cacharel founder, met the photographer Sarah Moon. Together they create a romantic and poetical universe. In fact, Sarah Moon was already famous for her dreamy and enchanting photographs, using fuzzy style, which fit completely Cacharel. She will be in charge of the add pictures of the brand for several years.
A last element is part of its identity: the print. At the end of the sixties, are created the first Liberty Blouses and it is during the seventies that they become a symbol of the house.
All the prints are unique. A team part of the creative division is dedicated to create prints so that every piece is special.
The target is easily recognisable: dynamic woman who wants to differentiate herself. She likes fashion and is fond of following new trends. She is dreamy but ambitious and wants to show her personality through her clothes and prints. She is a free woman.
Cacharel breathes of freedom and uniqueness. Its target reflects the spirit of the brand and represents a state of mind.
Audrey Hepburn : son rôle au sein des Maisons de couture.
Kappauf et Citizen K, 23 ans déjà !