text:
\section{Introduction}

Policy gradient methods~\cite{williams1992simple,sutton2000policy,kakade2002natural,peters2008natural,schulman2015trust,cheng2018predictor} are a popular class of model-free reinforcement learning (RL) algorithms. They have many advantages, including straightforward update rules and well-established convergence guarantees~\cite{sutton2000policy,konda2000actor,cheng2018fast,yang2019policy}.
However, basic policy gradient methods, like REINFORCE~\cite{williams1992simple}, are also notorious for their sample inefficiency.
This can be attributed, at least in part, to the high variance in Monte Carlo gradient estimates, which stems from both policy stochasticity necessary for exploration as well as stochastic environmental dynamics. The high variance is further exacerbated as the RL horizon becomes longer and higher dimensional.
If the variance of gradient estimates can be reduced, then the learning speed of policy gradient methods can be accelerated \cite{ghadimi2016mini,cheng2018predictor}.

The reduction of variance in policy gradients thus is an important research topic in RL, which has been studied since early work on the development of policy gradient methods.
For example, function approximators (critics) have been adopted to (partially) replace the Monte Carlo estimates of accumulated costs to  reduce variance but at the expense of introducing bias in the search direction ~\cite{sutton2000policy,kimura2000analysis,thomas2014bias,silver2014deterministic,schulman2015high,sun2018truncated}.
This bias-variance tradeoff can work well in practice, but can also potentially cause divergent behaviors and requires careful tuning
~\cite{schulman2015high,efroni2018beyond,yang2019policy}.

Another line of research uses the control variate (CV) method from statistics, designed for reducing variance in Monte Carlo methods without introducing bias~\cite{sutton2000policy,ng1999policy,greensmith2004variance,jie2010connection,gu2016q,liu2017action,grathwohl2017backpropagation,tucker2018mirage,pankov2018reward}. For policy gradient algorithms, specialized CV methods have been proposed in order to take advantage of structure inherent in RL problems.
The CV method works by defining a certain correlation function (called the control variate) that approximates the Monte Carlo samples and yet yields a closed-form (or low-variance approximation of) the expectation of interest.
For policy gradient especially, the state dependent CVs (also known as baselines or reward reshaping~\cite{ng1999policy,jie2010connection})  have been thoroughly investigated ~\cite{greensmith2004variance}.
Common state dependent CVs are constructed as approximators of the policy's value function, which admits update rules based on standard policy evaluation techniques.
Overall, state dependent CVs are simple to implement and have been found to be quite effective.
However, the resulting policy gradients can still posses detrimentally  high variance, especially in  problems that has a long horizon.
This  has motivated the recent development of state-action dependent CVs~\cite{gu2016q,liu2017action,grathwohl2017backpropagation,tucker2018mirage,wu2018variance}.
By using more elaborate CVs, these techniques can further reduce the variance due to randomness in the \emph{actions} in gradient estimates that the previous state-only CVs fails to manage.

Considering the decades-long development of CV methods, one might wonder if there is a need for new policy gradient CV techniques.
In this paper, we argue that the past development of CVs for policy gradients has overlooked an important fact that the Monte Carlo gradient estimates are generated by rolling out a policy and collecting statistics along a \emph{trajectory} of states and actions. Instead the focus has been on sampling \emph{pairs} of states and actions, ignoring the correlation between states and actions \emph{across} time steps.
Recently \citet{tucker2018mirage} empirically analyzed the variance of instantaneous state-action pairs and compared this to the variance correlations across time steps in multiple simulated robot locomotion tasks. They found that the variance due to long-term trajectories is often larger than the variance due to instantaneous state-action pairs.
This finding implies that there is potential room for improvement.

In this paper, we theoretically analyze the properties of previous CVs, and show that indeed the variance due to long-term trajectories has non-negligible effects. Motivated by this observation, we propose a family of trajectory-wise CVs, called TrajCV, which augment existing CVs with extra terms to \emph{additionally} cancel this long-term variance. We show that TrajCV is particularly effective when the transition dynamics, despite unknown, is close to deterministic.
Like existing CVs, TrajCV requires only knowledge of state-action value function (\ie Q-function) approximates, and can be computed recursively.
Furthermore, we prove that TrajCV is optimal for variance reduction under reasonable assumptions.
These theoretical insights are validated in simulation.

Upon finishing this work we discovered a recent technical report~\cite{pankov2018reward} that is motivated similarly and details exactly the equation~\eqref{eq:traj-wise difference estimator} that TrajCV uses. Their empirical results on simulated LQG tasks are encouraging too: TrajCV demonstrated superior performance compared with previous state and state-action dependent CVs.
By contrast, we derive TrajCV following a completely different route, which brings extra insight into the previous deficiency and suggests natural ways for improvement. In addition, we analyze other potential trajectory-wise CVs and prove the proposed idea is optimal.

\section{Problem Setup and Background} \label{sec:background}

We consider episodic policy optimization in a finite-horizon Markov Decision Process (MDP)~\cite{bellman1957markovian,bertsekas1995dynamic} with horizon $h$, state space $\Sc$, action space $\Ac$, instantaneous cost function $c: \Sc \times \Ac \to \Rbb}\def\Q{\Qbb}\def\Z{\Zbb}\def\N{\Nbb}\def\C{\Cbb$,
initial state distribution $p_1$, and
dynamics $d$.\footnote{We use one-based indexing throughout the manuscript.}
Given a parameterized \emph{stochastic} policy class $\Pi$
the goal is to search for a policy in $\Pi$
that achieves low accumulated costs averaged {over trajectories}
\begin{align} \label{eq:objective}
J(\pi) := \Exp   \br{ C_{1:h}},  \quad \text{where  } C_t := c(S_t, A_t), \quad S_1 \sim p_1, \quad A_t \sim \pi_{S_t}, \quad S_{t+1} \sim d_{S_t, A_t}
\end{align}
where
$d_{s,a}$ denotes the distribution of the next state after applying action $a\in\Ac$ at state $s\in\Sc$, and $\pi_s$ denotes  the distribution of action at state $s \in \Sc$.
Note that $S_t$ and $A_t$ are the sampled state and action at step $t$, and $_:$ denotes summation (\ie  $C_{1:h} = \sum_{t=1}^{h} C_t$).
For simplicity of writing, we embed the time information into the definition of state, \eg, $c(S_t, A_t)$ can represent non-stationary functions.
The randomness in \eqref{eq:objective} consists of
the randomness in the start state, policy, and dynamics.
In this work, we focus on the case where the dynamics  $d$ and the start state distribution $p_1$ are unknown, but the instantaneous cost $c$ is known.

For notation, we will use uppercases to denote random variables, such as $S_t$ and $A_t$, with the exception of $J$. We will be frequently manipulating conditional distributions.
 We adopt the subscript notation below to write conditional expectation and variance. For $\Exp_{X|Y}[f(X,Y)]$ of some function $f$, $X$ denotes the random variable over which the expectation is defined and $Y$ denotes the conditioned random variable. Furthermore, for $f(X_1, \dots, X_N, Y)$, we use $\Exp_{|Y}[f(X_1, \dots, X_N, Y)]$ as a shorthand to denote taking the expectation over all other random variables (\ie  $X_1, \dots, X_N$) conditioned on $Y$. This subscript notation also applies to variance, which is denoted as $\mathrm{Var}$.

\subsection{Policy Gradient Methods: Pros and Cons}

The goal of this paper is to improve the learning performance of policy gradient methods~\cite{williams1992simple,konda2000actor,sutton2000policy,kakade2002natural,peters2008natural,silver2014deterministic,schulman2015trust,cheng2018predictor}. These algorithms treat minimizing \eqref{eq:objective} as a first-order stochastic non-convex optimization problem, where noisy, unbiased gradient estimates of $J$ in \eqref{eq:objective} are used to inform policy search. The basic idea is to apply the likelihood-ratio method to derive the gradient of \eqref{eq:objective}. Let us define $ N_t  := \nabla \log \pi_{S_t}(A_t)$, where $\nabla$ is the derivative with respect to the policy parameters, and define $q^{\pi}$ as the Q-function of $\pi$; that is, $q^{\pi}(S_t, A_t) = \Exp[C_{t:h}]$ where the expectation is generated by taking $A_t$ at $S_t$ and then $\pi$ afterwards.
Define $G :=G_{1:h}$ and $G_t :=  N_t C_{t:h}$.
Then it follow
~\cite{williams1992simple}
\begin{align}  \label{eq:policy gradient}
\textstyle \nabla J(\pi) =\Exp[\sum_{t=1}^{h} N_t q^{\pi}(S_t, A_t)]  = \Exp\br{G},
\end{align}
where the second equality is due to  $q^{\pi}(S_t, A_t) = \Exp[C_{t:h}]$.
Equation \eqref{eq:policy gradient} is an expectation over trajectories generated by running $\pi$.
Therefore we can treat the random vector $G$ as an unbiased estimate of $\nabla J(\pi)$, which can be computed by executing the policy $\pi$ starting from distribution $p_1$ and then recording the statistics $G_t$, for $t\in \{1,\dots, h\}$.
This technique is known as the Monte Carlo estimate of the policy gradient, which samples i.i.d. trajectories from the trajectory distribution defined in \eqref{eq:objective} to approximate the expectation.

The policy gradient methods (\eg REINFORCE~\cite{williams1992simple}) optimize policies based on gradient estimates constructed using the above idea.
They have numerous advantages.
For example, they have straightforward update rules
and convergence guarantee, as they minimizes $J$ directly by updating parameters in descent directions in expectation~\cite{sutton2000policy,konda2000actor,cheng2018fast,yang2019policy}.
However, simply using the Monte Carlo
estimate $G$ in policy optimization (\ie the vanilla implementation of REINFORCE) can result in poor parameter updates due to excessive variance~\cite{silver2014deterministic,schulman2015high}.
Therefore, while ideally one can apply standard first-order optimization algorithms such as mirror descent~\cite{beck2003mirror} using $G$ to optimize the policy, this often is not viable in practice. They would in turn require a tremendous amount of trajectory rollouts in a single update step in order to attenuate the high variance, making learning sample inefficient.

The high variance of $G$ is due to the exploration difficulty in RL: in the worst-case, the variance of $G$ can grow exponentially in the problem's horizon $h$~\cite{kakade2003sample,vemula2019contrasting}, as it becomes harder for the policy to visit meaningful states and get useful update information. Intuitively we can then imagine that policy optimization progress can be extremely slow, when the gradient estimates are noisy.
From an optimization perspective, variance is detrimental to the convergence rate in stochastic optimization.
For example, the number of iterations for mirror descent to converge to an $\epsilon$-approximate stationary point is {$O ((\Tr \paren{\mathrm{Var}[G]}+1)/\epsilon^2)$}, increasing as the problem becomes more noisy~\cite{ghadimi2016mini}.
Therefore, if the variance of estimates of \eqref{eq:policy gradient} can be reduced, the policy gradient methods can be accelerated.

\subsection{Variance Reduction and Control Variate}

A powerful technique for reducing the variance in the Monte Carlo estimates is the CV method~\cite{ross1990course,mcbook}.
Leveraging correlation between random estimates, the CV method has formed the backbone of many state-of-the-art stochastic optimization algorithms
~\citep{schmidt2017minimizing,johnson2013accelerating,defazio2014saga}.
The use of CV methods has also proved to be critical to designing practical policy gradient methods for RL~\cite{greensmith2004variance}, because of the high-variance issue of $G$ discussed in the previous section.
Below, we review the basics of the CV method as well as previous techniques designed for reducing the variance of $G$.
Without loss of generality, we suppose only one trajectory is sampled from the MDP  to construct the estimate of \eqref{eq:policy gradient} and study the variance of different single-sample estimates.
We remind that the variance can be always further reduced, when more i.i.d. trajectories are sampled.

\subsection{The Control Variate Method} \label{sec:simple CV}

Consider the problem of estimating the expectation $\Exp [X]$, where $X$ is a
(possibly multivariate) random variable.
The CV method~\cite{ross1990course,mcbook} is a technique for synthesizing unbiased estimates of $\Exp [X]$ that potentially have lower variance than the naive sample estimate $X$.
It works as follows:
assume that we have access to another random variable $Y$,
called the \emph{CV},
whose expectation $\Exp [Y]$ is cheaper to compute than $\Exp [X]$. Then we can devise this new estimate by a linear combination:
\begin{align} \label{eq:estimate}
X - \alpha^\top (Y - \Exp [Y]),
\end{align}
where $\alpha$ is a properly-shaped matrix.
Due to the linearity of expectation, the estimate in \eqref{eq:estimate} is unbiased, and its trace of variance (\ie the size of variance) can be lower-bounded as~\cite{wang2013variance}
\begin{align} \label{eq:cv introduction}
&\eqsp
\Tr \paren{\mathrm{Var}[X - \alpha^\top(Y - \Exp [Y])]}
\ge
\Tr \paren{\mathrm{Var}[X] -
\mathrm{Cov}[X, Y] \mathrm{Var}[Y]^{-1} \mathrm{Cov}[Y, X]}
\end{align}
Suppose $Y$ is in the same dimension as $X$.
One can show that when $\alpha$ is optimally chosen as
$
\alpha^\star = \frac{1}{2}\mathrm{Var}[Y]^{-1} \paren{\mathrm{Cov}[X,Y] + \mathrm{Cov}[Y,X]}
$.
When data are too scarce to estimate $\alpha^\star$, a practical alternative is setting $\alpha$ as the identity matrix, which often works well when $Y$ is positively correlated with $X$.
The resulting estimate $X - (Y - \Exp \br{Y})$ is known as the \emph{difference estimator}~\cite{mcbook} and has variance
$
\mathrm{Var} \br{X - Y}
$, meaning that if $Y$ is close to $X$ then the variance becomes smaller.
In the following, we concentrate on the design of difference estimators; we note that designing a good $\alpha$ is an orthogonal research direction.

\subsection{Common Control Variates for Policy Gradient Methods} \label{sec:common cv}

The art to various CV methods lies in the design of the correlated random variable $Y$. The choice is often domain-dependent, based on how
$X$ is generated.
When estimating the policy gradient in~\eqref{eq:policy gradient}, many structures (\eg the Markov property) can be leveraged to design CVs, as we shall discuss.
Following previous works (\eg \cite{greensmith2004variance,tucker2018mirage}) here we focus on the policy gradient component $G_t$ of $G$ given in \eqref{eq:policy gradient} for simplicity of exposition.\footnote{
	Without any assumption of the MDP, the variance of $G$ can be bounded by the variance of $G_t$ (\cref{app:pg var bound}). Tighter bounds can be derived when assumptions on the MDP is made, \eg, faster mixing rate~\cite{greensmith2004variance}.
}
The most commonly used CVs for policy gradient~\cite{williams1992simple, ng1999policy,greensmith2004variance} are
state-dependent functions $\widehat{v}:\mathcal{S}}\def\TT{\mathcal{T}}\def\UU{\mathcal{U}\to\Rbb}\def\Q{\Qbb}\def\Z{\Zbb}\def\N{\Nbb}\def\C{\Cbb$, which leads to the difference estimator
\begin{align} \label{eq:state cv}
{\wt G^{\mathrm{S}, \wh v}}_t
:=  G_t - \paren{N_t \wh{V}_t  - \Exp_{A_t|S_t}[N_t \wh{V}_t]} =
G_t - N_t \wh V_t,
\quad \text{ where }\wh{V}_t := \wh{v}(S_t),
\end{align}
and the expectation vanishes as $\Exp_{A_t|S_t}[N_t \wh{V}_t] =  \wh{V}_t \nabla \Exp_{A_t|S_t}[ 1] = 0$.\footnote{State dependent functions naturally include non-stationary constant baselines in our notation.}
Recently, \emph{state-action CVs}  $\wh{q}: \Sc \times \Ac \to \Rbb}\def\Q{\Qbb}\def\Z{\Zbb}\def\N{\Nbb}\def\C{\Cbb$ have also been proposed~\cite{gu2016q, liu2017action,grathwohl2017backpropagation,tucker2018mirage,	wu2018variance,ciosek2018expected}, in an attempt to reduce more variance through CVs that better correlate with $G_t$.
The state-action CVs yields the difference estimator
\begin{align} \label{eq:state-action cv}
{\wt G^{\mathrm{SA}, \wh q}}_t
:=  G_t - \paren{N_t \wh{Q}_t  - \Exp_{A_t|S_t}[N_t \wh{Q}_t]},
\quad \text{ where } \wh{Q}_t := \wh{q}(S_t, A_t).
\end{align}
Usually $\wh{v}$ and $\wh{q}$ are constructed as function approximators of the value function $v^\pi$ and the Q-function $q^\pi$ of the current policy $\pi$, respectively, and learned by policy evaluation, \eg, variants of TD($\lambda$) \cite{singh1996reinforcement}, during policy optimization.
Therefore, these methods can also be viewed as unbiased actor-critic approaches.
In practice,
it has been observed
that these CVs indeed
 accelerate policy optimization, especially in challenging simulated robot control tasks ~\cite{gu2016q, liu2017action,tucker2018mirage,	wu2018variance,ciosek2018expected}.

\section{Why We Need New Control Variates}
\label{sec:why we need new cvs}

Given the decades-long development of CVs for policy gradient reviewed above, one might wonder if there is a need for new policy gradient CV techniques. If so, what is the additional gain we can potentially have?
To answer this question, let us first analyze the variance of policy gradient component  $G_t$ and how the CVs above reduce it.
By the law of total variance, $\mathrm{Var}[G_t]$ can be decomposed into \emph{three terms}
\begin{align} \label{eq:G_t variance}
\mathrm{Var}_{S_t}  \Exp_{|S_t} \br{ N_t C_{t:h}}
+ \Exp_{S_t}  \mathrm{Var}_{A_t|S_t} \br{  N_t \Exp_{|S_t, A_t} \br { C_{t:h}}}  + \Exp_{S_t, A_t} \mathrm{Var}_{|S_t, A_t} \br {N_t C_{t:h} },
\end{align}%
where the first term is due to the randomness of policy and dynamics before getting to  $S_t$, the second term is due to policy randomness alone at step $t$, \ie selecting $A_t$, and the third term is due to again both the policy and the dynamics randomness in the future trajectories, \ie after $S_t$ and $A_t$.\footnote{The law of total variance: $\mathrm{Var}[f(X,Y)]= \Exp_{X}\mathrm{Var}_{Y|X}[f(X,Y)] + \mathrm{Var}_{X}\Exp_{Y|X}[f(X,Y)]$~\cite{chung2001course}.}
Let us measure the size of these three terms by their trace and define
\begin{align}
\begin{split}
\V_{S_t} \coloneqq \Tr\paren{\mathrm{Var}_{S_t} \Exp_{|S_t} \br{ N_t C_{t:h}}}&, \quad
\V_{A_t|S_t} \coloneqq 	\Tr\paren{\Exp_{S_t} \mathrm{Var}_{A_t|S_t} \br{  N_t \Exp_{|S_t, A_t} \br { C_{t:h}} }},\\
\V_{|S_t, A_t} &\coloneqq \Tr\paren{\Exp_{S_t, A_t} \mathrm{Var}_{|S_t, A_t} \br {N_t C_{t:h}}}.
\end{split}
\end{align}
Hence,  $\Tr\paren{\mathrm{Var}[G_t]} = \V_{S_t} + \V_{A_t|S_t} + \V_{|S_t, A_t}$.
The following theorem shows the size of each term when the policy is Gaussian, which is commonly the case for problems with continuous actions.
\begin{theorem} \label{th:varaince size}
	Suppose the policy $\pi$ is Gaussian such that $\pi_{S_t}(A_t) = \NN(A_t|\mu_\theta(S_t), \sigma I )$, where $\mu_\theta$ is the mean function, and $\theta$ and $\sigma>0$ are learnable parameters.
	Assume the cost function $c$ is bounded and the Q-function $q^{\pi}(s,a)$ is analytic in $a$.
	Then for small enough $\sigma$, it holds that
	\begin{align*}
	\textstyle
	\V_{S_t} = O(h^2),
	\quad \V_{A_t|S_t}= O\left(\frac{h^2}{\sigma^4}\right),
	\quad
	\V_{|S_t, A_t}= O\left(\frac{h^2}{\sigma^4}\right).
	\end{align*}
\end{theorem}
Here we focus on the effects due to the problem horizon $h$ and the policy variance $\sigma$.  \cref{th:varaince size} shows that, when
the stochasticity in policy decreases
(\eg when it passes the initial exploration phase)
the terms $\V_{A_t|S_t}$ and $\V_{|S_t,  A_t}$ will dominate variance in policy gradients. An intuitive explanation to this effect is that, as the policy becomes more deterministic, it becomes harder to compute the derivative through zero-order feedback (\ie accumulated costs). In particular, one can expect that $\V_{|S_t, A_t}$ is likely to be larger than $\V_{A_t|S_t}$ when the variation of $C_{t:h}$ is larger than the variation of $q^{\pi}(S_t,A_t)= \Exp_{|S_t,A_t}[C_{t:h}]$.
After understanding the composition of  $\mathrm{Var} [G_t]$, let us analyze $\mathrm{Var}[{\wt G^{\mathrm{SA}, \wh q}}_t]$ to see why using Q-function estimates as CVs (in Section~\ref{sec:common cv}) can reduce the variance.\footnote{Discussion on $\mathrm{Var}[{\wt G^{\mathrm{S}, \wh v}}_t]$ is omitted in that ${\wt G^{\mathrm{S}, \wh v}}_t$ is subsumed by ${\wt G^{\mathrm{SA}, \wh q}}_t$.}
Akin to the derivation of \eqref{eq:G_t variance}, one can show that $\mathrm{Var}[{\wt G^{\mathrm{SA}, \wh q}}_t]$ can be written as
\begin{align} \label{eq:sa-cv variance}
\mathrm{Var}_{S_t}  \Exp_{|S_t} [ N_t C_{t:h}] +
\Exp_{S_t} \mathrm{Var}_{A_t|S_t} [  N_t (\Exp_{|S_t, A_t} [ C_{t:h}] - \wh{Q}_t)] +
\Exp_{S_t, A_t} \mathrm{Var}_{|S_t, A_t} [N_t C_{t:h} ].
\end{align}
Comparing \eqref{eq:G_t variance} and \eqref{eq:sa-cv variance}, we can see that the CVs in the literature have been focusing on reducing \emph{the second term} $\mathrm{Var}_{A_t|S_t}$.
Apparently, from the decomposition \eqref{eq:sa-cv variance}, the optimal choice of the state-action CV $\wh{q}$ is the Q-function of the current policy $q^\pi$, because
$q^\pi(S_t,A_t) := \Exp_{|S_t, A_t} [C_{t:h}]$,
which explains why $\wh{q}$ can be constructed by policy evaluation. When $\wh{q} = q^\pi$, the effect of  $\mathrm{Var}_{A_t|S_t}$ can be completely removed.
In practice, $\wh{q}$ is never perfect (let alone the state-dependent version); nonetheless, improvement in learning speed has been consistently reported.

However,  \cref{th:varaince size} suggests that $\mathrm{Var}_{|S_t, A_t}$ is a similar same size as $\mathrm{Var}_{A_t|S_t}$, implying that even when we completely remove the second term $\mathrm{Var}_{A_t|S_t}$, the variance of the gradient estimate can still be significant.
Indeed, recently~\citet{tucker2018mirage} empirically analyzed the three variance components in \eqref{eq:sa-cv variance} in LQG and simulated robot locomotion tasks. They found that the third term $\mathrm{Var}_{|S_t, A_t}$ is close to the second term $\mathrm{Var}_{A_t|S_t}$, and both of them are several orders of magnitude larger than the first term $\mathrm{Var}_{S_t}$.
Our \cref{th:varaince size} supports their finding and implies that there is a potential for improvement by reducing $\mathrm{Var}_{|S_t, A_t}$. We discuss exactly how to do this next.

\section{Trajectory-wise Control Variates}
\label{sec:trajectory}

We propose a new family of trajectory-wise CVs, called TrajCV, that improves upon existing state or state-action CV techniques by tackling \emph{additionally} $\mathrm{Var}_{|S_t, A_t}$, the variance due to randomness in trajectory after step $t$
(cf. \cref{sec:why we need new cvs}).
While this idea sounds intuitively pleasing, a technical challenge immediately arises. Recall in designing CVs, we need to know the expectation of the proposed CV function over the randomness that we wish to reduce (see \eqref{eq:estimate}).
In this case, suppose we propose a CV $g(S_t, A_t, \dots,A_{h-1}, S_{h})$, we would need to know its conditional expectation $\Exp_{|S_t, A_t}[g(S_t, A_t, \dots, S_{h}, A_h)]$.
This need makes reducing $\mathrm{Var} _{|S_t, A_t}$ fundamentally different from reducing $\mathrm{Var}_{A_t|S_t}$, the latter of which has been the main focus in the literature: Because the  dynamics $d$ is unknown, we do not have access to the distribution of trajectories after step $t$ and therefore cannot compute $\Exp_{|S_t, A_t}$; by contrast, reducing $\mathrm{Var}_{A_t|S_t}$ only requires knowing the policy $\pi$.

At first glance this seems like an impossible quest. But we will show that by a clever divide-and-conquer trick, an unbiased CV can actually be devised to reduce the variance $\mathrm{Var}_{|S_t, A_t}$.
The main idea is to
\textit{1)} decompose $\mathrm{Var}_{|S_t, A_t}$ through repeatedly invoking the law of total variance and then \textit{2)} attack the terms that are \emph{amenable} to reduction using CVs.
As expected, the future variance cannot be completely reduced, because of the  unknown dynamics. But we should be able to reduce the randomness due to known distributions, namely, the future uses of policy $\pi$.
\begin{figure}[t!]
	\centering

	\begin{subfigure}{.49\textwidth}
		\centering
		\includegraphics[trim={6cm 2cm 6cm 9cm}, clip, width=0.75\textwidth]{cv}
		\caption{CV on a toy problem}
		\label{fig:cv}
	\end{subfigure}
	\begin{subfigure}{.49\textwidth}
		\centering
		\includegraphics[trim={5.5cm 6cm 1cm 6cm}, clip, width=1.0\textwidth]{comparison}
		\caption{effects of state-action CV and TrajCV}
		\label{fig:comparison}
	\end{subfigure}
	\caption{
		(a) An illustration of the Divide-and-Conquer strategy on the toy problem in Section~\ref{sec:toy example} using color encoding.
		First row: ordering and coloring of the random variables.
		Second row:  the variance to reduce, \ie the variance due to the randomness of variables in red circle.
		Third row: the corresponding optimal CV.
		Last row: the expectation required to compute the difference estimator for the optimal CV.
		(b) An illustration of the effect of state-action CV \eqref{eq:state-action cv} and TrajCV \eqref{eq:traj-wise difference estimator} on each policy gradient component $G_t$. Each row corresponds to each $G_t$, and each column corresponds to one term in the total variance of $G_t$.
		For example, the $A_2$ column is associated with the term
		$\Exp_{S_{1,2}, A_1} \mathrm{Var}_{A_2 |S_2} \Exp_{| S_2, A_2} \br{G_t}$.

		The state-action CVs influence the terms in red, whereas TrajCV \emph{additionally} affect the terms in blue.
	}
	\label{fig:some visualization}
\end{figure}

\subsection{A Divide-and-Conquer Strategy} \label{sec:toy example}

Before giving the details, let us first elucidate our idea using a toy problem. Consider estimating $\Exp \br{f(X_{1,5})}$, the expectation of a function $f$ of $5$ random variables, where the subscript $_{i,j}$ denotes the collection of random variables, \ie $X_{1,5} = \{X_1, \dots, X_5\}$.
We can apply the law of total variance repeatedly, in the order indicated by the subscript, and decompose the variance into
\begin{align}
\textstyle
\mathrm{Var} \br{f(X_{1,5})} = \sum_{k=1}^{5} \Exp_{X_{1,k-1}} \mathrm{Var}_{X_k|X_{1,k-1}} \Exp_{X_{k+1, n}|X_{1,k}}[f(X_{1,5})]
\end{align}
For example, suppose we wish to reduce $\mathrm{Var}_{X_3|X_{1,2}}$ we simply need to consider a CV in the form $g \paren{X_{1,3}}$, which does not depends on random variables with larger indices. With the difference estimator $f(X_{1,5}) - g \paren{X_{1,3}} + \Exp_{X_3|X_{1,2}} [g \paren{X_{1,3}}]$, the variance $\mathrm{Var}_{X_3|X_{1,2}}$ changes into
\begin{align*}
\Exp_{X_{1,2}} \mathrm{Var}_{X_3|X_{1,2}}[ \Exp_{X_{4,5}|X_{1,3}}[f(X_{1,5})] - g(X_{1, 3})]
\end{align*}
Apparently when $g$ is optimally chosen as
$
g^\star (X_{1,3}) :=
\Exp_{X_{4, 5}|X_{1,3}}[f(X_{1,5})]
$, this term vanishes.

\textbf{Fact 1\quad }A key property of designing CVs by the recursive decomposition above is that
the inclusion of the extra term, \eg $g \paren{X_{1, 3}} - \Exp_{X_3 | X_{1,2}} \br{g(X_{1, 3})}$, in the difference estimator only affects a single component $\mathrm{Var}_{X_3|X_{1,2}}$ in the total variance, \emph{without influencing the other terms}.
This separation property hence allows for a divide-and-conquer strategy:
we can design CVs for each term separately and then combine them; the reduction on each term will add up and reduce the total variance.

\textbf{Fact 2\quad}There is still one missing piece before we can adopt the above idea to design CVs for estimating policy gradients: the ordering of random variables.
In the example above, we need to know $\Exp_{X_3|X_{1,2}} \br{g(X_{1,3})}$ to compute the difference estimator. Namely, it implicitly assumes the knowledge about
$p(X_3 |X_{1,2})$, which may or may not be accessible.
Suppose $p(X_3|X_{1,2})$ is not available but $p(X_3|X_{4,5})$ is. We can consider instead using
the law of total variance in a different order, \eg
$
X_4 \rightarrow X_5 \rightarrow X_3 \rightarrow X_1 \rightarrow  X_2
$,
and utilize the information $p(X_3|X_{4,5})$ to construct a difference estimator to reduce
$\mathrm{Var}_{X_3|X_{4,5}}$.
Therefore, the design of CVs hinges also on the information available. Recall that we only know about the policy but not the dynamics in RL.

\subsection{Design of TrajCV}

After fleshing out the idea in the example above, we are now ready to present
TrajCVs
for policy gradient.
Again let us focus on a policy gradient component $G_t$ for transparency.
Recall that $G_t$ is a function of the random variables $S_{t,h}$ and $A_{t,h}$.
Given the information we know about the random variables
(\ie the policy)
and the Markovian structure depicted in the Bayes network in Figure~\ref{fig: bayes net 1},
a natural ordering of them for applying law of total variance is
\begin{align} \label{eq:natural ordering}
S_t \rightarrow A_t \rightarrow S_{t+1} \rightarrow A_{t+1} \rightarrow \dots \rightarrow S_{h} \rightarrow A_{h}.
\end{align}
Thanks to the Markovian structure,
 $\mathrm{Var}_{A_k|S_{1,k}, A_{1,k-1}}$ can be simplified into $\mathrm{Var}_{A_k|S_k}$.
Thus, for reducing $\mathrm{Var}_{A_k|S_k}$, we may consider a CV, $\wh{g}^{(t)}(S_k, A_k)$, where the superscript $^{(t)}$ is a reminder of estimating $\Exp\br{G_t}$.
With the insights from \cref{sec:toy example}, we see the optimal choice of $\wh{g}^{(t)}$ is
\begin{align*}
\wh{g}^{(t)\star}(S_{k}, A_{k})
&= \Exp_{S_{k+1, h}, A_{k+1,h} | S_{k}, A_{k} }  \br{N_t C_{t:h}}
= N_t \paren{C_{t:k-1} + q^\pi(S_k, A_k)},
\end{align*}
suggesting practically we can use
$\wh{g}^{(t)}(S_k, A_k) := N_t(C_{t:k-1} + \wh Q_k)$,
where $\wh Q_k: = \wh q(S_k, A_k)$ and  $\wh q \approx q^\pi$
as was in \eqref{eq:state-action cv}.
In other words, we showed that finding the optimal CV for reducing $\mathrm{Var}_{A_k|S_k}$ can be reduced to learning the Q-function;
this enables us   to take advantage of existing policy evaluation algorithms.
Now we combine $\{\wh{g}^{(t)}(S_k, A_k)\}_{k=t}^h$ to build the CV for $G_t$. Because from \cref{sec:toy example} these terms do not interfere with each other, we can simply add them together, yielding
TrajCV:
$\sum_{k=t}^h \wh{g}^{(t)}(S_k, A_k)$.
Equivalently, we have derived a difference estimator:
\begin{align} \label{eq:traj-wise difference estimator}
{\wt G^{\mathrm{Traj}, \wh q}}_t &:= \textstyle G_t - \sum_{t = 1}^h \paren{\wh{g}^{(t)}(S_k, A_k) - \Exp_{A_k|S_k} [\wh{g}^{(t)}(S_k, A_k)]}
\nonumber \\
&= \textstyle G_t - \sum_{k=1}^h \paren{N_t \wh Q_k - \Exp_{A_k|S_k} [ N_t \wh Q_k]}
\end{align}
Comparing TrajCV in \eqref{eq:traj-wise difference estimator} and state-action CV in \eqref{eq:state cv}, we see that the state-action CV only contains the first term in the summation of TrajCV.\footnote{For brevity, we may use CV to mean the difference estimator of that CV when there is no confusion.}
The remaining terms, \ie $N_t (\wh Q_k - \Exp_{A_k|S_k} [\wh Q_k])$, for $t  < k \le h$, can be viewed as multiplying $N_t$ with  estimates of future advantage functions.
Appealing to law of total variance,
$\mathrm{Var} [{\wt G^{\mathrm{Traj}, \wh q}}_t]$ can be decomposed into
\begin{footnotesize}
\begin{align} \label{eq:traj variance}
\hspace{-3mm}
&\mathrm{Var}_{S_t}  \Exp_{|S_t} \br{ N_t C_{t:h}}+
\Exp_{S_t}  \mathrm{Var}_{A_t|S_t} [  N_t (\Exp_{|S_t, A_t} \br { C_{t:h}} - \wh{Q}_t)] + \\[-1mm]
&\sum_{k=t}^{h}
\Exp_{S_{k}, A_{k}}\mathrm{Var}_{S_{k+1}|S_k, A_k} \br { N_t \Exp_{|S_{k+1}}[ C_{k:h}] }
+
\sum_{k=t}^h \Exp_{S_{k+1}}  \mathrm{Var}_{A_{k+1}|S_{k+1} } [ N_t ( \Exp_{|S_{k+1},A_{k+1}}[ C_{k:h}] - \wh{Q}_{k+1} )] \nonumber
\end{align}
\end{footnotesize}%
where we further decompose the effect of $\mathrm{Var}_{|S_t, A_t}$ in the second line
into the randomness in dynamics and actions, respectively.
Therefore, suppose the underlying dynamics is deterministic (\ie $\mathrm{Var}_{S_{k+1}|S_k, A_k}$ vanishes), and $\wh{q} = q^{\pi}$,
then using TrajCV \eqref{eq:traj-wise difference estimator}
would completely remove  $\mathrm{Var}_{A_t|S_t}$  and  $\mathrm{Var}_{|S_t, A_t}$, the latter of which previous CVs \eqref{eq:state cv} and \eqref{eq:state-action cv} cannot affect.
In \cref{fig:comparison},
we visualize the effect of TrajCV and state-action CV on each policy gradient component $G_t$, for $1 \le t \le h$:
state-action CVs only influence the diagonal terms, while TrajCVs are able to affect the full upper triangle terms.
Note that in implementation of TrajCV for $G_{1:h}$, we only need to compute quantities
$\wh Q_t$,  $\Exp_{A_t|S_t} [ \wh Q_t]$ and
$\nabla \Exp_{A_t|S_t}[\wh Q_t]$
along a trajectory (done in $O(h)$ time) and they can be used to compute $\{{\wt G^{\mathrm{Traj}, \wh q}}_t\}_{t=1}^h$ \eqref{eq:traj-wise difference estimator}.%
\footnote{
	We have
	$
	\Exp_{A_t|S_t} [N_t \wh Q_t]=
	\Exp_{A_t|S_t}  [\nabla \log \pi_{S_t} \wh Q_t] =
	\Exp_{A_t|S_t} \br{\frac{\nabla \pi_{S_t}(A_t)}{\pi_{S_t}(A_t)}\wh Q_t}

	= \nabla \Exp_{A_t|S_t} [\wh Q_t]
	$.

}
In addition, we remark that when $\wh{q}(s, a) = \wh{v}(s)$, TrajCV
reduces to the state-dependent CVs.

\begin{figure}[t!]

	\centering
	\begin{subfigure}{.45\textwidth}
		\centering
		\includegraphics[trim={4cm 10cm 10cm 6.5cm}, clip, width=0.9\textwidth]{bayes_net1}
		\caption{before policy reparameterization}
		\label{fig: bayes net 1}
	\end{subfigure}
	\begin{subfigure}{.45\textwidth}
		\centering
		\includegraphics[trim={4cm 10cm 10cm 6.5cm}, clip, width=0.9\textwidth]{bayes_net2}
		\caption{after policy reparameterization}
		\label{fig: bayes net 2}
	\end{subfigure}
	\caption{
		Bayes networks for the random variables in $G_t$ \eqref{eq:policy gradient}, before and after reparameterization. After policy is reparameterized, action $A_k$ is decided by state $S_k$ and action randomness $R_k$.
	}
\end{figure}

\subsection{The Natural Ordering in \eqref{eq:natural ordering} is Optimal}
\label{sec:optimal ordering}

Recall in \cref{sec:toy example} we mentioned that the admissible ordering of random variables used in invoking the law of total variance depends on the information available. Here we show that the chosen ordering \eqref{eq:natural ordering} is indeed, the best ordering to adopt, as we only know the policy, not the dynamics.

We compare \eqref{eq:natural ordering} against potential orderings constructed by reparameterizing the policy such that its randomness in action becomes \emph{independent of the input state}.
We suppose the policy $\pi \in \Pi$ can be reparameterized by a function $\omega: \Sc \times \Rc \to \Ac$ and a distribution $p_R$, so that for all $s \in \Sc$,  $\omega(s, R)$
and $\pi_s$ are equal.
This is not a restricted assumption.
For example,
a Gaussian stochastic policy with mean $\mu_\theta(s)$ and covariance $\Sigma(s)$, widely used in continuous domains~\cite{williams1992simple,schulman2015trust,baxter2001infinite}, can be framed as
$\omega(s, R) = \mu(s) +\Sigma^{1/2}(s) R $, where $R$ is multivariate standard normal.
Even when $\Ac$ is discrete, a soft-max layer can be applied after $\omega$ to obtain a Gibbs (or Boltzmann) distribution~\cite{landau1958statistical,sutton2000policy}.

Reparameterization gives rise to the chance of designing a larger family of
TrajCVs.
By extracting the randomness in action selection into the random variable $R_{t,h}$,  the policy gradient component $G_t$ becomes a function of $R_{t,h}$ and $S_{t,h}$, as depicted in
the Bayes network in
Figure \ref{fig: bayes net 2}.
When applying the law of total variance,  the ordering the random variables now can have many possibilities.
For instance, in the extreme case,  the randomness of actions can be ordered before states (except $S_t$) as
\begin{align} \label{eq:action randomness first ordering}
S_t \rightarrow R_{t} \rightarrow \dots \rightarrow R_{h} \rightarrow  S_{t+1}\rightarrow \dots \rightarrow S_h,
\end{align}
leading to a CV
that's a function of $R_{t,h}$
bearing the optimal choice
$\Exp_{S_{t+1, h} |R_{t,h}, S_t} \br {N_t C_{t:h}}$.
Note that $\Exp_{S_{t+1, h} |R_{t,h}, S_t} \br {N_t C_{t:h}}$ is a function that inputs the observable action randomness $R_{t,h}$, not the randomness of the unknown dynamics.
Therefore, it can be approximated, \eg, if we have a biased simulator of the dynamics.\footnote{We sample all the action randomness $R_{t,h}$ first, execute the policy $\pi$ in simulation with fixed randomness $R_{t,h}$, and then collect the statistics $N_t C_{t:h}$.}
One might ask, given all possible orderings of random variables, which ordering we should pick to design the CV.
Interestingly, to this question, the most natural one and the optimal one coincide. The proof is deferred to \cref{app:proof}.
\begin{theorem} \label{thm: optimal ordering}
Suppose that policy specified by $\omega$ and $p_R$ is known, but the dynamics $d$ is unknown.
Assume the optimal CV of a given ordering of random variables $S_{t,h}$ and $R_{t,h}$ can be obtained.
The the optimal ordering that minimizes the residue variance is the natural ordering~\eqref{eq:natural ordering} .
\end{theorem}

Theorem~\ref{thm: optimal ordering} tells us that
if the optimal CVs are attainable (\ie we can estimate the Q-function exactly), then the natural ordering is optimal.
However, in practice, the CVs are almost always suboptimal due to error in estimation.
If the dynamics is relatively accurate and the computing resources for simulation are abundant, then although the residue is higher, the ordering \eqref{eq:action randomness first ordering} could actually be superior.
Therefore, the ordering of random variables based on the relative accuracy of different estimates is an interesting practical question to pursue in future work.
In the experiment section of this work, we will focus on the natural ordering \eqref{eq:natural ordering}.

\section{Experimental Results and Discussion}
\label{sec:result}

Although the focus of this paper is the theoretical insights, we illustrate our results with experiments in learning neural network policies to solve the CartPole balancing task in OpenAI Gym~\citep{brockman2016openai} with DART physics engine~\citep{Lee2018}.
In CartPole,
the reward function is the indicator function that equals to one when the pole is close to being upright and zero otherwise.
This is a delayed reward problem in that the effective reward signal is revealed only when the task terminates prematurely before reaching the horizon, \ie when the pole deviates from being upright.
The start state is perturbed from being vertical and still by an offset uniformly sampled from $[-0.01, 0.01]^{d_\Sc}$, and the dynamics is deterministic.\footnote{Symbol $d_\Sc$ denotes the dimension of $\Sc$.}
The action space is continuous and Gaussian policies are considered in the experiments.
The policy is optimized by natural gradient descent \cite{kakade2002natural} with a KL-divergence safe guard on the policy change to be robust to outliers in data collection. Below we report in rewards, negative of costs, which is the natural performance measure provided in OpenAI Gym.

\begin{figure}[t!]
	\centering
	\begin{subfigure}{.24\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{std_0-1}
		\caption{$\sigma = 0.1$}
		\label{fig:std_0-1}
	\end{subfigure}
	\begin{subfigure}{.24\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{std_0-3}
		\caption{$\sigma = 0.3$}
		\label{fig:std_0-3}
	\end{subfigure}
	\begin{subfigure}{.24\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{std_1-0}
		\caption{$\sigma = 1.0$}
		\label{fig:std_1-0}
	\end{subfigure}
	\begin{subfigure}{.24\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{std_3-0}
		\caption{$\sigma = 3.0$}
		\label{fig:std_3-0}
	\end{subfigure}
	\caption{
		The size of components of $\Tr \paren{\mathrm{Var}[G_t]}$ during training, for $t = 100$, evaluated at policies generated under the ``upper bound'' setting that emulates noiseless estimates for the CartPole problem with horizon $h = 1000$.
		Different initial values for $\sigma$ of the Gaussian policy (defined in \cref{th:varaince size}) are used.
		The $x$ axis denotes the iteration, and the $y$ axis is in $\log$ scale.
		The two vertical dashed lines mark the boundaries of iterations where the expected accumulated rewards is between $50$ and $900$.
	}
	\label{fig:sigmas}

\end{figure}

In \cref{fig:sigmas}, we corroborate the theoretical findings in \cref{th:varaince size} by empirically evaluating the components of
$\Tr\paren{\mathrm{Var}[G_t]}$ during learning, for $t = 100$.
The learning process on CartPole can be partitioned into \emph{three stages}:
1) initial exploration, when the policy performs very poorly and improves slowly,
2) rapid improvement, when the policy performance increases steeply,
3) near convergence, when the policy reaches and stays at the peak performance.
The dashed vertical lines in  \cref{fig:sigmas}  delimit these three stages.
In the rapid improvement stage, due to the variance in the accumulated rewards, \ie length of the trajectories, $\V_{|A_t, S_t}$ is large, close to $\V_{A_t|S_t}$ and about 10 times of $\V_{S_t}$
as predicted by \cref{th:varaince size}.
Furthermore, as $\sigma$ increases, the gap between the peak values of $\V_{S_t}$ and $\V_{A_t|S_t}$ narrows.

\begin{figure}[t]
	\centering
	\begin{subfigure}{.24\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{1k_mc}
		\caption{$h = 1000$ (MC)}
		\label{fig:cp1k}
	\end{subfigure}
	\begin{subfigure}{.24\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{2k_mc}
		\caption{$h = 2000$ (MC)}
		\label{fig:cp2k}
	\end{subfigure}
	\begin{subfigure}{.24\textwidth}
		\centering
		\includegraphics[ width=1.0\textwidth]{4k_mc}
		\caption{$h = 4000$ (MC)}
		\label{fig:cp4k}
	\end{subfigure}
	\begin{subfigure}{.24\textwidth}
		\centering
		\includegraphics[ width=0.8\textwidth]{mc-legend}
		\label{fig:cplegend}
	\end{subfigure}
	\caption{
		The  results of naive Monte Carlo estimate of policy gradient, and it augmented with state  CV, state-action CV, and TrajCV on CartPole problems with horizon $h=1000, 2000, 4000$.
		``MC" in the legend indicates that
		\emph{the expectation over actions for state-action CV and TrajCV is approximated using Monte Carlo (1000 samples)}.
		Upper bound emulates the results of noiseless estimates. The $x$ and $y$ axes are iteration number and accumulated rewards (the higher the better), respectively.
		The median of 8 random seeds is plotted, and the shaded area accounts for 25\% percentile.
	}
	\label{fig:cps}

\end{figure}

\begin{figure}[t]
	\centering
	\begin{subfigure}{.24\textwidth}
		\centering
		\includegraphics[width=1.0\textwidth]{4k_quad_diff}
		\caption{$h = 4000$ (diff)}
		\label{fig:cp4k-diff}
	\end{subfigure}
	\begin{subfigure}{.24\textwidth}
		\centering
		\vspace{-1em}
		\includegraphics[width=0.88\textwidth]{quad-diff-legend}

		\label{fig:cp4k-diff-legend}
	\end{subfigure}
	\begin{subfigure}{.24\textwidth}
		\centering
		\includegraphics[ width=1.0\textwidth]{4k_quad_next}
		\caption{$h = 4000$ (next)}
		\label{fig:cp4k-next}
	\end{subfigure}
	\begin{subfigure}{.24\textwidth}
		\centering
		\vspace{-1em}
		\includegraphics[ width=0.88\textwidth]{quad-next-legend}

		\label{fig:cp4k-next-legend}
	\end{subfigure}
	\caption{
		The exact same settings as \cref{fig:cps} except that
		\emph{the state-action CV and TrajCV are given by $\wh{q}^{(\text{diff})}$ and $\wh{q}^{(\text{diff-GN})}$  (\cref{fig:cp4k-diff}), and  $\wh{q}^{(\text{next})}$ and $\wh{q}^{(\text{next-GN})}$ (\cref{fig:cp4k-next})}.
	}
	\label{fig:cp4k-more}

\end{figure}

We compare naive Monte Carlo estimate \eqref{eq:policy gradient}, state dependent CV \eqref{eq:state cv}, state-action CV \eqref{eq:state-action cv}, and \mbox{TrajCV} \eqref{eq:traj-wise difference estimator}, in \cref{fig:cps} and \cref{fig:cp4k-more}.
To facilitate a fair comparison, we realize all algorithms with the same implementation
based on an on-policy value function approximator $\wh{v}$, so that the effects of value estimates' quality can be normalized.
 %
More concretely, $\wh{v}$ is learned by sampling abundant data from a biased dynamics model (which is obtained by perturbing the underlying physical parameters).

To construct the Q-function approximator $\wh q$ required by the state-action CV and TrajCV,
we further train a deterministic function $\wh d$ that maps the current state and action to next state using the same data from the environment that are used for computing the policy gradient estimates.
Combining the value function approximator $\wh v$ and the dynamics approximator $\wh d$ yields a natural Q-function approximator
$\wh{q}^{\text{(dyn)}}(s, a) = c(s, a) + \wh{v}(\wh{d}(s,a))$.
Based on this basic $\wh{q}^{\text{(dyn)}}$, we explored several options
of Q-function approximator for defining the state-action CV and TrajCV:
\begin{enumerate}
	\item Monte Carlo (MC) :  $\wh{q}^{\text{(dyn)}}(s,a)$. We use many samples of actions (1000 in the experiments) to approximate $\Exp_{A_t|S_t}  \br{\wh q^{\text{(dyn)}}(S_t, A_t)}$.
	To reduce variance,
	we use the same action randomness for different steps, \ie using the same 1000  i.i.d. samples from $p_R$ (defined in \cref{sec:optimal ordering}) in the evaluation for  $\Exp_{A_t|S_t}$ with different $t$.
	\item
	We also consider various  Q-function approximators that are quadratic in action, so that $\Exp_{A_t|S_t}$ can be evaluated in closed-form. They are 	derived by different linearizations of the Q-function approximator $\wh q$ as shown below.
	\begin{enumerate}
		\item
		$\wh{q}^{(\text{next})}(s, a) = c(s, a) + \wh{v}(\wh{s}') + (a-m)^\top \nabla_m \wh{d}(s,m) \nabla \wh{v}(\wh{s}'), $
		\item $\wh{q}^{(\text{next-GN})}(s, a) = \wh{q}^{(\text{next})}(s, a) + \frac{1}{2} (a-m)^\top \nabla_m \wh{d}(s,m) \nabla^2 \wh{v}(\wh{s}') \nabla_m \wh{d}(s,m)^\top (a-m), $
		\item $\wh{q}^{(\text{diff})}(s, a) = \wh{v}(s)  + (a-m)^\top \nabla_m (c(s, m) + \wh{v}(\wh{s}')) + \frac{1}{2}(a-m)^\top \nabla_m^2 c(s,m) (a-m),$
		\item $\wh{q}^{(\text{diff-GN})}(s, a) =\wh{q}^{(\text{diff})}(s, a) + \frac{1}{2} (a-m)^\top \nabla_m \wh{d}(s,m) \nabla^2 \wh{v}(\wh{s}') \nabla_m \wh{d}(s,m)^\top (a-m), $
	\end{enumerate}
	where $m = \mu_\theta(s)$ is the mean of the Gaussian policy, $\wh{s}' = \wh{d}(s,m)$,  and ``GN'' stands for Gauss-Newton.
	We assume $c(s,a)$ is quadratic in $a$ for $\wh{q}^{(\text{next})}$ and $\wh{q}^{(\text{next-GN})}$.
\end{enumerate}
The performance of different CVs using
MC for approximating $\Exp_{A_t|S_t}$ is reported in \cref{fig:cps}, where 5 rollouts are sampled for each iteration. We also emulate  noise-free gradients, denoted as upper bound, which is constructed by running the state dependent CV but with $100,000$ samples per iteration.\footnote{For the usual learners, the number of samples collected per iteration is often less than $5h$, and much less at the start of learning, because of early termination when the agent fails.}
Overall when more information is used to design the CVs (from state only, state-action, and then trajectory-wise) the convergence speed improves. In particular, as the problem horizon becomes longer, the gap becomes larger: the reward feedback becomes sparser, so the variance due to long-term trajectory starts to dominate, as shown in \cref{fig:cp4k}.
In \cref{fig:cp4k-more},
we observe that when $\Exp_{A_t|S_t}$ is approximated analytically using various techniques ($\wh{q}^{(\text{next})}$ and  $\wh{q}^{(\text{next-GN})}$ are the CVs suggested in~\cite{gu2016q} and \cite{pankov2018reward}, respectively), learning is accelerated when more information is leveraged in CV synthesis.

These preliminary experimental results support the theoretical insights provided in \cref{sec:why we need new cvs} and \cref{sec:trajectory}, suggesting the importance of considering long-term effects in designing CVs, especially for problems with a long horizon. The fix turns out to be quite simple: just padding additional terms (cf. \eqref{eq:traj-wise difference estimator}) onto the existing CVs, which can be done using Q-function approximators available in existing CVs without  new information. Interestingly we prove this simple idea is optimal.
 Important future work includes considering the different bias and variance trade-off discussed in \cref{sec:optimal ordering}, and learning the linear combination weights of the CVs for policy gradient components $\{G_t\}_{t=1}^h$.

\acknowledgments{This work was partially supported by NSF CAREER award 1750483.
}

\clearpage

redpajama_set_name: RedPajamaArXiv


text:
Q: Как сделать чтобы скрипт применялся к одному id Есть скрипт параллакс эффекта при движении мыши. Подскажите, как сделать, чтобы данный скрипт применить только к одной секции id (одному идентификатору) ??
let bg = document.querySelector('.mouse-parallax-bg');
    window.addEventListener('mousemove', function(e) {
        let x = e.clientX / window.innerWidth;
        let y = e.clientY / window.innerHeight;
        bg.style.transform = 'translate(-' + x * 20 + 'px, -' + y * 20 + 'px)';
    });

A: Вопрос понятен не до конца, но есть предположение, что Вам может подойти

*

*querySelectorAll('.mouse-parallax-bg')[i], где i - число, означающее позицию нужного Вам элемента.

*getElementById('id') вместо querySelector('.mouse-parallax-bg'), но в таком случае у всех нужных элементов нужны id

UPD учитывая комментарий:
Тогда наверное можно взять по id сначала через getElementById('id'), а потом обратиться с поиском не к document, а к результату по id
Что-то вроде document.getElementById('stages').querySelector('.mouse-parallax-bg') должно получиться

redpajama_set_name: RedPajamaStackExchange


Daylight Savings in the spring is bittersweet, isn't it? It means spring is just about here and it will stay light out later at night. But we lose an hour of sleep. Hey, I'll adjust if it means this cold weather is on its way out! To help ease the transition Krispy Kreme is offering up a FREE glazed doughnut on March 10 only. YUM!
This offer is good for one per person at participating locations. Enjoy!


text:
Q: Makefile: 23: the recipe for the target "allmodes" failed We have an issue installing OMNeT 4.6; everything proceed well but in the final part we have this error message:

Makefile: 23: the recipe for the target "allmodes" failed

The system doesn't ask us to tape OMNeT (like it should be). We taped it anyway and everything proceeded okay.
After that, we tried to install Castalia, but (like we had an error installing OMNeT - at least this is what we suppose-) we have another error message about liboppsim ( & we suppose that the cause is the failed OMNeT installation).
We tried the install the 4.6 version because Castalia doesn't support the OMNeT v5.x
any help guys please??
thanks in advance
OS : Windows 10
Error msg in terminal
Error in app

A: Regarding the error: you should successfully build and test OMNeT++ before proceeding to install any model. You screenshot shows only, that there was an error, but the important error messages are not shown at all.
As for Castalia compatibility, here is a port that works with the latest version of OMNET: https://github.com/rhornig/Castalia/tree/topic/omnetpp54-compatibility

A: Note that some people have ported Castalia 3.3 to OMNeT 5.x and they report it works on Windows 10. More info found here:
https://github.com/boulis/Castalia/issues/12
In general, Castalia 3.3 is not tested on Windows 10 (irrespective of the OMNeT version), so you are treading on unknown territory, even if some people have successfully used Castalia on Windows. If you do not want to bother with these complexities you could use VirtualBox to run an Ubuntu system.


Commitment takes courage. I began my Kundalini Yoga training in a small village in Israel. After the first week, my Visa expired and I had to return home to Florida. I thought I'd be going back to complete my training but with little money and no job, I didn't know how I was going to get there.
In the past, with my relationships, jobs, and school, I felt overwhelmed when I had to commit to something. Commitment seems so long-term, and to me, I feared the responsibility that came with commitment. Then, I realized that commitment didn't have to be so scary if I committed to my highest good. So, I trusted that with the power of a spiritual practice, I'd be able to let go of the "how" and listen to my inner guide.
Today people follow what society tells us is normal instead of listening to their internal guide. It is normal to commit to something that is often what your parents or friends wanted for you.
Usually, when people wake up in the morning they commit to their phones or their social media accounts. With the convenience of devices, people are under an illusion that they need this device from the moment they wake up. Technology takes our attention away from the present moment.
Directing my attention to something that spiritually aligned me in my morning allowed me to build the courage to align myself in other areas of my life. It turned out that there was another Kundalini Yoga teacher training course happening at the exact same time as the one I was taking in Israel, in Orlando where I was living. Instead of going back to Israel, I transferred my training over to the Orlando course and completed it a few months later. When this synchronicity happened in my life, I opened myself up to the power of universal flow.
It takes courage to wake up each morning, no matter how tired you are, and commit to a spiritual practice.
It takes courage to let go of what society thinks is normal and create your own normal.
It takes courage to trust in the path, no matter how many detours there are.
It takes courage to be yourself.
I learned at a young age to live by my spiritual name: Nirbho (Fearless) Atma (Soul). I encourage you to trust in yourself and do what it takes for you to commit to your spiritual essence. From that place, all else will align.


text:
Vi har bytt namn och varumärke från Aurora Music till Old Ox Music, mer info här
Old Ox Music LIVE
Biljetter – Festivaler
Old Ox Solo Classes
Antonio Meneses
Born in Recife, Brazil in 1957 into a family of musicians, Antonio Meneses began his cello studies at the age of ten. At the age of 16, he met the famous Italian cellist Antonio Janigro and was asked to join Janigro's classes in Düseldorf and later in Stuttgart. In 1977, Antonio won the first Prize at the ARD International Competition in Munich and in 1982; he was awarded first Prize and gold Medal at the Tchaikovsky Competition in Moscow.
Regularly appearing in the music capitals of Europe, the Americas, and Asia, Antonio Meneses has performed with most of the world's leading orchestras such as the Berlin Philharmonic Orchestra, London Symphony, BBC Symphony, Concertgebouw, Vienna Symphony, the Czech Philharmonic, the Moscow, the Saint Petersburg Philharmonic, the Israel Philharmonic, Orchestre de la Suisse Romande, the Bayerische Rundfunk Orchestra, New York Philharmonic, National Symphony Orchestra (Washington D.C.), in Buenos Aires, in Warsaw, in Brazil, and the NHK Symphony Orchestra in Tokyo. Among the conductors with whom he has collaborated are Herbert von Karajan, Riccardo Muti, Mariss Jansons, Claudio Abbado, Andre Previn, Andrew Davis, Semyon Bychkov, Herbert Blomstedt, Gerd Albrecht, Yuri Temirkanov, Kurt Sanderling, Neeme Järvi, Mstislav Rostropovitch, Vladimir Spivakov, and Riccardo Chailly.
Antonio is also a frequent guest at many important music festivals, including Puerto Rico (Festival Pablo Casals), Salzburg, Lucerne, the Vienna Festwochen, the Berlin Festwochen, the Prague Spring Festival, New York (Mostly Mozart Festival), Seattle, la Grange de Meslay, the Festival de Colmar, and the Jerusalem Chamber Music Festival.
A devoted chamber music performer, Antonio Meneses has collaborated with the Emerson Quartet and the Vermeer Quartet on tour as well as with pianists such as Nelson Freire, Menahem Pressler, Cristina Ortiz, and Gérard Wyss. Antonio has been a member of the Beaux Arts Trio since 1998. (beauxartstrio.org)
As a recording artist, Antonio Meneses made two recordings for Deutsche Grammophon with Herbert von Karajan and the Berlin Philharmonic Orchestra: Brahms' Double Concerto for Violin and Cello with Anne Sophie Mutter and Richard Strauss' "Don Quixote". Antonio has also recorded the D'Albert Concerto, works of David Popper—both with the Basel Symphony Orchestra—and Cello Concertos by Carl Philip Emanuel Bach with the Munich Chamber Orchestra (Pan Classics), The Six Suites of Bach (Nippon Phonogram), Tchaikovsky Piano Trio (EMI/Angel), Concertos and the Fantasy for Cello and Orchestra by Heitor Villa-Lobos (Auvidis France), complete works for Cello and Piano by Villa-Lobos with Cristina Ortiz, and Encore pieces with Gérard Wyss (Pan Classics), The Six Cello Suites by J.S. Bach (AVIE), and his most recent recording Schumann/Schubert (AVIE) was released in 2006. (www.avierecords.com)
In addition to a busy concert schedule, Antonio Meneses gives master classes in Europe, the Americas, and in Japan. Antonio will start teaching at Hochschule der Künste Bern in 2007. (www.hkb.bfh.ch)
Antonio Meneses plays ALESSANDRO GAGLIANO cello made in Naples ca. 1730.
NORDISKA KAMMARMUSIKFÖRENINGEN (NKMF)/Old Ox Music
Björngårdsgatan 17,
info@oldox.se
Om Old Ox
Om Old Ox Music
Organisation/Kontakt
Pressrummet
Organistaion/Styrelse
© Old Ox Music 2020
Leaders and Soloists
About Old Ox Music
Organisation/Contact

redpajama_set_name: RedPajamaCommonCrawl


text:
Julia Stiles brings "reckless determination" to her lead role in "Riviera" series
Actress Julia Stiles, who you'll remember from her teen movie roles in "Save the Last Dance" and "10 Things I Hate About You," and more recently, "Silver Linings Playbook," the Jason Bourne movie series and "Dexter," stopped by "Salon Talks" to discuss her lead role in "Riviera," now in its second season on Sundance Now on June 20.
Stiles plays Georgina, the rich widow of a Greek billionaire with a mystery on her hands. Full of everything today's short attention span audiences need to be captivated, "Riviera," set in the South of France, has beautiful people, gorgeous scenery, deceit, crime, an unsolved murder, and high fashion.
"Part of me was like, 'Ooh, the South of France, fancy dresses, high heels, sign me up,' but really, I was struck by this character because she is the only American in the show and she doesn't come from wealth," Stiles told SalonTV's Alli Joseph. "She starts out as a kind of window into this world and the eyes for the audiences of people who can't relate to people who have helicopters and have fancy yachts and drivers, but she becomes corrupted by this world."
After Stiles' character Georgina loses her billionaire husband Constantine Clios in a yacht explosion during season 1, she is shocked to discover the fortune that Clios had is, despite appearances, married by dishonesty, secrets, crime, and murder. Georgina's despair leads her to commit a murderous act herself, thereby drawing her into the intrigue further.
"She's doing the best that she can, and an otherwise good person, who does something that she's not proud of, but now is trying to figure out what she's going to do about it," Stiles said.
"I ask myself in the middle of a scene, why does this girl stay here...in the Riviera? Cause Julia, me, I would get on a plane to New York and get out of this family! But what appeals to me about her is that she's kind of reckless in her determination to keep things together, and keep her life in order the way she remembers it."
Stiles, known for playing strong, tough and resilient female characters, says she doesn't always realize her work is characterized that way, but appreciates the distinction. "One thing that is challenging about Riviera is showing Georgina's vulnerability, because she's not a sociopath, and she's not a stone-cold emotionless woman. For us, the challenge was how do we get inside of her heart and her inner world?"
Watch the episode above to hear more about what's in store for season 2 of "Riviera" and a preview of Stiles' next film, "Hustlers," where she stars opposite Jennifer Lopez and Cardi B, and plays the journalist who uncovers the story about a group of strippers who plan to steal tens of thousands of dollars from their Wall Street clients.
More From This Week on SalonTV
Kathy Griffin is back with a vengeance: "I have zero f**ks left to give"
Pastry chef Nicole Rucker shares the art of a good pie
Trendsetting fashion designer Dapper Dan defines style
GET THE BEST OF SALON
By signing up you agree to receive email newsletters or alerts from Salon.com. You may unsubscribe at any time.
Salon Talks Clips
Why Sherri Shepherd is still so thankful for "The View," five years after leaving the show
Democrat Amy McGrath is "hugely disappointed" with McConnell's Senate record
CNN's Jim Acosta on how he plans to cover Trump's 2020 campaign
Trump's war on "fake news" was just a taunt, until it changed politics


This was one tough 600-piece puzzle. My husband mostly wrestled with it. Many different sized pieces, and lots of pieces that didn't interlock. Then there was that piece leftover....hmm. It was a duplicate piece. What a trick!
That's simply not fair. How do you know you used the right duplicate?
I fear VP is right. Ack!!! Good job for you two, though.
A duplicate? Too bad someone doesn't get an extra bit of real land that easily!!


text:
11/19: Visionary Leadership, Dedicated Staff Help Grow FCA
As part of FCA's 30 Days of Caregiving blog during National Family Caregivers Month, former FCA staff members were asked to be guest bloggers as part of a unique "Throwback Thursday" series. They were asked to address how FCA has influenced their work in their current positions, what they learned at FCA that they have carried forward, and if they were King or Queen for a day, what they would make happen for family caregivers. FCA is pleased that so many of its former staff members, wherever they may be, have remained in the fields of health care, social services, research and public policy, and still are advocates for caregiving families.
I began working at Family Caregiver Alliance (although it wasn't called that then) in 1988. I had spent several years in publishing and public relations, and one of my clients was Family Survival Project, the organization's original name. I quickly realized how much I preferred working in the nonprofit sector, and when I was invited to join the staff in the newly-created position of Public Information Officer (eventually Manager of Communications and Media Relations) it was easy to say yes. My charge was to help articulate and increase public awareness of the critical services family caregivers provide.
The issues were clear to me, illuminated not only by the founders' stories of their struggles as caregivers, but also by personal experiences, as my relatives dealt with stroke and Alzheimer's, and a close friend suffered a brain hemorrhage at a young age, leaving her husband in the unexpected, overwhelming role of caring not only for his wife, but for their infant son. These families desperately needed support and services.
It seemed then that we could tell the future: we looked at the demographics, at the aging of the Baby Boomers and the increased longevity of our parents. It was clear 30 years ago, and it's clear now, that there would be a wave of care needs unlike anything seen before.
As the years passed, this grass-root project, an initiative of the San Francisco Mental Health Association, evolved and expanded guided by visionary leadership. After several years and much discussion, we changed the organization's name to Family Caregiver Alliance to better reflect its mission and impact. The decades saw impressive growth in its size, scope and influence:
California's innovative system of Caregiver Resource Centers (CRCs) was established with state legislative support. Each Center's services were modeled on FCA's, and the CRCs served every county in the state.
Later, the National Center on Caregiving was established at FCA, highlighting the organization's evolving national role. Management staff regularly received requests to speak at conferences and legislative hearings.
FCA was one of the first social service agencies with a presence on the web. We established online support groups open to anyone in the country, in addition to online support for clients of the CRCs. A virtual library of FCA publications was posted on the website, free to anyone who needed the information.
The award-winning FCA newsletter grew from a small local publication to one with a national, then international, subscriber base.
Local weekend camps for people with Alzheimer's began, as did popular retreats for caregivers.
Early on, we talked and wrote a lot about caregiving as a workplace issue. We were instrumental in the development and passage of California's Paid Family Leave Act, and it was an interesting experience to see how the initiative was molded and compromised on its way to passage. (And for me, a touch of Hollywood—actor/director Rob Reiner was at the press conference!)
It's common now to see stories on caregiving in the media, but in the organization's early days, it took a lot of educating to raise reporters' awareness of the complexities and breadth of caregiving. Now reporters call the office weekly, sometimes daily, for information, interviews and resources.
Publications like the FCA Fact Sheets were (and are) recognized as offering reliable information and advice on a multitude of topics in a multitude of languages. They're widely used by caregivers and social service providers, but also in legislative, medical, academic, corporate and research settings.
Many issues still remain of course, but they are gaining traction at national levels: paid family and medical leave for all Americans; work/family/caregiving balance; pay for family caregivers; Social Security credits for caregivers; adequate training for families when a loved one is discharged from the hospital with complicated care needs; recognition by medical staff of the important role families and caregivers play in healthcare. (Even recently [11/10/15], Gloria Steinem was on NPR mentioning the important role of family caregivers!)
When I left formal employment with FCA in 2009, it was with a profound appreciation for how this issue touches all of us in one way or another, and how much there is to learn.
At that time I was offered an interesting opportunity to explore a very different side of aging: as Communications Manager with Coming of Age, a nonprofit that worked to capture the talents and energy of people aged 50+ and to promote volunteer work and lifelong learning among active retirees. I took that message to heart, and after three years, left full-time work to pursue other interests and projects.
Still, my relationship with FCA has continued (they say no one ever entirely leaves!). I have recently participated in a variety of interesting projects and publications, including, with FCA's Operations Director Leah Eskenazi, writing a series of columns on long-term caregiving for PBSNewshour.com.
The relationship continued in another way as well: my husband and I became caregivers for my mother-in-law with Alzheimer's, who died at age 96, and for my mother, who recently passed away at age 91. Any time we had questions on caregiving, on assisted living, on memory impairments…we knew exactly whom to call: the smart, dedicated and responsive staff at FCA. With all the changes of the past decades, one thing has remained constant and set the agency apart: the professional skills of its knowledgeable staff.
— Bonnie Lawrence, Principal, Lawrence Communications, providing writing,
editing and PR consultation to nonprofit organizations
#30DaysCaregiving
#CaregiversMonth
#NationalFamilyCaregiversMonth
View all 30 Days of Caregiving blogs (to date) at caregiver.org/blog.
The Quiet Seasons of Life
Autumn, that time of year when there is a chill in the air and the evening sky appears gray with strokes of purple and orange painted through it. A time when the horizon glows with the vivid colors of red, orange and gold much like the colors on an artist's palate.


[S508] Evelyn Beran, family tree titled "Sanford-Shulsen Family", published by Ancestry.com, wc.rootsweb.ancestry.com, from database named sanford-shulsen, updated Nov 2006, viewed Dec 2007.
***Research Conclusion: Obadiah is NOT the son of Robert #7243 & Mary #7246.
[S600] Don Lindsay, family tree titled "Lindsay/Methvin Ancestors", published by Ancestry.com, wc.rootsweb.ancestry.com, from database named lindsrels, updated Sep 2004, viewed Dec 2007.
[S601] Shannon Jackson, family tree titled "Shannons Family Tree", published by Ancestry.com, wc.rootsweb.ancestry.com, from database named liberty_68123, updated Jul 2004, viewed Dec 2007.
[S602] Kacy Davis, family tree titled "ReVeal - Davis Family Tree Nov 2007", published by Ancestry.com, wc.rootsweb.ancestry.com, updated Nov 2007, viewed Dec 2007 , Not Available as of Mar 2015.
[S508] Evelyn Beran, family tree titled "Sanford-Shulsen Family", published by Ancestry.com, wc.rootsweb.ancestry.com, from database named sanford-shulsen, updated Jun 2009, viewed Aug 2009.
[S615] Susan Wycoff, family tree titled "Wyckoff - Palen - Sexton - Nevius - Sanford - Tilton - Chambers - Ewing - Williams - Doudna", published by Ancestry.com, wc.rootsweb.ancestry.com, from database named srw123, updated Nov 2007, viewed Jan 2008.
Abigail Holt married Nathaniel Barnes, son of Thomas Barnes and Abigail Frost, on Monday, 22 Mar 1744/45 in New Haven, Connecticut.
[S508] Evelyn Beran, family tree titled "Sanford-Shulsen Family", published by Ancestry.com, wc.rootsweb.ancestry.com, from database named sanford-shulsen, updated Nov 2006, viewed Jan 2008.
[S621] Jennifer Yoch, family tree titled "Jensen Family", published by Ancestry.com, wc.rootsweb.ancestry.com, from database named j_yoch, updated Jun 2004, viewed Jan 2008.


text:
Exploration, Adventures, and Maps
Fastfood Undead
Houses of the Hallows
California Fool's Gold
Greater Streets
High Rising
Homes Fit for Heroes
Mini-Mallism
Mist & Iron
No Enclave
Nobody Drives in LA
Office Park Life
Pan-Asian Metropolis
Pendersleigh & Sons Cartography
Red Deserts
Southland Parks
Take 'em to the Bridge
Those Useless Trees
Urban Rambles
There It Is, Revitalize It
Where Fools Fear to Tread
Radio, Podcasts, & Television
Aesthetes, Artists, Dandies, Decadents, Poets, Style Tribes, & Symbolists
BUY MAP MERCHANDISE AND PRINTS
Pan-Asian Metropolis — Asian-American Murals in Los Angeles
26/05/2016 08/07/2016 ~ Eric Brightwell
The landscape of Los Angeles is full of murals. Occupying as they do public rather than private space, their creators have to assume at their creation that their existence is temporary. They're rarely respected by artless taggers and once sufficiently damaged, graffiti removers usually obliterate them with their own artless paint jobs. Increasing they're annihilated by community colonizers, who move into old businesses and whose bourgeois aesthetic sensibilities prefer blank walls to murals. The lucky ones crack and fade in the sun unless they're preserved or restored, which occasionally they are.
Because Los Angeles is the world's great pan-Asian city and because it's currently Asian Pacific American Heritage Month, I thought it would be nice to highlight the murals created by APA artists. I've listed and mapped the works of which I'm aware but I'd love to add more, whether they're commissioned artworks, street art, commercial murals, &c. So please leave suggestions in the comments.
Chinese Celestial Dragon — Tyrus Wong, 1941 (951 N. Broadway)
The Chinese Celestial Dragon mural was painted in 1941 by artist Tyrus Wong, a Chinese-born artist and kite maker celebrated for his work on the Disney film, Bambi (1942). In 2015, when he was 104, he was the subject of a biographical documentary titled Tyrus. The mural is located on a tower at New Chinatown Center Plaza and was restored by Fu Ding Cheng in 1984. It's served by Metro's Gold, 28, 45, 83 and LA DOT's DASH Downtown B lines.
Epoch — Patrick Nagatani, 1996 (800 N. Alameda St.)
Image: DTLA Rendezvous
Patrick Nagatani's collage Epoch (1996) depicts ships, trains, and other forms of transit. It also includes a series of Eadweard Muybridge photos from 1887, which though safe enough for Victorians proved so offensive to some in the 1990s that they were briefly covered with a plastic tarp. Epoch is located inside Union Station and is thus served by all of the transit lines which pass through Union Station.
Gintong Kasaysayan, Gintong Pamana — Eliseo Silva, 1995 (1600 Beverly Blvd.)
Image: a.k.a. DJ AFOS
Eliseo Silva's large Gintong Kasaysayan is located in Unidad Park, which began life as the the Candy Chuateco Community Garden in 1993. The mural and park are served by Metro's 14/37 line.
Home is Little Tokyo — Tony Osumi, Sergio Diaz, and Jorge Diaz, 2005 (S. Central Avenue & E. 1st St.)
The mural Home is Little Tokyo was designed by Tony Osumi with substantial input from the Little Tokyo community and depicts several eras and aspects of the neighborhood. It's served by Metro's Gold and LA DOT's DASH A lines.
Los Angeles Wholesale Produce Market mural — Thomas Suriya, 1986 (1601 E. Olympic Blvd.)
Image: Fine Art Conservation Lab
Thomas Suriya's two large murals at the Wholesale Produce Market cover 613 square meters and depict giant floating produce. Most produce in the Southwest passes through the Produce Market, a vast complex the new owners of which hope to transform into a mixed-use community known as Row DTLA. The market and mural are served by Metro's 53 and 66 lines.
The Party at Lan-Ting (Chinatown Public Library)
Shiyan Zhang's 1991 mural, The Party at Lan-Ting, is located on the exterior wall of Chinatown Public Library. The mural depicts 4th century calligrapher Wang Xi Zhi drinking wine, writing a poem, and hosting a party. It's served by Metro's Gold, 81, 90/91, 94, 96, and Rapid 794 lines as well as LA DOT's Commuter Express 419 line.
Picture of Viewing Waterfalls in Summer Mountains, Palace in Heaven, and Four Beauties Catching Swimming Fish, anonymous, 1968 (913 N. Broadway)
Three scroll-like tile murals, in three different styles, were created by an unknown artist in 1968 and adorn a wall of a building that's been home to Plum Tree Inn since 1979. The mural is accessible via Metro's Gold, 28, 45, 83, and LA DOT's DASH Lincoln Heights/Chinatown lines.
Sen Bana No Saki (Thousand Blossoms) — Kazuko Kayasuga Matthews, 1981 (400 E. 2nd St.)
Kazuko Kayasuga Matthews's Sen Bana no Saki mural adorns the western wall of Little Tokyo's Honda Plaza. The abstract designs represent emblematic Japanese family crests known as mondokoro (紋所) or kamon (家紋). It's served by Metro's Gold and LA DOT's DASH A lines.
Senzo — Jerry Matsukuma, 1981 (356 E. 1st St.)
Jerry Matsukuma's Senzo is a six meter tall tile mural that incorporates text and photography to relate the experience of Little Tokyo's Issei pioneers. It's served by Metro's Gold Line.
Shades of Chinatown — Steven Wong and others, 2003 (421 W. College St.)
Image: Mural Conservancy of Los Angeles
Shades of Chinatown was created by Steven Wong with help from the Youth Leadership Council of Chinatown and Lincoln Heights and is designed to portray Chinatown's diversity. It's served by Metro's Gold, 81, 90/91, 94, 96, and Rapid 794 lines.
Lost Asian-Angeleno murals include Dong-in Park's Koreans (1989 – 690 Wilshire Pl.) and Hitoshi Yoshida's Flight of the Angel (1989 – 407 E. Third St.). Carol Nye's Chinese American Women in L. A. (1994 – 711 N. Main St.), was taken down from Metro Plaza Hotel in 2000.
Special thanks: Public Art in LA
See also: Mural Conservancy of Los Angeles
Eric Brightwell is an adventurer, writer, rambler, explorer, cartographer, and guerrilla gardener who is always seeking writing, speaking, traveling, and art opportunities — or salaried work. He is not interested in writing advertorials, clickbait, listicles, or other 21st century variations of spam. Brightwell's written work has appeared in Amoeblog, diaCRITICS, and KCET Departures. His work has been featured by the American Institute of Architects, the Architecture & Design Museum, Boom: A Journal of California, the Craft & Folk Art Museum, Form Follows Function, Los Angeles County Store, Skid Row Housing Trust, and 1650 Gallery. Brightwell has been featured in the Los Angeles Times, Huffington Post, Los Angeles Magazine, LAist, Eastsider LA, Boing Boing, Los Angeles, I'm Yours, and on Notebook on Cities and Culture. He has been a guest speaker on KCRW's Which Way, LA? and at Emerson College. Art prints of his maps are available from 1650 Gallery and on other products from Cal31. He is currently writing a book about Los Angeles and you can follow him on Facebook, Instagram, and Twitter.
Click here to offer financial support and thank you!
Posted in Asian Pacific American, Pan-Asian Metropolis, Southland Art, Uncategorized	Asian Pacific American Heritage MonthEliseo SilvaGraffitiJerry MatsukumaKazuko Kayasuga MatthewsMuralsPan-Asian MetropolisPatrick NagataniPublic ArtSouthland ArtSteven WongStreet ArtThomas SuriyaTony OsumiTyrus Wong
Published by Eric Brightwell
Eric Brightwell is an essayist, rambler, explorer, cartographer, and guerrilla gardener. He lives in Los Angeles because he loves it -- not because he was born there. He doesn't really care about street art, sleb culture, sunshine, and prefers mass and active transit to cars.	View all posts by Eric Brightwell
‹ PreviousNo Enclave — Exploring Laotian Los Angeles
Next ›Mini-Mallism — Los Angeles's Asian Malls
3 thoughts on "Pan-Asian Metropolis — Asian-American Murals in Los Angeles"
This is a great collection, but I notice that no female Asian American artists are included. Can you update the article with some?
Eric Brightwell says:
Ah thanks! While I included female Asian American artists in the pieces on public art, sculpture, and artchitecture; I'm afraid that I don't know of any female Asian American muralists. Do you? If you or anyone else does, let me know.
I'm also working on a piece about Asian Angeleno artists, in general, but that probably won't be published for a bit.
Pingback: Pendersleigh & Sons Cartography — The Google Maps | Eric Brightwell
The Most Popular Spotify Playlists
Daphne Oram
Seasons in Los Angeles
Christmas Ghost Stories On Film
𝕷𝖔𝖘 𝕬𝖓𝖌𝖊𝖑𝖊𝖘 𝕲𝖔𝖙𝖍𝖎𝖈, 𝖆 𝕳𝖆𝖑𝖑𝖔𝖜𝖊𝖊𝖓 𝖆𝖓𝖉 𝕯í𝖆 𝖉𝖊 𝖑𝖔𝖘 𝕸𝖚𝖊𝖗𝖙𝖔𝖘 𝕱𝖆𝖓𝖙𝖆𝖘𝖎𝖆
Follow Eric Brightwell on WordPress.com


Very interesting. I think testing second cousins and half cousins ends up being more valuable in phasing data than with first cousins. It is possible for us to determine a narrower inheritance path. I have another second cousin in the cue to be tested (number 3). She should provide some interesting results.
I completely agree, Jim. I have one more second cousin to write about. This time from my dad's side.
I've got a paper trail 2nd cousin who shares 1.23% over 6 segments. 23and me says she is most likely a 3rd cosuin, range 2nd-3rd. My suspicious mind says she could be somewhere in between. Yet to put my mind to that issue of what cousinship that is if my grandmother's much older sister was her actual mother which is what I have thought for a long time now. If that's the case that would certainly help to explain why none of her father's ancestors have shown up and my abundance of Rhineland folk on RF and AF.
Since second cousins once removed would be expected to average 1.563% of their DNA in common that could indeed be what you are seeing in your family. However, the cousin above is definitely my second cousin and I only share 1.07% of my DNA with her, while my first cousins who share the same relationship with her have 4.49% and 4.69% in common with her. This tells us that the inheritance at this level is already so random that the actual percentages can vary widely and one cannot read too much into a low sharing with one member of the family without testing other family members.
It is very interesting to see the differences between the second cousins. Thanks for sharing the results.