text:
\subsubsection*{\bibname}}

\bibliographystyle{apalike}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{amsfonts}
\usepackage{nicefrac}
\usepackage{microtype}
\usepackage{xcolor}

\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{wrapfig}

\input{math_commands.tex}

\makeatletter
\newtheorem*{rep@theorem}{\rep@title}
\newcommand{\newreptheorem}[2]{%
\newenvironment{rep#1}[1]{%
 \def\rep@title{#2 \ref{##1}}%
 \begin{rep@theorem}}%
 {\end{rep@theorem}}}
\makeatother

\newtheorem{theorem}{Theorem}
\newreptheorem{theorem}{Theorem}
\newtheorem{acknowledgement}{Acknowledgement}
\newtheorem{axiom}{Axiom}
\newtheorem{assumption}{Assumption}
\newtheorem{case}[theorem]{Case}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{conclusion}{Conclusion}
\newtheorem{condition}{Condition}
\newtheorem{conjecture}{Conjecture}
\newtheorem{corollary}{Corollary}
\newreptheorem{corollary}{Corollary}
\newtheorem{criterion}[theorem]{Criterion}
\newtheorem{definition}{Definition}
\newtheorem{example}{Example}
\newtheorem{exercise}{Exercise}
\newtheorem{lemma}{Lemma}
\newreptheorem{lemma}{Lemma}
\newtheorem{notation}{Notation}
\newtheorem{problem}{Problem}
\newtheorem{proposition}{Proposition}
\newtheorem{remark}{Remark}
\newtheorem{solution}{Solution}
\newtheorem{summary}{Summary}
\newcommand{\mathcal{A}}{\mathcal{A}}
\newcommand{\mathcal{S}}{\mathcal{S}}
\newcommand{p}{p}

\newcommand{0.9\textwidth}{0.9\textwidth}
\newcommand{0.48\textwidth}{0.48\textwidth}
\newcommand{0.355\textwidth}{0.355\textwidth}
\newcommand{0.235\textwidth}{0.235\textwidth}
\newcommand{0.18\textwidth}{0.18\textwidth}
\newcommand{0.15\textwidth}{0.15\textwidth}
\newcommand{LR estimator}{LR estimator}
\newcommand{RP estimator}{RP estimator}
\newcommand{RP estimators}{RP estimators}

\usepackage[utf8]{inputenc}
\usepackage{pgfplots}
\DeclareUnicodeCharacter{2212}{−}
\usepgfplotslibrary{groupplots,dateplot}
\usetikzlibrary{patterns,shapes.arrows}
\pgfplotsset{compat=newest}

\usepackage{sectsty
\usepackage{titlecaps

\begin{document}

\twocolumn[
\aistatstitle{Model-free Policy Learning with Reward Gradients}
\aistatsauthor{Qingfeng Lan \And Samuele Tosatto \And Homayoon Farrahi \And A. Rupam Mahmood}
\aistatsaddress{University of Alberta \And University of Alberta \And University of Alberta \And University of Alberta \\ CIFAR AI Chair, Amii}
]

\begin{abstract}
Despite the increasing popularity of policy gradient methods, they are yet to be widely utilized in sample-scarce applications, such as robotics.
The sample efficiency could be improved by making best usage of available information.
As a key component in reinforcement learning, the reward function is usually devised carefully to guide the agent.
Hence, the reward function is usually known, allowing access to not only scalar reward signals but also reward gradients.
To benefit from reward gradients, previous works require the knowledge of environment dynamics, which are hard to obtain.
In this work, we develop the \textit{Reward Policy Gradient} estimator, a novel approach that integrates reward gradients without learning a model.
Bypassing the model dynamics allows our estimator to achieve a better bias-variance trade-off, which results in a higher sample efficiency, as shown in the empirical analysis.
Our method also boosts the performance of Proximal Policy Optimization on different MuJoCo control tasks.
\footnote{The code is available at \url{https://github.com/qlan3/Explorer/tree/RPG}.}
\end{abstract}

\sectionfont{\MakeUppercase}
\section{Introduction}

Policy gradient methods are increasingly popular in the reinforcement learning (RL) community due to their robustness and ability to deal with continuous action spaces.
They have been employed in a large variety of domains, including computer games~\citep{ vinyals2019grandmaster, berner2019dota,zha2021douzero,badia2020agent57}, simulated robotic tasks~\citep{haarnoja2018soft,lillicrap2015continuous}, and recommendation systems~\citep{zheng2018drn,zhao2018deep,afsar2021reinforcement}.
Despite their popularity, policy gradient methods still suffer from low sample efficiency, which hinders their applicability to complex real-world situations.
One way to mitigate this inefficiency, is to make best use of prior knowledge into the learning system.

In most real-world applications, reward functions are manually designed~\citep{dulac2019challenges}, and therefore known in advance.
Most of state-of-the-art techniques, however, do not benefit from such prior knowledge. One way to take full advantage of the reward function is to use its gradient w.r.t.\ the policy parameters.
Attempts to use reward gradients are already observed in prior works, such as Deterministic Value-Policy Gradient (DVPG) algorithm~\citep{cai2020deterministic}, Stochastic Value Gradient (SVG) methods~\citep{heess2015learning}, and Dreamer~\citep{hafner2019dream}.
However, all these methods require a learned transition model, which is well-known to be a harder task than learning value functions; an inaccurate model may even bring negative effect into training~\citep{van2019use,jafferjee2020hallucinating}.
In this work, we ask this question: \textit{how can we benefit from reward gradients without learning a model?}

To answer this question, we develop a new policy gradient estimator---the \textit{Reward Policy Gradient} (RPG) estimator---that incorporates reward gradients.
Specifically, in order to apply reward gradients while avoiding the use of a transition model, our approach incorporates two distinct gradient estimators that are popularly used in model-free learning --- the likelihood ratio (LR) estimator and the reparameterization (RP) estimator.
Broadly speaking, the LR estimator{} is generally applicable to both discrete and continuous variables, but often suffers from high variance and thus is frequently used in conjunction with variance reduction techniques \citep{williams1992simple,schulman2017proximal}.
The LR estimator{} is also a key component in the policy gradient theorem~\citep{sutton2000policy}.
On the other hand, the RP estimator{} exhibits lower variance in practice~\citep{greensmith2004variance}, and its recent utilization in policy gradient methods that improved sample efficiency in several benchmark tasks \citep{haarnoja2018soft,lillicrap2015continuous,fujimoto2018addressing}.

Our RPG estimator combines LR and RP gradients to avoid the need for a functional approximation of the model, while taking full advantage of reward gradients.
Based on this new estimator, we propose a new on-policy policy gradient algorithm---the RPG algorithm---that utilizes and takes advantage of both LR and RP gradients.
We empirically show that by incorporating reward gradients into the gradient estimator, the bias and variance of estimated gradient decrease significantly.
To analyze the benefit of reward gradients and the properties of our estimator, we test our algorithm on bandit and simple Markov decision processes, where the ground truth gradient is known in closed-form.
Moreover, we compare RPG with a state-of-the-art actor-critic algorithm---Proximal Policy Optimization---on challenging problems, showing that our algorithm outperforms the baseline algorithm.

In the rest of the paper, we first introduce some background knowledge for the LR and RP estimators. We then move to our major theoretical result --- the Reward Policy Gradient theorem. Then we present RPG algorithm as well as the experimental results.

\section{Background}

Consider a Markov decision process (MDP), $M=(\mathcal{S}, \mathcal{A}, p, p_0, r, \gamma)$, where $\mathcal{S}$ is the continuous state space, $\mathcal{A}$ is the continuous action space, $p: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0, \infty)$ is the state transition probability distribution, $p_0: \mathcal{S} \rightarrow [0, \infty)$ is the initial state probability distribution, $r: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ is the reward function,\footnote{For simplicity, we assume $r$ is a deterministic function. In the stochastic case, $r$ can be defined as the expected reward for a given state-action pair.} and $\gamma \in [0,1)$ is the discount factor.
In a given MDP $M$, an agent interacts with the environment to generate a trajectory based on a policy distribution $\pi: \mathcal{S} \times \mathcal{A} \rightarrow [0,\infty)$. Specifically, the agent starts at some state $S_0\sim p_0(\cdot)$; at each time-step $t = 0, 1, 2, \dots$, the agent samples an action $A_t \in \mathcal{A}$ according to the policy $\pi$ (i.e. $A_t \sim \pi(\cdot|S_t)$), receives the immediate reward $R_t = r(S_t, A_t)$, and observes the next state $S_{t+1}$ sampled from the transition probability distribution (i.e. $S_{t+1} \sim p(\cdot | S_t, A_t)$). A trajectory up to time-step $T$ is defined as $\tau_T = (S_0, A_0, R_1, S_1, \cdots, S_T)$. We define the total discounted reward from time-step $t$ over $\tau_T$ as return $G_t = \sum_{k=t}^{T-1} \gamma^{k-t} r(S_k, A_k)$. Value functions are defined to be the expected return under policy $\pi$, $v_{\pi}(s) = \mathbb{E}_{\pi}[G_t | S_t=s]$; similarly, action-value functions are defined as $q_{\pi}(s, a) = \mathbb{E}_{\pi}[G_t | S_t=s, A_t=a]$.

Furthermore, the state-values and action-values can be related as $q_{\pi}(s,a) = r(s,a) + \gamma \int p(s'|s,a) v_{\pi}(s') \mathrm{d}s'$ and $v_{\pi}(s) = \int \pi(a | s) q_{\pi}(s,a) \mathrm{d}a$. The goal of the agent is to obtain a policy $\pi$ that maximizes the expected return starting from initial states. Let a policy $\pi_{\theta}$ be a differentiable function of a weight vector $\theta$. Our goal is to find $\theta$ that maximizes
\begin{align} \label{eq_obj}
J(\theta) = \int p_0(s) v_{\pi_{\theta}}(s) \mathrm{d}s.
\end{align}
To this end, we can apply gradient ascent techniques.
Since the true gradient $\nabla_{\theta} J(\theta)$ is not typically available, we resort to Monte Carlo methods~\citep{mohamed2020monte}. This gradient estimation problem can be formalized as computing the unbiased gradient of an expectation of a function with respect to some parameters of a distribution.
Specifically, let $p_{\theta}(x)$ be the probability distribution of $x$ with parameters $\theta$.~Define $F(\theta) = \int p_{\theta}(x) \phi(x) \mathrm{d}x$. Then the key step of the problem can be formally defined as estimating $\nabla_{\theta} F(\theta)$:
\begin{align}
\nabla_{\theta} F(\theta) = \nabla_{\theta} \mathbb{E}_{X \sim p_{\theta}}[\phi(X)].
\end{align}
The gradient estimation problem is fundamental in many machine learning areas, such as reinforcement learning~\citep{williams1992simple}, variational inference~\citep{hoffman2013stochastic}, evolutionary algorithms~\citep{conti2018improving}, and variational auto-encoders~\citep{kingma2013auto}. In general, there are many approaches for gradient estimation, such as likelihood-ratio gradient estimators, reparameterization gradient estimators, finite difference methods, and mean-valued derivative methods~\citep{pflug1989sampling,carvalho2021empirical}. Next, we introduce two major approaches for solving this problem.

\subsubsection*{Likelihood-Ratio Gradient Estimators}

The likelihood-ratio (LR) gradient estimator~\citep{glynn1990likelihood} is one of the most popular gradient estimators.
This estimator applies the log-derivative technique $\nabla_{\theta} \log{p_{\theta}(x)} = \frac{\nabla_{\theta} p_{\theta}(x)}{p_{\theta}(x)}$ to obtain the unbiased gradient estimations:
\begin{align}
&\nabla_{\theta} F(\theta) = \nabla_{\theta} \mathbb{E}_{p_{\theta}(x)}[\phi(x)]
= \nabla_{\theta} \int p_{\theta}(x) \phi(x) \mathrm{d}x \nonumber \\
&= \int \phi(x) \nabla_{\theta} p_{\theta}(x) \mathrm{d}x
= \int \phi(x) p_{\theta}(x) \nabla_{\theta} \log{p_{\theta}(x)} \mathrm{d}x \nonumber \\
&= \mathbb{E}_{X \sim p_{\theta}}[\phi(X) \nabla_{\theta} \log{p_{\theta}(X)}].
\end{align}

The LR estimator{} is a fundamental component of the policy gradient theorem~\citep{sutton2011reinforcement}.
Partly because of its generality of being applicable to both continuous and discrete action spaces, the LR estimator{} has been used in many policy gradient methods in RL.
Many actor-critic algorithms also use this estimator to estimate gradient, such as Asynchronous Advantage Actor-Critic~\citep{mnih2016asynchronous}, Trust Region Policy Optimization~\citep{schulman2015trust}, Proximal Policy Optimization (PPO)~\citep{schulman2017proximal}, and Actor-Critic with Experience Replay~\citep{wang2016sample}.

In practice, variance reduction~\citep{greensmith2004variance} is usually necessary to fully exert the power of this estimator. Subtracting a baseline~\citep{williams1992simple} from $\phi(x)$, applying eligibility traces~\citep{singh1996reinforcement}, and utilizing the Generalized Advantage Estimator (GAE)~\citep{schulman2015high} are three effective approaches to mitigate the variance issue of policy gradient based on the LR estimator{}.

\subsubsection*{Reparameterization Gradient Estimators}

The reparameterization (RP) gradient estimator is also known as the pathwise gradient estimator or the reparameterization technique~\citep{mohamed2020monte}. Given the underlying probability distribution $p_{\theta}(x)$, this estimator takes the advantage of the knowledge of distribution $p_{\theta}(x)$ and reparameterize $p_{\theta}(x)$ with a simpler base distribution $p({\epsilon})$ that makes two equivalent sampling processes:
\begin{align}
X \sim p_{\theta}(\cdot) \iff X = f_{\theta}({\epsilon}), \, {\epsilon} \sim p(\cdot),
\end{align}
where $f_{\theta}$ is a function that maps ${\epsilon}$ to $x$. In other words, there are two equivalent ways to sample $X \sim p_{\theta}(x)$: one is to sample it directly; the other way is to first sample ${\epsilon}$ from a base distribution $p({\epsilon})$ and then apply a function $f$ to transform ${\epsilon}$ to $X$. For example, assume $X$ is a Gaussian variable where $X \sim {\mathcal{N}}(\mu, \sigma)$ and $\theta=[\mu, \sigma]$. Let the base distribution be $p({\epsilon}) = {\mathcal{N}}(0,1)$. Then $X$ can be reparameterized as $X = f_{\theta}({\epsilon}) = \mu + \sigma {\epsilon}$. For many common continuous distributions (e.g., the Gaussian, Log-Normal, Exponential, and Laplace), there exists such ways to reparameterize them with a simpler base distribution. Finally, we can write the gradient estimation as:
\begin{align}
&\nabla_{\theta} F(\theta)
= \nabla_{\theta} \int p_{\theta}(x) \phi(x) \mathrm{d}x \nonumber \\
&= \nabla_{\theta} \int p({\epsilon}) \phi(f_{\theta}({\epsilon})) \mathrm{d}{\epsilon}
= \int p({\epsilon}) \nabla_{\theta} \phi(f_{\theta}({\epsilon})) \mathrm{d}{\epsilon}.
\end{align}

Note that $p_{\theta}(x) = p({\epsilon}) |\nabla_{{\epsilon}} f_{\theta}({\epsilon})|^{-1}$ due to integration by substitution. RP estimators{} are only applicable to known and continuous distributions.\footnote{An application to discrete random variables is possible by reparameterizing the Gumble distribution---the continuous counterpart of the categorical distribution \citep{jang2016categorical}.} In general, there is no guarantee that RP outperforms LR~\citep{gal2016uncertainty,parmas2018pipps}.
However, RP estimators{} have a lower variance under certain assumptions~\citep{xu2019variance}, which usually leads to great benefits in many areas~\citep{mohamed2020monte}.
\citet{kingma2013auto} applied the RP estimator{} to obtain a differentiable estimator of the variational lower bound that can be optimized directly using standard stochastic gradient methods. \citet{rezende2014stochastic} used RP estimators{} for deep generative models and proposed Deep Latent Gaussian Models that are able to generate realistic images.

In many RL algorithms, RP estimators{} play a crucial role to reparameterize actions and decouple the randomness from a policy, such as Soft Actor-Critic (SAC)~\citep{haarnoja2018soft}, SVG~\citep{heess2015learning}, and RELAX~\citep{grathwohl2018backpropagation}. Deterministic Policy Gradient algorithm (DPG)~\citep{silver2014deterministic}, and Deep Deterministic Policy Gradient algorithm (DDPG)~\citep{lillicrap2015continuous}.
Deterministic Value-Policy Gradient algorithm~\citep{cai2020deterministic} can also be viewed as special cases of these algorithms, where the probability density function of the base distribution is a Dirac delta function. Finally, \citet{wang2019generalization} proposed a class of RL algorithms called Reparameterizable RL, where the randomness of the environment is decoupled from the trajectory distribution via the reparameterization technique.

\iffalse
Generalized RG:
    Implicit reparameterization gradients: it is hard to implement it to our method in pytorch

    The Generalized Reparameterization Gradient: [Ruiz et al. introduced generalized reparameterization gradients (GRG) that expand the applicability of the reparameterization technique by using a standardization function that allows the underlying base distribution to depend weakly on the parameter vector (e.g. only through the higher moments). The resulting gradient estimator, which in addition to the the reparameterized gradients term includes a score-function gradient term that takes into account the dependence of the base distribution on the parameter vector, was applied to the Gamma, Beta, and log-Normal distributions. The challenge of using this approach lies in finding an effective approximate standardization function, which is nontrivial yet essential for obtaining low-variance gradients.]

    A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme. One can also combine the best of both LR and RP gradients by dynamically reweighting them (Parmas et al., 2018; Metz et al., 2019).
\fi

\section{Reward Policy Gradient Theorem}

In this section, we first present our main theoretical result---the \textit{Reward Policy Gradient Theorem}---a new policy gradient theorem that incorporates the gradient of the reward function without using a state transition function explicitly. The reward policy gradient theorem requires perfect knowledge of the reward and the value function.
We show that an unbiased estimate can be obtained using approximated reward and state-value functions, by defining a set of compatible features, similarly to~\citet{sutton2000policy}.

We begin by assuming that the action $A$ is sampled from the policy $\pi$ parameterized with $\theta$ given the current state $S$: $A \sim \pi_{\theta}(\cdot | S)$. We reparameterize the policy with a function $f$, $A = f_{\theta}({\epsilon};S), \, {\epsilon} \sim p(\cdot)$. Let function $g$ be the inverse function of $f$, that is, ${\epsilon} = g_{\theta}(A;S)$ and $A = f_{\theta}(g_{\theta}(A;S);S)$. Furthermore, following~\citet{imani2018off}, we make two common assumptions on the MDP.

\begin{assumption}\label{assum1}
$\mathcal{S}$ and $\mathcal{A}$ are closed and bounded.
\end{assumption}

\begin{assumption}\label{assum2}
$p(s' | s,a)$, $f_{\theta}({\epsilon}; s)$, $g_{\theta}(a | s)$, $\pi_{\theta}(a|s)$, $p({\epsilon})$, $r(s,a)$, $p_0(s)$ and their derivatives are continuous in all variables $s$, $a$, $s'$, $\theta$, and ${\epsilon}$.
\end{assumption}

\begin{remark}
The two assumptions above allow us to exchange derivatives and integrals, and the order of multiple integrations, using Fubini's theorem and Leibniz integral rule.
\end{remark}

We use the same objective $J(\theta)$ as in Eq.~\ref{eq_obj}. Then under these two assumptions, we have the following results. The details of all proofs can be found in the appendix.

\begin{theorem}[Reward Policy Gradient]\label{thm_rpg}
Suppose that the MDP satisfies Assumption~\ref{assum1} and \ref{assum2}, then
\begin{align*}
&\nabla_{\theta} J(\theta) \\
&= \int\!\!\!d^{\pi_{\theta}}\!(s) \pi_{\theta}(a|s) p(s'|s,a)\!\! \left[\nabla_{\theta} r(s,f_{\theta}({\epsilon}; s))|_{\epsilon=g_{\theta}(a;s)} \right. \\
&\left. + \gamma v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a | s)} \right] \mathrm{d}s \mathrm{d}a \mathrm{d}s',
\end{align*}
where $d^{\pi_{\theta}}(s') = \int \sum_{t=0}^{\infty} \gamma^t p_0(s) p(s \to s', t, \pi_{\theta}) \mathrm{d} s$ is the (discounted) stationary state distribution for policy $\pi_{\theta}$ and $p(s \to s', t, \pi_{\theta})$ is the transition probability from $s$ to $s'$ with $t$ steps under policy $\pi_{\theta}$.
\end{theorem}

\textit{Proof sketch of Theorem~\ref{thm_rpg}.} We first apply the policy gradient theorem~\citep{sutton2000policy} and get an intermediate result in form of the action-value function. Next, we split the action-value function in the policy gradient theorem into two parts: the immediate reward and the state-value of the next state. To incorporate reward gradients, we use the RP technique to the immediate reward part; to avoid the knowledge of the model, we apply the LR estimator to the state-value part. Finally, we combine both parts into an unbiased gradient estimation.

This theorem provides a new way of computing the objective gradients.
Specifically, it presents the objective gradient in terms of both LR and RP gradients as additive components.
The theorem also presents the first model-free unbiased gradient estimator of the objective function that utilizes gradients of the reward function. Some algorithms also estimate the gradient of the objective using gradients of the reward function, such as DVPG~\citep{cai2020deterministic} and SVG~\citep{heess2015learning}. However, they are model-based algorithms in the sense that they require the knowledge of the state transition function to estimate an unbiased gradient, while our theorem points out a model-free approach without the knowledge of the state transition.

\begin{theorem}[Reward Policy Gradient with Function Approximation] \label{thm_rpg_appro}
Consider a parametric approximation of the reward function $\hat{r}_\omega(s, a)$ and a parametric approximation of the value function $\hat{v}_\phi(s)$ such that
\begin{align*}
&\nabla_a \hat{r}_\omega(s, a) = \nabla_\theta^\intercal f_\theta(\epsilon; s)|_{\epsilon = g_{\theta}(a; s)} \omega, \\
&\int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) \nabla_{\phi} \hat{v}_{\phi}(s')  \mathrm{d} s \mathrm{d} a \\
=& \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) \nabla_{\theta} \log \pi_{\theta}(a|s) \mathrm{d} s \mathrm{d} a,
\end{align*}
where $\phi = \argmin_{\phi}{\mathbb{E}}\left[(\hat{v}_\phi(s') - v(s'))^2\right]$ and
\begin{align*}
\omega = \argmin_{\omega} \sum_{i}\mathbb{E} &\left[\left(\frac\partial{\partial\theta_i} \hat{r}_\omega(s, a) - \frac\partial{\partial\theta_i}r(s, a)\right)^2\right].
\end{align*}
Then,
\begin{align*}
 \nabla_\theta J(\theta)
= \mathbb{E} & \left[\nabla_\theta\hat{r}_{\omega}(s, f_\theta(\epsilon; s))|_{\epsilon = g_\theta(a; s)} \right. \\
& \left. + \gamma \hat{v}_{\phi}(s')\nabla_\theta \log \pi_{\theta}(a | s) \right].
\end{align*}
Note that all expectations are computed under $s \sim d^{\pi_{\theta}}$, $a \sim \pi_\theta(\cdot | s)$, $s' \sim p(\cdot|s,a)$.
\end{theorem}
By this theorem, we show that when the reward function and the state-value function are approximated by sufficiently good function approximators (e.g., neural networks), we can obtain an unbiased gradient estimation under certain assumptions. The functions $\hat{r}_\omega$ and $\hat{v}_\phi$ are also known as compatible approximators~\citep{sutton2000policy,peters2008reinforcement}. Note that $\phi$, $\omega$, and $\theta$ have same dimensions.

\section{A Reward Policy Gradient Algorithm based on PPO}

\begin{algorithm}[tb]
\caption{RPG} \label{algo_rpg}
\begin{algorithmic}
    \STATE Input: initial policy parameters $\theta$, value estimate parameters $\phi$, and reward estimate parameters $w$.
    \FOR{$k=1,2,\dots$}
        \STATE Collect trajectories $\mathcal{D}=\{\tau_{i}\}$ with policy $\pi_{\theta}$. \\
        \STATE Compute $G_t$ and $G_{t}^{\lambda} = H_{t}^{\text{GAE}(\lambda)} + \hat{v}_{\phi}(S_t)$. \\
        \STATE Compute PPO advantage $H_{t}=H_{t}^{\text{GAE}(\lambda)}$ and normalize.
        \FOR{epoch $=1,2,\dots$}
        \STATE Slice trajectories $\mathcal{D}$ into mini-batches.
            \FOR{each mini-batch $B$}
                \STATE Set $\hat{\rho}_t(\theta)$ by Eq.~\ref{eq_clip_new} and detach it from the computation graph.
                \STATE Reparameterize the action $A_t=f_{\theta}(\epsilon_t; S_t)$.
                \STATE Compute the predicted reward $\hat{R}_{t+1} = \hat{r}_{w}(S_t, f_{\theta}(\epsilon_t; S_t))$. \\
                \STATE Update $\theta$ by maximizing $\mathbb{E}_{B}[\hat{\rho}_t(\theta) H_{t}^{\text{RPG}}]$,
                where $H_{t}^{\text{RPG}} = \hat{R}_{t+1} + (\gamma G_{t+1}^{\lambda} - \hat{v}_{\phi}(S_t))\times\log{\pi_{\theta}(A_t|S_t)}$.
                \STATE Update $\phi$ by minimizing $\mathbb{E}_{B}[(\hat{v}_{\phi}(S_{t})-G_t)^{2}]$.
                \STATE Update $w$ by minimizing $\mathbb{E}_{B}[(\hat{r}_{w}(S_{t}, A_{t})-R_{t+1})^{2}]$.
            \ENDFOR
        \ENDFOR
    \ENDFOR
\end{algorithmic}
\end{algorithm}

Based on the reward policy gradient theorem, many different policy gradient algorithms can be developed that benefit from reward gradients without using a transition model. In this section, we develop a new policy gradient algorithm based on an existing deep policy gradient method called Proximal Policy Optimization (PPO). First, we introduce the baseline subtraction technique which is generally used in policy gradient algorithms to reduce variance. To be specific, we subtract the baseline $v_{\pi_{\theta}}(s)$ from $\gamma v_{\pi_{\theta}}(s')$ in the RPG estimator:
\begin{align}
\nabla_{\theta} r(s,f_{\theta}({\epsilon}; s)) + (\gamma v_{\pi_{\theta}}(s') - v_{\pi_{\theta}}(s)) \nabla_{\theta} \log{\pi_{\theta}(a | s)}. \label{baseline}
\end{align}
Note that no bias is introduced in this step~\citep{sutton2011reinforcement}.

In practice, when the state-value function and the reward function are unknown to the agent, we could use function approximators (e.g., neural networks) to approximate them. Furthermore, we use $\lambda$-return~\citep{sutton2011reinforcement} $G_{t+1}^{\lambda}$ to replace $v_{\pi_{\theta}}(s')$ in Eq.~\ref{baseline}, which has a close relationship to GAE $H_t^{\text{GAE}(\lambda)}$~\citep{schulman2015high}, i.e. $G_{t}^{\lambda} = H_{t}^{\text{GAE}(\lambda)} + v_{\pi_{\theta}}(s_t)$, where $\lambda \in [0,1]$.
Using $\lambda$-returns significantly reduce the variance of gradient estimations while retaining tolerable biases~\citep{schulman2015high}.

\begin{figure}[t]
\input{figures/lqg.tex}
\vspace{-.2in}
\caption{The bias, variance, and mean squared error (MSE) of the estimated gradient w.r.t. the number of samples for the PG estimator (left plot) and the RPG estimator (right plot). The shaded area representing a 95\% interval using bootstrapping techniques. The values of bias, variance, and MSE for the RPG estimator are significantly smaller than the values for the PG estimator, which is clear from the Y-axis ranges.}
\label{fig_lqg}
\end{figure}

We first briefly describe PPO, which is the basis of the new RPG algorithm.
PPO is simple to implement, closely related to on-policy learning, and achieves good performance on many tasks~\citep{schulman2017proximal}. The key idea behind PPO is to constrain the policy update by using a clipped surrogate objective $\mathbb{E}_{\pi_{\theta_{\text{old}}}}[l_t(\theta)]$, where $l_t(\theta) = \min(\rho_t(\theta) H_{t}, \text{clip}(\rho_t(\theta), 1-{\epsilon}, 1+{\epsilon}) H_{t})$, where $H_{t}$ is an estimator of the advantage function, and $\rho_t(\theta) = \pi_{\theta}(a_t | s_t) / \pi_{\theta_{\text{old}}}(a_t | s_t)$ is the importance sampling ratio. Then the gradients of $l_t(\theta)$ can be written as
\begin{equation} \label{eq_clip}
\nabla_{\theta} l_t(\theta)=\left\{
\begin{aligned}
& 0, & H_{t} > 0, \rho_t(\theta) > 1+{\epsilon} \\
& 0, & H_{t} < 0, \rho_t(\theta) < 1+{\epsilon} \\
& \nabla_{\theta} \rho_t(\theta) H_{t}, & \text{otherwise.}
\end{aligned}
\right.
\end{equation}

\begin{figure*}[htb]
\vspace{-.1in}
\begin{center}
  \subfigure[Peaks. RPG converges much faster than PPO in the whole training stage, when the variance is not so large.]{
    \includegraphics[width=0.9\textwidth]{figures/peak.pdf}
    \label{fig_peak}
  }
\subfigure[Holes. RPG converges much faster than PPO at the early training stage, under various variance settings. Then it slows down at the later stage.]{
    \includegraphics[width=0.9\textwidth]{figures/hole.pdf}
    \label{fig_hole}
  }
\end{center}
\vspace{-.2in}
\caption{The reward landscapes of Peaks and Holes as well as learning curves for PPO and RPG during training. The initial Gaussian policies are also visualized. All results were averaged over 30 runs; the shaded areas represent standard errors.}
\end{figure*}

Inspired by Eq.~\ref{eq_clip}, we can use a modified ratio $\hat{\rho}_t(\theta)$,
\begin{equation} \label{eq_clip_new}
\hat{\rho}_t(\theta)=\left\{
\begin{aligned}
& 0, & H_{t} > 0, \rho_t(\theta) > 1+{\epsilon} \\
& 0, & H_{t} < 0, \rho_t(\theta) < 1+{\epsilon} \\
& \rho_t(\theta), & \text{otherwise.}
\end{aligned}
\right.
\end{equation}
Define $l_t(\theta) = \hat{\rho}_t(\theta) H_{t}$. It is easy to verify that this $\nabla_{\theta} l_t(\theta)$ is exactly the same as Eq.~\ref{eq_clip}. For a complete algorithm description of PPO, please check the appendix.

The RPG algorithm builds on PPO with two major modifications. First, we replace the original LR estimator{} in PPO with the RPG estimator. Second, in order to use reward gradients, we have a neural network to learn the reward function. Basically, RPG can be viewed as a version of PPO but using the RPG estimator to do gradient estimation. The detailed algorithm description for RPG is listed in Algorithm~\ref{algo_rpg}.

\begin{figure}[t]
\vspace{-.2in}
\includegraphics[width=0.48\textwidth]{figures/mountain.pdf}
\vspace{-.2in}
\caption{The learning curves for PPO and RPG during training on Mountain Climbing. The results were averaged over 20 runs, with the shaded area representing one standard error. In terms of convergence rate, RPG (true reward) > RPG (learned reward) > PPO.}
\label{fig_mountain}
\end{figure}

\begin{figure*}[!ht]
\vspace{-.2in}
\begin{center}
  \subfigure[HalfCheetah-v2]{\includegraphics[width=0.355\textwidth]{figures/HalfCheetah-v2.pdf}}\hspace{-0.7cm}\vspace{-0.1cm}
  \subfigure[Hopper-v2]{\includegraphics[width=0.355\textwidth]{figures/Hopper-v2.pdf}}\hspace{-0.7cm}\vspace{-0.1cm}
  \subfigure[Walker2d-v2]{\includegraphics[width=0.355\textwidth]{figures/Walker2d-v2.pdf}}\vspace{-0.1cm}
  \subfigure[Swimmer-v2]{\includegraphics[width=0.355\textwidth]{figures/Swimmer-v2.pdf}}\hspace{-0.7cm}
  \subfigure[Ant-v2]{\includegraphics[width=0.355\textwidth]{figures/Ant-v2.pdf}}\hspace{-0.7cm}
  \subfigure[Reacher-v2]{\includegraphics[width=0.355\textwidth]{figures/Reacher-v2.pdf}}
\end{center}
\vspace{-.2in}
\caption{Learning curves of evaluations on six benchmark tasks for PPO and RPG. The results were averaged over 30 runs, with the shaded area representing one standard error.}
\label{fig_mujoco}
\end{figure*}

\section{Experiments}

In this section, we first analyze the bias-variance trade-off for the RPG estimator on a Linear Quadratic Gaussian (LQG) control task. We then gain more understanding to the advantages and drawbacks of using reward gradients on two bandit tasks.
Furthermore, we investigate the benefit of RPG when the reward function is known. Finally, we evaluate our algorithm on six MuJoCo control tasks~\citep{todorov2012mujoco} and one robot task compared to the baseline method.

\subsection*{A Bias-variance Analysis of the RPG Estimator}

Environments can have highly stochastic rewards~\citep{brockman2016openai}. Knowing the reward function in advance allows reducing the stochasticity of the gradient estimator. Furthermore, in most cases, reparameterization gradients exhibit lower variance w.r.t. likelihood ratio gradients~\citep{xu2019variance}. However, the variance of our estimator depends on many compounding factors (e.g. reward shapes and environment dynamics) that complicate a theoretical analysis. To compensate for this gap, we empirically investigate this aspect on a LQG problem.

The LQG control problem is one of the most classical control problems in control theory,  with linear dynamics, quadratic reward, and Gaussian noise. Using the Riccati equations, we can solve the LQG problem in closed form. This property makes the LQG task an ideal benchmark to do bias-variance analysis for different policy gradient estimators. Specifically, we consider the discrete-time LQG problem, defined as
\begin{align*}
& \max_{\theta} \sum_{t=0}^{\infty}  \gamma^t r_t \\
& \text{s.t. } \textbf{s}_{t+1} = A \textbf{s}_t + B \textbf{a}_t; \quad r_t = -{\textbf{s}_t}^\top Q \textbf{s}_t - {\textbf{a}_t}^\top Z \textbf{a}_t \\
& \quad\quad \textbf{a}_{t+1} = \Theta \textbf{s}_t + \Sigma {\epsilon}_t; \quad {\epsilon}_t \sim {\mathcal{N}}(0, I), \\
\end{align*}
where $A,B,Q,Z,\Sigma$, and $\Theta$ are diagonal matrices and $\Theta = \text{diag}(\theta)$. Our policy is Gaussian: $\textbf{a} \sim {\mathcal{N}}(\Theta \textbf{s}; \Sigma)$, where $\theta \in {\mathbb{R}}^2$.

Given an LQG task, we study how the bias, the variance, and the mean squared error (MSE) of the estimated gradient vary w.r.t. the number of samples for both the policy gradient (PG) estimator and the RPG estimator. A similar study can also be found in~\cite{tosatto2021batch}. Note that we do not subtract a baseline term in either estimator as in Eq.~\ref{baseline}. In our experiment, we assume that the true reward function and the true value function are provided to both gradient estimators. Once the reward function is given, the RPG estimator is able to use not only the reward signals but also reward gradients. However, the PG estimator can only use the reward signals since it is not designed to use reward gradients. For further details, please refer to the appendix.
In Figure~\ref{fig_lqg}, the left and right plots show the values of the PG estimator and the RPG estimator, respectively. Clearly, both the bias and the variance of the RPG estimator are significantly smaller than the ones of the PG estimator, which can be noticed from the Y-axis ranges of the two plots.
This result indicates that by incorporating reward gradients, the RPG estimator is able to reduce the bias and the variance of gradient estimation substantially.

\subsection*{Benefits and Drawbacks of Reward Gradients}

To further understand the role of reward gradients, we design two kinds of bandit tasks---\textit{Peaks} and \textit{Holes}---with continuous action spaces $\mathbb{R}$. The reward functions of Peaks and Holes are defined as $r(a)=\exp\left(-\frac{(a-1)^2}{b^2}\right)$ and $r(a)=1-\exp\left(-\frac{a^2}{b^2}\right)$, as shown in Figure~\ref{fig_peak} and Figure~\ref{fig_hole}, respectively. In our experiments, a small ($0.01\times$) Gaussian noise is added to all reward signals and $b^2$ is chosen from $[2, 8, 32]$.

The initial policy distribution is a Gaussian distribution ${\mathcal{N}}(0, 0.69)$. For RPG, we use a neural network to approximate the true reward function, denoted by $\hat{r}_w(a)$. In bandit cases, there are no value functions and we have the following single-sample gradients for PPO and RPG, without considering normalization for simplicity:
\begin{align} \label{eq_bandit_grad}
&\nabla_{\theta} l_t^{\text{PPO}}(\theta) = \hat{\rho}_t(\theta) R_t \nabla_{\theta} \log \pi_{\theta}(A_t)
\text{ and } \nonumber \\
&\nabla_{\theta} l_t^{\text{RPG}}(\theta) = \hat{\rho}_t(\theta) \nabla_{\theta} f_{\theta}({\epsilon}_t) \nabla_{a} \hat{r}_w(a)|_{a=A_t},
\end{align}
where $\theta = [\mu, \sigma]$. We test PPO and RPG on these bandits over 30 runs. The gradient is clipped by 1.

The learning curves of PPO and RPG are visualized in Figure~\ref{fig_peak} and \ref{fig_hole}. We observe that as $b^2$ increases, the learning speed decreases for both PPO and RPG, on Peaks and Holes. This is because the magnitude of reward's gradient decreases as $b^2$ increases which reduces $\nabla_{\theta} l_t(\theta)$ and slows down the learning update, as suggested by Eq.~\ref{eq_bandit_grad}.

Furthermore, RPG converges much faster than PPO on Peaks in the whole training period when $b^2$ is not so large. This shows that RPG is able to find the optimal action quicker than PPO, with the help of a relatively large reward's gradient. On Holes, RPG escapes from the sub-optimal action faster than PPO at the early training stage, but it slows down later and performs worse than PPO in the end. This is not surprising since the reward function is still sharp at the early training stage (when $|a|$ is small) but tends to become flat later (i.e. $|\nabla_{a} \hat{r}_w(a)|$ is smaller.) which slows down the learning process of RPG.

Therefore, we conclude that in bandits the reward's gradient accelerates learning when it is relatively large but hurts learning when the reward function is too flat.
Reward and action-value gradients are identical in bandits; to compensate that, and we conduct the rest of the studies using simple and complex MDPs.

\subsection*{The Benefit of Knowing the Reward Function}

To explore the role of reward gradients in MDP settings, we design a simple MDP---\textit{Mountain Climbing}---with continuous state and action spaces. Specifically, $\mathcal{S}=[-8,8]^2$, $\mathcal{A}=[-1,1]^2$, $S_0=(0,0), R_{t+1} = r(S_t, A_t)$, and $S_{t+1} = S_t + A_t + {\epsilon}$ where $r(s,a) = \exp(-||s+a-\nu||_2^2)$, ${\epsilon} \sim {\mathcal{U}}(-0.005,0.005)$, and $\nu=(1,-1)$. Every episode ends in 10 steps.
We test PPO (LR gradient) and RPG (LR gradient + reward gradient) on this MDP for $80,000$ steps. We set the number of epochs in PPO and RPG to 1 and the batch and  mini-batch size to 40. The gradient is clipped by 0.5.

The average returns over 20 runs for each algorithm during training are shown in Figure~\ref{fig_mountain}. By utilizing reward gradients, RPG (LR gradient + reward gradient) outperforms PPO (LR gradient) significantly. Moreover, the reward gradients of the true reward function are more helpful to accelerate learning than the reward gradients of a learned one, probably due to a higher accuracy of gradient estimation; although two versions of RPG reach a similar performance in the end.

\subsection*{Evaluation on Simulated Benchmark Tasks}

To further evaluate our algorithm, we measure its performance on six MuJoCo control tasks~\citep{todorov2014convex} through OpenAI Gym~\citep{brockman2016openai}.
Our PPO implementation is based on \citet{deeprl} and \citet{SpinningUp2018}. Then we implement RPG based on this version of PPO. The appendix contains all hyper-parameters which are largely borrowed from some popular implementations. The code is also submitted as a supplementary file. For the simulated tasks, each algorithm was trained on every task for 3 million steps. For every 5 epochs, we evaluated the agent's test performance using a deterministic policy for one episode. Our results were reported by averaging over 30 runs with different random seeds. The learning curves during evaluation are presented in Figure~\ref{fig_mujoco}.
Overall, RPG outperforms PPO significantly on three tasks -- HalfCheetah, Hopper, and Walker2d. It is slightly better on Reacher and worse on Ant, compared to PPO. On Swimmer, however, PPO has a clear advantage.

\subsection*{Evaluation on a Real-Robot Task}
We include results on a real-robot task to showcase RPG's ability to effectively learn in a real-world scenario.
We use the Real-Robot Reacher task based on a UR robotic arm in which an agent must move the fingertip of a robot arm as close and as fast as possible to a randomly selected target by rotating the base and elbow joints.
Each episode is 4 seconds long followed by a reset, bringing the fingertip to a start position.
The appendix contains the full description of the task.
RPG runs on Real-Robot Reacher for 90,000 time steps at 40ms action-cycle time, resulting in an hour of robot-experience time excluding resets.
The undiscounted episodic returns were stored during training without measuring a separate test performance.

Figure~\ref{fig:ur5} shows the resulting learning curves averaged over five independent runs.
RPG continues to increase performance throughout learning.
For comparison, we also included the performance of PPO, which is known to learn an effective reaching behavior on this task, specifically at this performance level of about 300 average return~\citep{mahmood2018benchmarking,farrahi2020making}.
Both RPG and PPO achieved the same performance on this task, which is not surprising as we did not perform a hyper-parameter search.
Instead, our results confirm that RPG is at least as effective as PPO on a well-studied real-robot task.

\begin{figure}[t]
\vspace{-.2in}
\includegraphics[width=0.48\textwidth]{figures/ur5.pdf}
\vspace{-.2in}
\caption{The learning curves for RPG and PPO on the Real-Robot Reacher task. The results were averaged over 5 runs, with the shaded area representing one standard error. The bottom-right image depicts the task setup. RPG is as effective as PPO on this task.}
\label{fig:ur5}
\end{figure}

\section{Discussion}

Our work focuses more on understanding the properties of the RPG estimator by conducting a series of analysis on simple environments.
Moreover, the RPG theorem only provides a new way to estimate policy gradient; the implementation of actual algorithms can vary. For example, the RPG version of the na\"ive actor-critic would be a straight-forward implementation. Our current implementation builds on PPO, and it benefits from PPO's techniques as well. To further explore and exploit the advantage of the RPG estimator, we may develop more advanced implementations by combining some modern techniques in the future, such as entropy regularization~\citep{haarnoja2018soft}, parallel training~\citep{mnih2016asynchronous}, separating training phases for policy and value functions~\citep{cobbe2021phasic}, Retrace~\citep{munos2016safe}, and V-trace~\citep{espeholt2018impala}, etc. Our current implementation is just one approach; exploring more possibilities are among the potential future works resulting from this work.

\section{Conclusion}

In this paper, we introduced a novel strategy to compute the policy gradient which uses reward gradients without a model. Based on this strategy, we developed---RPG---a new on-policy policy gradient algorithm. We showed that our method of using reward gradients is beneficial over the PG estimator in terms of the bias-variance trade-off and sample efficiency. Experiments showed that RPG generally outperformed PPO on several simulation tasks.
RPG is limited to the same class of tasks where reparameterization applies and, thus, not directly applicable to tasks with discrete actions. However, combined with the Gumbel-Softmax technique~\citep{jang2016categorical}, it is possible to apply RPG on discrete control tasks as well.

\subsubsection*{Acknowledgements}
We gratefully acknowledge funding from the Canada CIFAR AI Chairs program, the Reinforcement Learning and Artificial Intelligence (RLAI) laboratory, the Alberta Machine Intelligence Institute (Amii), and the Natural Sciences and Engineering Research Council (NSERC) of Canada.
We also thankfully acknowledge the donation of the UR5 arm from the Ocado Group.

\section{Theorems and Proofs}

\subsection{Proof of Reward Policy Gradient Theorem}

Let $d^{\pi_{\theta}}(s') = \int \sum_{t=0}^{\infty} \gamma^t p_0(s) p(s \to s', t, \pi_{\theta}) \mathrm{d} s$ be the (discounted) stationary state distribution for policy $\pi_{\theta}$; and $p(s \to s', t, \pi_{\theta})$  the transition probability from $s$ to $s'$ with $t$ steps under policy $\pi_{\theta}$.

\begin{reptheorem}{thm_rpg}[Reward Policy Gradient Theorem]
Suppose that the MDP satisfies Assumption~\ref{assum1} and \ref{assum2}, then
\begin{align*}
\nabla_{\theta} J(\theta) = \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) \left[\nabla_{\theta} r(s,f_{\theta}({\epsilon}; s))|_{\epsilon=g_{\theta}(a;s)} + \gamma v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a | s)} \right] \mathrm{d} s \mathrm{d} a \mathrm{d} s'.
\end{align*}
\end{reptheorem}

\begin{proof}

By the policy gradient theorem, we have
\begin{align*}
\nabla_{\theta} J(\theta)
= \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) q_{\pi_{\theta}}(s,a) \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} a \mathrm{d} s.
\end{align*}

Next, we split the action-value function $q_{\pi_{\theta}}(s,a)$ into two parts:
\begin{align*}
\nabla_{\theta} J(\theta)
=& \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) \left[r(s,a) + \gamma \int p(s' | s,a) v_{\pi_{\theta}}(s') \mathrm{d} s' \right] \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} a \mathrm{d} s \\
=& \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) r(s,a) \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} s \mathrm{d} a + \gamma \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} s \mathrm{d} a \mathrm{d} s' \\
=& \int d^{\pi_{\theta}}(s) r(s,a) \nabla_{\theta} \pi_{\theta}(a|s) \mathrm{d} s \mathrm{d} a + \gamma \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} s \mathrm{d} a \mathrm{d} s' \\
=& \int d^{\pi_{\theta}}(s) \nabla_{\theta} \left( \int \pi_{\theta}(a|s) r(s,a) \mathrm{d} a \right) \mathrm{d} s + \gamma \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} s \mathrm{d} a \mathrm{d} s'.
\end{align*}

Now, we apply the reparameterization technique to the first part,
\begin{align*}
\nabla_{\theta} J(\theta)
=& \int d^{\pi_{\theta}}(s) \nabla_{\theta} \left( \int \pi_{\theta}(a|s) r(s,a) \mathrm{d} a \right) \mathrm{d} s + \gamma \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} s \mathrm{d} a \mathrm{d} s' \\
=& \int d^{\pi_{\theta}}(s) \nabla_{\theta} \left( \int p({\epsilon}) r(s,f_{\theta}({\epsilon}; s)) \mathrm{d} {\epsilon} \right) \mathrm{d} s + \gamma \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} s \mathrm{d} a \mathrm{d} s' \\
=& \int d^{\pi_{\theta}}(s) \left( \int p({\epsilon}) \nabla_{\theta} r(s,f_{\theta}({\epsilon}; s)) \mathrm{d} {\epsilon} \right) \mathrm{d} s + \gamma \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} s \mathrm{d} a \mathrm{d} s'.
\end{align*}

We then apply the reverse operation of reparameterization to the first part,
\begin{align*}
&\nabla_{\theta} J(\theta) \\
=& \int d^{\pi_{\theta}}(s) \left( \int p({\epsilon}) \nabla_{\theta} r(s,f_{\theta}({\epsilon}; s)) \mathrm{d} {\epsilon} \right) \mathrm{d} s + \gamma \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} s \mathrm{d} a \mathrm{d} s' \\
=& \int d^{\pi_{\theta}}(s) \left( \int \pi_{\theta}(a|s) \nabla_{\theta} r(s,f_{\theta}({\epsilon}; s))|_{\epsilon=g_{\theta}(a;s)} \mathrm{d} a \right) \mathrm{d} s + \gamma \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} s \mathrm{d} a \mathrm{d} s' \\
=& \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) \left[\nabla_{\theta} r(s,f_{\theta}({\epsilon}; s))|_{\epsilon=g_{\theta}(a;s)} + \gamma v_{\pi_{\theta}}(s') \nabla_{\theta} \log{\pi_{\theta}(a | s)} \right] \mathrm{d} s \mathrm{d} a \mathrm{d} s'.
\end{align*}

\end{proof}

\subsection{Proof of Reward Policy Gradient with Function Approximation}

\begin{reptheorem}{thm_rpg_appro}[Reward Policy Gradient with Function Approximation]
Consider a parametric approximation of the reward function $\hat{r}_\omega(s, a)$ and a parametric approximation of the value function $\hat{v}_\phi(s)$ such that
\begin{enumerate}
\item $\nabla_a \hat{r}_\omega(s, a) = \nabla_\theta^\intercal f_\theta(\epsilon; s)|_{\epsilon = g_{\theta}(a; s)} \omega$,
\item $\int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) \nabla_{\phi} \hat{v}_{\phi}(s') \mathrm{d} s \mathrm{d} a = \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) p(s'|s,a) \nabla_{\theta} \log \pi_{\theta}(a|s) \mathrm{d} s \mathrm{d} a$,
\end{enumerate}
where $\phi = \argmin_{\phi} \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta \\ s' \sim p(\cdot|s, a)}}{\mathbb{E}}\left[(\hat{v}_\phi(s') - v(s'))^2\right]$ \\
and $\omega = \argmin_{\omega} \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta }}{\mathbb{E}} \left[(\nabla_a \hat{r}_\omega(s, a) - \nabla_a r(s, a))^\intercal (\nabla_a \hat{r}_\omega(s, a) - \nabla_a r(s,a)) \right]$. \\
Then,
$\nabla_\theta J(\theta)
= \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta \\ s' \sim p(\cdot|s,a)}}{\mathbb{E}} \left[\nabla_\theta\hat{r}_{\omega}(s, f_\theta(\epsilon; s))|_{\epsilon = g_\theta(a; s)} + \gamma \hat{v}_{\phi}(s')\nabla_\theta \log \pi_{\theta}(a | s) \right]$.
\end{reptheorem}

\begin{proof}
By the definitions of $\phi$ and $\omega$, we have
\begin{align*}
& \omega  = \argmin_{\omega}\underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta }}{\mathbb{E}}\left[(\nabla_a \hat{r}_\omega(s, a) - \nabla_a r(s, a))^\intercal(\nabla_a \hat{r}_\omega(s, a) - \nabla_a r(s, a)) \right] \\
\implies & \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta }}{\mathbb{E}}\left[(\nabla_a \hat{r}_\omega(s, a) - \nabla_a r(s, a))\nabla_\omega \nabla_a \hat{r}_\omega(s, a) \right] = 0 \\
\implies & \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta }}{\mathbb{E}}\left[(\nabla_a \hat{r}_\omega(s, a) - \nabla_a r(s, a))\nabla_\theta   f_\theta(\epsilon; s)|_{\epsilon = g(a; s)} \right] = 0 \\
\implies & \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta }}{\mathbb{E}}\left[\nabla_a \hat{r}_\omega(s, a)\nabla_\theta f_\theta(\epsilon; s)|_{\epsilon = g(a; s)} \right] =  \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta }}{\mathbb{E}}\left[\nabla_a r(s, a)\nabla_\theta f_\theta(\epsilon; s)|_{\epsilon = g(a; s)} \right] \\
\implies & \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta }}{\mathbb{E}}\left[\nabla_\theta\hat{r}_{\omega}(s, f_\theta(\epsilon; s))|_{\epsilon = g_\theta(a; s)} \right] =  \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta }}{\mathbb{E}}\left[\nabla_\theta r(s, f_\theta(\epsilon; s))|_{\epsilon=g_\theta(a; s)} \right],
\end{align*}
and
\begin{align*}
& \phi = \argmin_\phi \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta \\
s' \sim p(\cdot | s, a)}}{\mathbb{E}}\left[(\hat{v}_{\phi}(s') - v(s') )^2\right] \\
\implies &  \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta \\
s' \sim p(\cdot | s, a)}}{\mathbb{E}}\left[(\hat{v}_{\phi}(s') - v(s'))\nabla_\phi \hat{v}_{\phi}(s')\right] = 0 \\
\implies &  \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta \\
s' \sim p(\cdot | s, a)}}{\mathbb{E}}\left[(\hat{v}_{\phi}(s') - v(s'))\nabla_\theta \log \pi_\theta (a | s)\right] = 0 \\
\implies &  \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta \\
s' \sim p(\cdot | s, a)}}{\mathbb{E}}\left[\hat{v}_{\phi}(s')\nabla_\theta \log \pi_\theta (a | s)\right] = \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta \\
s' \sim p(\cdot | s, a)}}{\mathbb{E}}\left[v(s')\nabla_\theta \log \pi_\theta (a | s)\right].
\end{align*}

Combine the above results with Theorem~\ref{thm_rpg}, we have
\begin{align*}
\nabla_\theta J(\theta)
&= \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta \\
s' \sim p(\cdot|s,a)}}{\mathbb{E}}\left[\nabla_\theta r(s, f_\theta(\epsilon; s))|_{\epsilon = g_\theta(a; s)} + \gamma v(s')\nabla_\theta \log \pi_{\theta}(a|s)\right] \\
&= \underset{\substack{s \sim d^{\pi_{\theta}} \\ a \sim \pi_\theta \\ s' \sim p(\cdot|s,a)}}{\mathbb{E}}\left[\nabla_\theta \hat{r}_{\omega}(s, f_\theta(\epsilon; s))|_{\epsilon = g_\theta(a; s)} + \gamma \hat{v}_{\phi}(s')\nabla_\theta \log \pi_{\theta}(a|s)\right].
\end{align*}
\end{proof}

\subsection{Other Related Theorems}

\begin{theorem}[Reparameterization Policy Gradient Theorem]\label{thm_repara}
Suppose that the MDP satisfies Assumption~\ref{assum1} and \ref{assum2}, then
\begin{align*}
\nabla_{\theta} J(\theta)
= \int d^{\pi_{\theta}}\!(s) p({\epsilon}) \nabla_{\theta} f_{\theta}({\epsilon};s) \nabla_{a} q_{\pi_{\theta}}(s, a)|_{a=f_{\theta}({\epsilon};s)} \mathrm{d} {\epsilon} \mathrm{d} s.
\end{align*}
\end{theorem}

\begin{proof}

By the policy gradient theorem, we have
\begin{align*}
\nabla_{\theta} J(\theta)
= \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) q_{\pi_{\theta}}(s,a) \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} a \mathrm{d} s.
\end{align*}

Thus
\begin{align*}
&\nabla_{\theta} J(\theta) \\
=& \int d^{\pi_{\theta}}(s) \pi_{\theta}(a|s) q_{\pi_{\theta}}(s,a) \nabla_{\theta} \log{\pi_{\theta}(a|s)} \mathrm{d} a \mathrm{d} s \\
=& \int d^{\pi_{\theta}}(s) \left( \int q_{\pi_{\theta}}(s,a) \nabla_{\theta} \pi_{\theta}(a|s) \mathrm{d} a \right) \mathrm{d} s \\
=& \int d^{\pi_{\theta}}(s) \left[ \int \nabla_{\theta} ( q_{\pi_{\theta}}(s,a) \pi_{\theta}(a|s) ) \mathrm{d} a - \int \pi_{\theta}(a|s) \nabla_{\theta} q_{\pi_{\theta}}(s,a) \mathrm{d} a \right] \mathrm{d} s \\
=& \int d^{\pi_{\theta}}(s) \left[\nabla_{\theta} \left(\int q_{\pi_{\theta}}(s,a) \pi_{\theta}(a|s) \mathrm{d} a \right) - \int \pi_{\theta}(a|s) \nabla_{\theta} q_{\pi_{\theta}}(s,a) \mathrm{d} a \right] \mathrm{d} s \\
=& \int d^{\pi_{\theta}}(s) \left[\nabla_{\theta} \left(\int p({\epsilon}) q_{\pi_{\theta}}(s,f_{\theta}({\epsilon}; s)) \mathrm{d} {\epsilon} \right) - \int p({\epsilon}) \nabla_{\theta} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \mathrm{d} {\epsilon} \right] \mathrm{d} s \text{\quad (by reparameterization)} \\
=& \int d^{\pi_{\theta}}(s) \left[\int p({\epsilon}) \left(\nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} + \nabla_{\theta} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \right) \mathrm{d} {\epsilon} - \int p({\epsilon}) \nabla_{\theta} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \mathrm{d} {\epsilon} \right] \mathrm{d} s \\
=& \int d^{\pi_{\theta}}(s) \; p({\epsilon}) \nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \mathrm{d} {\epsilon} \mathrm{d} s.
\end{align*}
\end{proof}

\begin{remark}
This theorem is a direct application of reparameterization to the gradient of the policy objective. It is understood to be known but is not formally presented or derived for policy gradients in any existing work.
We include it here for completeness as well as a useful theoretical tool.
\end{remark}

\begin{corollary}[Deterministic Policy Gradient Theorem]\label{cor_dpg}
Suppose that the MDP satisfies Assumption~\ref{assum1} and \ref{assum2}, for a deterministic policy $\mu_{\theta}(s)$, we have
\begin{align*}
\nabla_{\theta} J(\theta)
= \int d^{\mu_{\theta}}(s) \nabla_{\theta} \mu_{\theta}(s) \nabla_{a} q_{\mu_{\theta}}(s,a)|_{a=\mu_{\theta}(s)} \mathrm{d} s.
\end{align*}
\end{corollary}

\begin{proof}
Let $p({\epsilon})$ be the delta function. Thus $\int_{{\epsilon}} p({\epsilon}) f_{\theta}({\epsilon}; s) \mathrm{d} {\epsilon} = f_{\theta}(0;s)$. Furthermore, let $f_{\theta}(0;s)=\mu_{\theta}(s)$. By Theorem~\ref{thm_repara},
\begin{align*}
\nabla_{\theta} J(\theta)
=& \int d^{\pi_{\theta}}(s) \left(\int p({\epsilon}) \nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \mathrm{d} {\epsilon} \right) \mathrm{d} s \\
=& \int d^{\pi_{\theta}}(s) \nabla_{\theta} f_{\theta}(0;s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}(0;s)} \mathrm{d} s \\
=& \int d^{\pi_{\theta}}(s) \nabla_{\theta} \mu_{\theta}(s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=\mu_{\theta}(s)} \mathrm{d} s.
\end{align*}
\end{proof}

\begin{remark}
Both DPG and DDPG are proposed based on this theorem, which is first proved by~\citet{silver2014deterministic}. It is limited to deterministic policies and thus can be deduced as a corollary of Theorem~\ref{thm_repara}, which is applicable to stochastic policies as well.
\end{remark}

\begin{corollary}[Entropy-regularized Reparameterization Policy Gradient Theorem]\label{cor_sac}
Consider the entropy-regularized values $v_{\pi_{\theta}}(s_0) = \mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty} \gamma^{t}(r(s_t, a_t) + \alpha {\mathcal{H}}(\pi_{\theta}(\cdot | s_t)))\right]$ and $q_{\pi_{\theta}}(s_0, a_0) = \mathbb{E}_{\pi} \left[\sum_{t=0}^{\infty} \gamma^{t} r(s_t, a_t) + \alpha \sum_{t=1}^{\infty} \gamma^{t} {\mathcal{H}}(\pi_{\theta}(\cdot | s_t)) \right]$, where ${\mathcal{H}}(p) = - \int_{x} p(x) \log{p(x)} \mathrm{d} x$ is the differential entropy for probability density function $p(x)$, and $\alpha$ is a positive constant. Suppose that the MDP satisfies Assumption~\ref{assum1} and \ref{assum2}, then
\begin{align*}
\nabla_{\theta} J(\theta)
= \int d^{\pi_{\theta}}(s) p({\epsilon}) \left[\nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s, a)|_{a=f_{\theta}({\epsilon}; s)} - \alpha \nabla_{\theta} \log{\pi_{\theta}(f_{\theta}({\epsilon}; s)|s)} \right] \mathrm{d} {\epsilon} \mathrm{d} s.
\end{align*}
\end{corollary}

\begin{proof}
By definition, we have
\begin{align*}
v_{\pi_{\theta}}(s) &= \mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty} \gamma^{t}(r(s_t, a_t) + \alpha {\mathcal{H}}(\pi_{\theta}(\cdot | s_t))) \mid s_0=s \right], \\
q_{\pi_{\theta}}(s,a) &= \mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty} \gamma^{t} r(s_t, a_t) + \alpha \sum_{t=1}^{\infty} \gamma^{t} {\mathcal{H}}(\pi_{\theta}(\cdot | s_t)) \mid s_0=s, a_0=a \right],
\end{align*}
where ${\mathcal{H}}(p) = - \int_{x} p(x) \log{p(x)} \mathrm{d} x$ is the differential entropy for probability density function $p(x)$.

Then $v_{\pi_{\theta}}$ and $q_{\pi_{\theta}}$ are connected by
\begin{align*}
v_{\pi_{\theta}}(s)
&= \mathbb{E}_{\pi}[q_{\pi_{\theta}}(s,a)] + \alpha {\mathcal{H}}(\pi_{\theta}(\cdot | s)) \\
&= \int \pi_{\theta}(a|s) (q_{\pi_{\theta}}(s,a) - \alpha \log \pi_{\theta}(a|s)) \mathrm{d} a \\
&= \int p({\epsilon}) \left(q_{\pi_{\theta}}(s,f_{\theta}({\epsilon}; s)) - \alpha \log{\pi_{\theta}(f_{\theta}({\epsilon}; s) | s)} \right) \mathrm{d} {\epsilon},
\end{align*}
and the Bellman equation for $q_{\pi_{\theta}}$ is
\begin{align*}
q_{\pi_{\theta}}(s,a)
&= \mathbb{E}_{\pi}\left[r(s, a) + \gamma(q_{\pi_{\theta}}(s',a') + \alpha {\mathcal{H}}(\pi_{\theta}(\cdot | s'))) \right] \\
&= r(s,a) + \gamma \mathbb{E}[v_{\pi_{\theta}}(s')]
= r(s,a) + \gamma \int p(s' | s,a) v_{\pi_{\theta}}(s') \mathrm{d} s'.
\end{align*}

Then
\begin{align*}
& \nabla_{\theta} v_{\pi_{\theta}}(s) \\
=& \nabla_{\theta} \left( \int \pi_{\theta}(a|s) q_{\pi_{\theta}}(s,a) \mathrm{d} a \right) + \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)) \\
=& \nabla_{\theta} \left( \int p({\epsilon}) q_{\pi_{\theta}}(s,f_{\theta}({\epsilon}; s)) \mathrm{d} {\epsilon} \right) + \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)) \\
=& \int p({\epsilon}) \nabla_{\theta} q_{\pi_{\theta}}(s,f_{\theta}({\epsilon}; s)) \mathrm{d} {\epsilon} + \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)) \\
=& \int p({\epsilon}) \left(\nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} + \nabla_{\theta} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \right) \mathrm{d} {\epsilon} + \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)) \\
=& \int p({\epsilon}) \left(\nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} + \gamma \int p(s' | s,f_{\theta}({\epsilon}; s)) \nabla_{\theta} v_{\pi_{\theta}}(s') \mathrm{d} s' \right) \mathrm{d} {\epsilon} + \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)) \\
=& \int p({\epsilon}) \nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \mathrm{d} {\epsilon} + \gamma \int p({\epsilon}) p(s' | s,f_{\theta}({\epsilon}; s)) \nabla_{\theta} v_{\pi_{\theta}}(s') \mathrm{d} {\epsilon} \mathrm{d} s' + \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)) \\
=& \int p({\epsilon}) \nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \mathrm{d} {\epsilon} + \gamma \int p(s \to s', 1, \pi_{\theta}) \nabla_{\theta} v_{\pi_{\theta}}(s') \mathrm{d} s' + \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)),
\end{align*}
where $p(s \to s', 1, \pi_{\theta}) = \int p({\epsilon}) p(s' | s,f_{\theta}({\epsilon}; s)) \mathrm{d} {\epsilon}$.

Now iterating this formula we have
\begin{align*}
& \nabla_{\theta} v_{\pi_{\theta}}(s) \\
=& \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)) + \int p({\epsilon}) \nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \mathrm{d} {\epsilon} + \gamma \int p(s \to s', 1, \pi_{\theta}) \nabla_{\theta} v_{\pi_{\theta}}(s') \mathrm{d} s' \\
=& \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)) + \int p({\epsilon}) \nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \mathrm{d} {\epsilon} + \gamma \int p(s \to s', 1, \pi_{\theta}) \\
& \left(\alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s')) + \int p({\epsilon}') \nabla_{\theta} f_{\theta}({\epsilon}'; s') \nabla_{a} q_{\pi_{\theta}}(s',a)|_{a=f_{\theta}({\epsilon}'; s')} \mathrm{d} {\epsilon}' + \gamma \int p(s' \to s'', 1, \pi_{\theta}) \nabla_{\theta} v_{\pi_{\theta}}(s'') \mathrm{d} s''\right) \mathrm{d} s' \\
=& \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)) + \int p({\epsilon}) \nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \mathrm{d} {\epsilon} \\
&+ \alpha \gamma \int p(s \to s', 1, \pi_{\theta}) \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s')) \mathrm{d} s' + \gamma \int p(s \to s', 1, \pi_{\theta}) p({\epsilon}') \nabla_{\theta} f_{\theta}({\epsilon}'; s') \nabla_{a} q_{\pi_{\theta}}(s',a)|_{a=f_{\theta}({\epsilon}'; s')} \mathrm{d} {\epsilon}' \mathrm{d} s' \\
&+ \gamma^2 \int p(s \to s', 1, \pi_{\theta}) p(s' \to s'', 1, \pi_{\theta}) \nabla_{\theta} v_{\pi_{\theta}}(s'') \mathrm{d} s'' \mathrm{d} s' \\
=& \alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s)) + \int p({\epsilon}) \nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} \mathrm{d} {\epsilon} \\
&+ \alpha \gamma \int p(s \to s', 1, \pi_{\theta}) \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s')) \mathrm{d} s' + \gamma \int p(s \to s', 1, \pi_{\theta}) p({\epsilon}') \nabla_{\theta} f_{\theta}({\epsilon}'; s') \nabla_{a} q_{\pi_{\theta}}(s',a)|_{a=f_{\theta}({\epsilon}'; s')} \mathrm{d} {\epsilon}' \mathrm{d} s' \\
&+ \gamma^2 \int p(s \to s', 2, \pi_{\theta}) \nabla_{\theta} v_{\pi_{\theta}}(s') \mathrm{d} s' \\
=& \hdots\\
=& \int \sum_{t=0}^{\infty} \gamma^{t} p(s \to s', t, \pi_{\theta}) \left(\alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s')) + \int p({\epsilon}') \nabla_{\theta} f_{\theta}({\epsilon}'; s') \nabla_{a} q_{\pi_{\theta}}(s',a)|_{a=f_{\theta}({\epsilon}'; s')} \mathrm{d} {\epsilon}' \right) \mathrm{d} s',
\end{align*}
where $p(s \to s'', t+1, \pi_{\theta}) = \int p(s \to s', t, \pi_{\theta}) p(s' \to s'', 1, \pi_{\theta}) \mathrm{d} s'$.

Furthermore,
\begin{align*}
\nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s))
= - \nabla_{\theta} \int \pi_{\theta}(a|s) \log{\pi_{\theta}(a|s)} \mathrm{d} a
= - \nabla_{\theta} \int p({\epsilon}) \log{\pi_{\theta}(f_{\theta}({\epsilon}; s) | s)} \mathrm{d} {\epsilon}
= - \int p({\epsilon}) \nabla_{\theta} \log{\pi_{\theta}(f_{\theta}({\epsilon}; s) | s)} \mathrm{d} {\epsilon}.
\end{align*}

Then
\begin{align*}
\nabla_{\theta} v_{\pi_{\theta}}(s)
=& \int \sum_{t=0}^{\infty} \gamma^{t} p(s \to s', t, \pi_{\theta}) \left(\alpha \nabla_{\theta} {\mathcal{H}}(\pi_{\theta}(\cdot | s')) + \int p({\epsilon}') \nabla_{\theta} f_{\theta}({\epsilon}'; s') \nabla_{a} q_{\pi_{\theta}}(s',a)|_{a=f_{\theta}({\epsilon}'; s')} \mathrm{d} {\epsilon}' \right) \mathrm{d} s' \\
=& \int \sum_{t=0}^{\infty} \gamma^{t} p(s \to s', t, \pi_{\theta}) p({\epsilon}') \left(\nabla_{\theta} f_{\theta}({\epsilon}'; s') \nabla_{a} q_{\pi_{\theta}}(s',a)|_{a=f_{\theta}({\epsilon}'; s')} - \alpha \nabla_{\theta} \log{\pi_{\theta}(f_{\theta}(s,{\epsilon}') | s)} \right) \mathrm{d} {\epsilon}' \mathrm{d} s'.
\end{align*}

Finally,
\begin{align*}
\nabla_{\theta} J(\theta)
=& \nabla_{\theta} \int p_0(s) v_{\pi_{\theta}}(s) \mathrm{d} s = \int p_0(s) \nabla_{\theta} v_{\pi_{\theta}}(s) \mathrm{d} s \\
=& \int \sum_{t=0}^{\infty} \gamma^{t}  p_0(s) p(s \to s', t, \pi_{\theta}) p({\epsilon}) \left(\nabla_{\theta} f_{\theta}({\epsilon}; s') \nabla_{a} q_{\pi_{\theta}}(s',a)|_{a=f_{\theta}({\epsilon}; s')} - \alpha \nabla_{\theta} \log{\pi_{\theta}(f_{\theta}({\epsilon}; s) | s)} \right) \mathrm{d} {\epsilon} \mathrm{d} s' \mathrm{d} s \\
=& \int d^{\pi_{\theta}}(s) \; p({\epsilon}) \left(\nabla_{\theta} f_{\theta}({\epsilon}; s) \nabla_{a} q_{\pi_{\theta}}(s,a)|_{a=f_{\theta}({\epsilon}; s)} - \alpha \nabla_{\theta} \log{\pi_{\theta}(f_{\theta}({\epsilon}; s) | s)} \right)  \mathrm{d} {\epsilon} \mathrm{d} s.
\end{align*}
\end{proof}

\begin{remark}
There is a similar result presented by~\cite{haarnoja2018soft} for SAC. They obtain it by minimizing the Kullback-Leibler divergence between the new policy and the policy derived from the exponential of the soft Q-function. However, we derive it by directly minimizing the objective with the help of the RP gradient. This corollary provides an alternative way to understand SAC.
\end{remark}

\section{Hyper-parameter Settings for Simulation Tasks}

All our experiments were performed only with CPUs (AMD EPYC 7601 32-Core Processor).

\subsection{LQG}

The parameters used for the LQG task are
\begin{align*}
A=
\begin{bmatrix}
0.01 & 0\\
0 & 0.01
\end{bmatrix}
;\quad
B=
\begin{bmatrix}
1e-4 & 0\\
0 & 1e-4
\end{bmatrix}
;\quad
Q=
\begin{bmatrix}
1 & 0\\
0 & 1
\end{bmatrix}
;\quad
R=
\begin{bmatrix}
1 & 0\\
0 & 1
\end{bmatrix}
;\quad
\Sigma=
\begin{bmatrix}
0.1 & 0\\
0 & 0.1
\end{bmatrix}
;\quad
\textbf{s}_0 = [0.5, 0.5].
\end{align*}

The discount factor is $\gamma=0.99$. The number of steps for each episode is 100. The policy parameter is chosen randomly to be $\theta=[-1.1104430687690852, -1.3649958298432607]$.

\subsection{MuJoCo}

\begin{table}[thbp]
\caption{The hyper-parameter settings for PPO and RPG on MuJoCo tasks.} \label{tb_hyper_ppo_rpg}
\begin{center}
\begin{tabular}{lcc}
\toprule
\bf{Hyper-parameter} & \bf{PPO} & \bf{RPG} \\
\midrule
Policy network LR & $3 \times 10^{-4}$ & $3 \times 10^{-4}$ \\
Value network LR & $10^{-3}$ & $10^{-3}$ \\
Reward network LR & None & $10^{-3}$ \\
Hidden layers & [64, 64] & [64, 64] \\
Optimizer & Adam & Adam \\
Time-steps per iteration & 2048 & 2028 \\
Number of epochs & 10 & 10 \\
Mini-batch size & 64 & 64 \\
Discount factor ($\gamma$) & 0.99 & 0.99 \\
GAE parameter ($\lambda$) & 0.95 & 0.95 \\
PPO Clipping (${\epsilon}$) & 0.2 & 0.2 \\
Target KL divergence & 0.01 & 0.01 \\
State Clipping & [-10, 10] & [-10, 10] \\
Gradient clipping & 2 & 2 \\
\bottomrule
\end{tabular}
\end{center}
\end{table}

\begin{algorithm}[b]
\caption{PPO} \label{algo_ppo}
\begin{algorithmic}
    \STATE Input: initial policy parameters $\theta$, initial value estimate parameters $\phi$.
    \FOR{$k=0,1,2,\dots$}
        \STATE Collect set of trajectories $\mathcal{D}=\{\tau_{i}\}$ by running policy $\pi_{\theta}$ in the environment. \\
        \STATE Compute returns $G_t$. \\
        \STATE Compute advantage estimates $H_t = H_t^{\text{GAE}(\lambda)}$.
        \STATE Normalize $H_{t}$ using the sample mean and standard deviation of all advantage estimates.
        \FOR{epoch $=0,1,2,\dots$}
        \STATE Shuffle and slice trajectories $\mathcal{D}$ into mini-batches.
            \FOR{each mini-batch $B$}
                \STATE Set $\hat{\rho}_t(\theta)$ according to Equation~\ref{eq_clip_new}.
                \STATE Update policy parameters $\theta$ by maximizing the objective: $\mathbb{E}_{B}[\hat{\rho}_t(\theta) H_{t}]$.
                \STATE Update value estimate parameters $\phi$ by minimizing: $\mathbb{E}_{B}[(\hat{v}_{\phi}(S_{t})-G_t)^{2}]$.
            \ENDFOR
        \ENDFOR
    \ENDFOR
\end{algorithmic}
\end{algorithm}

\section{Real-Robot Task Description}

For the real-robot experiment in this paper, we adopted the \textit{UR-Reacher-2} task from Mahmood et al.\ (2018).
The task runs on a UR5 robot arm from Universal Robots.
By moving only two joints of the robot---the base and the elbow joints---an agent can move the robot around in two dimensions above a table to which the base joint is attached.
The elbow joint connects the base joint to the fingertip.

Each episode lasts for 100 time steps of 40ms duration or 4 seconds in total.
At the start of each episode the fingertip is reset to a fixed position above the robot, and a random target is drawn uniformly from a rectangular area around the fingertip.
The goal of the agent is to get the fingertip as close and as fast as possible to the target by moving the base and elbow joints.
The reward at each time step has two terms.
The first term is the negative of the distance from fingertip to target.
The second term rewards the agent equal to a Gaussian function that has its maximum value when the fingertip is on the target and reduces gradually as the fingertip moves away.
In other words, the Gaussian function is centered at the target and evaluated at the distance from fingertip to target.
This second reward term is used to encourage precision when fingertip and target are close by.
The observation vector includes the angular position and velocity of the base and elbow joints, the vector from the fingertip to the target, and the previous action.
The action vector contains the target angular velocity that the base and elbow joints should be set to.

In the original environment, whenever the learning agent pushed the fingertip outside a 2-dimensional rectangular boundary above the robot, a manually written script would move the fingertip to be inside the boundary.
To reduce the frequency of scripted position corrections, we relaxed their movement constraints to a much larger area similarly to Farrahi and Mahmood (2020).
The fingertip can now move freely as long as it is not too close to the table.
The base and elbow joints can now rotate farther as well.
We additionally made a few changes to the environment's code, without changing the task setup, to eliminate the protective stops that occurred when operating the joints at high speeds.

redpajama_set_name: RedPajamaArXiv


There is a Muriel Louise Nelson (also recorded as Louise Muriel) listed in the 1939 register. She was living in Temple House, High Street, Meldreth. Her date of birth is given as 19 March 1907. I hope this helps.


The CDC and FDA are worried decades worth of work in reducing youth tobacco use has been erased by e-cigarettes.
About 4.9 million middle and high school students used some type of tobacco product in 2018, according to the CDC. The CDC believes the rise in teen tobacco use is driven by e-cigarettes.
The number of U.S. high school students using e-cigarettes went up 78% between 2017 and 2018, according to the FDA. There are now 3.05 million high school students using e-cigarettes, a 78% increase from 2017. In middle schools there are 570,000 students using e-cigarettes, or 4.9% of middle-school students.
Officials are worried about the effect e-cigarettes will have on the nation's health.
"These new data show that America faces an epidemic of youth e-cigarette use, which threatens to engulf a new generation in nicotine addiction," says Health and Human Services Secretary Alex Azar.
The CDC and FDA believe the rise in e-cigarette use is tied to the trendiness of things like Juul and flavored vapes. Many e-cigarettes are slim, discreet, and come in flavors like mango and mint. Juuls and other e-cigarettes get plenty of free marketing from social media. One Juul pod has the same amount of nicotine as a pack of cigarettes, according to Fortune magazine.
When teenagers use nicotine they might become addicted early in life, according to the FDA. The young developing brain doesn't handle nicotine and other chemicals. Additionally, teens who start smoking with e-cigarettes are more likely to transition to regular cigarettes.
E-cigarettes are useful for helping cigarette smokers slowly wean themselves off of nicotine. Officials are trying to figure out how to keep nicotine products out of teenager's hands while still keeping e-cigarettes available to the general market. The CDC says strategies to reduce youth tobacco use include prohibit the sale of flavored tobacco products, starting a hard-hitting media campaign against tobacco, or increasing the prices of tobacco products.


Efficiency and organization seem to go hand in hand these days. When you are dealing with dozens of network devices, and most of the time simultaneously, it is vital that they are being regulated and ordered.
This is why Modern IT Job Scheduling Software is becoming more important than ever. The more tasks a server needs to handle, the more sophisticated your scheduler should be.
Skybot is offering a free trial download of their scheduling software, which is a perfect opportunity to realize the convenience of organization.
eBooks, white papers and essential guides.


This article is a guide on how to change your NetWeaver password for the control panel.
See our How To Find The cPanel article if you are struggling.
Using the search bar near the top of the page search for 'Password & Security'.
Select the 'Password & Security' link.
You should now be on the page shown above. Type your old password and desired new password in the required fields or if you are struggling to meet the strength requirements, use the password generator on the right.


Nothing Can Rain on Our Parade: Am Yisrael Chai!
Flyers and ads for today's Salute to Israel parade announced that the parade would proceed, rain or shine. For a while, it looked as though we would have no shine, since the weather forecast ominously predicted rain for today. Since something like a little rain would not stop me from coming to support Israel and cheer on those marching, I packed an umbrella and went off to 5th Avenue. It became apparent to me soon enough that not only was my umbrella unnecessary, but that bringing sunglasses and sunscreen would have been much more worthwhile. The threatening clouds were only present for the beginning of the parade, and they soon disappeared, allowing the sun to shine down and illuminate the cheerful proceedings.
Men, women, students, children, and dogs were all there to display their pride for Israel. Both those marching and those observing waved Israeli flags and cheered. And one of the things I love most about these parades for Israel is the incredible sense of unity. There is a connection, a special bond, between all the people who show up to the parade. It does not matter who they are or how they identify themselves -- if they are religious, non-religious, Orthodox, Conservative, wearing a kippa or bareheaded, wearing a short skirt or pants -- they are all there together to show their love for Israel and the Jewish nation. And that's beautiful. Seeing all these people -- thousands of them -- together in one place and for a common cause is beautiful. I could have cried out of pride.
I spent a couple of hours at the parade and then made my way through Central Park to the Summerstage, where the Israel Day Concert was taking place. At that point, the sun came out full force and I felt myself baking in its heat. It cooled down eventually though, so I stayed for a while. There was music, dancing, and singing as everyone celebrated together.
Next time though, I am bringing sunscreen even if rain is expected. My face is not looking happy right now -- I don't think I've been this burnt since last summer!
Their signs said things like "I found my heart/brain/courage in Israel"
To quote my niece: "Why don't you just quickly choose a chosson and get married?"
And I also sense that I won't be hearing the end of it until I hand her an invitation to my wedding... which still doesn't have anyone in the main role.
Operation Nifty Package was an operation launched by Navy SEALs to prevent the escape of Noriega. They sank Noriega's boat and destroyed his jet at a cost of 4 killed and 9 wounded. Military operations continued for several weeks, mainly against military units of the Panamanian Army. Noriega remained at large for several days, but realizing he had few options in the face of a massive manhunt, with a one million dollar reward for his capture, he obtained refuge in the Vatican diplomatic mission in Panama City. The US military's psychological pressure on him and diplomatic pressure on the Vatican mission, however, was relentless, as was the playing of loud rock-and-roll music day and night in a densely populated area. The report of the Office of the Chairman of the Joint Chiefs of Staff maintains that the music was used principally to prevent parabolic microphones from being used to eavesdrop on negotiations, and not as a psychological weapon based around Noriega's supposed loathing of rock music. Noriega finally surrendered to the U.S. military on January 3, 1990. He was immediately put on an MC-130 Combat Talon aircraft and extradited to the United States.
This is one of those things that just make you smile as you study for a history test. :] Who says there is no humor in history?
My niece and nephew were at my house for Shabbat, and at one point, my niece turned to me and asked with a straight face, "When are you getting married?" My father and I exchanged looks, amused by the question and by the fact that it was my eight-year-old niece who was asking it. I cautiously replied that I might first have to find someone to marry before I can start planning the wedding, and I inquired why she was asking this all of a sudden. Her reply? She wants to wear a beautiful gown and wants to know when she can do that. I promised her that I would let her know as soon as I knew the answer myself.
It was her next question though that really surprised me. She wanted to know at what age I want to get married -- young or older. She elaborated that young was at 19 or 20 and older is basically anything above that number.
So this is just to let all you girls know -- if you're above 20, you are apparently older. Who knew?


Living in the modern world, we all got so accustomed to working WIFI, Instagram, running water and fridges we can barely imagine what it would feel like to live a day in a life of, say, Tsaatan people. They move 5 to 10 times per year, building huts when the temperature is -40 and herd reindeer for transportation, clothing and food.
That's why British photographer Jimmy Nelson decided to travel the world for three years, visiting 35 tribes in all five continents, documenting their lives and customs before they completely disappear for his latest book "Before They Pass Away".
These tribes serve as a reminder that we live in a very colorful world, one with rich histories, differences, unique identities and stories. A world which needs respect, conservation and preservation.
Be amazed at the environments, cultural diversity, and colors of these tribes from around the globe.


text:
Starts Mar 09 2h 10m View in admin
Life of Pi (Broadway)
Buy tickets from $49 on TodayTix
Score it
Life of Pi (Broadway) NYC Reviews and Tickets
This show has not been reviewed yet
Members say
The award-winning play, based on the novel by Yann Martel, comes to Broadway after a hit London run.
After a cargo ship goes under in the middle of the Pacific Ocean, a sixteen-year-old boy named Pi is left stranded on a lifeboat with the most unusual of companions: a Royal Bengal tiger. With time and nature against them, the pair must learn to trust each other if they are to survive the ordeal.
Featuring puppetry and stunning visuals, this new play brings to life Yann Martel's classic novel (previously the basis for an Academy Award-winning film) in a never before seen way.
The London production garnered five Olivier Awards including Best New Play.
Daryl Roth
Hal Luftig
Lolita Chakrabarti

redpajama_set_name: RedPajamaCommonCrawl


text:
Win a Pair of Tickets to See Whiskey Myers in Lubbock, TX
We're giving away a pair of tickets to see Whiskey Myers live in Lubbock in 2022. Wanna win? Just click on any of the boxes below. We'll pick five random winners* on Friday, November 19th, 2021. Good luck! *This is a multi-station contest running on lonestar995fm...
Whiskey Myers Is Bringing the Drinks & Fun to Lubbock
Country act Whiskey Myers is coming to Lubbock next year and you could be going.
Whiskey Myers Guitarist John Jeffers Fractures Skull in ATV Accident
Jeffers also lost hearing in one ear and received many stitches.
Whiskey Myers Officially Cancel Lubbock Concert
Whiskey Myers have announced they're canceling their upcoming Lubbock concert due to the coronavirus.
Whiskey Myers Announce Lubbock Show
Whiskey Myers just added a Lubbock date to their tour.
Whiskey Myers Announce Massive U.S. Tour for 2017
Whiskey Myers are just back in the states from a sold out UK tour, but have zero plans to chill. The East Texas rockers will be taking their latest album, Mud, to dance halls and clubs across the country beginning in January. Mud was recorded with Grammy winning producer Dave Cobb and debuted at No...
Ever Wondered 'Where Does The Name 'Whiskey Myers' Come From?'


Walex Products Company is committed to protecting your privacy and using technology that gives you the most powerful and safe online experience. This Statement of Privacy applies to the Walex website and governs data collection and usage. By using the Walex website, you consent to the data practices described in this statement.
Walex and third-party vendors, including Google, use analytics data and the DoubleClick cookie to serve ads based on a user's prior visits to our website. You may opt out of the DoubleClick cookie or customize Google Display Network ads using the Ads Preferences Manager. You may also opt out of Google Analytics by using the Google Analytics Opt-out Browser Add-on. Google has additional information available about their Remarketing Privacy Guidelines, Policies, and Restrictions.
This policy was last modified on July 22, 2015.


This is a basic example of a 1500-calorie diet for a period of one week. You can modify the diet every week to your personal taste.
This is a very radical calorie diet and it might not have the sufficient nutrients your body needs to function properly and stay healthy. That is why it is important to be monitored and to follow the diet properly after the approval of a specialist. The lack of nutrients must be controlled because it can cause major physical and mental health disruption to people who do not follow it properly or without any control.
Nice Weight Loss tips you have given here. I hope you will post some more.


We take on all the elements of finding, planning, designing and managing your new office.
Whether you are expanding, downsizing or refurbishing, let us know and we'll do the thinking for you !
Our areas of search includes but not limited to office space, commercial or industrial land & retail space.
If you're into buying or leasing, allow us to source your ideal space at the location of your choice and present the options to you. From negotiations with the landlord up until the agreement is signed, we are with you all the way.
During this stage our team can start the test fit planning to ensure the size meets your requirements. Our Solution Professionals will advise you on the preferred square footage that is required and ensure efficient space utilization.
Start saving on your property personnel & administrators handling non-core, non-revenue generating duties.
The Office Management & Maintenance is an extension of our continued commitment after handing over the office. As a single point of contact, our team are at hand with whatever problems we may be presented with, from warm A/C systems, faulty phone lines, power restoration, system backup's right up to the blinking light bulb !
Our Solution Professionals are one call away at all times whether you are a single office unit or a multi branch set up.
The virtual office is the most cost effective way to provide a professional image to your business. Driven by the need for more flexible ways of working, businesses can be done from anywhere today.
The virtual office is ideally suited to entrepreneurs, freelancers and small businesses that requires a professional working support environment on an all-inclusive monthly rate.
'BORED' meetings will never be the same again.
We understand business isn't about the time you spend at your desk. That's why our unrivalled selection of rooms are available for all requirements, from meetings, conferences, team training to product launches, workshops and private events.
Let us know the dates and your preferences – we promise to get you the best deal in town.
Our room caters for businesses of all sizes, industries and requirements, from conference suites and seminar rooms to training centres and auditoriums.


text:
Download About Help
v 1.1 TRY NEW BETA SITE
Enter dbSNP IDs, 0-based coordinates, BED files, VCF files, GFF3 files (hg19).
Use RegulomeDB to identify DNA features and regulatory elements in non-coding regions of the human genome by entering ...
dbSNP IDs
Single nucleotides
A chromosomal region
Enter dbSNP ID(s) (example) or upload a list of dbSNP IDs to identify DNA features and regulatory elements that contain the coordinate of the SNP(s).
Enter hg19 coordinates for a single nucleotide as 0-based (example) coordinates or in a BED file (example), VCF file (example), or GFF3 file (example). These coordinates will be mapped to a dbSNP IDs (if available) in addition to identifying DNA features and regulatory elements that contain the input coordinate(s).
Enter hg19 chromosomal regions, such as a promoter region upstream of a gene, as 0-based (example) coordinates or in a BED file (example) or GFF3 file (example). All dbSNP IDs with an allele frequency >1% that are found in this region will be used to identify DNA features and regulatory elements that contain the coordinate of the SNP(s).
A project of the Center for Genomics and Personalized Medicine at Stanford University.
RegulomeDB (TM) Copyright ©2011 The Board of Trustees of Leland Stanford Junior University. Permission to use the information contained in this database was given by the researchers/institutes who contributed or published the information. Users of the database are solely responsible for compliance with any copyright restrictions, including those applying to the author abstracts. Documents from this server are provided "AS-IS" without any warranty, expressed or implied. The RegulomeDB project at Stanford University is supported by a Genome Research Resource Grant from the US National Human Genome Research Institute, part of the US National Institutes of Health.


text:
Go to page : 1 ... 5, 6, 7 ... 23 ... 40
GarryB on Sun Feb 16, 2014 9:43 am
Hey thanx Garry . So , just to be clear are the 76 mm guns been upgraded to 100mm , 130 mm or are they replacing the 76mm with news 100 mm , 130 mm guns ?
The corvettes that are the subject of this thread would have gotten an armament in Soviet times of a single 76mm gun and probably either two 30mm gatling turrets or perhaps a single turret and an SA-N-4 naval OSA equivalent and perhaps 4-6 large anti ship missiles like SS-N-22 Moskit.
Fitting a 100mm gun would have been out of the question because of the weight of the 100mm guns.
The Corvettes being made now will be fitted with 100mm guns that weigh about the same as the old 76mm gun but with a much heavier shell fired over much greater range, plus much more heavily armed in terms of other weapons and choices.
For instance the single UKSK launcher will allow it to carry 8 Onyx supersonic anti ship missiles, but could just as easily carry anti submarine missiles that deliver torpedoes to targets 40km away at mach 2.5, or land attack cruise missiles able to hit targets 2,500km away or any combination of these weapons.
They have developed new light weight 100mm and 130mm guns that are the weight of 76mm and 100mm guns respectively.
The new 152mm gun was developed with the Army and in the army it is entering service as the Koalition. It has an effective range of about 70km and has Glonass/laser guidance to allow precision targeting.
TR1 on Tue Mar 25, 2014 8:54 am
https://www.youtube.com/watch?v=QVgMgUk2V70
Great photos of the frigates on patrol! You can see the combat information center.
Stealthflanker
Location : Indonesia
Stealthflanker on Tue Mar 25, 2014 10:16 am
TR1 wrote: https://www.youtube.com/watch?v=QVgMgUk2V70
Cool :3
TR1 on Fri Apr 04, 2014 1:47 am
http://www.flotprom.ru/news/?ELEMENT_ID=164194
What the.
The contract with Aur over 20380 construction had DOUBLE the price the MOD is paying for one ship. The first ship was for 6.5 billion rubles, now the contract is for 13 billion rubles.
Yes, you heard that right. For that sort of money Amur better turn into a magic yard.
And this isn't even for 20385!!!
xeno on Fri Apr 04, 2014 2:29 am
Amur Shipyard has been totally ruined during past 20 years. MOD knows it very well and is actually rebuilding the shipyard(infrastructure and staff recruiting and training, etc).
BTW I would rather pay 400M dollars for a 20381 instead of a 11356M...
sepheronx
sepheronx on Fri Apr 04, 2014 3:39 am
TR1 wrote: http://www.flotprom.ru/news/?ELEMENT_ID=164194
Well, if it comes with restructuring the facility, modernize it, and training staff, then I have to say it will be money well spent. If not, then wow.
Location : India
Austin on Fri Apr 04, 2014 7:38 am
Looking at the way SY rips of money from MOD ......I am quite certain they wont be able to meet Navy SAP 2020 needs within alloted money ...... not to mentioned the economy problem does not help either.
Generally for production batches the cost is lower over lead ship but here is exactly the opposite.
TR1 on Tue Apr 08, 2014 8:50 am
TR1 on Sun Apr 20, 2014 8:26 am
http://bmpd.livejournal.com/821840.html#comments
First 20380 from Amur is expected at the earliest in late 2015.
So probably 2016.
xeno on Tue Apr 22, 2014 12:27 pm
So, finally 9M96 from 20381...
Location : Slovenia
medo on Tue Apr 22, 2014 8:17 pm
xeno wrote:
So Redut now work normally as planed? I hope it will soon become operational.
calripson
Stoikiy
calripson on Sat May 24, 2014 2:59 am
When is the Stoikiy supposed to be commissioned ? She has been in trials for a long time.
TR1 on Mon May 26, 2014 10:11 pm
The superstructure of the lead 20835 ship, Gremyshy, is completed and being sent by barge to Severnaya.
http://www.wrk.ru/forums/attachment.php?item=389551&download=2&type=.jpg
TR1 on Thu May 29, 2014 11:03 am
Stoiky almost done.
TR1 on Thu May 29, 2014 9:07 pm
calripson wrote: When is the Stoikiy supposed to be commissioned ? She has been in trials for a long time.
It has been handed over finally.
GarryB on Fri May 30, 2014 9:39 am
That really only applies to a shipyard ready for production.
In this case the shipyard was in poor condition and would scramble to build a ship properly.
Getting a second order wont help that much unless there is a profit padded in there to pay for infrastructure.
Very simply when you are paid to build something you work out materials and labour costs and use any profits to reinvest into the shipyard with new equipment, extra training and personnel.
When you are tragically underfunded you take the first money you are paid and try to pay off some debt so the subcontractors you need to build components will continue to work with you. You scrimp and scrape parts together at the lowest cost so you can invest more of the money into the infrastructure so future contracts can be tendered for.
the dramatic increase in price is likely largely for infrastructure and tooling and training costs which should lead to improved production performance without the scrimping and lead to the costs going down later and production speed increasing.
TR1 on Mon Jun 02, 2014 9:44 pm
http://tvzvezda.ru/news/forces/content/201406020307-p0oe.htm#
Finally footage of Soobrazitelny firing Redut.
1.) You can clearly see its one tube per cell.
2.) Man that side thruster makes a hideous smoke.
navyfield on Mon Jun 02, 2014 11:33 pm
so just 12 redut missiles on that ship ?
TR1 on Tue Jun 03, 2014 12:44 am
navyfield wrote: so just 12 redut missiles on that ship ?
Yes, as I thought from the start.
It is a small ship, Redut is a pretty potent system on a corvette.
Morpheus Eberhardt
Redut
Morpheus Eberhardt on Tue Jun 03, 2014 12:17 pm
xeno on Tue Jun 03, 2014 2:42 pm
Awesome....
But vid is too short...
Morpheus Eberhardt on Wed Jun 04, 2014 4:08 am
GarryB on Wed Jun 04, 2014 1:10 pm
In that second video you can see the tube for the missile does not take up the entire area for the "tube" under the hatch.
TheArmenian
TheArmenian on Wed Jun 04, 2014 3:29 pm
From the original TvZvezda video (which is now deleted from their website):
I paused the video when the missile came out and was surprised to see that it looked different than the 9M96 that where shown many times before. The layout was similar, but the forward fins were placed further back.
I wish I had made a screenshot.
In the video posted by Morpheue Eberhardt, I could not get to see the missile properly because of the smoke.
Viktor on Wed Jun 04, 2014 3:39 pm
Something like this perhaps?
Create a forum | Invision | Free forum support | Report an abuse | Forumotion.com


text:
Fladnitz im Raabtal war bis 31. Dezember 2014 eine selbständige Gemeinde mit 757 Einwohnern (Stand: 31. Oktober 2013) im Süd-Osten der Steiermark im Gerichtsbezirk Feldbach bzw. Bezirk Südoststeiermark. Der Gemeinderat hatte sich in seiner Sitzung vom 19. Februar 2013 einstimmig für die Fusion mit Kirchberg an der Raab ausgesprochen. Im Rahmen der Gemeindestrukturreform in der Steiermark ist die Gemeinde ab 2015 mit den Gemeinden Kirchberg an der Raab, Oberdorf am Hochegg und Studenzen zusammengeschlossen, die neue Gemeinde führt den Namen Kirchberg an der Raab weiter. Grundlage dafür ist das Steiermärkische Gemeindestrukturreformgesetz – StGsrG.

Geografie

Geografische Lage
Fladnitz im Raabtal liegt ca. 28 km östlich von Graz und ca. 8 km nordwestlich der Bezirkshauptstadt Feldbach im Oststeirischen Hügelland. Der Ort ist über den Bahnhof Studenzen-Fladnitz der Steirischen Ostbahn an den Eisenbahnverkehr angebunden.

Nachbargemeinden bis 2014
 im Norden: Eichkögl
 im Osten: Edelsbach bei Feldbach
 im Süden: Kirchberg an der Raab
 im Westen: Studenzen

Gemeindegliederung
Keine Ortschaften

Einwohnerentwicklung

Politik

Letzter Gemeinderat
Der Gemeinderat bestand aus 9 Mitgliedern und setzte sich nach der Gemeinderatswahl 2010 aus Mandaten der folgenden Parteien zusammen:
 7 ÖVP – stellte den Bürgermeister und den Vizebürgermeister
Bgm Mag. Katharina Tabloner, Vzbgm.Walter Loidl, Agnes Steiner, Walter Binder, Johann Weninger-Bachmann, Raimund Thiele, Erich Rainer
 2 SPÖ – stellte den Kassier, Lotte Fuchs, Stefan Zipper

Wappen
Die Verleihung des Gemeindewappens erfolgte mit Wirkung vom 1. Oktober 1998.
Wappenbeschreibung: Im Schild ein goldener Balken, begleitet von goldenen Lilien, oben in Rot von zwei, unten in Schwarz von einer.

Kultur und Sehenswürdigkeiten

Museen
Dorfmuseum Fladnitz

Musik
Singkreis Fladnitz

Sport
USV Fladnitz

Regelmäßige Veranstaltungen
Frühschoppen der FF Fladnitz (Juli)

Fladnitzer Mordaffäre
Im Jahre 1950 wurde der Ort überregional bekannt durch eine Serie von Raubmorden, die sich immer freitags ereigneten, da an diesem Tag die Männer ihren Wochenlohn erhielten und als Bargeld nach Hause trugen. Die Morde wurden nie aufgeklärt. Ein mutmaßlicher Täter wurde zwar 1951 rechtskräftig verurteilt, einige Jahre später jedoch in einem Wiederaufnahmeverfahren freigesprochen, nachdem die beiden Hauptbelastungszeugen ihre Aussagen widerrufen hatten.

Weblinks

Einzelnachweise

Kirchberg an der Raab
Ort im Bezirk Südoststeiermark
Katastralgemeinde im Bezirk Südoststeiermark
Ehemalige Gemeinde in der Steiermark

redpajama_set_name: RedPajamaWikipedia


hand-crafted invitations and party favors for our clientele.
details and will be on their way to you in just a few days.
"I've used Fantastic Favors & Invitations for over four years in my wedding, Christmas and business.


text:
Home » Jay-Z leads list of most-nominated artists in Grammys history
Jay-Z leads list of most-nominated artists in Grammys history
LOS ANGELES, Nov 23 (Reuters) – Rapper Jay-Z rose to the highest of the record of artists with essentially the most Grammy nominations on Tuesday, hitting a lifetime whole of 83 that put him simply forward of former Beatle Paul McCartney.
The three newest nominations for Jay-Z got here for collaborations with different artists. He made visitor appearances on DMX music "Tub Salts" and Kanye West's "Jail," which additionally earned Jay-Z a nod for West's album "Donda."
Jay-Z has gained 23 Grammys total, together with a greatest rap album award in 1999 and honors for songs "Loopy in Love" along with his spouse, Beyonce, in 2004 and "Empire State of Thoughts" with Alicia Keys in 2011.
Register now for FREE limitless entry to reuters.com
Earlier this yr, Beyonce set the report for many Grammy wins by a lady with 28 over her profession.
McCartney's Grammy nomination whole climbed to 81 on Tuesday with two nods for greatest rock album "McCartney III" and greatest rock music for "Discover My Method."
Producer Quincy Jones, who had been tied with Jay-Z for many nominations earlier than Tuesday, has acquired 80 Grammy nominations over his lifetime.
Winners of the Grammys, the very best honors within the music trade, might be introduced Jan. 31 at a ceremony in Los Angeles that might be broadcast on CBS.
Reporting by Lisa Richwine, Enhancing by Nick Zieminski
artistsGrammyshistoryJayZLeadsListmostnominated
Arlene Dahl, Movie Star Turned Entrepreneur, Is Dead at 96
The Best Edible Gifts You Can Ship (That...
These Private Jet NFTs Are Meant For Your...
Cannabis & Psychedelics Experts Reflect On Cali Sober...
The Reaction To Netflix's 'Don't Look Up' Is...
Tesla Says It Delivered Record 936,172 Electric Vehicles...
Luxury Gym And Lifestyle Brand, Workshop Opens Debut...
Vacationing At An All-Inclusive Resort? Why You Still...
Create Luck For The New Year With This...
Stylish Neutral Homewares And Accessories
For Its Annual Métiers D'Art Collection, Chanel Unveils New Building...
How A Former Hollywood Mogul Created An...
Jeff Hyland, Co-Founder Of Top L.A. Brokerage...
The Best CBD Gifts For Lowering Stress