text:
\section{Introduction}
Stochastic control problems arise naturally in a range of applications in engineering, economics, and finance.
Apart from very specific cases such as linear-quadratic control in engineering or the Merton
portfolio optimization task in finance,  stochastic control problems typically have no closed form
solutions and have to be solved numerically.
In this paper we consider the policy iteration algorithm and gradient iteration
algorithm; see Algorithms~\ref{alg pia} and~\ref{alg via}.
These are effectively a linearization method for the inherently nonlinear problem
and play an essential role in numerical solutions of stochastic control problems.

We will consider the continuous space, continuous time problem where
the controlled system is modeled by an $\mathbb R^d$-valued diffusion process.
Let $W$ be a $d'$-dimensional Wiener martingale on a filtered probability space $(\Omega, \mathcal{F}, (\mathcal{F}_t)_{t\ge 0}, \ensuremath{\mathbb{P}})$.
Let us fix a finite time $T\in(0,\infty)$ and consider the controlled SDE
\begin{equation}\label{sde}
dX_s=b^{\alpha}(s,X_s)\,ds+\sigma(s,X_s)\,dW_s\,,\,\, s\in [t,T]\,,\,\,\, X_t = x\,.
\end{equation}
Here $\alpha=(\alpha_s)$ is a control belonging to the space of admissible controls
$\mathcal A$, valued in $A\subseteq\ensuremath{\mathbb{R}}^m$, and we will write $X^{t,x,\alpha}$ to denote the solution of~\eqref{sde}
which starts from $x$ at time $t$ while being controlled by $\alpha$.
We shall consider the gain functional in the form
\begin{equation}\label{problem}
J(t,x,\alpha):=\E\left[\int_{t}^{T}f^{\alpha}(s,X_s^{t,x,\alpha})ds+g(X_T^{t,x,\alpha})\right]
\end{equation}
for all $(t,x)\in[0,T]\times\ensuremath{\mathbb{R}}^d$ and $\alpha\in\mathcal{A}$.
The value function $v=v(t,x)$ is given for all $t\in[0,T]$ and $x\in \mathbb R^d$ by
\begin{equation}\label{cp}
v(t,x)=\sup_{\alpha\in\mathcal{A}}J(t,x,\alpha).
\end{equation}
We wish to solve the optimization problem, i.e., to find either the value function $v$
or the optimal control $\alpha^*$ which achieves the maximum (or, if the supremum
cannot be reached by $\alpha \in \mathcal A$, then an $\varepsilon$-optimal control $\alpha^\varepsilon \in \mathcal A$ such that $v(t,x) \leq J(t,x,\alpha^\varepsilon) + \varepsilon$).
It is well known that (see, e.g., Krylov~\cite{krylov controlled}) that under reasonable assumptions the value function satisfies the Bellman PDE:
\begin{align}\label{HJB general}
\begin{split}
\partial_tv + \frac{1}{2}\text{tr}(\sigma \sigma^\top D^2_xv)+ \sup_{a\in A}\left(b^aD_xv+f^a\right)&=0\text{ on }[0,T)\times\ensuremath{\mathbb{R}}^d,\\
v(T,x)&=g(x)\,\,\text{ on } x\in\ensuremath{\mathbb{R}}^d\,.
\end{split}
\end{align}
Moreover (again see Krylov~\cite{krylov controlled}), it is sufficient to consider
Markovian controls, i.e., processes $\alpha_s=a(s,X_s^{t,x,\alpha})$ for some measurable function $a:[0,T]\times\ensuremath{\mathbb{R}}^d \to A$.
Thus if we have obtained the value function, then we can find the optimal control
(if it exists) as
\[
a^\ast(t,x) = \arg \max_{a\in A}\big( b^a(t,x) (D_x v)(t,x) + f^a(t,x)\big)\,.
\]

It is rarely possible to find a closed form solution to~\eqref{HJB general}
and so various approximations have to be employed.
One may, for example, choose to use a finite difference method
to discretize~\eqref{HJB general} and indeed this has been widely studied;
see, e.g.,~\cite{dong krylov rate} or~\cite{gyongy siska finite} and references therein.
This results in a high dimensional nonlinear system of equations that still retains
the structure of~\eqref{HJB general}.
To solve this nonlinear system one may apply the Howard's policy improvement
algorithm.
The rate of convergence would then follow from results available on discrete
space-time control problems.
However, to check that the assumptions required for convergence are satisfied
is not straightforward and moreover it is dependent on the discretization scheme used.

An alternative approach is to linearize~\eqref{HJB general} and to iterate.
The classical approach is the Bellman--Howard policy improvement/iteration algorithm.
The algorithm is initialized with a ``guess'' of the Markovian control.
Given a Markovian control strategy at step $n$ one solves a linear PDE with
the given control fixed and then one uses the solution to the linear PDE to update
the Markovian control.
In this paper we will show that this policy improvement algorithm (see Algorithm~\ref{alg pia}) and a variant
which we call the gradient iteration algorithm (see Algorithm~\ref{alg via}) converge, under appropriate assumptions, exponentially fast.

Iterative algorithms for the solution of optimal control problems go back to the work of Bellman~\cite{Bellman,Bellman:1957} where the value iteration algorithms
for finite space-time problems are developed and their convergences are shown.
Howard~\cite{howard:dp} proposed the policy improvement algorithm in the context
of the discrete space-time Markovian decision rocess.
Puterman and Brumelle's paper~\cite{puterman79} was one of the first results on the convergence
properties for the policy iteration for MDP problems.
The abstract function space setting employed in the paper applies to both discrete
and continuous settings.
Their main observation is that the policy iteration can be viewed as a type of Newton's method.
Hence similar convergence results to those known for Newton's method follow: in particular, if the initial guess is in a neighborhood of the true solution, then the convergence will be quadratic.
Puterman~\cite{Puterman:1981} applied this in a setting very similar
to that of this paper to prove quadratic convergence in the neighborhood of the
limit.
Santos and Rust~\cite{Santos2004ConvergencePO} consider the discrete time but continous space and controls setting.
They extend the results of Puterman and Brumelle~\cite{puterman79} to show global convergence, but without global rate, and
quadratic local convergence rate of policy iteration and superlinear local convergence under more general conditions. In the case of stochastic control problems with jump-diffusion processes, B\"{a}uerle and Rieder~\cite{Rieder} have proved a  convergence result of the Howard's policy improvement algorithm with the help of martingale techniques.
In the fully discrete space and time setting
Bokanowski, Maroso, and Zidani~\cite{Bokanowski2009} have shown global superlinear convergence, under a monotonicity assumption on the matrices defining the control problem.
Convergence of policy iteration has been recently proved by Jacka and Mijatovi\'c~\cite{jacka miatovic policy} and Jacka, Mijatovi\'c, and Siraj~\cite{jacka miatovic siraj coupling}.
Further, Maeda and Jacka~\cite{maeda jacka evaluation} have shown quadratic local convergence of the policy iteration algorithm for the time-independent control problem.
The local quadratic convergence is similar to the result of Puterman~\cite{Puterman:1981} but the specific control problem is different
and moreover they employ a completely different technique based on Schauder estimates for linear PDEs.

The main contributions of this paper are to establish a global rate of convergence
and stability
for the policy iteration algorithm and a variant, which we call the gradient iteration algorithm.
The analysis is carried out using backward stochastic differential equations (BSDEs) and to the best knowledge of the authors this is the first
time BSDEs have been used to study convergence of the policy iteration algorithm.
The assumptions required for this are effectively Lipschitz dependence in the drift, diffusion, instantaneous payoff, and terminal payoff functions and independence of the diffusion matrix on the control; see~\eqref{sde}.
The stability results show that the policy iteration remains stable even if the linear PDE is solved
only approximately and even if the maximization is step performed approximately.
Moreover they allow one to devise computationally efficient algorithms as they
show that in the initial steps it is sufficient to solve the linear PDE with very
low accuracy, and a highly accurate PDE solver is only required for the final few iterations of the algorithms.

The paper is organized as follows. In Section~\ref{sec main} we introduce all the assumptions and notation used throughout the paper.
In Sections~\ref{sec conv gia} and~\ref{sec conv pia}
we state and prove the results concerning convergence of the gradient iteration algorithm and the policy improvement algorithm, respectively.
Section~\ref{sec policy improvement} justifies the name ``policy improvement algorithm'' in that it shows that the value functions increase monotonically
with iterations and it also shows that the algorithm converges under weaker assumptions than those required for obtaining the rate.
Sections~\ref{sec stability lin PDE} and~\ref{sec stability max} prove
the stability of the algorithms. In Section~\ref{sec example} we present an example that fits the setting of this paper.
Finally, in Appendix~\ref{sec BSDEs}, we collect several known results
from the theory of BSDEs that are essential for the proofs.

\begin{algorithm}[H]
\caption{Policy improvement algorithm.}
\label{alg pia}
\begin{algorithmic}
\STATE{Initialization: make a guess of the control $a^0 = a^0(t,x)$.}
\WHILE{difference between $v^{n+1}$ and $v^n$ is large}
\STATE{Given a control $a^n = a^n(t,x)$ solve the {\em linear} PDE
\begin{equation}
\label{eq policy iteration pde}
\begin{split}
\partial_t v^n + \frac{1}{2}\text{tr}(\sigma \sigma^\top D^2_x v^n)+b^{a^n} D_x v^n + f^{a^n} & = 0\text{ on }[0,T)\times\ensuremath{\mathbb{R}}^d\,,\\
v^n (T,\cdot) & = g\,\,\text{ on } x\in\ensuremath{\mathbb{R}}^d\,.
\end{split}
\end{equation}
}
\STATE{Update the control
\begin{equation}
\label{eq def anplus1}
a^{n+1}(t,x) = \arg \max_{a\in A}\left[\left(b^a D_x v^n+f^a\right)(t,x)\right]\,.
\end{equation}
}
\ENDWHILE
\RETURN $v^n, a^{n+1}$.
\end{algorithmic}
\end{algorithm}

\begin{algorithm}
\caption{Gradient iteration algorithm.}
\label{alg via}
\begin{algorithmic}
\STATE{Initialization: make a guess of the value function $v^0 = v^0(t,x)$.
}
\WHILE{difference between $v^n$ and $v^{n-1}$ is large}
\STATE{Given value function $v^{n-1} = v^{n-1}(t,x)$ update the control
	\begin{equation}
	\label{eq gia update control}
	a^{n}(t,x) = \arg \max_{a\in A}\left[ \left(b^a D_x v^{n-1}+f^a\right)(t,x)\right]\,.
	\end{equation}
}
\STATE{Solve the {\em linear} PDE
\begin{equation}
\label{eq value iteration pde}
\begin{split}
\partial_t v^n + \frac{1}{2}\text{tr}(\sigma \sigma^\top D^2_x v^n)
+ b^{a^n} D_x v^{n-1} + f^{a^n}  & = 0\text{ on }[0,T)\times\ensuremath{\mathbb{R}}^d\,,\\
v^n (T,\cdot) & = g\,\,\text{ on } x\in\ensuremath{\mathbb{R}}^d\,.
\end{split}
\end{equation}
}
\ENDWHILE
\RETURN $v^n, a^{n}$.
\end{algorithmic}
\end{algorithm}

We would like to emphasize that Algorithm~\ref{alg pia} and Algorithm~\ref{alg via} are different, although they look rather similar.
In Algorithm~\ref{alg pia}, $v^n$ is the value function for the Markov control $a^{n}$, since it solves the PDE~\eqref{eq policy iteration pde}.
In Algorithm~\ref{alg via} $v^n$ is not the value function for the Markov control $a^n$.
This is due to the term $b^{a^n}D_x v^{n-1}$ in the linear PDE~\eqref{eq value iteration pde}.

\section{Assumptions and Notation}
\label{sec main}
We fix a finite horizon $T\in (0,\infty)$.
We assume that for some $m\in \mathbb N$ we have $A \subseteq R^m$ such that $0\in A$.
This is the space where the control processes $\alpha$ take values.
We fix a filtered probability space $(\Omega, \mathcal{F}, \F=(\mathcal{F}_t)_{0\le t\le T}, \ensuremath{\mathbb{P}})$.
Let  $W=(W_t)_{t\in [0,T]}$ be a $d'$-dimensional Wiener martingale on this space. Moreover, we have the following:
\begin{enumerate}[(i)]
\item
For $\gamma > 0$ and a predictable process $\phi$ let us define
\[
\|\phi\|_{\mathbb{H}^2_\gamma} := \left(\E\int_{0}^{T}e^{\gamma s}|\phi_s|^2\,ds\right)^{\frac12}\,.
\]
For $\gamma = 0$ we will write $\|\cdot\|_{\mathbb H^2}$.
We will use $\mathbb{H}^2$ to denote the set of all
predictable processes $\phi$ such that $\|\phi\|_{\mathbb H^2} < \infty$.
Note that the norm $\|\cdot\|_{\mathbb H^2}$ is equivalent to the norm $\|\cdot\|_{\mathbb H^2_\gamma}$ for any $\gamma \geq 0$.
\item Let $\mathcal{S}^{2}$ be the set of real valued $\mathbb F$-adapted
continuous processes $\phi$ on $[0,T]$ such that
	\[
	\|\phi\|_{\mathcal{S}^{2}}:=\E\left[\sup_{0\le r\le T}|\phi_r|^2\right]<\infty\,.
	\]
\item For adapted processes $\phi$ such that $\int_0^t |\phi_s|^2\, ds < \infty$ almost surely we will define
\[
(\phi \bullet W)_t := \int_0^t \phi_s \, dW_s\,.
\]
\item For any continuous local martingale $M$ let with $(\langle M \rangle_t)_{t\in [0,T]}$ denote the quadratic variation process and moreover let
\[
\mathcal E(M)_t := \exp\left(M_t - \frac12 \langle M \rangle_t\right)\,.
\]
\end{enumerate}

We are given measurable functions
\[
b:A\times [0,T]\times\ensuremath{\mathbb{R}}^d\to \ensuremath{\mathbb{R}}^d\text{ and } \sigma:[0,T]\times\ensuremath{\mathbb{R}}^d\rightarrow\ensuremath{\mathbb{R}}^{d\times d'}\,.
\]
The state of the system is governed by the controlled SDE~\eqref{sde}.

\begin{assumption}
\label{assumption controlled SDE}
The functions $b$ and $\sigma$ are continuous in $t$.
There exists $K\ge 0$
and such that $\forall x,y\in\ensuremath{\mathbb{R}}^d,\forall a\in A,\forall t\in[0,T]$,
\begin{equation}\label{lipchitz}
|b^a(t,x)-b^a(t,y)|+|\sigma(t,x)-\sigma(t,y)|\le K|x-y|
\end{equation}
and
\begin{equation}\label{linear growth}
|\sigma(t,x)|\le K(1+|x|)\,,\,\,\,|b^a(t,x)|\le K(1+|x|+|a|)\,.
\end{equation}
\end{assumption}
\noindent Under Assumption~\ref{assumption controlled SDE} we know that for any $(t,x)\in[0,T]\times \mathbb R^d$
and for any progressively measurable $A$-valued control process $\alpha=(\alpha_s)$
there is a unique strong solution to~\eqref{sde} which we denote $(X^{t,x,\alpha}_s)_{s\in[t,T]}$.
Let
\[
f:A\times [0,T]\times\ensuremath{\mathbb{R}}^d \to \ensuremath{\mathbb{R}}\text{  and  } g:\ensuremath{\mathbb{R}}^d\rightarrow\ensuremath{\mathbb{R}}
\]
be two given measurable functions. Let us assume the following for the running gain function $f$ and the terminal gain function $g$ appearing in~\eqref{problem}.
\begin{assumption}
\label{assumption rewards}
There is a constant $K\ge0$ such that  $\forall x,y\in\ensuremath{\mathbb{R}}^d,\forall a\in A,\forall t\in[0,T]$
\begin{equation}\label{loc_Lip}
|g(x)-g(y)|+|f^a(t,x)-f^a(t,y)|\le K|x-y|
\end{equation}
and
\begin{equation}\label{bound_f}
|f^a(t,x)|\le K(1+|x|+|a|)\,,\,\,\,
|g(x)|\le K\,.
\end{equation}

\end{assumption}
\noindent Under Assumption~\ref{assumption rewards} the gain functional $J$ given
by~\eqref{problem} and the value function $v$ given by~\eqref{cp} are well defined.
Moreover, the value function $v$ satisfies the Bellman equation (with derivatives existing almost everywhere, see Krylov~\cite[Chapter 4]{krylov controlled}, or in the sense of viscosity solutions, see, e.g., Pham~\cite{pham book} or Fleming and Soner~\cite{fleming soner})
\begin{equation}\label{HJB}
\begin{split}
\partial_tv+\frac{1}{2}\text{tr}(\sigma\sigma^\top D^2_xv)+\sup_{a\in A}\left(b^aD_xv+f^a\right)&=0\text{ on }[0,T)\times\ensuremath{\mathbb{R}}^n,\\
v(T,x)&=g(x)\,\,\text{ on } x\in\ensuremath{\mathbb{R}}^d\,.
\end{split}
\end{equation}

Let us now state the additional assumptions required for our convergence result.
\begin{assumption}
\label{ass convexity}
Let us define for each fixed $(t,x,z) \in [0,T] \times \mathbb R^d \times \mathbb R^d$  the function
\begin{equation}
\label{control function}
a(t,x,z) := \arg\max_{a\in A} \left(b^a(t,x)\sigma^{-1}(t,x)z + f^a(t,x)\right).
\end{equation}
We assume that the function $a(t,x,z)$ is measurable.
\end{assumption}

If the function $a\mapsto \left(b^a(t,x)\sigma^{-1}(t,x)z + f^a(t,x)\right)$ is convex for each fixed $(t,x,z)$, which is in $ [0,T] \times \mathbb R^d \times \mathbb R^d$, one can immediately see that Assumption \ref{ass convexity} holds.
More generally, this assumption can be verified using an appropriate measurable selection theorem.
For example, if $A$ is compact, then~\cite[Proposition D.5]{Hernandez1996} shows that an appropriate measurable selection exists.
If $A$ is not compact but $f$ is bounded, then~\cite[Proposition D.6]{Hernandez1996} gives the same conclusion (using also that $z=D_x v(t,x)$ and Remark~\ref{rem grad bdd}).

\begin{assumption}
\label{ass extra for pia}
There are constants $K, \theta \geq 0$ such that the following hold:
\begin{enumerate}
\item (On the drift)
For all $t\in[0,T]$, $x\in\ensuremath{\mathbb{R}}^d$, $a,a'\in A$,
\begin{equation} \label{ass b lip in a}
|b^a(t,x) - b^{a'}(t,x)| \leq \sqrt{\theta}|a-a'|
\end{equation}
and for all $t\in[0,T]$, $x\in\ensuremath{\mathbb{R}}^d$, $a\in A$ we have
\begin{equation} \label{change_of_measure}
|(b^a\sigma^{-1})(t,x)|<K\,.
\end{equation}
\item (On the control function)
For all $t\in[0,T]$, $x,x',z,z' \in\ensuremath{\mathbb{R}}^d$, $a,a'\in A$ we have that
\begin{equation*}
|a(t,x,z)-a(t,x,z')| \leq \sqrt\theta|z-z'|\,,
\end{equation*}
\begin{equation*}
|a(t,x,z)-a(t,x',z)| \leq K|x-x'|\,\,\,\,\text{and}\,\,\,\, |a(t,0,0)| \leq K\,.
\end{equation*}
\item (On the running reward)
\begin{equation*}
|f^{a}(t,x)-f^{a'}(t,x)| \leq \sqrt{\theta}|a-a'| \,\,\,\, \forall t\in[0,T], \forall x\in\ensuremath{\mathbb{R}}^d, \forall a,a'\in A\,.
\end{equation*}
\end{enumerate}
\end{assumption}
\begin{remark}
Under Assumptions~\ref{assumption rewards} and~\ref{ass extra for pia} we have that
for all $t\in[0,T]$, $x,z,z'\in \ensuremath{\mathbb{R}}^d$ the following hold:
\[
|f^{a(t,x,z)}(t,x)-f^{a(t,x,z')}(t,x)|\le \theta |z-z'|
\]
and
\[
|f^{a(t,x,0)}(t,x)| \leq (K+K^2)(1+|x|)\,.
\]
\end{remark}

Under Assumptions~\ref{assumption controlled SDE}, \ref{assumption rewards},
\ref{ass convexity}, and~\ref{ass extra for pia}  there is an optimal control process and this
fact will be used to prove the main results.
\begin{remark}
\label{remark alpha star}
Due to results of Krylov~\cite{krylov controlled} we know that~\eqref{HJB general} has a unique solution and moreover
the map $[0,T] \times \mathbb R^d \ni (t,x) \mapsto D_x v(t,x) \in \mathbb R^d$ is bounded;
see~\cite[Chapter 4, section 1, Theorem 1]{krylov controlled}.
Hence, by Assumptions~\ref{ass convexity} and~\ref{ass extra for pia}
we know that $(t,x) \mapsto a(t,x,\sigma(t,x)D_x v(t,x))$ is jointly
measurable and Lipschitz in $x$.
Thus, for each $(t,x)\in [0,T] \times \mathbb R^d$, the SDE
\[
dX_s = b^{a(s,X_s,\sigma(s,X_s)D_x v(s,X_s))}(s,X_s)\,ds + \sigma(s,X_s)\,dW_s\,,\,\,\,s\in[t,T]\,, X_t = x
\]
has a unique solution $X^{t,x}$.
Then by the verification theorem, the process $\alpha^\ast_s := a(s,X_s,\sigma(s,X_s)D_x v(s,X_s))$ is the optimal control process for~\eqref{cp}.
\end{remark}

All the proofs will be completed in a new measure $\hat{\mathbb P}$ on $(\Omega, \mathcal F)$
given in the
following lemma.
We will use $\hat{\mathbb E}$ to denote the expectation under the measure $\hat{\mathbb P}$.

\begin{lemma}
\label{lemma P hat}
Let Assumptions~\ref{assumption controlled SDE} and~\ref{assumption rewards}
together with~\eqref{change_of_measure} hold.
Let $(t,x) \in [0,T]\times \mathbb R^d$.
Let $X = X^{t,x,\alpha^\ast}$ be the solution to the SDE~\eqref{sde} started from $(t,x)$ and controlled by the optimal control process $\alpha^\ast$.
Then $d\hat{\mathbb P} := \mathcal E((b^{\alpha^\ast}\sigma^{-1})(\cdot,X) \bullet W)_T \, d\mathbb P$
is a probability measure equivalent to $\mathbb P$ and the process
\[
\widehat{W}_s := W_s + \int_0^s b^{\alpha^\ast_r}(r,X_r)\sigma^{-1}(r,X_r) \,dr
\]
is a $\hat{\mathbb P}$-Wiener process.
\end{lemma}
\begin{proof}
This is an immediate consequence of~\eqref{change_of_measure} and Girsanov's theorem.
\end{proof}
\begin{remark}
\label{rem grad bdd}
From Krylov~\cite[Chapter 4, section 1, Theorem 1]{krylov controlled}
we get that there is a constant $C>0$ such that for all $(t,x)\in [0,T)\times \mathbb R^d$ we have that $|D_x v(t,x)| \leq C$.
\end{remark}

\section{Convergence of gradient iteration algorithm}
\label{sec conv gia}

The following theorem gives the convergence result for Algorithm~\ref{alg via}.

\begin{theorem}\label{main theorem alg 1}
Let Assumptions~\ref{assumption controlled SDE}, \ref{assumption rewards},
\ref{ass convexity}, and~\ref{ass extra for pia} hold.
Let $v$ be the solution to~\eqref{HJB general} and let $(v^n)_{n\in \mathbb N}$ be the approximation sequence given by Algorithm~\ref{alg via}.
Then there is $q\in (0,1)$ depending only on $ K,\theta,T$ and the initial guess $v^0=v^0(t,x)$ such that for all
$(t,x)\in [0,T]\times \mathbb R^d$ there exists $C=C(t,x)$ such that
\begin{equation*}
\begin{split}
|v(t,x)-v^n(t,x)|^2\le C(t,x) q^n\,.
\end{split}
\end{equation*}
\end{theorem}

The main idea of the proof consists of noticing that Algorithm~\ref{alg via}
can be seen as an iteration on the level of BSDEs.
Using Lemma~\ref{lem classic bsde iteration} we see that on the level of
BSDEs this iteration is contractive.
Finally we need to use known results on the connection between BSDEs and
solutions to the HJB equation.

\begin{proof}[Proof of Theorem~\ref{main theorem alg 1}]

We prove the main result in several steps.
First, we show how to rewrite the gradient iteration algorithm as
an iteration on the level of BSDEs.
On the $n$th step of the algorithm we need to solve the linear PDE with Lipschitz continuous coefficients~\eqref{eq value iteration pde}.
Let $v^n$ be the solution to~\eqref{eq value iteration pde} and recall that
\[
a^{n}(t,x) = \arg \max_{a\in A}\left((b^aD_x v^{n-1})(t,x)+f^a(t,x)\right) = a(t,x,\sigma(t,x) D_x v^{n-1}(t,x))\,.
\]
Since we are working with the linear
PDE with Lipschitz continuous coefficients, we have $v^n$ in
$C^{1,2}([0,T)\times \mathbb R^d)$.
Let $X = X^{t,x,\alpha^\ast}$ be the solution to the SDE~\eqref{sde} started from $(t,x)$ and controlled by the optimal control process $\alpha^\ast$; see Remark~\ref{remark alpha star}.
From It\^o's formula we then get that
\begin{equation*}
\begin{split}
dv^n(s,X_s) = & \left[\partial_t v^n(s,X_s) + \frac{1}{2}\text{tr}(\sigma\sigma^\top D^2_xv^n)(s,X_s) + (b^{\alpha^\ast_s} D_x v^n)(s,X_s) \right]\,ds \\
& + (D_x v^n \sigma ) (s,X_s) \, dW_s\\
= & \left[(b^{\alpha^\ast_s} D_x v^n)(s,X_s) - (b^{a^{n}}D_xv^{n-1})(s,X_s)-f^{a^{n}}(s,X_s)\right]\,ds\\
& + (D_x v^n \sigma )(s,X_s) \, dW_s	\,.
\end{split}
\end{equation*}
Let
\[
F_s(z) := b^{a(s,X_s,z)}(s,X_s)\sigma^{-1}(s,X_s) z + f^{a(s,X_s, z)}(s,X_s)
\]
and
\begin{equation}
\label{eq def of Y}
Y^n_t:=v^n(t,X_t)\,,\,\,\,\,Z^n_t:=\sigma(t,X_t)D_xv^n(t,X_t)\,,\,\,\,\xi:=g(X_T).
\end{equation}
Then we may write
\begin{equation}
\label{eq bsde val iter in P}
Y_t^n=\xi-\int_{t}^{T}\left[(b^{\alpha^\ast_s}\sigma^{-1})(s,X_s) Z^n_s - F_s( Z^{n-1}_s)\right]ds-\int_{t}^{T}Z^n_s\,dW_s\,.
\end{equation}
Let $\hat{\mathbb P}$ and $\widehat{W}$ be given by Lemma~\ref{lemma P hat}.
Hence~\eqref{eq bsde val iter in P} becomes
\begin{equation}
\label{eq bsde val iter in Phat}
Y_t^n=\xi + \int_{t}^{T} F_s(Z^{n-1}_s)\,ds-\int_{t}^{T}Z^n_s\,d\widehat W_s\,.
\end{equation}
Consider now the following BSDE:
\begin{equation}
\begin{split}\label{eq: BSDE for value function}
Y_{t'}^{t,x}&=\xi + \int_{t'}^{T} b^{a(s, X_s^{t,x},Z_s^{t,x})}(s,X_s^{t,x})\sigma^{-1}(s,X_s^{t,x})Z_s^{t,x}+f^{a(s, X_s^{t,x},Z_s^{t,x})}(s, X_s^{t,x})\,ds\\
&\qquad- \int_{t'}^{T}Z_s^{t,x}\,d\widehat W_s,\,\,\,t'\in[t,T]\,,
\end{split}
\end{equation}
where the superscript means that the forward process started from $(t,x)$.
Hence, we can define
\begin{equation}
\label{eq def of w}
w(t,x):=Y^{t,x}_t\,\,\text{ and }\,\,\sigma(t,x)D_xw(t,x):=Z^{t,x}_t.
\end{equation}
Therefore by \eqref{control function} we have
\begin{equation*}
\begin{split}
b^{a(t, x,z)}(t,x)\sigma^{-1}(t,x)z+f^{a(t, x,z)}(t, x)=\max_{a\in A}\left(b^{a}(t,x)\sigma^{-1}(t,x)z+f^{a}(t,x)\right).
\end{split}
\end{equation*}
Thus, by Pham~\cite[Theorem 6.3.3]{pham book}, the function $w=w(t,x)$ solves the HJB equation~\eqref{HJB general}.
Notice that here is the crucial point where the fact that we use the optimal control $\alpha^*$ plays a role.
Indeed with other control processes we couldn't claim that $w$ solves the HJB equation.
By uniqueness of the viscosity solution to the HJB equation (see the strong comparison principle from~\cite[Theorem 4.4.5]{pham book}), we can conclude that $w=v$ and therefore $w$ is the value function of our stochastic control problem. Therefore, the BSDE~\eqref{eq: BSDE for value function} is the BSDE corresponding to the value function. Notice that~\eqref{eq: BSDE for value function} is a quadratic BSDE, since in the generator we have a product of two Lipschitz functions which depend on $Z$. The existence of the solution to~\eqref{eq: BSDE for value function} under our assumptions can be obtained by applying Theorem~\ref{thr well-defined qBSDE from Adrien's paper} in the case when the terminal cost is bounded for our stochastic control problem.

Using Remark~\ref{rem grad bdd}, the fact that $\sigma^{-1}(s,X_s) Z_s = D_x v(s,X_s)$, and Assumption~\ref{ass extra for pia}, for all $s\in[t,T]$ we get that
\begin{equation}
\label{eq F lip val it}
\begin{split}
|F_s(Z_s)& - F_s(Z^{n-1}_s)|\\
&\le
|\sigma^{-1}(s,X_s) Z_s||b^{a(s,X_s, Z_s)}(s,X_s)-b^{a(s,X_s,Z^{n-1}_s )}(s,X_s)|\\
& \qquad +|b^{a(s,X_s,Z^{n-1}_s )}(s,X_s)\sigma^{-1}(s,X_s)||Z_s-Z^{n-1}_s|+\theta|Z_s-Z^{n-1}_s|\\
&\le
(C\theta+K+\theta)|Z_s-Z^{n-1}_s|\,.
\end{split}
\end{equation}

Moreover, recalling $\xi=g(X_T)$, we get that $\xi\in L^2(\Omega, \mathcal F_T, \hat{\mathbb P})$ from the higher moment estimates for the solution of the SDE and from the Lipschitz property of $g$.
Similarly $F_s(0)\in\hat{\mathbb H}^2$ by Assumption~\ref{ass extra for pia}.
We may thus apply Lemma~\ref{lem classic bsde iteration} and hence,
due to~\eqref{eq bsde val iter in Phat} and~\eqref{eq F lip val it}, we have $q \in (0,1)$ and $\gamma \geq 0$ such that for all $t\in [0,T]$
\begin{equation}
\label{eq conclusion of contr lem grad}
e^{\gamma t}\hat{\mathbb E} |Y_t - Y^n_t|^2 + \|Z - Z^n\|_{\hat{\mathbb H}^2_\gamma}^2 \leq q \|Z - Z^{n-1}\|_{\hat{\mathbb H}^2_\gamma}^2\,.
\end{equation}
Therefore, from~\eqref{eq def of w} and~\eqref{eq def of Y}, we have $v(t,x)=Y^{t,x}_t$ and $v^n(t,x)=Y^{n,t,x}_t$ and by~\eqref{eq conclusion of contr lem grad} we obtain
\begin{equation*}
\begin{split}
e^{\gamma t}|v(t,x)-v^n(t,x)|^2&\le \hat{\mathbb E}[e^{\gamma t}|Y^{t,x}_t-Y^{n,t,x}_t|^2]+\|Z^{t,x}-Z^{n,t,x}\|^2_{\hat{\mathbb H}^2_{\gamma}}\\
&\le q^n \|Z^{t,x}-Z^{0,t,x}\|^2_{\hat{\mathbb H}^2_{\gamma}}\,.
\end{split}
\end{equation*}
Hence
\begin{equation*}
\begin{split}
&|v(t,x)-v^n(t,x)|^2\\
&\qquad\le q^n\hat{\mathbb E}\int_t^T e^{\gamma(T-t)}|\sigma(s,X^{t,x,\alpha^\star}_s)|^2|D_xv(s,X^{t,x,\alpha^\star}_s)-D_xv^0(s,X^{t,x,\alpha^\star}_s)|^2\,ds\,.
\end{split}
\end{equation*}
This finishes the proof.
\end{proof}

\section{Convergence of policy improvement}
\label{sec conv pia}

\begin{theorem}\label{main theorem alg 2}
Let Assumptions~\ref{assumption controlled SDE}, \ref{assumption rewards},
\ref{ass convexity}, and~\ref{ass extra for pia} hold.
Let $v$ be the solution to~\eqref{HJB general} and let $(v^n)_{n\in \mathbb N}$ be the approximation sequence given by Algorithm~\ref{alg pia}.
Then there is $q\in (0,1)$ depending only on $ K,\theta,T$ and the initial guess $v^0=v^0(t,x)$ such that for all
$(t,x)\in [0,T]\times \mathbb R^d$ there exists $C=C(t,x)$ such that
\begin{equation*}
\begin{split}
|v(t,x)-v^n(t,x)|^2\le C(t,x) q^n\,.
\end{split}
\end{equation*}
\end{theorem}

The proof of Theorem~\ref{main theorem alg 2} is similar to that of
Theorem~\ref{main theorem alg 1} except that the iteration on the level of
BSDEs is nonstandard.

\begin{proof}[Proof of Theorem~\ref{main theorem alg 2}]
Let $v^n$ be the solution to~\eqref{eq policy iteration pde} and recall that
\[
a^{n}(t,x) = \arg \max_{a\in A}\left((b^a D_x v^{n-1})(t,x)+f^a(t,x)\right) = a(t,x,\sigma(t,x)D_x v^{n-1}(t,x)) \,.
\]
As before, let $X = X^{t,x,\alpha^\ast}$ be the solution to the SDE~\eqref{sde} started from $(t,x)$ and controlled by the optimal control process $\alpha^\ast$; see Remark~\ref{remark alpha star}.
By It\^o's formula
\begin{equation*}
\begin{split}
dv^n(s,X_s) = & \left[\partial_t v^n(s,X_s) + \frac12\text{tr}(\sigma\sigma^\top D^2_x v^n)(s,X_s) + (b^{\alpha^\ast_s} D_x v^n)(s,X_s) \right]\,ds \\
& + (D_x v^n\sigma) (s,X_s) \, dW_s\\
= & \big[(b^{\alpha^\ast_s} D_x v^n)(s,X_s)  - (b^{a^n}D_xv^n)(s,X_s)
- f^{a^n}(s, X_s) \big]\,ds\\
& + (D_x v^n\sigma)(s,X_s) \, dW_s	\,.
\end{split}
\end{equation*}
Let
\[
F_s(z,Z):=b^{a(s,X_s,z)}(s,X_s)\sigma^{-1}(s,X_s)Z+f^{a(s,X_s,z)}(s,X_s)\,.
\]
Recalling that the control $\alpha^\ast$ and the associated diffusion $X$ are fixed
we can write
\begin{equation}
\label{eq pia bsde iter in P2}
Y_t^n=\xi-\int_{t}^{T}\left[(b^{\alpha^\ast_s}\sigma^{-1})(s,X_s)Z^n_s-F_s(Z_s^{n-1},Z^n_s)\right]ds-\int_{t}^{T}Z^n_sdW_s.
\end{equation}
Let $\hat{\mathbb P}$ and $\widehat{W}$ be given by Lemma~\ref{lemma P hat}.
Then~\eqref{eq pia bsde iter in P2} becomes
\begin{equation}
\label{eq pia bsde iter in P hat2}
Y_t^n=\xi+\int_{t}^{T}F_s(Z_s^{n-1},Z^n_s)ds-\int_{t}^{T}Z^n_sd\widehat{W}_s\,.
\end{equation}
Similarly as in Theorem~\ref{main theorem alg 1}, consider the BSDE
\begin{equation}
\begin{split}\label{eq: BSDE for value function policy iteration}
Y_{t'}^{t,x}&=\xi + \int_{t'}^{T} b^{a(s, X_s^{t,x},Z_s^{t,x})}(s,X_s^{t,x})\sigma^{-1}(s,X_s^{t,x})Z_s^{t,x}+f^{a(s, X_s^{t,x},Z_s^{t,x})}(s, X_s^{t,x})\,ds\\
&\qquad- \int_{t'}^{T}Z_s^{t,x}\,d\widehat W_s,\,\,\,t'\in[t,T]\,.
\end{split}
\end{equation}
In same way we can show that $v(t,x)=Y^{t,x}_t$ is the value function of our stochastic control problem.
As before, from Krylov~\cite[Chapter 4, section 1, Theorem 1]{krylov controlled},
we get that there is a constant $C>0$ such that for all $(t,x)\in [0,T)\times \mathbb R^d$ we have that $|D_x v(t,x)| \leq C$.
Moreover, as before, using Remark~\ref{rem grad bdd}, the fact that $\sigma^{-1}(s,X_s) Z_s = D_x v(s,X_s)$, and Assumption~\ref{ass extra for pia}, for all $s\in[t,T]$ we get that
\begin{equation}\label{lipschitz F policy iteration}
\begin{split}
|F_s(Z_s,Z_s) & - F_s(Z^{n-1}_s,Z^{n}_s)|  \\
& \le\theta|\sigma^{-1}(s,X_s)Z_s||Z_s-Z^{n-1}_s|\\
&\qquad +|b^{a(s,X_s,Z^{n-1}_s)}(s,X_s)\sigma^{-1}(s,X_s)||Z_s-Z^n_s|+\theta|Z_s-Z^{n-1}_s|\\
& \le\theta C|Z_s-Z^{n-1}_s| + K|Z_s-Z^{n}_s|+\theta|Z_s-Z^{n-1}_s|\,.
\end{split}
\end{equation}
Finally we note that $\xi \in L^2(\Omega, \mathcal F_T, \hat{\mathbb P})$ and $F_s(0,0)\in\hat{\mathbb{H}}^2$,
so by Lemma~\ref{policy convergence estimate}, together with~\eqref{eq pia bsde iter in P hat2} and~\eqref{eq: BSDE for value function policy iteration},
we have $q \in (0,1)$ and $\gamma \geq 0$ such that for all $t\in [0,T]$
\begin{equation}
\label{eq conclusion of contr lem}
e^{\gamma t}\hat{\mathbb E} |Y_t - Y^n_t|^2 + \|Z - Z^n\|_{\hat{\mathbb H}^2_\gamma}^2 \leq q \|Z - Z^{n-1}\|_{\hat{\mathbb H}^2_\gamma}^2\,.
\end{equation}
Similarly as before, using~\eqref{eq conclusion of contr lem}, we conclude that
\begin{equation*}
\begin{split}
&|v(t,x)-v^n(t,x)|^2 \\
&\qquad \leq q^n \hat{\mathbb E} \int_t^T  e^{\gamma (T-t)}|\sigma(s,X^{t,x,\alpha^\ast}_s)|^2|D_x v(s,X^{t,x,\alpha^\ast}_s) - D_x v^0(s,X^{t,x,\alpha^\ast}_s)|^2\,ds\,.
\end{split}
\end{equation*}
This concludes the proof of the theorem.
\end{proof}

\begin{remark}
Consider briefly the situation where the diffusion coefficient
also depends on the control, i.e., $\sigma = \sigma^a(t,x)$.
After applying It\^o's formula to $v^n(s,X_s)$ and substituting the solution to the linear PDE for $v^n$ we get
	\begin{equation*}
	\begin{split}
	dv^n(s,X_s)
	= & \big[(b^{\alpha^\ast_s} D_x v^n)(s,X_s)+\frac12\text{tr}(\sigma^{\alpha^\ast_s}(\sigma^{\alpha^\ast_s})^\top D^2_x v^n)(s,X_s)\\
	&  - (b^{a^n}D_xv^n)(s,X_s)-\frac12\text{tr}(\sigma^{a^n}(\sigma^{a^n})^\top D^2_x v^n)(s,X_s)
	- f^{a^n}(s, X_s) \big]\,ds\\
	& + (D_x v^n\sigma)(s,X_s) \, dW_s	\,.
	\end{split}
	\end{equation*}
	The resulting object can be seen as a second order BSDE (2BSDE). Analysis of 2BSDEs goes beyond the scope of this paper.
\end{remark}
\begin{remark}
Let us briefly consider the infinite-time-horizon control problem.
In this case we consider a constant $\lambda > 0$ and the gain functional:
\begin{equation*}
	J(x,\alpha)=\E\left[\int_0^{\infty}e^{-\lambda s}f^{\alpha}(X^x_s)\,ds\right]\,.
\end{equation*}
It is known that the Bellman PDE for the value function is
\begin{equation*}
	\lambda v-\frac{1}{2}\text{tr}(\sigma\sigma^\top D_x^2 v)-\sup_{a\in A}\left(b^aD_xv+f^a\right)=0\,\,\,\text{on} \,\,\, \ensuremath{\mathbb{R}}^d\,.
\end{equation*}
The linear PDE from the iteration of the policy improvement algorithm then is
\begin{equation*}
	\lambda v^n-\frac{1}{2}\text{tr}(\sigma\sigma^\top D_x^2 v^n)-b^{a^n}D_x v^{n}-f^{a^n}=0\,\,\,\text{on} \,\,\, \ensuremath{\mathbb{R}}^d\,,
\end{equation*}
	where
\begin{equation*}
	a^n(x)=\arg\max_{a \in A}\left(b^a(x)D_xv^{n-1}(x)+f^a(x)\right)\,.
\end{equation*}
After applying It\^o's formula we get
\begin{equation*}
\begin{split}
	dv^n(X_s) = & \left[ \frac12\text{tr}(\sigma\sigma^\top D^2_x v^n)(X_s) + (b^{\alpha^\ast_s} D_x v^n)(X_s) \right]\,ds + (D_x v^n\sigma) (X_s) \, dW_s\\
	= & \big[(b^{\alpha^\ast_s} D_x v^n)(X_s)  - (b^{a^n}D_xv^n)(X_s)- f^{a^n}(X_s) +\lambda v^n(X_s) \big]\,ds\\
	& + (D_x v^n\sigma)(X_s) \, dW_s	\,.
\end{split}
\end{equation*}
Let $Y^n_t:=v^n(X_t)$ and 	$Z^n_t:=\sigma(X_t)D_xv^n(X_t)$. Then after change of measure we may write
\begin{equation}
\label{eq: BSDE in infinite horizon}
	dY^n_s=-\big[(b^{a(X_s,Z^{n-1}_s)}\sigma^{-1})(X_s)Z^n_s+ f^{a(X_s,Z^{n-1}_s)}(X_s) -\lambda Y^n_s \big]\,ds + Z^n_s \, d\widehat{W}_s\,.
\end{equation}
Let
\begin{equation*}
F_s(z,Z):=(b^{a(X_s,z)}\sigma^{-1})(X_s)Z+ f^{a(X_s,z)}(X_s)\,.
\end{equation*}
Hence~\eqref{eq: BSDE in infinite horizon} becomes
\begin{equation}
\label{eq: BSDE in infinite horizon 2}
dY^n_s=\left[-F_s(Z^{n-1}_s,Z^n_s) +\lambda Y^n_s \right]ds + Z^n_s \, d\widehat{W}_s\,,\,\,\,s\in [0,\infty)\,.
\end{equation}
To proceed, we need a suitable contraction-type inequality for this infinite time horizon BSDE.
Buckdahn and Peng~\cite{Buckdahn1999Ergodic} studied infinite time horizon BSDEs and have proved existence and uniqueness of their solutions for sufficiently large values of $\lambda$.
To get the required contraction-type inequality we can use similar calculations as in Fuhrman and Tessitore~\cite[Theorems 3.2 and 3.7]{Fuhrman2004Infinite}, where they use Banach's fixed point theorem to show existence and uniqueness of solutions to infinite time horizon BSDEs.
Hence, for sufficiently large $\lambda > 0$ we would obtain results analogous to Theorem~\ref{main theorem alg 2} as well as the other theorems in the article.

\end{remark}

\section{Policy improvement}
\label{sec policy improvement}

We want to show that the policy obtained at each step of Algorithm~\ref{alg pia}
is an improvement on the one from the previous step.
This is formulated as Theorem~\ref{thm pia improves} below.
Note that we do not require Assumption~\ref{ass extra for pia} here.
\begin{theorem}
\label{thm pia improves}
Let Assumptions~\ref{assumption controlled SDE}, \ref{assumption rewards}, and \ref{ass convexity} hold.
Assume that there exists $K\ge 0$ such that $\forall t\in[0,T]$, $\forall x\in\ensuremath{\mathbb{R}}^d$, and $\forall a\in A$
\[
|b^{a}\sigma^{-1}(t,x)|<K.
\]
Fix $n \in \mathbb N$. Let $v^n$ and $v^{n+1}$ be the solutions of~\eqref{eq policy iteration pde} at
steps $n$ and $n+1$ of the algorithm.
Then for all $t\in[0,T]$, $x\in\ensuremath{\mathbb{R}}^d$ it holds that
\[
v^{n+1}(t,x)\ge v^{n}(t,x).
\]
\end{theorem}
\begin{proof}
Let $X = X^{t,x,\alpha^\ast}$ be the solution to the SDE~\eqref{sde} started from $(t,x)$ and controlled by the optimal control process $\alpha^\ast$; see Remark~\ref{remark alpha star}.
Then, as in the proof of Theorem~\ref{main theorem alg 2}, we get that for $k=n,n+1$ with
$Y^k=Y^{k,t,x}=v^k(\cdot, X^{t,x,\alpha^\ast})$ and with $Z^k=Z^{k,t,x}=(\sigma D_x v^k)(\cdot, X^{t,x,\alpha^\ast})$
we have the BSDE representation
\begin{equation*}
Y_t^k= \xi+\int_{t}^{T}F_s(Z^{k-1}_s,Z^{k}_s)ds-\int_{t}^{T}Z^k_s\,d\widehat{W}_s\,,\,\,k=n,n+1,
\end{equation*}
where
\[
F_s(z,Z):=b^{a(s,X_s,z)}(s,X_s)\sigma^{-1}(s,X_s)Z+f^{a(s,X_s,z)}(s,X_s)\,.
\]
Let us denote for $s\in[t,T]$ and $z\in\ensuremath{\mathbb{R}}^d$
\[
\phi^2_s(z):=F_s(Z^{n}_s,z)\,\,\,\text{and }\,\,\, \phi^1_s(z):=F_s(Z^{n-1}_s,z)\,.
\]
Hence, notice that by the definition of the $a^{n+1}$ (see~\eqref{eq def anplus1}), we have for all $s\in [t,T]$ that
\begin{equation*}
\begin{split}
\phi^2_s(Z^{n}_s)=&F_s(Z^n_s,Z^n_s)=b^{a^{n+1}}(s,X_s)\sigma^{-1}(s,X_s)Z^n_s+f^{a^{n+1}}(s,X_s)\\
=&\max_{a\in A}\left((b^aD_xv^{n})(s,X_s)+f^a(s,X_s)\right)\geq (b^{a^n}D_xv^{n})(s,X_s)+f^{a^n}(s,X_s)\\
=&F_s(Z^{n-1}_s,Z^{n}_s)=\phi^1_s(Z^n_s).
\end{split}
\end{equation*}
Therefore by the comparison principle for BSDEs (see Lemma~\ref{comparison principle}), we get
\[
Y_{t'}^{n+1}\ge Y^{n}_{t'}\,\,\, \forall t'\in [t,T].
\]
Hence, we have
\[
v^{n+1}(t,x) = Y^{n+1,t,x}_t \ge  Y^{n,t,x}_t = v^n(t,x)\,.
\]
\end{proof}

\begin{remark}
It is perhaps interesting to note that the comparison principle for BSDEs cannot be used to deduce that in the gradient iteration algorithm we have
an ``improvement'' at each step.
Indeed, let us write the BSDE representation of the two steps of gradient iteration for $n,n+1\in\mathbb{N}$,
\begin{equation*}
Y_t^n= \xi+\int_{t}^{T} F_s(Z^{n-1}_s)ds-\int_{t}^{T}Z^n_s\,d\widehat{W}_s\,
\end{equation*}
and
\begin{equation*}
Y_t^{n+1}= \xi+\int_{t}^{T} F_s(Z^{n}_s)ds-\int_{t}^{T}Z^{n+1}_s\,d\widehat{W}_s\,,
\end{equation*}
where
\[
F_s(z) := b^{a(s,X_s,z)}(s,X_s)\sigma^{-1}(s,X_s) z + f^{a(s,X_s,z)}(s,X_s).
\]
In order to apply a comparison principle for BSDEs (see Lemma~\ref{comparison principle}), we would need to have $F_s(Z^{n-1}_s)\le F_s(Z^{n}_s)$.
Nevertheless we observe that
\begin{equation*}
\begin{split}
F_s(Z^{n-1}_s)&=b^{a(s,X_s,Z^{n-1}_s)}\sigma^{-1}(s,X_s)Z^{n-1}_s+f^{a(s,X_s,Z^{n-1}_s)}\\
&=b^{a^n}(s,X_s)\sigma^{-1}(s,X_s)Z^{n-1}_s+f^{a^n}(s,X_s)
\\&=\max_{a\in A}(b^{a}(s,X_s)D_xv^{n-1}(s,X_s)+f^a(s,X_s)).
\end{split}
\end{equation*}
Similarly,
\begin{equation*}
\begin{split}
F_s(Z^{n}_s)&=b^{a(s,X_s,Z^{n}_s)}\sigma^{-1}(s,X_s)Z^{n}_s+f^{a(s,X_s,Z^{n}_s)}\\
&=b^{a^{n+1}}(s,X_s)\sigma^{-1}(s,X_s)Z^{n}_s+f^{a^{n+1}}(s,X_s)
\\&=\max_{a\in A}(b^{a}(s,X_s)D_xv^{n}(s,X_s)+f^a(s,X_s)).
\end{split}
\end{equation*}
From the above calculations we have no way to conclude that $F_s(Z^{n-1}_s)\le F_s(Z^{n}_s)$.
Thus the gradient iteration algorithm is not guaranteed to be improving the policy with each step.
\end{remark}

\section{Stability under Perturbations to Solution of the Linear PDE}
\label{sec stability lin PDE}
In this section we study a stability property of the policy improvement algorithm
under perturbations to solutions of the linear PDE~\eqref{eq policy iteration pde}
since in practical applications one will only solve this equation approximately.
Of course the maximization step~\eqref{eq def anplus1} of Algorithm~\ref{alg pia} can
now be performed only with this approximate solution, thus feeding the errors into further iterations.

Let $\varepsilon$ be a parameter (or a set of parameters),
which determines the accuracy of our approximation to the solution of the linear PDE~\eqref{eq policy iteration pde}.
Let $\pi^n_\varepsilon$ be the policy at iteration $n$ obtained from an approximate
solution to the linear PDE.
Let $v_\varepsilon^n$ denote the solution to
\begin{equation}
\label{eq pde for perturbations}
\begin{split}
\partial_t v_\varepsilon^n + \frac{1}{2}\text{tr}(\sigma \sigma^\top D^2_x v_\varepsilon^n)+b^{\pi^n_\varepsilon} D_x v^n_\varepsilon + f^{\pi^n_\varepsilon} & = 0\text{ on }[0,T)\times\ensuremath{\mathbb{R}}^d\,,\\
v_\varepsilon^n (T,\cdot) & = g\,\,\text{ on } x\in\ensuremath{\mathbb{R}}^d\,.
\end{split}
\end{equation}
At step $n$ of Algorithm~\ref{alg pia} we approximate the solution to the
equation above (this is PDE~\eqref{eq policy iteration pde} but with $\pi^n_\varepsilon$ replacing $a^n$ everywhere).
We will denote such approximation by $\tilde v^n_\varepsilon$.
The policy function for the next iteration step is then given by
\[
\pi^{n+1}_\varepsilon(t,x)=a(t,x,(\sigma D_x\tilde{v}^n_\varepsilon)(t,x))=
\text{arg}\max_{a \in A} \left[(b^a D_x\tilde{v}^n_\varepsilon)(t,x) + f^a(t,x) \right]\,,
\]
recalling that the function $a=a(t,x,z)$ was defined in~\eqref{control function}.
We need to assume that $(t,x)\mapsto D_x \tilde v_\varepsilon^n$ is bounded so that $\pi_\varepsilon^{n+1}$ is Lipschitz in $x$
so that the solution to~\eqref{eq pde for perturbations} is $C^{1,2}([0,T]\times \mathbb R^d)$.
This assumption is not really a restriction as we know that the gradient of
the value function is bounded under our assumptions; see Krylov~\cite[Chapter 4, section 1, Theorem 1]{krylov controlled} and also Remark~\ref{remark alpha star}.
Any reasonable approximation should retain this property.

\begin{theorem}
Let Assumptions \ref{assumption controlled SDE}, \ref{assumption rewards}, \ref{ass convexity}, and \ref{ass extra for pia} hold.
Let $(v^n)_{n\in \mathbb N}$ be the approximation sequence given by Algorithm~\ref{alg pia}.
Let $(v^n_\varepsilon)_{n\in \mathbb N}$ be the approximation sequence given by~\eqref{eq pde for perturbations}.
Let $\alpha^*$ and $X^{t,x,\alpha^*}$ be the optimal control process for~\eqref{cp}
and the associated diffusion started from $(t,x) \in  [0,T] \times \ensuremath{\mathbb{R}}^d$.
Assume that $D_x \tilde v_\varepsilon^n$ is uniformly bounded.
Define
\[
\begin{split}
E^{k}_{t,x} := \bigg\|\Big[(\sigma (D_x v^k_\varepsilon - D_x \tilde v^k_\varepsilon))(\cdot, X^{t,x,\alpha^*})\Big]
\mathcal E^{-1/2}((b^{\alpha^*}\sigma^{-1})(\cdot,X^{t,x,\alpha^*}) \bullet W)_T
\bigg \|_{\mathbb H^2}\,.
\end{split}
\]
Then there is $q\in(0,1)$ and $\gamma > 0$, depending only on $K, \theta, T$, such that for all $(t,x)\in[0,T]\times\ensuremath{\mathbb{R}}^d$  there exists $C=C(t,x)$ such that
\begin{equation*}
\begin{split}
& |v^n(t,x) -  v^n_\varepsilon(t,x)|^2 \le  C(t,x)q^{n}+2e^{\gamma (T-t)}\sum_{k=1}^nq^kE^{n-k}\,.
\end{split}
\end{equation*}
\end{theorem}

\begin{proof} Let $X = X^{t,x,\alpha^\ast}$ be the solution to the SDE~\eqref{sde} started from $(t,x)$ and controlled by the optimal control process $\alpha^\ast$; see Remark~\ref{remark alpha star}.
By applying It\^o's formula  to $v^n_\varepsilon$ we get
\begin{equation*}
\begin{split}
dv^n_\varepsilon(s,X_s) = & \left[\partial_t v^n_\varepsilon(s,X_s) + \frac{1}{2}\text{tr}(\sigma\sigma^\top D^2_xv^n_\varepsilon)(s,X_s) + (b^{\alpha_s^*} D_x v^n_\varepsilon)(s,X_s) \right]\,ds \\
& + (D_x v^n_\varepsilon \sigma ) (s,X_s) \, dW_s\\
= & \left[(b^{\alpha_s^*} D_x v^n_\varepsilon)(s,X_s) - \left(b^{\pi^n_\varepsilon}D_x v^n_\varepsilon + f^{\pi^n_\varepsilon}\right)(s,X_s)
\right]\,ds\\
& + (D_x v^n_\varepsilon \sigma )(s,X_s) \, dW_s.	\,
\end{split}
\end{equation*}
Let us denote
\[
Y^n_{t,\varepsilon}:=v^n_\varepsilon(t,X_t)\,,\,\,\,\,Z^n_{t,\varepsilon}:= \sigma(t,X_t)D_xv^n_\varepsilon(t,X_t)\,,\,\,\,\xi:=g(X_T)\,,
\]
\[
F_s(z,Z):=b^{a(s,X_s,z)}(s,X_s)\sigma^{-1}(s,X_s)Z+f^{a(s,X_s,z)}(s,X_s)
\]
and
\[
\tilde{Z}^{n-1}_{t,\varepsilon}:=\sigma(t,X_t)D_x\tilde{v}^{n-1}_\varepsilon(t,X_t),
\]
where $\tilde{v}^{n-1}_\varepsilon$ is an approximate solution to corresponding PDE.
Then using this notation, we may write
\begin{equation*}
\begin{split}
Y^n_{t,\varepsilon}=\xi-\int_{t}^{T}\left[(b^{\alpha_s^*}\sigma^{-1})(s,X_s)Z^n_{s,\varepsilon} -F_s(\tilde{Z}^{n-1}_{s,\varepsilon},Z^n_{s,\varepsilon}) \right]ds-\int_{t}^{T}Z^n_{s,\varepsilon}\,dW_s\,.
\end{split}
\end{equation*}
Let $\hat{\mathbb P}$ and $\widehat{W}$ be given by Lemma~\ref{lemma P hat}.
Then the above equation becomes
\begin{equation*}
Y^n_{t,\varepsilon}= \xi+\int_{t}^{T}F_s(\tilde{Z}^{n-1}_{s,\varepsilon},Z^n_{s,\varepsilon}) ds-\int_{t}^{T}Z^n_{s,\varepsilon}\,d\widehat{W}_s\,.
\end{equation*}
We want to study the difference of $(Y^n_{\varepsilon},Z^n_{\varepsilon})$ with $(Y^n,Z^n)$, where $(Y^n,Z^n)$ solves the BSDE~\eqref{eq pia bsde iter in P hat2}.
\begin{equation}
\begin{split}
\label{eq stability estimate}
\hat{\mathbb E}[e^{\gamma t}|Y^n_t-Y^n_{t,\varepsilon}|^2]+&\|Z^n-Z^n_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\le 2\hat{\mathbb E}[e^{\gamma t}|Y_t-Y^n_{t}|^2]+2\|Z-Z^n\|_{\hat{\mathbb H}^2_\gamma}^2\\
&+2\hat{\mathbb E}[e^{\gamma t}|Y_t-Y^n_{t,\varepsilon}|^2]+2\|Z-Z^n_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\,,
\end{split}
\end{equation}
where $(Y,Z)$ solves the BSDE~\eqref{eq: BSDE for value function policy iteration}. Due to~\eqref{lipschitz F policy iteration} and
\begin{equation*}
|F_s(Z_s,Z_s)-F_s(\tilde{Z}^{n-1}_{s,\varepsilon},Z^n_{s,\varepsilon})|\le (C\theta+\theta)|Z_s-\tilde{Z}^{n-1}_{s,\varepsilon}|+K|Z_s-Z^n_{s,\varepsilon}|\,,
\end{equation*}
we can apply Lemma~\ref{policy convergence estimate}. Hence, there is $\tilde{q}\in (0,1/2)$ and $\gamma >0$ such that
\begin{equation*}
\hat{\mathbb E}[e^{\gamma t}|Y_t-Y^n_{t}|^2]+\|Z-Z^n\|_{\hat{\mathbb H}^2_\gamma}^2\le \tilde{q}\|Z-Z^{n-1}\|_{\hat{\mathbb H}^2_\gamma}^2
\end{equation*}
and
\begin{equation*}
\hat{\mathbb E}[e^{\gamma t}|Y_t-Y^n_{t,\varepsilon}|^2]+\|Z-Z^n_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\le \tilde q \|Z-\tilde{Z}^{n-1}_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\,.
\end{equation*}
Therefore we continue the estimate~\eqref{eq stability estimate}
\begin{equation*}
\begin{split}
\hat{\mathbb E}&[e^{\gamma t}|Y^n_t-Y^n_{t,\varepsilon}|^2]+\|Z^n-Z^n_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\\
&\le 2\hat{\mathbb E}[e^{\gamma t}|Y_t-Y^n_{t}|^2]+2\|Z-Z^n\|_{\hat{\mathbb H}^2_\gamma}^2+2\hat{\mathbb E}[e^{\gamma t}|Y_t-Y^n_{t,\varepsilon}|^2]+2\|Z-Z^n_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\,\\
&\le 2\tilde q\|Z-Z^{n-1}\|_{\hat{\mathbb H}^2_\gamma}^2+2\tilde q \|Z-\tilde{Z}^{n-1}_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\\
&\le 2\tilde q\|Z-Z^{n-1}\|_{\hat{\mathbb H}^2_\gamma}^2+4\tilde q\|Z-Z^{n-1}_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2+4\tilde q\|Z^{n-1}_{\varepsilon}-\tilde{Z}^{n-1}_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\\
&\le 2\tilde q^2\|Z-Z^{n-2}\|_{\hat{\mathbb H}^2_\gamma}^2+4\tilde q^2\|Z-\tilde{Z}^{n-2}_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2+4\tilde q\|Z^{n-1}_{\varepsilon}-\tilde{Z}^{n-1}_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\\
&\le 2\tilde q^2\|Z-Z^{n-2}\|_{\hat{\mathbb H}^2_\gamma}^2+8\tilde q^2\|Z-Z^{n-2}_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2+8\tilde q^2\|Z^{n-2}_{\varepsilon}-\tilde{Z}^{n-2}_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\\
&\qquad+4\tilde q\|Z^{n-1}_{\varepsilon}-\tilde{Z}^{n-1}_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\le\cdots\\
&\le 2\tilde q^n\|Z-Z^{0}\|_{\hat{\mathbb H}^2_\gamma}^2+2^{n+1}\tilde q^n\|Z-Z^0_\varepsilon\|_{\hat{\mathbb H}^2_\gamma}^2+\sum_{k=1}^n2^{k+1}\tilde q^k\|Z^{n-k}_{\varepsilon}-\tilde{Z}^{n-k}_{\varepsilon}\|_{\hat{\mathbb H}^2_\gamma}^2\,.
\end{split}
\end{equation*}
This concludes the proof of the theorem.
\end{proof}

\section{Stability under Perturbation of the Maximization}
\label{sec stability max}
In this section we study a stability property of the gradient iteration algorithm under perturbations to maximization procedure~\eqref{eq gia update control}.
Let $\bar{v}^{n}$ be the solution to  corresponding PDE at iteration $n$ of the gradient iteration algorithm, where instead of obtaining the control function corresponding to the exact maximum
\[
a^n(t,x)
= a(t,x,(\sigma D_x\bar{v}^{n})(t,x))
=\text{arg}\max_{a \in A}((b^a D_x\bar{v}^{n}+f^a)(t,x))\,
\]
we only solve this maximization problem approximately and so we are dealing with
a control function of the form
\[
\bar{a}(t,x,(\sigma D_x\bar{v}^{n})(t,x))
:=a(t,x,(\sigma D_x\bar{v}^{n})(t,x))+\varepsilon(t,x,(\sigma D_x\bar{v}^{n})(t,x))\,,
\]
where the function $\varepsilon=\varepsilon(t,x,z)$ determines the accuracy of our approximation.

\begin{theorem}
Let Assumptions \ref{assumption controlled SDE}, \ref{assumption rewards}, \ref{ass convexity}, and \ref{ass extra for pia} hold.
Let $(v^n)_{n\in\mathbb{N}}$ be the approximation sequence given by Algorithm \ref{alg via}. Let $(\bar{v}^n)_{n\in\mathbb{N}}$ be the approximation sequence given by the perturbations to the maximization procedure and assume that $v^0 = \bar v^0$.
Let $\alpha^\ast$ and $X^{t,x,\alpha^\ast}$ be the optimal control process for~\eqref{cp} and the associated diffusion started from $(t,x)\in[0,T]\times\ensuremath{\mathbb{R}}^d$. Define
\[
E^{k+1}_{t,x}:=\left\|\left[1+|D_x\bar{v}^k(\cdot,X^{t,x,\alpha^*})|\right]\mathcal{E}^{-1/2}((b^{\alpha^*}\sigma^{-1})(\cdot,X^{t,x,\alpha^\ast}) \bullet W)_T\right\|^2_{\mathbb{H}^2}\,,
\]
\[
\varepsilon^{k+1}=\sup_{(s,y)\in[t,T]\times\ensuremath{\mathbb{R}}^d}|\varepsilon(s,y,(\sigma D_x\bar{v}^k)(s,y))|^2\,.
\]
Then there is $q\in(0,1)$ and $\gamma>0$, depending only on $K,\theta,T$, such that for all $(t,x)\in[0,T]\times\ensuremath{\mathbb{R}}^d$ there exists $C=C(t,x)$ such that
\begin{equation*}
|\bar{v}^n(t,x)-v^n(t,x)|^2\le  C(t,x)q^n+ e^{\gamma (T-t)}\frac{2\theta}{C\theta+K+\theta}\sum_{k=1}^{n}q^{n-k+1}\varepsilon^{k}E_{t,x}^{k}\,.
\end{equation*}
\end{theorem}
\begin{proof} Let $X = X^{t,x,\alpha^\ast}$ be the solution to the SDE~\eqref{sde} started from $(t,x)$ and controlled by the optimal control process $\alpha^\ast$; see Remark~\ref{remark alpha star}.
As in the proof of Theorem~\ref{main theorem alg 1} we can  write two BSDEs we get after the change of measure given by Lemma~\ref{lemma P hat}.
The first BSDE arises from the perturbations of the maximization:
\[
\bar{Y}_t^n=\xi+\int_{t}^{T}\bar{F}_s(\bar{Z}^{n-1}_s)\,ds-\int_{t}^T\bar{Z}_s^n\,d\widehat{W}_s\,,
\]
where
\[
\bar{F}_s(z)=b^{\bar{a}(s,X_s,z)}\sigma^{-1}(s,X_s)z+f^{\bar{a}(s,X_s,z)}(s,X_s)\,.
\]
The second BSDE arises from the gradient iteration algorithm with the maximization performed exactly:
\[
Y^n_t=\xi+\int_{t}^TF_s(Z^{n-1}_s)\, ds-\int_{t}^TZ^n_s\,d\widehat{W}_s\,,
\]
where
\[
F_s(z)=b^{a(s,X_s,z)}\sigma^{-1}(s,X_s)z+f^{a(s,X_s,z)}(s,X_s)\,.
\]
We want to study the difference of $(\bar{Y}^n,\bar{Z}^n)$ with $(Y^n,Z^n)$. Hence, notice that
\begin{equation}\label{eq: diff of bar Y^n-Y^n}
\begin{split}
e^{\gamma t}\hat\E|\bar{Y}^n_{t}-Y^n_{t}|^2&+\|\bar{Z}^n-Z^n\|^2_{\hat{\mathbb H}^2_\gamma}
\le 2e^{\gamma t}\hat\E|Y_{t}-Y^n_{t}|^2+2\|Z-Z^n\|^2_{\hat{\mathbb H}^2_\gamma}\\
&+2e^{\gamma t}\hat\E|Y_t-\bar{Y}^n_{t}|^2+2\|Z-\bar{Z}^n\|^2_{\hat{\mathbb H}^2_\gamma}\,,
\end{split}
\end{equation}
where $(Y,Z)$ solves~\eqref{eq: BSDE for value function}.
Therefore, since
\begin{equation*}
\begin{split}
|F_s(Z_s)-F_s(\bar{Z}^{n-1}_s)|\le (C\theta+K+\theta)|Z_s-\bar{Z}^{n-1}_s|\,,
\end{split}
\end{equation*}
we can apply Lemma \ref{lem grad itr pert max} so that there is $q\in(0,1)$ and $\gamma>0$ such that
\begin{equation}
\label{eq outcome of lemma pert max}
\begin{split}
e^{\gamma t}\hat\E|Y_t-\bar{Y}^n_{t}|^2+\|Z-\bar{Z}^n\|^2_{\hat{\mathbb H}^2_\gamma}
\le& q \|Z-\bar{Z}^{n-1}\|^2_{\hat{\mathbb H}^2_\gamma}\\
&+\frac{q}{C\theta+K+\theta}\|\bar{F}(\bar{Z}^{n-1})-F(\bar{Z}^{n-1})\|^2_{\hat{\mathbb H}^2_\gamma}.
\end{split}
\end{equation}
Now we need to estimate the second term of the right-hand side (RHS). Notice that by Assumption \ref{ass extra for pia} the following holds:
\begin{equation}
\label{eq diff of generators pert max}
\begin{split}
&|\bar{F}_s(\bar{Z}^{n-1}_s)-F_s(\bar{Z}^{n-1}_s)|\\
&\le |\sigma^{-1}(s,X_s)\bar{Z}^{n-1}_s||b^{\bar{a}(s,X_s,\bar{Z}^{n-1}_s)}(s,X_s)-b^{a(s,X_s,\bar{Z}^{n-1}_s)}(s,X_s)|\\
&\qquad+|f^{\bar{a}(s,X_s,\bar{Z}^{n-1}_s)}(s,X_s)-f^{a(s,X_s,\bar{Z}^{n-1}_s)}(s,X_s)|\\
&\le \sqrt{\theta}|\sigma^{-1}(s,X_s)\bar{Z}^{n-1}_s||\varepsilon(s,X_s,\bar{Z}^{n-1}_s)|+\sqrt{\theta}|\varepsilon(s,X_s,\bar{Z}^{n-1}_s)|\,.
\end{split}
\end{equation}
Hence by~\eqref{eq diff of generators pert max} we have
\begin{equation}
\label{eq estimate for pert max}
\|\bar{F}(\bar{Z}^{n-1})-F(\bar{Z}^{n-1})\|^2_{\hat{\mathbb H}^2_\gamma}\le \theta\|(1+|\sigma^{-1}(\cdot,X)\bar{Z}^{n-1}|)\varepsilon(\cdot,X,\bar{Z}^{n-1})\|^2_{\hat{\mathbb H}^2_\gamma}\,.
\end{equation}
By inequalities~\eqref{eq: diff of bar Y^n-Y^n},~\eqref{eq outcome of lemma pert max},~\eqref{eq estimate for pert max}, and the result of Theorem~\ref{main theorem alg 1} and since $\bar{Y}^{t,x,n}_t=\bar{v}^n(t,x),\,Y^{t,x,n}_t=v^n(t,x)$
as well as $Z^0 = \bar Z^0$, we conclude that
\begin{equation*}
\begin{split}
e^{\gamma t}&|\bar{v}^n(t,x)-v^n(t,x)|^2
\\
&\le 2e^{\gamma t}\hat{\E}|Y_t-Y^n_t|^2+\|Z-Z^n\|^2_{\hat{\mathbb H}^2_\gamma}+2e^{\gamma t}\hat{\E}|Y_t-\bar{Y}^n_t|^2+\|Z-\bar{Z}^n\|^2_{\hat{\mathbb H}^2_\gamma}\\
& \le 2q\|Z-Z^{n-1}\|^2_{\hat{\mathbb H}^2_\gamma}+2q\|Z-\bar{Z}^{n-1}\|^2_{\hat{\mathbb H}^2_\gamma}+\frac{2q}{C\theta+K+\theta}\|\bar{F}(\bar{Z}^{n-1})-F(\bar{Z}^{n-1})\|^2_{\hat{\mathbb H}^2_\gamma}\\
&\le 4q^n\|Z-Z^0\|^2_{\hat{\mathbb H}^2_\gamma}+ \sum_{k=1}^{n}q^{k}\frac{2\theta}{C\theta+K+\theta}\|(1+|\sigma^{-1}(\cdot,X)\bar{Z}^{n-k}|)\varepsilon(\cdot,X,\bar{Z}^{n-k})\|^2_{\hat{\mathbb H}^2_\gamma}\,.
\end{split}
\end{equation*}
\end{proof}

We obtain the same result for the policy improvement algorithm.

\begin{theorem}
Let Assumptions \ref{assumption controlled SDE}, \ref{assumption rewards}, \ref{ass convexity}, and \ref{ass extra for pia} hold.
Let $(v^n)_{n\in\mathbb{N}}$ be the approximation sequence given by Algorithm \ref{alg pia}. Let $(\bar{v}^n)_{n\in\mathbb{N}}$ be the approximation sequence given by the perturbations to the maximization procedure.
Let $\alpha^\ast$ and $X^{t,x,\alpha^\ast}$ be the optimal control process for~\eqref{cp} and the associated diffusion started from $(t,x)\in[0,T]\times\ensuremath{\mathbb{R}}^d$. Define
\[
E^{k}_{t,x}:=\left\|\left[1+D_x\bar{v}^k(\cdot,X^{t,x,\alpha^\ast})\right]\mathcal{E}^{-1/2}((b^{\alpha^*}\sigma^{-1})(\cdot,X^{t,x,\alpha^\ast}) \bullet W)_T\right\|^2_{\mathbb{H}^2}\,,
\]
\[
\varepsilon^{k+1}=\sup_{(s,y)\in[t,T]\times\ensuremath{\mathbb{R}}^d}|\varepsilon(s,y,(\sigma D_x\bar{v}^k)(s,y))|^2\,.
\]
Then there is $q\in(0,1)$ and $\gamma>0$, depending only on $K,\theta,T$, such that for all $(t,x)\in[0,T]\times\ensuremath{\mathbb{R}}^d$ there is $C=C(t,x)$ such that
\begin{equation*}
|\bar{v}^n(t,x)-v^n(t,x)|^2\le C(t,x)q^n+ \frac{2\theta}{\max(C\theta+\theta,K)}e^{\gamma (T-t)} \sum_{k=1}^{n}q^{k}\varepsilon^{n-k+1}E_{t,x}^{n-k+1}\,.
\end{equation*}
\end{theorem}
\begin{proof} Let $X = X^{t,x,\alpha^\ast}$ be the solution to the SDE~\eqref{sde} started from $(t,x)$ and controlled by the optimal control process $\alpha^\ast$; see Remark~\ref{remark alpha star}.
Due to Theorem \ref{main theorem alg 2} we can  write two BSDEs we get after the change of measure: first from the perturbation and second from the gradient iteration
\[
\bar{Y}_t^n=\xi+\int_{t}^{T}\bar{F}_s(\bar{Z}^{n-1}_s,\bar{Z}_s^n)\,ds-\int_{t}^T\bar{Z}_s^n\,d\widehat{W}_s\,,
\]
\[
Y^n_t=\xi+\int_{t}^TF_s(Z^{n-1}_s,Z^n_s)\, ds-\int_{t}^TZ^n_s\,d\widehat{W}_s\,,
\]
where
\[
\bar{F}_s(z,Z):=b^{\bar{a}(s,X_s,z)}(s,X_s)\sigma^{-1}(s,X_s)Z+f^{\bar{a}(s,X_s,z)}(s,X_s)\,,
\]
\[
F_s(z,Z):=b^{a(s,X_s,z)}(s,X_s)\sigma^{-1}(s,X_s)Z+f^{a(s,X_s,z)}(s,X_s)\,.
\]
Similarly, we want to study the difference of $(\bar{Y}^n,\bar{Z}^n)$ with $(Y^n,Z^n)$. Hence, notice that
\begin{equation}\label{eq: difference of Y bar and Y^n policy iteration}
\begin{split}
e^{\gamma t}\E|\bar{Y}^n_{t}-Y^n_{t}|^2&+\|\bar{Z}^n-Z^n\|^2_{\hat{\mathbb H}^2_\gamma}
\le 2e^{\gamma t}\E|Y_{t}-Y^n_{t}|^2+\|Z-Z^n\|^2_{\hat{\mathbb H}^2_\gamma}\\
&+2e^{\gamma t}\E|Y_t-\bar{Y}^n_{t}|^2+\|Z-\bar{Z}^n\|^2_{\hat{\mathbb H}^2_\gamma}\,,
\end{split}
\end{equation}
where $(Y,Z)$ solves~\eqref{main theorem alg 2}. Therefore, since
\begin{equation*}
\begin{split}
|F_s(Z_s,Z_s)-F_s(\bar{Z}^{n-1}_s,\bar{Z}^{n}_s)|\le \theta C|Z_s-\bar{Z}^{n-1}_s|+K|Z_s-\bar{Z}^{n}_s|+\theta|Z_s-\bar{Z}^{n-1}_s|\,,
\end{split}
\end{equation*}
we can apply Lemma \ref{lem policy impr pert max} so that there is $q\in(0,1)$ and $\gamma>0$ such that
\begin{equation}
\label{eq outcome of lemma pert max policy iteration}
\begin{split}
e^{\gamma t}\E|Y_t-\bar{Y}^n_{t}|^2&+\|Z-\bar{Z}^n\|^2_{\hat{\mathbb H}^2_\gamma}
\le q \|Z-\bar{Z}^{n-1}\|^2_{\hat{\mathbb H}^2_\gamma}\\
&+\frac{q}{\max(C\theta+\theta,K)}\|\bar{F}(\bar{Z}^{n-1},\bar{Z}^n)-F(\bar{Z}^{n-1},\bar{Z}^n)\|^2_{\hat{\mathbb H}^2_\gamma}.
\end{split}
\end{equation}
Now we need to estimate the second term of the RHS. Notice that by Assumption \ref{ass extra for pia} we have that
\begin{equation}
\label{eq diff of generators pert max policy iteration}
\begin{split}
&|\bar{F}_s(\bar{Z}^{n-1}_s,\bar{Z}^{n}_s)-F_s(\bar{Z}^{n-1}_s,\bar{Z}^{n}_s)|\\
&\le\left|b^{\bar{a}(s,X_s,\bar{Z}^{n-1}_s)}(s,X_s)\sigma^{-1}(s,X_s)\bar{Z}^{n}_s-b^{a(s,X_s,\bar{Z}^{n-1}_s)}(s,X_s)\sigma^{-1}(s,X_s)\bar{Z}^{n}_s\right|\\
 &\qquad+\left|f^{\bar{a}(s,X_s,\bar{Z}^{n-1}_s)}(s,X_s)-f^{a(s,X_s,\bar{Z}^{n-1}_s)}(s,X_s)\right|\\
 &\le \sqrt{\theta}|\sigma^{-1}(s,X_s)\bar{Z}^{n}_s||\varepsilon(s,X_s,\bar{Z}^{n-1}_s)|+\sqrt{\theta}|\varepsilon(s,X_s,\bar{Z}^{n-1}_s)|\,.
\end{split}
\end{equation}
Hence by~\eqref{eq diff of generators pert max policy iteration} we have
\begin{equation}
\label{eq estimate for pert max policy iteration}
\|\bar{F}(\bar{Z}^{n-1},\bar{Z}^n)-F(\bar{Z}^{n-1},\bar{Z}^n)\|^2_{\hat{\mathbb H}^2_\gamma}\le \theta\|(1+|\sigma^{-1}(s,X)\bar{Z}^{n}|)\varepsilon(\cdot,X,\bar{Z}^{n-1})\|^2_{\hat{\mathbb H}^2_\gamma}\,.
\end{equation}
By inequalities~\eqref{eq: difference of Y bar and Y^n policy iteration},~\eqref{eq outcome of lemma pert max policy iteration},~\eqref{eq estimate for pert max policy iteration}, by the result of Theorem \ref{main theorem alg 2}, and by $\bar{Y}^{t,x,n}_t=\bar{v}^n(t,x)$, $Y^{t,x,n}_t=v^n(t,x)$ we conclude that
\begin{equation*}
\begin{split}
e^{\gamma t}&|\bar{v}^n(t,x)-v^n(t,x)|^2
\\
&\le 2e^{\gamma t}\hat{\E}|Y_t-Y^n_t|^2+\|Z-Z^n\|^2_{\hat{\mathbb H}^2_\gamma}+2e^{\gamma t}\hat{\E}|Y_t-\bar{Y}^n_t|^2+\|Z-\bar{Z}^n\|^2_{\hat{\mathbb H}^2_\gamma}\\
& \le 2q\|Z-Z^{n-1}\|^2_{\hat{\mathbb H}^2_\gamma}+2q\|Z-\bar{Z}^{n-1}\|^2_{\hat{\mathbb H}^2_\gamma}\\
&\qquad+\frac{2q}{\max(C\theta+\theta,K)}\|\bar{F}(\bar{Z}^{n-1})-F(\bar{Z}^{n-1})\|^2_{\hat{\mathbb H}^2_\gamma}\\
&\le 2q^n\|Z-Z^0\|^2_{\hat{\mathbb H}^2_\gamma}+2q^n\|Z-\bar Z^0\|^2_{\hat{\mathbb H}^2_\gamma}\\
&\qquad+ \sum_{k=1}^{n}q^{k}\frac{2\theta}{\max(C\theta+\theta,K)}\|(1+|\sigma^{-1}(\cdot,X)\bar{Z}^{n-k+1}|)\varepsilon(\cdot,X,\bar{Z}^{n-k})\|^2_{\hat{\mathbb H}^2_\gamma}\,.
\end{split}
\end{equation*}
\end{proof}

\section{Example}\label{sec example}
In this section we would like to consider an example when Assumptions~\ref{assumption controlled SDE}, \ref{assumption rewards},
\ref{ass convexity}, and~\ref{ass extra for pia} hold. Let $t\mapsto s(t)$ and $t\mapsto k(t)$ be continuous functions for $t\in[0,T]$. Consider the state which is governed by the controlled SDE
\begin{equation*}
dX_t=s(t)\sin \alpha_t\,dt+\sqrt{2}\,dW_t\,,\,\,\,t\in[0,T]\,,
\end{equation*}
and consider the cost functional
\begin{equation*}
J(t,x,\alpha)=\E\left[\int_t^Tk(s)\cos \alpha_s\,ds+g(X_T)\right]\,.
\end{equation*}
The aim is to maximize $J$ over admissible controls $\alpha\in\mathcal{A}$.
The value function $v=\sup_{\alpha\in\mathcal{A}}J(t,x,\alpha)$ satisfies the Bellman PDE
\begin{equation*}
\partial_t v+D_x^2v+\sup_{a\in A}\left[s(t)\sin a\, D_xv+k(t)\cos a\right]=0\,,\,\,\,\text{on}\,\,[0,T)\times\ensuremath{\mathbb{R}}\,,
\end{equation*}
with the terminal condition $v(T,x)=g(x):=\arctan(x)$. Hence, the optimal control is
\begin{equation*}
a(t,x)=\arctan\left(\frac{s(t)D_xv}{k(t)}\right)\,.
\end{equation*}
It is easy to check that Assumptions~\ref{assumption controlled SDE}, \ref{assumption rewards},
\ref{ass convexity}, and~\ref{ass extra for pia} hold for this problem.
Therefore, the Bellman PDE becomes
\begin{equation}\label{eq: pde for value function example}
\partial_t v+D_x^2v+\frac{\frac{(s(t)D_xv)^2}{k(t)}}{\sqrt{1+\left(\frac{s(t)D_xv}{k(t)}\right)^2}} +\frac{k(t)}{\sqrt{1+\left(\frac{s(t)D_xv}{k(t)}\right)^2}}=0\,.
\end{equation}

We can solve this problem using the policy improvement algorithm by approximating the Bellman PDE with a sequence of linear PDEs:

\textit{Step} 1. Make an initial choice of control $a^0(t,x)$.

\textit{Step} 2. For $n=0,1,\dots$:
\begin{itemize}
	\item Evaluation step: Find a solution $v^n=v^n(t,x)$ to the linear PDE
	\begin{equation}\label{eq: pde for v^n example}
	\partial_t v^n+D_x^2 v^n+s(t)\sin a^nD_xv^n+\cos a^n=0\,.
	\end{equation}
	\item Improvement step: Find a new policy $a^{n+1}=a^{n+1}(t,x)$ such that
	\begin{equation*}
	a^{n+1}(t,x)=\arctan\left(\frac{s(t)D_xv^n}{k(t)}\right)\,.
	\end{equation*}
\end{itemize}

\textit{Step} 3. Iterate the process until no changes occur in the controls updates.

One can do similar calculations in the case of the gradient iteration algorithm.

We will solve~\eqref{eq: pde for value function example} and~\eqref{eq: pde for v^n example} by the finite difference method. For simplicity, let us choose $s(t)=1$ and $k(t)=1$ for all $t\in[0,T]$. In Figure~\ref{plot for errors example}, one can see the logarithm of the error between the value function obtained by the iterative methods, by the policy improvement algorithm, and by the gradient iteration algorithm at every step and the value function obtained by the solution of the Bellman PDE. This shows the fast convergence of the policy improvement method for our example in one dimension. In Figure~\ref{figure from policy iteration}, we can see that after only a few steps the policies obtained from the policy improvement algorithm are close to the exact one. Finally, in Figure~\ref{figure from the true solution of the example}, we plot the value function and the policy from the solution of the Bellman PDE.

\begin{figure}[h]
\centering
\minipage{0.49\textwidth}
\includegraphics[width=\linewidth]{images/error_plot_policy_iteration_true.pdf}
\endminipage\hfill
\minipage{0.49\textwidth}
\includegraphics[width=\linewidth]{images/error_plot_gradient_iteration_true.pdf}
\endminipage
\caption{\label{plot for errors example} Plot for the logarithm of the error between the value function obtained by the iterative methods, by the policy improvement, and by the gradient iteration algorithms (from left to right) at every step and the value function obtained by the solution of the Bellman PDE.
Note that the convergence stops once we have reached the accuracy of the finite-difference solver.
}
\end{figure}

\begin{figure}[!htb]
	\minipage{0.32\textwidth}
	\includegraphics[width=\linewidth]{images/policy_iteration_initial.pdf}
	\endminipage\hfill
	\minipage{0.32\textwidth}
	\includegraphics[width=\linewidth]{images/policy_iteration1.pdf}
	\endminipage\hfill
	\minipage{0.32\textwidth}%
	\includegraphics[width=\linewidth]{images/policy_iteration6.pdf}
	\endminipage
	\caption{\label{figure from policy iteration}Plot of the initial policy and policies obtained by the policy improvement algorithm at Steps 1 and 5}
\end{figure}

\begin{figure}[!htb]
	\minipage{0.49\textwidth}
	\includegraphics[width=\linewidth]{images/value_function.pdf}
	\endminipage\hfill
	\minipage{0.49\textwidth}
	\includegraphics[width=\linewidth]{images/policy.pdf}
	\endminipage\hfill
	\caption{\label{figure from the true solution of the example}Plot of the true value function and the policy}
\end{figure}

redpajama_set_name: RedPajamaArXiv


OUR products are used in-kind shooting, due to light and camera problems will have a little color.
Receipt cycle 8-15 days,If you have any questions, please contact the customer service in time.
Please look at the product details, a detailed understanding of the product will be helpful.
Size: 51 * 51cm; only sell napkins, plates do not sell.
Cotton fabric, made of cotton, soft skin, pure color printing and dyeing, color concise and beautiful, elegant package, solid and durable.
Please rest assured to buy, we will provide quality after-sales service.


text:
Chelsea vs Manchester City Predictions
Select Fixture Carlisle vs BarrowSalford City vs Walsall Newcastle vs Southampton Arouca vs Benfica Inter Milan vs Atalanta
Manchester City -1
City beat Chelsea 2-0 in the EFL Cup before the World Cup break, despite resting Erling Haaland, and can beat a -1 handicap at Stamford Bridge. Chelsea have struggled recently, winning just two of their last seven games, and the visitors should be looking to make a statement as they have a chance to move back to within five points of pacesetters Arsenal if they can secure a win.
Manchester City 3-0
Chelsea have failed to score in their last three meetings with City and in three of their last five games. The Stamford Bridge outfit appear low on confidence so back the visitors to record a comfortable 3-0 win.
Erling Braut Haaland To Score Anytime
Erling Haaland has picked up where he left off with four goals in three games since the World Cup break came to an end. Man City's Norwegian star can continue his fine form with a goal at Chelsea.
Manchester City beat Chelsea 2-0 in the EFL Cup in November.
Chelsea were held to a 1-1 draw by Nottingham Forest last time out.
Chelsea have failed to score in the last three meetings between these sides.
Erling Haaland has four goals in his last three games.
Chelsea face tough task against the champions
There is just one Premier League game to look forward to on Thursday but it is arguably the standout tie in this round of fixtures as Manchester City head to Chelsea.
City fell further behind leaders Arsenal last time out as Everton somehow managed to claim a point at the Etihad Stadium, but the Gunners' 0-0 draw with Newcastle on Tuesday means the defending champions have a chance to move back to within five points of top spot with a win at Chelsea.
The hosts have managed four points from two games since the Premier League returned but they were hardly convincing in a win over Bournemouth and a draw at Nottingham Forest and they look set for a difficult evening on Thursday.
City can win with room to spare
City dropped points last time out as they were uncharacteristically wasteful in front of goal, and they can put that right at Stamford Bridge.
Pep Guardiola's men overcame Chelsea 2-0 in the EFL Cup in November and won both league meetings 1-0 last season. Despite resting a few players, including Erling Haaland, City could perhaps have won the last meeting by a bigger margin and with Chelsea looking out of sorts and away win with a -1 handicap appeals.
The London outfit lost their final three games without scoring before the World Cup and they may struggle to find a way through on Thursday. They have rediscovered their scoring touch in their last two, but both of those games were against teams in the bottom six and City have a chance to record a clean sheet.
Potter struggling to settle
Graham Potter already finds himself under pressure as Chelsea manager and the visit of the champions comes at a difficult time. The Blues initially improved after he replaced Thomas Tuchel in mid-September, embarking on a nine game unbeaten run, but they have picked up just two wins in seven games since and occasionally look devoid of ideas in attack, while leaking goals at the other end.
City need a result here if they are to close in on Arsenal and as Chelsea have drawn a blank in the last three meetings between these teams a 3-0 win for the visitors stands out. Once again, it is impossible to ignore Haaland as an anytime scorer option as he has four goals in three games since the action resumed after Qatar and if you are planning a bet builder on this game it could be worth backing the Norwegian to score two or more goals or possibly first – as he did against Everton and Liverpool in the EFL Cup – for extra value.
@TheGFreeman

redpajama_set_name: RedPajamaCommonCrawl


text:
On My Bookshelf: Milkweed by Jerry Spinelli
The basic plot from Amazon: He's a boy called Jew. Gypsy. Stopthief. Filthy son of Abraham.
He's a boy who lives in the streets of Warsaw. He's a boy who steals food for himself, and the other orphans. He's a boy who believes in bread, and mothers, and angels.
He's a boy who wants to be a Nazi, with tall, shiny jackboots of his own-until the day that suddenly makes him change his mind.
And when the trains come to empty the Jews from the ghetto of the damned, he's a boy who realizes it's safest of all to be nobody.
Newbery Medalist Jerry Spinelli takes us to one of the most devastating settings imaginable-Nazi-occupied Warsaw during World War II-and tells a tale of heartbreak, hope, and survival through the bright eyes of a young Holocaust orphan.
Why I liked it: I loved Jerry Spinelli's book Maniac Magee, so I couldn't believe I hadn't read Milkweed sooner. The narrators in the two novels are equally interesting, but Milkweed is set in the Warsaw ghetto (rather than my hometown Philadelphia; Spinelli grew up just outside the city). Similar to The Boy in the Striped Pajamas, the narrator of Milkweed is a young boy confused by the events happening around him. However, Milkweed is much more historically accurate and realistic than John Boyne's book.
At the start of the novel, Misha knows nothing: where his family is, where he came from, not even his own name. He joins a group of ragtag orphan boys, most of them Jewish who roam the city, eating and sleeping where they can, until the Warsaw ghetto is built and closed off from the rest of the city. Uri, one of the other boys in the group, takes Misha under his wing, names him, and makes up a story for him of what his past life was life. Uri thinks Misha is likely Roma (Gypsy) and as an adult reader, I inferred that Misha was suffering from extreme PTSD as a result of whatever terrible thing had happened to his family.
Misha becomes close with a Jewish family who has a young girl his own age named Janina. Before moving into the ghetto, Misha leaves food for Janina and her family. After the ghetto is closed up, Misha begins living with them in their quarters and sneaks out of the ghetto at night to steal food for them. Misha tries to save Janina when the ghetto is liquidated, but she is unwilling to leave her family. I especially appreciated the section of the book focused on Misha's life after he emigrated to the United States, in which he struggles to maintain his sanity and normal relationships as a result of the trauma he has experienced.
Classroom application: We chose Milkweed as one of the six options for our literature circles this spring. All of the texts were set during the Holocaust and WWII and connected to the theme of "decisions that matter." There is some violence and death, but because Misha avoids being sent to a concentration camp, the content was overall less disturbing than some other Holocaust books.
If you are interested in purchasing a copy of Milkweed for yourself, you can find it on Amazon here.
September 30, 2019 history, holocaust, my bookshelf, young adult literature Leave a Comment
#2ndaryELA Twitter Chat on Sunday 9/29: Social Emo...
On My Bookshelf: Hitler's Canary by Sandi Toksvig
#2ndaryELA Twitter Chat on Sunday 9/22: Curriculum...
On My Bookshelf: The Boy on the Wooden Box by Leon...
#2ndaryELA Twitter Chat on Sunday 9/15: Student Ch...
On My Bookshelf: The Boy Who Dared by Susan Campbe...
Turning Students Into Readers with First Chapter F...
#2ndaryELA Twitter Chat on Sunday 9/8: First Chapt...
On My Bookshelf: Sunny by Jason Reynolds


text:
Jalan Bahar () is a major arterial road from Lim Chu Kang Road to Jurong West Avenue 2; originally a Jurong Road/Lim Chu Kang Road link. It was opened on 9 April 1965 to connect the rural areas of Lim Chu Kang to the Jurong Industrial Estate. It also connects through the Pan Island Expressway.

Jalan Bahar is located at the western part of Singapore , and is in close proximity to Tuas and Lim Chu Kang districts. A large portion of Jalan Bahar's land area is restricted and designated for the Singapore Armed Forces' live firing exercises and the Singapore Civil Defence' Basic Rescue Training exercises.

Primarily a wooded, forested area, the land at Jalan Bahar has been cleared over the years and is left undeveloped till today. Only a few commercial buildings stand here, mainly industrial buildings and educational institutions.

Etymology
In Malay, bahar refers to a large lake or river. A flyover also takes its name from the road.

History
Besides the Civil Defence Academy, Pusara Aman Muslim Cemetery as well as the entrance to Nanyang Technological University campus at Nanyang Road, there are few ongoing developments surrounding Jalan Bahar. The road had recently completed its road widening project in late 2016. The purpose of this project was to improve the current traffic situation and to support future developments and economic activities in the West Region.

References

Places in Singapore
Roads in Singapore

redpajama_set_name: RedPajamaWikipedia


text:
Seasonal Gifts Coming to Your Television
Not sure what holiday classics are on TV and when? Sarah Hope breaks down the highlights of the final 10 days leading up to Christmas day.
It's a weird time of year for television.
The fall season is over for most (though not all) new series, many of which won't return until January (or at all). I am personally working hard to fend off symptoms of withdrawal, while I wait a solid six weeks to find out what Annalise knew and when.
For students, the semester is over. For corporate folks, business is slow. It's the holiday season! Whether or not you're religious, it's time to enjoy some good cheer, stare through the steam on a cup of tea at the lights on the Christmas tree, the solstice tree, the menorah or the kinara, and enjoy the company of friends and family.
While some of us soak up holiday TV fare like it will never come again, those who work in industries that thrive on cheer (ahem, retail) may not want to come home and watch Elf for the fourteen bajillionth time. But there's still plenty of good TV out there this week if you know where to look.
On ABC, The Great Christmas Light Fight (8 p.m.) is back for a second season. This yearly short-run reality show features 24 families from across the United States competing to assemble the greatest holiday home lightshow. Catch up on last week's season premiere online at ABC.com or on Hulu Plus before you tune in.
For the Scrooges, there's a new episodes of Jane the Virgin on The CW at 9 p.m. Based on a Venezuelan telenovela, this show, despite its still struggling viewership, has received rave reviews. Plus, it continues through the new year – no mid-season break! There's plenty of time to catch up and get on board before the finale.
Bonus: For those who get truTV, our neighbors to the west will see some hometown heros on the new series, Fake Off. Rochester's PUSH Physical Theatre will be competing in the final round of this creative pop culture reconstruction show at 10 p.m. Catch up on their earlier performances with season reruns all afternoon.
Filmed in 1947, the classic flick Miracle on 34th Street received a worthy reboot in 1994 starring Mara Wilson and Richard Attenborough (AMC 8 p.m.). If you're not in a mood to believe in Santa, the MasterChef Junior finale will air on Fox at 8 p.m. Gordon Ramsay is shockingly charming on this entertaining show, but he doesn't hold a candle to its adorable kid stars.
In a nostalgic and raucous night of back-to-back cheer, ABC Family is airing Home Alone (1990) followed by The Santa Clause (1994), beginning at 6:30 p.m. For those who'd rather see friction than farce, the 29th season of Survivor concludes Thursday night. The Survivor: San Juan del Sur – Blood vs. Water finale will air on CBS at 8 p.m.
Each year, iHeartRadio celebrates the year's best in pop music with the Jingle Ball. The annual holiday concert, formerly hosted by NYC's Z100, was filmed this past Friday at Madison Square Garden in New York City. This year features a host of powerful young ladies (Taylor Swift, Iggy Azalea, Meghan Trainor, and Ariana Grande) and a few fellas (Maroon 5, Nick Jonas, Pharrell Williams, OneRepublic). It will air on The CW at 8 p.m. Thursday. Not too in tune with the current pop sugar? Still get a little nostalgic when it snows, sighing Matthew's proposal to Mary? Never fear – PBS will air a Downton Abbey Recap at 9 p.m., looking back at the first four seasons. It's just in time, too – the fifth season will finally come stateside on January 4.
Will Ferrel as Buddy the Elf
Photo by Alan Markfield
Who doesn't love Buddy the Elf? Well, I'm not a huge Will Farrell fan (sorry, not sorry), but a lot of people are! So, for their sake, I'm happy to say that Buddy will grace your screen once again this year as Elf airs on ABC Family on Friday at 8 p.m. If you're in the mood for more serious holiday fare, or have seen squeal one too many times, turn over to CBS for A Home for the Holidays at 9 p.m. The annual telecast raises awareness and funds for foster and adoption programs, this year with performances by Jennifer Judson, Katharine McPhee, Train, Earth, Wind and Fire, and others. This year among the uplifting adoption stories will be the story of Rich Valenza and Jared Gee, a gay couple with two adopted children.
Five days away and still not feeling the cheer? Check out the series premiere of Caught on Camera with Nick Cannon (8 p.m.), NBC's answer to Tosh.0.
Saturday night, gather your family – Mom, Dad, Gramps, Cousin Mary and the kids – all of whom have come to town a week early (surprise!), and settle in for an evening of fun for the whole gang. Dr. Suess' How The Grinch Stole Christmas (2000, the Jim Carrey version) on ABC Family at 8 p.m. might make Uncle Lou's cold, cold heart grow three sizes. Alternately, in case you missed it, NBC is re-airing last year's live musical disaster The Sound of Music Live! at 8 p.m.
Especially if you caught The Santa Clause on Wednesday, catch The Santa Clause 2 (2002) on AMC at 8 p.m. Sunday.
If not, or if your Scroogy side is starting to show and you're sick of all this cheer business, why not check out what TV was like when it first started? The Disneyland TV series is a multi-format anthology series that has aired consistently, under different names, since 1954. It remains one of the longest-running television series in history. On Sunday night, TCM will air the 1954 special Walt Disney Presents (8:30 p.m.), featuring Walt Disney himself, unveiling the upcoming series.
John Payne (left), Natalie Wood (middle), & Edmund Gwenn (right) in Miracle on 34th Street.
Photo from imdb.com
By now, you'll obviously be hooked on The Great Christmas Light Fight. The season finale airs on ABC 8 p.m. Let there be light! No? Well, Monday has plenty more to offer. Try NBC, where A Saturday Night Live Christmas (originally aired on December 3) will revisit the series' best holiday sketches throughout its 40 seasons. You might also catch a rerun of The iHeartRadio Jingle Ball on The CW at 8 p.m., or have another hot cocoa with The Santa Clause on ABC Family at 8:30 p.m. On AMC, the original 1947 Miracle on 34th Street with Natalie Wood and Edmund Gwenn starts at 6:30 p.m. immediately followed by the 1994 remake at 8:45 p.m.
Not interested in lights, miracles or pop stars? The Syracuse v. Colgate men's basketball game will surely be on at every bar in Syracuse. Tipoff is at 7 p.m.
Two nights before Christmas and all through the house, not a creature will be stirring – except for me, with excitement, because Nickelodeon has a classic Full House Christmas double feature. First at 8 p.m., "A Very Tanner Christmas" (1992), when Steve and DJ break up. Next, "Our Very First Christmas Show" (1988) when the Tanners get snowed in at the airport.
Over at ABC, Robin Roberts will host The Year: 2014 (9 p.m.), a retrospective on the past year in news and pop culture.
Peter Billingsley (right) as Ralphie and Jeff Gillen (left) as Santa Clause in A Christmas Story
Photo from imbd.com
Sorry Scrooges, TV doesn't have much to offer you on Christmas Eve. The airwaves are chock full of classics: It's A Wonderful Life (1946) on both NBC and USA at 8 p.m.; A Christmas Story (1983) on both TBS and TNT at 8 p.m.; and the I Love Lucy (1956) Christmas special on CBS at 8 p.m. For a less savory evening, put the kids to bed early, grab some liquor and watch Billy Bob Thornton at (arguably) his slimiest in Bad Santa (2003) on MTV at 8 p.m.
By the time 11 p.m. rolls around, it'll be time to turn off the TV and put out the cookies and milk. Don't forget to leave the bedroom door cracked so you can listen for sleigh bells on the roof.
And hey, even if you're not into holiday cheer, there's cause to be glad: on Sunday the 21st, these dark Syracuse days start getting longer, and we begin the long journey to springtime.
Happy watching and happy holidays to all!
Header image render by zeroX3A on Deviant Art.
Peter Pan Live: fun, disappointing, poignant
Top 10 Best Television Moments of 2014


text:
Category: Senate Judiciary Committee
Vancouver, BC – Exclusive new video published to social media on September 28, 2018, explores multidimensional facets of Senate Judiciary Committee testimony from Stanford University School of Medicine research psychologist Dr. Christine Blasey Ford. Granular audio analysis appears to partially confirm this Palo Alto University professor's allegations she was sexually assaulted during a high school party in 1982. U.S. President Donald Trump called Dr. Ford a credible witness and has now directed the Federal Bureau of Investigation to conduct a supplemental background check of controversial United States Supreme Court nominee Judge Brett Kavanaugh, named by Dr. Ford as one of her alleged attackers. Supreme Court Justice Ruth Bader Ginsburg is said to be encouraged "by the number of women coming forward to call out sexual misconduct" in MeToo America.
"Speech is a multidimensional stimulus" with "numerous ways you can extract a message from it," explains Dr. Kourosh Saberi, Professor at the University of California, Irvine School of Social Sciences. Dr. Saberi coauthored research (reported by the journal Nature in 1999) using granular synthesis methods that time-reversed and resequenced human speech to test for intelligibility under conditions simulating transmission of audio signals over digital networks. This research informed now-widely implemented Voice over Internet Protocol (VoIP) telephony and was headlined by the Associated Press as "Human Brain Understands Backwards Speech".
This reporter's expert independent study replicated the granular audio synthesis methods used by scientific researchers (including sampling, granular time-reversal and resequencing) in an attempt to detect sleep talking-like unconscious messages reflected in the spoken word testimony of Dr. Christine Blasey Ford during her appearance before the U.S. Senate Judiciary Committee Thursday. The study was not in vain and in fact did detect traces of what appear to be a survivor's disclosures of sexual assault at the hands of one or more men, substantiating public testimony of the witness.
The following results highlight these findings in bold followed by transcripts of the statements from which they were derived. Expert commentary is embedded between each of the results.
Saw that. Speaks it: I remember being on the street and feeling an enormous sense of relief that I had escaped from the house and that Brett and Mark were not coming after me. Brett's assault on me drastically altered my life. For a very long time, I was too afraid and ashamed to tell anyone the details. I did not want to tell my parents that I, at age 15, was in a house without any parents present, drinking beer with boys.
Comment: This message appears to amplify Dr. Ford's remarks, expressing how what she witnessed in the past is what was described during her appearance before the Senate Judicial Committee. It seems to be telling us that she saw those events of which she spoke at that time.
I saw this. Know what I am. 2) Heal victim. 3) He wounded. That knife. Had to have it. 4) You had a block. An exam: The reason this came up in counseling is that my husband and I had completed an extensive remodel of our home, and I insisted on a second front door, an idea that he and others disagreed with and could not understand. In explaining why I wanted to have a second front door, I described the assault in detail. I recall saying that the boy who assaulted me could someday be on the U.S. Supreme Court and spoke a bit about his background. My husband recalls that I named my attacker as Brett Kavanaugh.
Comment: The speaker becomes verbose expressing a burst of multiple messages during this part of her statement. The first message emphasizes standing as a witness while validating cognition of her identity. This is followed by a call for relief directed towards a perceived injured party, as if such a party did exist in this case. The messages then describe a past injury occurring due to the actions of a male perpetrator.
The instrument used to cause injury is described as a knife. During her testimony Dr. Ford stated that a then-teenage Judge Kavanaugh began "grinding his hips into me" suggesting the possibility she was able to feel his erect genital organ. The phallus as an instrument of assault is here rendered by the speaker as a dangerously sharp pointed and bladed symbolic weapon capable of causing injury.
In the language of symbols this weaponized attribution can describe male fertility and sexual arousal whose outcome leads to death. Use of such symbolism further equates rape as an expression of human sacrifice. Insistence of perpetrators intent upon obtaining a trophy in sexual conquest is emphasized by the final part of the message. Sequential references to a block and an exam are consistent with Dr. Ford's testimony in which she told the committee "I struggled academically" both immediately after the alleged assault and during college.
Penis shoves. He's very close. 2) Law save us. You remember. Their own use. Abandoned you: After that May 2012 therapy session, I did my best to suppress memories of the assault because recounting the details caused me to relive the experience, and caused panic attacks and anxiety. Occasionally I would discuss the assault in individual therapy, but talking about it caused me to relive the trauma, so I tried not to think about it or discuss it.
Comment: Here the symbolic phallus from earlier is now revealed as the male reproductive organ asserted by "grinding" where the sense of proximity is highlighted as well. By closeness the speaker may be describing physical proximity ("on top of me") during a course of violent activities perceived as leading to sexual penetration and male orgasm during which Dr. Ford feared for her life. Opening remarks emphasized "I thought that Brett was accidentally going to kill me."
An appeal to justice suggests Dr. Ford's fundamental belief in the system of laws represented by the Supreme Court emphasized in her citation of civic duty as a key reason for her appearance before the committee. Affirming her recollection the messages goes on to cite feelings of exploitation and isolation, aligned with testimony stating "Both Brett and Mark were drunkenly laughing during the attack. They both seemed to be having a good time. Mark was urging Brett on, although at times he told Brett to stop. A couple of times I made eye contact with Mark and thought he might try to help me, but he did not."
They rape her: But over the years, I went through periods where I thought about Brett's attack. I confided in some close friends that I had an experience with sexual assault. Occasionally I stated that my assailant was a prominent lawyer or judge but I did not use his name. I do not recall each person I spoke to about Brett's assault, and some friends have reminded me of these conversations since the publication of The Washington Post story on September 16, 2018.
Comment: Use of infinitive here suggests an incident that perpetuates in the mind of the victim long after the alleged assault is over. While Dr. Ford does not claim to be have been physically penetrated, her feelings of diminished personhood and psychological injury can be understood as consequences of trespasses of personal boundaries resulting from a non-consensual physical attack. It seems that during the reported incident Dr. Ford perceived intent to rape and that she was confronted by perpetrators who had already raped her in their minds.
Citations of terror offered to the committee also suggested Dr. Ford's concerns that her appearance Thursday to discuss these allegations could potentially (and already had in the public sphere) serve to further abuse resulting in the social and psychological rape of her reputation and personhood. "I am here today not because I want to be. I am terrified. I am here because I believe it is my civic duty to tell you what happened to me while Brett Kavanaugh and I were in high school," Dr. Ford stated in opening remarks.
My nerve. I'm damaged. 2) He rapes. The death bar. 3) The man broke me. 4) Give us some soda: But until July 2018, I had never named Mr. Kavanaugh as my attacker outside of therapy. This all changed in early July 2018. I saw press reports stating that Brett Kavanaugh was on the "short list" of potential Supreme Court nominees. I thought it was my civic duty to relay the information I had about Mr. Kavanaugh's conduct so that those considering his potential nomination would know about this assault.
Comment: Dr. Ford's fragility was evident in her presentation and testimony where for example she claimed to have insisted against her husband's wishes on renovating the Ford family home to include two front doors as a means of alleviating apparent lifelong feelings of claustrophobia resulting from the reported assault.
Rape again appears in the infinitive ascribed to a male assailant within a near-fatal context consistent with witness testimony. The words "death" and "bar" are paired together in sequence. Psychological implications of a reported sexual assault's aftermath are here expressed in a phallic way that alludes to the male organ's geometry in terms of language common to both alcohol dispensaries and legal institutions of which the alleged assailant is a highly visible consumer, a member and nominee.
The notion of ruptured identity as consequence of a male perpetrator's crimes against a minor is here expressed using language also found in testimony ascribed to another victim who survived severe sexual and physical abuse proven in a court of law. In a recent report published by People an anonymous survivor of New Mexico's Aggressive Christianity Missions Training Corps was said to have explained how "Emotionally, she broke me as a child to the point where I still today struggle with my own self-confidence, my self-esteem, my sense of worth." Based upon such testimony founding leader Deborah Greene was sentenced to 72 years in prison by a Thirteenth Judicial District Court judge.
The final reference to soda is consistent with the large bottle of Coca Cola from which Dr. Ford poured soda into her glass during Thursday's appearance before the committee. This reference appears to bring the content of her messages into compliance with then-present-time events during which she accused Judge Brett Kavanaugh of sexual assault. Parallelism with present events gives rise to the suggestion that the otherwise unnamed 1982 male rapist described in her messages may have indeed been the accused nominee of today.
'She's a Spy': Alleged Russian Operative Maria Butina Disclosed Secret Agent Status at 2015 FreedomFest
'Sex with Daddy': Helsinki Secret Messages Confirm Trump a Putin Proxy