\chapter{Mathematics 1}

Lambda calculus is a set of rules for manipulating expressions.

\section*{Accepting mathematics}

If you love mathematics, skip this section.
This is written for people who think
they can understand physics without mathematics.

\paragraph{Mathematics is not only formulas}
Mathematics is about thinking.
Ideas are important.

\paragraph{Mathematics is an unavoidable tool for physics}

Mathematics has advanced physics.
Mathematics is here to stay.

If you don't want to learn at least some mathematics,
don't waste your time learning physics.
Without math, you will forever be physicist-wannabe.

For physicists, math is a tool.
Physicists use math.
Carpenters use saws.
Blacksmiths use hammers.

You don't need to love math.
Just don't hate it.
A carpenter neither loves nor hates his hammer;
he is simply not attached to it.
He is dispassionate, indifferent, neutral, uncaring.
He has no reason to hate it as long as it does what he expects it to do.

If you can understand English,
then you can understand the math used in this book.
It will take time, and you will have to think hard, but it is surmountable.
If you have tried but you still can't understand,
let us know that we have failed to write clearly.

\paragraph{Mathematics has become a necessity for physics}

We can really understand 20th century physics only if
we are somewhat fluent in the needed mathematics.
Without mathematics, our understanding is superficial and vague,
and all we can do is impress others who are even more clueless.

\paragraph{Mathematics needs mental effort}

Understanding triangles is like lifting a 1 kg weight.
Understanding vectors is like lifting a 2 kg weight.
Understanding manifolds is like lifting a 20 kg weight.
Lifting is uncomfortable,
but getting used to it makes it less comfortable,
until it becomes no big deal.
The only way to get used to something is by repeating it.
Don't try too hard to understand mathematics; just try to get used to it.
If you are hell-bent on building your body,
you won't minding lifting weights, no matter how heavy they are.
Now you just have to get yourself hell-bent on building your brain.
Physical training builds muscles.
Mental training builds brains.
Muscles can be sexy.
Brains can be sexy too.
A book is a mental gym.
If that motivates you.

Fluffy text ends here.
The rest of the book requires hard thinking.
Begin the journey!

\section{Reading mathematical notation}

\paragraph{Mathematical notation is concise English}

When you read an English sentence mixed with mathematical notation,
replace the notation with an English fragment that makes the sentence grammatical.

\index{definitions!overloading a symbol}%
\paragraph{Overloading}

To overload a symbol is to give it different meanings in different contexts.
For example, the same letter \(\Real\) means a set in \enquote{\(x \in \Real\)}
but means a type in \enquote{\(x : \Real\)}.
We write \(\Real\) for both the set of all real numbers and the type of a real number.

\TableRef{tab:overloading} shows overloading.
Note how the same notation \(x,y:\Real\) reads differently.

\Table{
    \Label{tab:overloading}
    \Caption{Reading mathematical notation}
    \Columns{p{0.4\textwidth}p{0.4\textwidth}}
    \Head{notation & English}
    \Body{
        Let \(x,y:\Real\).
        &
        Let \(x\) be a real number and \(y\) be a real number.
        \\
        \midrule
        Given \(x, y:\Real\), we can compute \(z:\Real\) between them.
        &
        Given a real number \(x\) and a real number \(y\), we can compute a real number \(z\) between them.
    }
}

\section{Lambda calculus: rules for forming and reducing expressions}

(Do we really need to tell the reader about lambda calculus?)

In mathematics, a calculus is a set of rules.
(\enquote{Calculus} means \enquote{pebble}.
How did its meaning change that much?)

Lambda calculus is a set of rules for forming and reducing expressions.
An expression is a description of a mathematical object.

\paragraph{Formation rules}

An expression is a variable, a function, or an application.

A variable is an irreducible expression.

If \(a\) is a variable and \(b\) is an expression,
then \(a \to b\) is a function.

If \(a\) and \(b\) are expressions,
then \((a)(b)\) is an application.
We omit the parentheses if such omission doesn't confuse us.
We do application from left to right: \(abcd\) means \(((ab)c)d\).

\paragraph{Substitution}

The notation \( E[x:=y] \) means the expression \(E\)
but with all free occurrences of \(x\) replaced by \(y\).

\paragraph{Reduction rules}

We can reduce an application \((a)(x \to y)\) to \(y[x:=a]\).

We can't reduce anything else.

\paragraph{Question}

Why do we bother to make up expressions, only to reduce them?
(Hint: Computers.)

\paragraph{Typed lambda calculus}
We can further constraint the rules using types in the next section.

\section{Describing existence using type theory}

Every value has a type.

Every expression has a type.

Every variable has a type.

The type of an expression reminds us about what it is.
For example, we write \(x : \Nat\) to mean that \(x\) is a natural number.

\index{definitions!type}%
We write \( x : T \) to mean ``the value \(x\) has the type \(T\)''.
We can read \(x:T\) as ``\(x\) is a \(T\)''.

We write \( x, y : T \) to shorten ``\( x : T \) and \( y : T \)''.

Variable...

Free variable and bound variable...

Let \(x:T\).
Then the answer to the question \enquote{What is \(x\)?} is \enquote{\(x\) is a \(T\)}.
We say that \(x\) inhabits \(T\).
We also say that \(x\) is an inhabitant of \(T\).

\paragraph{Evaluating an expression to a value}

Example: the expression \(1+2\) evaluates to the value \(3\)
according to high school arithmetic.

\paragraph{Tuples}

\index{definitions!tuple}%
A tuple has a fixed number of components.
An \(n\)-tuple has \(n\) components.
Some examples of tuples are \( (1,2,3) \) and \( (t_1,t_2) \).

\index{definitions!tuple component}%
The \(k\)th component of the tuple \(t\) is written \( t_k \) with \(k\) beginning from one.
For example, if \( t = (0,1) \), then \( t_1 = 0 \) and \( t_2 = 1 \).

\section{Converting between English and logic}

Every popular human language contains logic.

\( p \wedge q \)

\section{Describing collections using set theory}

\index{definitions!``practically''}%
In this book, \enquote{practically} means \enquote{wrong but useful}.
We can be practical when we aren't dealing with the foundation of math.

\index{definitions!set}%
We can think of a \enquote{set} as an unordered collection of things without duplicates.%
\footnote{Mathematicians don't define \enquote{set} as such because it would lead to Russell's paradox.}
\enquote{Unordered} means that \( \{a,b\} \) and \( \{b,a\} \) are the same set.
We say that \( a \) is an element of that set.
We notate that as \( a \in \{a,b\} \).

We can also define a set by a predicate or an English description.
An example is the set of all 26 Latin capital letters;
some of its elements are A, B, and C.

\paragraph{Working with sets}

TODO intersection \(A \cap B\),
union \(A \cup B\),
subtraction \(A - B\), ...

\paragraph{Using the set-builder notation to define sets using predicates}

\footnote{\url{https://en.wikipedia.org/wiki/Set-builder_notation}}

\( \{ x ~|~ P(x) \} \)

\( \{ x : P(x) \} \)

\paragraph{Relating set operations and logic operations}

Set intersection corresponds to logical conjunction.
\[
\{ x ~|~ a(x) \} \cap \{ x ~|~ b(x) \} = \{ x ~|~ a(x) \wedge b(x) \}
\]

Set union corresponds to logical disjunction.
\[
\{ x ~|~ a(x) \} \cup \{ x ~|~ b(x) \} = \{ x ~|~ a(x) \vee b(x) \}
\]

\section{Working with equations}

An equation \(a = b\) means that every \(a\) can be replaced with \(b\)
and every \(b\) can be replaced with \(a\).

Equations help us program computers.

An equation relates several parameters.

An equation can also be thought as describing a relationship, a constraint, something that may or may not be satisfied.

\paragraph{Substituting equal things}

\section{Working with numbers}

\paragraph{Natural numbers}

\paragraph{Real numbers}

% https://www.quora.com/Why-are-real-numbers-called-%E2%80%98real%E2%80%99
\index{definitions!real number}%
We write \( \Real \) to mean the set of real numbers\footnote{%
It's not that other numbers are fake. It's just a name that has stuck.%
}.
Example elements are \( 123 \), \( -12.34 \), \( 1/2 \),
\( \sqrt{2} \), \( 2^{0.1} \), \( \log 5 \), \( \sin 1 \), and \( \sum_{k=0}^\infty 3^{-k} \).
We assume that you know real number arithmetics.

\paragraph{Basic arithmetics}

We expect you to be able to add, subtract, multiply, and divide two one-digit numbers (from 0 to 9).
Primary education should have taught you this.

\footnote{\url{https://en.wikipedia.org/wiki/Arithmetic\#Arithmetic_operations}}%
\footnote{\url{https://en.wikipedia.org/wiki/Addition\#Addition_table}}%
\footnote{\url{https://en.wikipedia.org/wiki/Multiplication_table}}%

\paragraph{Using the summation notation}

If \(a < b\), then
\[
\sum_{k=a}^b f(k) = f(a) + f(a+1) + \ldots + f(b-1) + f(b)
\]

If \(K = \{ k_0, k_1, \ldots, k_n \}\) is a set, then
\[
\sum_{k \in K} f(k) = f(k_0) + f(k_1) + \ldots + f(k_n)
\]

\( \sum_k f(k) \) is a notation of the sum of all \(f(k)\) for each value that \(k\) can take. The meaning depends on context. For example, if \(v\) is a tuple with three components, then \(\sum_k v_k\) means \(v_1 + v_2 + v_3\).


\part{Lecture 02: Markov Decision Processes}
\title[RL Lecture 02]{Lecture 02: Markov Decision Processes}
\date{}
\frame{\titlepage}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Preface / Motivation %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Preface}
\begin{itemize}
	\onslide<2->\item Markov decision processes (MDP) are a \hl{mathematically idealized form of RL problems}.
	\onslide<2->\item They allow precise theoretical statements (e.g., on optimal solutions).
	\onslide<2->\item They deliver insights into suitable RL algorithms since many real-world problems can be abstracted as MDPs.
	\onslide<3->\item In the following \hl{we'll focus on}:
	\onslide<3->\begin{itemize}
		\item fully observable MDPs (i.e., $\bm{x}_k=\bm{y}_k$) and
		\item finite MDPs (i.e., finite number of states \& actions).
	\end{itemize}
\end{itemize}
\hfill
\onslide<1->\begin{table}
	\centering
		\begin{tabular}{|M{0.5cm}|M{1cm}||M{3cm}|M{3cm}|}
		\cline{3-4}
			\multicolumn{2}{c||}{\multirow{2}{*}{}}& \multicolumn{2}{c|}{All states observable?}\\
			\cline{3-4}
			\multicolumn{2}{c||}{} & Yes & No\\
			\hline\hline
			 \multirow{2}{*}{\rotatebox{90}{Actions?}} & No & Markov chain &Hidden Markov model\\
			\cline{2-4}
			& Yes & Markov decision process (MDP) & Partially observable MDP\\
			\hline
		\end{tabular}
	\caption{Different Markov models}
	\label{tab:DifferentMarkovModels}
\end{table}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Remark: Scalar and Vectorial State/Action Representation %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Scalar and Vectorial Representations in Finite MDPs}
\begin{itemize}
	\onslide<1->\item The position of a chess piece can be represented in two ways:
	\begin{itemize}
		\onslide<1->\item Vectorial: $\bm{x}=\begin{bmatrix}x_h & x_v\end{bmatrix}\T$, i.e., a two-element vector with horizontal and vertical information (e.g., $\bm{x}=\begin{bmatrix}c & 1 \end{bmatrix}\T$) or
		\onslide<2->\item \hl{Scalar: simple enumeration of all available positions (e.g., $x=3$).}
	\end{itemize}
	\onslide<3->\item Both ways represent the same amount of information and can be easily converted into each other.
	\onslide<3->\item For sake of readability \hl{we will stick to the scalar representation of states and actions in finite MDPs} whenever possible.
\end{itemize}
\onslide<1->\begin{figure}
	\includegraphics[width=4cm]{fig/lec02/Chess_Board.pdf}
\end{figure}
}

\frame{\frametitle{Table of Contents}\tableofcontents}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Finite Markov Chains}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Recap Markov Property %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Recap on Markov Property}

\begin{block}{Information state}
A state $X_k$ is called an information or Markov state if and only if
\begin{equation}
	\Pb{X_{k+1}|X_k}=\Pb{X_{k+1}|X_0, X_1,\ldots, X_k}\, .
\end{equation}
\end{block}
\pause
\begin{itemize}
	\item History is fully condensed in the sample $X_k$, i.e., $X_{k+1}$ is only depending on $X_k$.
	\item A given system can be fully described by $X_k$.
	\item Further past observations $X_{k-1},X_{k-2},\ldots$ are irrelevant.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% State Transistion Matrix %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{State Transition Matrix}

\begin{defi}{State transition matrix}{STM}
Given a Markov state $X_k=x$ and its successor $X_{k+1}=x'$ , the \hl{state transition probability} $\forall \left\{x,x'\right\}\in \mathcal{X}$ is defined by the matrix
\begin{equation}
	\bm{\mathcal{P}}_{xx'}=\Pb{X_{k+1}=x'|X_{k}=x} .
\end{equation}
 \end{defi}
\pause
Here,  $\bm{\mathcal{P}}_{xx'}\in\mathbb{R}^{n\times n}$ has the form
\begin{equation*}
	\bm{\mathcal{P}}_{xx'} = \begin{bmatrix} p_{11} & p_{12} & \cdots & p_{1n}\\ p_{21} &  &  & \vdots\\ \vdots & & & \vdots\\ p_{n1} & \cdots & \cdots & p_{nn}\end{bmatrix}
\end{equation*}
with $p_{ij}\in\left\{\mathbb{R}|0 \leq p_{ij} \leq 1 \right\}$ being the specific probability to go from state $x=X_i$ to state $x'=X_j$. Obviously, $\sum_{j} p_{ij}=1 \, \forall i$ must hold.
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Markov Chain %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Markov Chain}

\onslide<1->\begin{defi}{Finite Markov chain}{Markov_chain}
A \hl{finite Markov chain} is a tuple $\left\langle\mathcal{X}, \bm{\mathcal{P}} \right\rangle$ with
\begin{itemize}
	\item $\mathcal{X}$ being a finite set of discrete-time states $X_k\in\mathcal{X}$,
	\item $\bm{\mathcal{P}}=\bm{\mathcal{P}}_{xx'}=\Pb{X_{k+1}=x'|X_{k}=x}$ is the state transition probability.
\end{itemize}
\end{defi}
\begin{itemize}
	\onslide<2->\item Specific stochastic process model
	\onslide<2->\item Sequence of random variables $X_k, X_{k+1},\ldots$
	\onslide<2->\item 'Memoryless'
	\onslide<3->\item In continuous-time framework: Markov process\only<3->{\footnote[1]{However, this results in a literature nomenclature inconsistency with Markov decision/reward 'processes'.}}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Chapman–Kolmogorov Equation %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Chapman-Kolmogorov Equation}
Define $\bm{p}_k\in\mathbb{R}^n$ as a \hl{probability row vector} where $p_{i,k}$ gives the probability of being in state $X_k \in\mathcal{X}$ at time-step $k$.
\begin{defi}{Chapman-Kolmogorov equation for finite Markov chains}{Chapman_Kolmogorov_Equation}
The probability of being in state $X_{k+m}$ at time-step $(k+m)$ starting from state $x_{k}$ is given by:
\begin{equation}
	\bm{p}_{k+m} = \bm{p}_{k}\bm{\mathcal{P}}_{xx'}^{m} .
\end{equation}
\end{defi}
\pause
Hence, for $m\rightarrow\infty$ ('\textit{steady state}') the following equation must hold:
\begin{equation}
	\bm{p} = \bm{p}\bm{\mathcal{P}}_{xx'} .
	\label{eq:Chapman_steady}
\end{equation}
Solving \eqref{eq:Chapman_steady} for $\bm{p}$ given the transition probability $\bm{\mathcal{P}}_{xx'}$ gives insights which states of the Markov chain are visited in the long-run.
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Forest Markov Chain Example (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Example of a Markov Chain (1)}
\begin{minipage}[t]{0.53\linewidth}
		\vspace{0pt}
		\begin{figure}		\includegraphics[width=6cm]{fig/lec02/Forest_Markov_Chain.pdf}
		\caption{Forest tree Markov chain}
		\label{fig:Forest_Markov_Chain}
	\end{figure}
\end{minipage}
\begin{minipage}[t]{0.45\linewidth}
\vspace{-0.7cm}
\begin{align*}
  x &= x\in\left\{1,2,3,4\right\}\\
				 &=\left\{\mbox{small},\mbox{medium},\mbox{large},\mbox{gone}\right\}\\[1em]
	\bm{\mathcal{P}} &= \begin{bmatrix}0 & 1-\alpha & 0 & \alpha  \\ 0 & 0 &1-\alpha & \alpha \\ 0 & 0 &1-\alpha & \alpha \\ 0 & 0 & 0 & 1\end{bmatrix}
\end{align*}
\end{minipage}
\begin{itemize}
	\item At $x=1$ a small tree is planted ('starting point').
	\item A tree grows with $(1-\alpha)$ probability.
	\item If it reaches $x=3$ (large) its growth is limited.
	\item With $\alpha$ probability a natural hazard destroys the tree.
	\item The state $x=4$ is terminal ('infinite loop').
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Forest Markov Chain Example (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Example of a Markov Chain (2)}
\begin{figure}
	\includegraphics[width=6cm]{fig/lec02/Forest_Markov_Chain.pdf}
\end{figure}
Possible \hl{samples} for the given Markov chain example starting from $x=1$ (small tree):
\begin{itemize}
	\item Small $\rightarrow$ gone
	\item Small $\rightarrow$ medium $\rightarrow$ gone
	\item Small $\rightarrow$ medium $\rightarrow$ large $\rightarrow$ gone
	\item Small $\rightarrow$ medium $\rightarrow$ large $\rightarrow$ large $\rightarrow\ldots$
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Forest Markov Chain Example (3)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Example of a Markov Chain (3)}
For $k=0$ the initial state probability is given $\bm{p}_{k=0}=\begin{bmatrix} 1 & 0 & 0 & 0\end{bmatrix}$, i.e., we are in state $x=1$. What is the state probability after $k=1,2,\ldots$ steps with $\alpha = 0.2$?
\begin{alignat*}{2}
	\onslide<2->{\bm{p}_{1}&=\begin{bmatrix}0 & 0.8 & 0 & 0.2\end{bmatrix}&&=\underbrace{\begin{bmatrix} 1 & 0 & 0 & 0\end{bmatrix}}_{\bm{p}_{0}}\underbrace{\begin{bmatrix}0 & 0.8 & 0 & 0.2  \\ 0 & 0 &0.8 & 0.2 \\ 0 & 0 &0.8 & 0.2 \\ 0 & 0 & 0 & 1\end{bmatrix}}_{\bm{\mathcal{P}}}},\\
	\onslide<3->{\bm{p}_{2}&=\begin{bmatrix}0 & 0 & 0.64 & 0.36\end{bmatrix}&&=\underbrace{\begin{bmatrix} 0 & 0.8 & 0 & 0.2\end{bmatrix}}_{\bm{p}_{1}}\underbrace{\begin{bmatrix}0 & 0.8 & 0 & 0.2  \\ 0 & 0 &0.8 & 0.2 \\ 0 & 0 &0.8 & 0.2 \\ 0 & 0 & 0 & 1\end{bmatrix}}_{\bm{\mathcal{P}}}},\\
	\onslide<4->{\bm{p}_{3}&=\begin{bmatrix}0 & 0 & 0.51 & 0.49\end{bmatrix}, &&\quad \bm{p}_{10}=\begin{bmatrix}0 & 0 & 0.11 & 0.89\end{bmatrix},\\
	\bm{p}_{\infty}&=\begin{bmatrix}0 & 0 & 0 & 1\end{bmatrix}.}
\end{alignat*}

}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Finite Markov Reward Processes}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Markov Reward Process %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Markov Reward Process}

\begin{defi}{Finite Markov reward process}{Markov_reward_process}
A \hl{finite Markov reward process (MRP)} is a tuple $\left\langle\mathcal{X}, \bm{\mathcal{P}}, \hl{\mathcal{R}, \gamma} \right\rangle$ with
\begin{itemize}
	\item $\mathcal{X}$ being a finite set of discrete-time states $X_k\in\mathcal{X}$,
	\item $\bm{\mathcal{P}}=\bm{\mathcal{P}}_{xx'}=\Pb{X_{k+1}=x'|X_{k}=x}$ is the state transition probability,
	\item \hl{$\mathcal{R}$ is a reward function, $\mathcal{R}=\mathcal{R}_x=\E{R_{k+1}|X_k=x_k}$ and}
	\item \hl{$\gamma$ is a discount factor, $\gamma\in\left\{\mathbb{R}|0 \leq \gamma \leq 1\right\}$}.
\end{itemize}
\end{defi}
\pause
\begin{itemize}
	\item Markov chain extended with rewards \pause
	\item Still an autonomous stochastic process without specific inputs \pause
	\item Rewards $R_k$ only dependent on state $X_k$
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Forest Markov Reward Process %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Example of a Markov Reward Process}
\begin{figure}
	\includegraphics[width=10cm]{fig/lec02/Forest_Markov_Reward_Process.pdf}
	\caption{Forest Markov reward process}
	\label{fig:Forest_Markov_Reward_Process}
\end{figure}

\begin{itemize}
	\item Growing larger trees is rewarded, since it will be
	\begin{itemize}
		\item appreciated by hikers and
		\item useful for wood production.
	\end{itemize}
	\item Loosing a tree due to a hazard is unrewarded.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Recap Return %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Recap on Return}

\begin{block}{Return}
The return $G_k$ is the total discounted reward starting from step $k$ onwards. For \hl{episodic tasks} it becomes the finite series
\begin{equation}
	G_k = R_{k+1} + \gamma R_{k+2} + \gamma^2 R_{k+3} + \cdots = \sum_{i=0}^N \gamma^i R_{k+i+1}
\end{equation}
terminating at step $N$ while it is an infinite series for \hl{continuing tasks}
\begin{equation}
	G_k = R_{k+1} + \gamma R_{k+2} + \gamma^2 R_{k+3} + \cdots = \sum_{i=0}^\infty \gamma^i R_{k+i+1}\, .
\end{equation}
\end{block}
\begin{itemize}
	\item The discount $\gamma$ represents the value of future rewards.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Remark on notation: Episodic and Continuing Tasks %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Remark on Notation: Episodic and Continuing Tasks}
\begin{itemize}
	\item In episodic tasks there is not only one (long/infinite) sequence of time steps.
	\item Instead there is a sequence of time steps of an arbitrary episode $j$.
	\begin{itemize}
		\item Formally: $X_{k,j},U_{k,j}, R_{k,j},\ldots$
	\end{itemize}
	\onslide<2->{
	\item Nevertheless, to increase the readability the episode index is normally dropped in this course:
	\begin{itemize}
		\item Simplified: $X_{k}\leftarrow X_{k,j},U_{k}\leftarrow U_{k,j}, R_{k}\leftarrow R_{k,j},\ldots$
	\end{itemize}
	\item Only in certain cases the episode indexing will be re-activated.
\end{itemize}}
\begin{figure}
	\includegraphics[width=5.5cm]{fig/lec02/chess_elderly_men.jpg}
	\caption{RL in practice (source: Vlad Sargu on \href{https://unsplash.com/photos/ItphH2lGzuI}{Unsplash})}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Value in MRP %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Value Function in MRP}
\vspace{-0.25cm}
\begin{defi}{Value function in MRP}{value_function_MRP}
The \hl{state-value function} $v(x_k)$ of an MRP is the expected return starting from state $x_k$
\begin{equation}
	v(x_k) = \E{G_k|X_k=x_k} .
\end{equation}
\end{defi}
\begin{itemize}
	\item Represents the long-term value of being in state $X_k$.
\end{itemize}
\vspace{0.25cm}
\begin{figure}
	\includegraphics[width=6cm]{fig/lec02/Value_Golf.pdf}
	\caption{Isolines indicate state value of different golf ball locations (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Value_Golf}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% State-Value Samples Forest Markov Reward Process %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{State-Value Samples of Forest MRP}
\begin{figure}
	\includegraphics[width=8cm]{fig/lec02/Forest_Markov_Reward_Process.pdf}
\end{figure}
Exemplary \hl{samples} $\hat{v}$ with $\gamma=0.5$ starting in $x=1$:
\begin{alignat*}{2}
	&x=1 \rightarrow 4, &&\hat{v}= 1,\\
	&x=1 \rightarrow 2 \rightarrow 4, &&\hat{v}= 1 + 0.5\cdot 2=2.0,\\
	&x=1 \rightarrow 2 \rightarrow 3  \rightarrow 4, \quad&&\hat{v}=1+ 0.5\cdot 2+ 0.25\cdot 3=3.75,\\
	&x=1 \rightarrow 2 \rightarrow 3 \rightarrow 3 \rightarrow 4, \quad&&\hat{v}=1+ 0.5\cdot 2+0.25 \cdot 3+ 0.125\cdot3=4.13.
\end{alignat*}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Equation for MRPs (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Equation for MRPs (1)}
Problem: How to calculate all state values in closed form?\\
	\onslide<2->Solution: Bellman equation.
\begin{equation}
\onslide<3->{
\label{eq:Bellman_MRP}
\begin{split}
		v(x_k) &= \E{G_k|X_k=x_k}}\\
							&	\onslide<4->{= \E{R_{k+1} + \gamma R_{k+2} + \gamma^2 R_{k+3}+\ldots|X_k=x_k}}\\
							&\onslide<5->{= \E{R_{k+1} + \gamma \left(R_{k+2} + \gamma R_{k+3}+\ldots\right)|X_k=x_k}}\\
							&\onslide<6->{= \E{R_{k+1} + \gamma G_{k+1}|X_k=x_k}}\\
							&\onslide<7->{= \E{R_{k+1} + \gamma 	v(X_{k+1})|X_k=x_k}}
\end{split}
\end{equation}
\vspace*{\fill}
\onslide<8->{
\begin{figure}
		 \hspace*{-1.5cm}
		\includegraphics[width=6.1cm]{fig/lec02/Back_Up_MRP.pdf}
		\caption{Backup diagram for $v(x_k)$}
		\label{fig:Back_Up_MRP}
\end{figure}
}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Equation for MRPs (2) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Equation for MRPs (2)}
Assuming non-random rewards for every state $X=x\in\mathcal{X}$
\begin{equation}
	\bm{r}_{\mathcal{X}}= \begin{bmatrix} \mathcal{R}(x_1) & \cdots & \mathcal{R}(x_n)\end{bmatrix}\T = \begin{bmatrix} \mathcal{R}_1 & \cdots & \mathcal{R}_n\end{bmatrix}\T
\end{equation}
for a finite number of $n$ states with state-values
\begin{equation}
	\bm{v}_{\mathcal{X}}= \begin{bmatrix} v(x_1) & \cdots & v(x_n)\end{bmatrix}\T = \begin{bmatrix} v_1 & \cdots & v_n\end{bmatrix}\T
\end{equation}
one can derive a linear equation system based on \figref{fig:Back_Up_MRP}:\pause
\begin{equation}
\label{eq:Bellman_MRP_linear}
\begin{split}
	\bm{v}_{\mathcal{X}}&=\bm{r}_{\mathcal{X}}+\gamma\bm{\mathcal{P}}_{xx'}\bm{v}_{\mathcal{X}},\\
	\begin{bmatrix} v_1 \\ \vdots \\ v_n \end{bmatrix} &= \begin{bmatrix} \mathcal{R}_1 \\ \vdots \\ \mathcal{R}_n \end{bmatrix} + \gamma\begin{bmatrix} p_{11} & \cdots & p_{1n}\\ \vdots &  & \vdots\\ p_{n1} & \cdots & p_{nn}\end{bmatrix}\begin{bmatrix} v_1 \\ \vdots \\ v_n \end{bmatrix} .
\end{split}
\end{equation}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Sovling the MRP Bellman Equation %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Solving the MRP Bellman Equation}
Above, \eqref{eq:Bellman_MRP_linear} is a normal equation in $\bm{v}_{\mathcal{X}}$:
\begin{equation}
\label{eq:MRP_normal_equation}
\begin{split}
\bm{v}_{\mathcal{X}}&=\bm{r}_{\mathcal{X}}+\gamma\bm{\mathcal{P}}_{xx'}\bm{v}_{\mathcal{X}},\\
\Leftrightarrow \underbrace{\left(\bm{I}-\gamma\bm{\mathcal{P}}_{xx'}\right)}_{\bm{A}}\underbrace{\bm{v}_{\mathcal{X}}}_{x}&=\underbrace{\bm{r}_{\mathcal{X}}}_{\bm{b}}.
\end{split}
\end{equation}
Possible solutions:
\begin{itemize}
	\item Direct inversion (Gaussian elimination, $\mathcal{O}(n^3)$)\pause
	\item Matrix decomposition (QR, Cholesky, etc. , $\mathcal{O}(n^3)$)\pause
	\item Iterative solutions (e.g., Krylov-subspaces, often better than $\mathcal{O}(n^3)$)\pause
\end{itemize}
\vspace{0.5cm}
In RL identifying and solving \eqref{eq:MRP_normal_equation} is a key task, which is often realized only approximately for high-order state spaces.
}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Forest Markov Reward Process with State-Values%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Example of a Markov Reward Process with State Values}
\begin{figure}
	\includegraphics[width=10cm]{fig/lec02/Forest_Markov_Reward_Process_State_Value.pdf}
	\caption{Forest Markov reward process including state values}
	\label{fig:Forest_Markov_Chain_State_Value}
\end{figure}
\begin{itemize}
	\item Discount factor $\gamma=0.8$
	\item Disaster probability $\alpha=0.2$
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Finite Markov Decision Processes}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Markov Decision Process %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Markov Decision Process}

\begin{defi}{Finite Markov decision process}{Markov_decision_process}
A \hl{finite Markov decision process (MDP)} is a tuple $\left\langle\mathcal{X}, \hl{\mathcal{U}}, \bm{\mathcal{P}}, \mathcal{R}, \gamma \right\rangle$ with
\begin{itemize}
	\item $\mathcal{X}$ being a finite set of discrete-time states $X_k\in\mathcal{X}$,
	\item \hl{$\mathcal{U}$ as a finite set of discrete-time actions $U_k\in\mathcal{U}$,}
	\item $\bm{\mathcal{P}}=\bm{\mathcal{P}}_{xx'}^{\hl{u}}$ is the state transition probability $\bm{\mathcal{P}}=\Pb{X_{k+1}=x'|X_{k}=x_k, \hl{U_{k}=u_k}}$,
	\item $\mathcal{R}$ is a reward function, $\mathcal{R}=\mathcal{R}_x^{\hl{u}}=\E{R_{k+1}|X_k=x_k, \hl{U_{k}=u_k}}$ and
	\item $\gamma$ is a discount factor, $\gamma\in\left\{\mathbb{R}|0 \leq \gamma \leq 1\right\}$.
\end{itemize}
\end{defi}
\pause
\begin{itemize}
	\item Markov reward process is extended with actions / decisions.
	\item Now, rewards also depend on action $U_k$.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Forest Markov Decision Process (1) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Example of a Markov Decision Process (1)}
\begin{figure}
	\includegraphics[width=11cm]{fig/lec02/Forest_Markov_Decision_Process.pdf}
	\caption{Forest Markov decision process}
	\label{fig:Forest_Markov_Decision_Process}
\end{figure}

\begin{itemize}
	\item Two actions possible in each state:
	\begin{itemize}
		\item Wait $u=w$: let the tree grow.
		\item Cut $u=c$: gather the wood.
	\end{itemize}
	\item With increasing tree size the wood reward increases as well.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Forest Markov Decision Process (2) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Example of a Markov Decision Process (2)}
For the previous example the state transition probability matrix and reward function are given as:
\begin{alignat*}{2}
	\bm{\mathcal{P}}_{xx'}^{u=c} &= \begin{bmatrix}0 & 0 & 0 & 1  \\ 0 & 0 &0 & 1 \\ 0 & 0 &0 & 1 \\ 0 & 0 & 0 & 1\end{bmatrix}, \quad \bm{\mathcal{P}}_{xx'}^{u=w} &&= \begin{bmatrix}0 & 1-\alpha & 0 & \alpha  \\ 0 & 0 &1-\alpha & \alpha \\ 0 & 0 &1-\alpha & \alpha \\ 0 & 0 & 0 & 1\end{bmatrix},\\
	\bm{r}_{\mathcal{X}}^{u=c} &= \begin{bmatrix}1 & 2 & 3 & 0 \end{bmatrix}\T, \quad \bm{r}_{\mathcal{X}}^{u=w} &&= \begin{bmatrix}0 & 0 & 1 & 0 \end{bmatrix}\T .
\end{alignat*}
\begin{itemize}
	\item The term $\bm{r}_{\mathcal{X}}^u$ is the abbreviated form for receiving the output of $\mathcal{R}$ for the entire state space $\mathcal{X}$ given the action $u$.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Policy (1) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Policy (1)}
\vspace{-0.2cm}
\onslide<1->{
\begin{defi}{Policy in MDP (1)}{Policy_MDP}
In an MDP environment, a \hl{policy} is a distribution over actions given states:
\begin{equation}
\label{eq:Policy_MDP}
	\pi(u_k|x_k)=\Pb{U_k=u_k  | X_k=x_k}\, .
\end{equation}
\end{defi}
}
\vspace{-0.05cm}
\begin{itemize}
	\onslide<2->{\item In MDP, policies depend only on the current state.}
	\onslide<3->{\item A policy fully defines the agent's behavior.}
	\onslide<4->{\item Might relax to a deterministic policy for certain applications.}
\end{itemize}
\onslide<1->{
\begin{figure}
	\includegraphics[width=5.75cm]{fig/lec02/monopoly.jpg}
	\caption{What is you best Monopoly policy? (source: Ylanite Koppens on \href{https://www.pexels.com/de-de/foto/begrifflich-brettspiel-drinnen-farben-776654/}{Pexels})}
\end{figure}
}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Policy (2) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Policy (2)}
Given an MDP $\left\langle\mathcal{X}, \mathcal{U}, \bm{\mathcal{P}}, \mathcal{R}, \gamma \right\rangle$ and a policy $\bm{\pi}$:
\begin{itemize}
	\item The state sequence $X_k, X_{k+1},\ldots$ is a Markov chain $\left\langle\mathcal{X}, \bm{\mathcal{P}}^{\pi} \right\rangle$ since the state transition probability is only depending on the state:
	\begin{equation}
		\bm{\mathcal{P}}_{xx'}^{\pi}=\sum_{u_k\in\mathcal{U}}\bm{\pi}(u_k|x_k)\bm{\mathcal{P}}_{xx'}^{u}\, .
	\end{equation}\pause
	\item Consequently, the sequence $X_k, R_{k+1}, X_{k+1},R_{k+2},\ldots$ of states and rewards is a Markov reward process $\left\langle\mathcal{X}, \bm{\mathcal{P}}^{\pi}, \mathcal{R}^{\pi}, \gamma \right\rangle$:
	\begin{equation}
		\mathcal{R}_{xx'}^{\pi}=\sum_{u_k\in\mathcal{U}}\bm{\pi}(u_k|x_k)\mathcal{R}_{x}^{u}\, .
	\end{equation}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Recap on Value functions %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Recap on MDP Value Functions}

\begin{defi}{State-value function}{state_value_MDP}
The state-value function of an MDP is the expected return starting in $x_k$ following policy $\pi$:
\begin{equation*}
		v_\pi(x_k) = \El{G_k|X_k=x_k}{\pi}=\El{\left. \sum_{i=0}^\infty \gamma^i R_{k+i+1}\right|X_k}{\pi}\,.
	\end{equation*}
\end{defi}
\pause
\begin{defi}{Action-value function}{action_value_MDP}
The action-value function of an MDP is the expected return starting in $x_k$ taking action $u_k$ following policy $\pi$:
\begin{equation*}
		q_\pi(x_k,u_k) = \El{G_k|X_k=x_k, U_k=u_k}{\pi}=\El{\left.\sum_{i=0}^\infty \gamma^i R_{k+i+1}\right|X_k, U_k}{\pi}\, .
	\end{equation*}
\end{defi}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Expectation Equation (1) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Expectation Equation (1)}
Analog to \eqref{eq:Bellman_MRP}, the state value of an MDP can be decomposed into a Bellman notation:
\begin{equation}
	v_\pi(x_k)	= \El{R_{k+1} + \gamma 	v_{\pi}(X_{k+1})|X_k=x_k}{\pi}\, .
\end{equation}
\pause
In finite MDPs the state value can be directly linked to the action value (cf.~\figref{fig:Back_Up_Value_MDP}):
\begin{equation}
\label{eq:v_MDP_finite}
	v_\pi(x_k)	= \sum_{u_k\in\mathcal{U}}\pi(u_k|x_k)q_\pi(x_k,u_k) \, .
\end{equation}
\pause
\vspace{0.5cm}
\begin{figure}
		 \hspace*{-1.5cm}
		\includegraphics[width=6.5cm]{fig/lec02/Back_Up_Value_MDP.pdf}
		\caption{Backup diagram for $v_{\pi}(x_k)$}
		\label{fig:Back_Up_Value_MDP}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Expectation Equation (2) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Expectation Equation (2)}
Likewise, the action value of an MDP can be decomposed into a Bellman notation:
\begin{equation}
	q_\pi(x_k,u_k)	= \El{R_{k+1} + \gamma 	q_{\pi}(X_{k+1},U_{k+1})|X_{k}=x_{k},U_{k}=u_{k}}{\pi}\, .
\end{equation}
\pause
In finite MDPs the action value can be directly linked to the state value (cf.~\figref{fig:Back_Up_Action_Value_MDP}):
\begin{equation}
\label{eq:q_MDP_finite}
	q_\pi(x_k,u_k)	= \mathcal{R}^u_x + \gamma \sum_{x_{k+1}\in\mathcal{X}}p_{xx'}^u v_\pi(x_{k+1}) \, .
\end{equation}
\pause
\vspace{0.5cm}
\begin{figure}
		 \hspace*{-1.5cm}
		\includegraphics[width=6.5cm]{fig/lec02/Back_Up_Action_Value_MDP.pdf}
		\caption{Backup diagram for $q_{\pi}(x_k,u_k)$}
		\label{fig:Back_Up_Action_Value_MDP}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Expectation Equation (3) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Expectation Equation (3)}
Inserting \eqref{eq:q_MDP_finite} into \eqref{eq:v_MDP_finite} directly results in:
\begin{equation}
	v_\pi(x_k)	= \sum_{u_k\in\mathcal{U}}\pi(u_k|x_k)\left(\mathcal{R}^u_x + \gamma\sum_{x_{k+1}\in\mathcal{X}}p_{xx'}^u v_\pi(x_{k+1})\right) \, .
\end{equation}
\pause
Conversely, the action value becomes:
\begin{equation}
	q_\pi(x_k,u_k)	= \mathcal{R}^u_x + \gamma\sum_{x_{k+1}\in\mathcal{X}}p_{xx'}^u \left(\sum_{u_{k+1}\in\mathcal{U}}\pi(u_{k+1}|x_{k+1})q_\pi(x_{k+1},u_{k+1})\right) \, .
\end{equation}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Expectation Equation in Matrix Form %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Expectation Equation in Matrix Form}
Given a policy $\pi$ and following the same assumptions as for \eqref{eq:Bellman_MRP_linear}, the Bellman expectation equation can be expressed in Matrix form:
\begin{equation}
\label{eq:Bellman_MDP_linear}
\begin{split}
	\bm{v}_{\mathcal{X}}^{\pi}&=\bm{r}_{\mathcal{X}}^{\pi}+\gamma\bm{\mathcal{P}}_{xx'}^{\pi}\bm{v}_{\mathcal{X}}^{\pi},\\
	\begin{bmatrix} v_{1}^{\pi} \\ \vdots \\ v_{n}^{\pi} \end{bmatrix} &= \begin{bmatrix} \mathcal{R}_{1}^{\pi} \\ \vdots \\ \mathcal{R}_{n}^{\pi} \end{bmatrix} + \gamma\begin{bmatrix} p_{11}^{\pi} & \cdots & p_{1n}^{\pi}\\ \vdots &  & \vdots\\ p_{n1}^{\pi} & \cdots & p_{nn}^{\pi}\end{bmatrix}\begin{bmatrix} v_{1}^{\pi} \\ \vdots \\ v_{n}^{\pi} \end{bmatrix}.
\end{split}
\end{equation}
Here, $\bm{r}_{\mathcal{X}}^{\pi}$ and $\bm{\mathcal{P}}_{xx'}^{\pi}$ are the rewards and state transition probability following policy $\pi$. \pause Hence, the state value can be calculated by solving \eqref{eq:Bellman_MDP_linear} for $\bm{v}_{\mathcal{X}}^{\pi}$, e.g., by direct matrix inversion:
\begin{equation}
	\bm{v}_{\mathcal{X}}^{\pi}=\left(\bm{I}-\gamma\bm{\mathcal{P}}_{xx'}^{\pi}\right)^{-1}\bm{r}_{\mathcal{X}}^{\pi}.
\end{equation}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Expectation Equation Applied to Forest Tree Example (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Expectation Equation \& Forest Tree Example (1)}
Let's assume following very simple policy ('\textit{fifty-fifty}')
\begin{equation*}
\pi(u = \mbox{cut}|x) = 0.5, \quad \pi(u = \mbox{wait}|x) = 0.5 \quad\forall x\in\mathcal{X} \, .
\end{equation*}
\pause
Applied to the given environment behavior
\begin{alignat*}{2}
	\bm{\mathcal{P}}_{xx'}^{u=c} &= \begin{bmatrix}0 & 0 & 0 & 1  \\ 0 & 0 &0 & 1 \\ 0 & 0 &0 & 1 \\ 0 & 0 & 0 & 1\end{bmatrix}, \quad \bm{\mathcal{P}}_{xx'}^{u=w} &&= \begin{bmatrix}0 & 1-\alpha & 0 & \alpha  \\ 0 & 0 &1-\alpha & \alpha \\ 0 & 0 &1-\alpha & \alpha \\ 0 & 0 & 0 & 1\end{bmatrix},\\
	\bm{r}_{\mathcal{X}}^{u=c} &= \begin{bmatrix}1 & 2 & 3 & 0 \end{bmatrix}\T, \quad \bm{r}_{\mathcal{X}}^{u=w} &&= \begin{bmatrix}0 & 0 & 1 & 0 \end{bmatrix}\T ,
\end{alignat*}
\pause
one receives:
\begin{equation*}
	\bm{\mathcal{P}}_{xx'}^{\pi} = \begin{bmatrix}0 & \frac{1-\alpha}{2} & 0 & \frac{1+\alpha}{2}  \\ 0 & 0 &\frac{1-\alpha}{2} & \frac{1+\alpha}{2} \\ 0 & 0 &\frac{1-\alpha}{2} & \frac{1+\alpha}{2} \\ 0 & 0 & 0 & 1\end{bmatrix}, \quad\bm{r}_{\mathcal{X}}^{\pi} = \begin{bmatrix}0.5 \\ 1 \\ 2 \\ 0 \end{bmatrix} .
\end{equation*}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Expectation Equation Applied to Forest Tree Example (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Expectation Equation \& Forest Tree Example (2)}
\begin{figure}
	%\hspace*{-1.5cm}
	\includegraphics[width=10cm]{fig/lec02/Forest_Markov_Decision_Process_State_Value.pdf}
	\caption{Forest MDP with fifty-fifty-policy including state values}
	\label{fig:Forest_Markov_Decision_Process_State_Value}
\end{figure}
\begin{itemize}
	\item Discount factor $\gamma=0.8$
	\item Disaster probability $\alpha=0.2$
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Expectation Equation Applied to Forest Tree Example (3)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Expectation Equation \& Forest Tree Example (3)}
Using the Bellman expectation eq. \eqref{eq:q_MDP_finite} the action values can be directly calculated:
\vspace{0.5cm}
\begin{figure}
	\hspace*{0.5cm}
	\includegraphics[width=10cm]{fig/lec02/Forest_Markov_Decision_Process_Action_Value.pdf}
	\caption{Forest MDP with fifty-fifty-policy plus action values}
	\label{fig:Forest_Markov_Decision_Process_Action_Value}
\end{figure}
Is the '\textit{fifty-fifty-policy}' maximizing our long-term reward? How can you evaluate your answer?
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Optimal Policies and Value Functions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optimal Value Functions in MDP %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Optimal Value Functions in an MDP}
\vspace{-0.35cm}
\begin{defi}{Optimal state-value function}{optimal_state_value_MDP}
The optimal state-value function of an MDP is the maximum state-value function over all polices:
\begin{equation}
		v^*(x) = \max_{\pi} v_{\pi}(x)\,.
	\end{equation}
\end{defi}
\vspace{-0.2cm}
\pause
\begin{defi}{Optimal action-value function}{optimal_action_value_MDP}
The optimal action-value function of an MDP is the maximum action-value function over all polices:
\begin{equation}
		q^*(x,u) = \max_{\pi} q_\pi(x,u)\, .
	\end{equation}
\end{defi}
\vspace{-0.1cm}
\pause
\begin{itemize}
	\item The optimal value function denotes the best possible agent's performance for a given MDP / environment. \pause
	\item A (finite) MDP can be easily solved in an optimal way if $q^*(x,u)$ is known.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optimal Policy in MDP %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Optimal Policy in an MDP}
Define a partial ordering over polices
\begin{equation}
	\pi \geq \pi' \quad \mbox{if} \quad v_{\pi}(x) \geq v_{\pi'}(x) \quad \forall x\in\mathcal{X} \,.
\end{equation}
\pause
\begin{theo}{Optimal policies in MDPs}{Optimal_policies_MDP}
For any MDP
\begin{itemize}
	\item there exists an optimal policy $\pi^*\geq\pi$ that is better or equal to all other policies,
	\item all optimal policies achieve the same optimal state-value function $v^*(x)=v_{\pi^*}(x)$,
	\item all optimal policies achieve the same optimal action-value function $q^*(x, u)=q_{\pi^*}(x, u)$.
\end{itemize}
\end{theo}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Optimality Equation (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Optimality Equation (1)}
\onslide<1->{
\begin{theo}{Bellman's principle of optimality}{bellman_principle_optimality}
``\textit{An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision.}''(R.E. Bellman, Dynamic Programming, 1957)
\end{theo}
}
\begin{itemize}
	\onslide<2->{\item Any policy (i.e., also the optimal one) must satisfy the self-consistency condition given by the Bellman expectation equation.}
	\onslide<3->{\item An optimal policy must equal the expected return for the best action of a given state:}
\end{itemize}
\onslide<4->{
\begin{equation}
\begin{split}
		v^*(x_k) &= \max_{u} q_{\pi^*}(x_k, u)\, ,}\\
									&\onslide<5->{= \max_{u} \El{G_k|X_k=x_k, U=u}{\pi^*}\, ,}\\
									&\onslide<6->{= \max_{u} \El{R_{k+1} + \gamma G_{k+1}|X_k=x_k, U=u}{\pi^*}\, ,}\\
									&\onslide<7->{= \max_{u} \E{R_{k+1} + \gamma v^*(X_{k+1})|X_k=x_k, U=u}\, .}
\end{split}
\end{equation}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Optimality Equation (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Optimality Equation (2)}
Again, the Bellman optimality equation can be visualized by a backup diagram:
\begin{figure}
		 %\hspace*{-1.5cm}
		\includegraphics[width=4.5cm]{fig/lec02/Back_Up_Value_Optimal_MDP.pdf}
		\caption{Backup diagram for $v^*(x_k)$}
		\label{fig:Back_Up_Value_Optimal_MDP}
\end{figure}
\pause
For a finite MDP the following expression results:
\begin{equation}
\label{eq:Bellman_Optimal_State_Value_MDP}
	v^*(x_k) = \max_{u_k}\, \mathcal{R}^u_x+ \gamma \sum_{x_{k+1}\in\mathcal{X}}p_{xx'}^u v_{\pi^*}(x_{k+1}) .
\end{equation}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Bellman Optimality Equation (3)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Bellman Optimality Equation (3)}
Likewise, the Bellman optimality equation is applicable to the action value:
\begin{equation}
	q^*(x_k,u_k) = \E{R_{k+1}+ \gamma \max_{u_{k+1}} q^*(X_{k+1},U_{k+1})|X_k=x_k, U_k=u_k} .
\end{equation}
\pause
And, in the finite MDP case:
\begin{equation}
\label{eq:Bellman_Optimal_action_Value_MDP}
	q^*(x_k,u_k) = \mathcal{R}^u_x + \gamma \sum_{x_{k+1}\in\mathcal{X}}p_{xx'}^u \max_{u_{k+1}}q^*(x_{k+1}, u_{k+1}) .
\end{equation}
\pause
\begin{figure}
		 %\hspace*{-1.5cm}
		\includegraphics[width=4.7cm]{fig/lec02/Back_Up_Action_Value_Optimal_MDP.pdf}
		\caption{Backup diagram for $q^*(x_k,u_k)$}
		\label{fig:Back_Up_Action_Value_Optimal_MDP}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Solving the Bellman Optimality Equation %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Solving the Bellman Optimality Equation}
\begin{itemize}
	\item In finite MDPs with $n$ states, \eqref{eq:Bellman_Optimal_State_Value_MDP} delivers an \hl{algebraic equation system} with $n$ unknowns and $n$ \hl{state-value equations}. \pause
	\item Likewise, \eqref{eq:Bellman_Optimal_action_Value_MDP} delivers an algebraic equation system with up to $n\cdot m$ unknowns and $n \cdot m$ \hl{action-value equations} ($m=$number of actions). \pause
	\item Due to $\max$ operator the equation set is generally \hl{nonlinear}. Direct, closed form solution rarely available. \pause
	\item Hence, often  approximate / iterative solutions are required (upcoming lectures).\pause
	\item If environment is exactly known, solving for $v^*$ or $q^*$ directly delivers optimal policy.
	\begin{itemize}
		\item If $v(x)$ is known, a one-step-ahead search is required to get $q(x)$.
		\item If $q(x, u)$ is known, directly choose $q^*$. \pause
	\end{itemize}
	\item Even though above decisions are very short sighted (one-step-ahead search for $v$ or direct choice of $q$), by Bellman's principle of optimality one receives the long-term maximum of the expected reward.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optimal Forest MDP %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Optimal Policy for Forest Tree MDP}
Remember the forest tree MDP example:
\begin{figure}
	\includegraphics[width=11cm]{fig/lec02/Forest_Markov_Decision_Process_No_Color.pdf}
	\label{fig:Forest_Markov_Decision_Process_No_Color}
\end{figure}

\begin{itemize}
	\item Two actions possible in each state:
	\begin{itemize}
		\item Wait $u=w$: let the tree grow.
		\item Cut $u=c$: gather the wood.
	\end{itemize}
	\item Lets first calculate $v^*(x)$ and then $q^*(x,u)$.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optimal Policy for Forest Tree MDP: State-Value (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Optimal Policy for Forest Tree MDP: State Value (1)}
\onslide<1->{Start with $v(x=4)$ ('gone') and then continue going backwards:}
\onslide<2->{
\begin{align*}
	v^*(x=4) &= 0\, ,}\\
	\onslide<3->{v^*(x=3) &= \max \begin{cases} 1 + \gamma\left[(1-\alpha)v^*(x=3) + \alpha v^*(x=4)\right]\, ,\\ 3+\gamma v^*(x=4)\, ,\end{cases}}\\
					 &\onslide<4->{= \max \begin{cases} 1 + \gamma\left[(1-\alpha)v^*(x=3)\right]\, ,\\ 3\, ,\end{cases}}\\\
	\onslide<5->{v^*(x=2) &= \max \begin{cases} 0 + \gamma\left[(1-\alpha)v^*(x=3) + \alpha v^*(x=4)\right]\, ,\\ 2+\gamma v^*(x=4)\, ,\end{cases}}\\
					 &\onslide<6->{= \max \begin{cases} \gamma\left[(1-\alpha)v^*(x=3)\right]\, ,\\ 2\, ,\end{cases}}\\
	\onslide<7->{v^*(x=1) &= \max \begin{cases} \gamma\left[(1-\alpha)v^*(x=2)\right]\, ,\\ 1\, .\end{cases}}
\end{align*}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optimal Policy for Forest Tree MDP: State-Value (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Optimal Policy for Forest Tree MDP: State Value (2)}
\begin{itemize}
	\item Due to presence of terminal state ($x=4$) only three unknowns
	\item Solution: numerical optimization approach (e.g., simplex method, gradient descent,...)
\end{itemize}
\begin{figure}
	\includegraphics[width=11cm]{fig/lec02/Forest_Markov_Decision_Process_Optimal_State_Value.pdf}
		\caption{State values under optimal policy ($\gamma=0.8$,	$\alpha=0.2$)}
	\label{fig:Forest_Markov_Decision_Process_Optimal_State_Value}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optimal Policy for Forest Tree MDP: State-Value (3)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Optimal Policy for Forest Tree MDP: State Value (3)}
\begin{figure}
	\includegraphics[width=11cm]{fig/lec02/Forest_Markov_Decision_Process_Optimal_State_Value_Gamma09.pdf}
		\caption{State values under optimal policy (\hl{$\gamma=0.9$},	$\alpha=0.2$)}
	\label{fig:Forest_Markov_Decision_Process_Optimal_State_Value_Gamma09}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optimal Policy for Forest Tree MDP: Action-Value (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Optimal Policy for Forest Tree MDP: Action Value (1)}
\onslide<1->{Use $u_{k+1}=u'$ to set up equation system:
\begin{align*}
q^*(x=1,u=c) &= 1\, ,}\\
\onslide<2->{q^*(x=1,u=w) &= \gamma(1-\alpha)\max_{u'}q^*(x=2,u')\, ,}\\
\onslide<3->{q^*(x=2,u=c) &= 2\, ,}\\
\onslide<4->{q^*(x=2,u=w) &= \gamma(1-\alpha)\max_{u'}q^*(x=3,u')\, ,}\\
\onslide<5->{q^*(x=3,u=c) &= 3\, ,}\\
\onslide<6->{q^*(x=3,u=w) &= 1+\gamma(1-\alpha)\max_{u'}q^*(x=3,u')\, .}
\end{align*}
\onslide<7->{
\begin{itemize}
	\item There are six action-state pairs in total.
	\item Three of them can be directly determined.
	\item Three unknowns and three equations remain.
\end{itemize}
}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optimal Policy for Forest Tree MDP: Action-Value (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Optimal Policy for Forest Tree MDP: Action Value (2)}
\onslide<1->{
Rearrange $\max$ expressions for unknown action values:
\begin{align*}
q^*(x=1,u=w) &= \gamma(1-\alpha)\max\begin{cases}\gamma(1-\alpha)\max\begin{cases}1+\gamma(1-\alpha)q^*(3,w)\\3\end{cases}\\2\end{cases}}\\
\onslide<2->{q^*(x=2,u=w) &= \gamma(1-\alpha)\max\begin{cases}1+\gamma(1-\alpha)q^*(3,w)\\3\end{cases}}\\
\onslide<3->{q^*(x=3,u=w) &= 1+\gamma(1-\alpha)\max\begin{cases}q^*(3,w)\\3\end{cases}\, .}
\end{align*}
\onslide<4->{Again, retrieve unknown optimal action values by numerical optimization solvers.}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optimal Policy for Forest Tree MDP: Action-Value (3)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Optimal Policy for Forest Tree MDP: Action Value (3)}
\begin{figure}
	\includegraphics[width=11cm]{fig/lec02/Forest_Markov_Decision_Process_Optimal_Action_Value.pdf}
		\caption{Action values under optimal policy ($\gamma=0.8$,	$\alpha=0.2$)}
	\label{fig:Forest_Markov_Decision_Process_Optimal_Action_Value}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optimal Policy for Forest Tree MDP: Action-Value (4)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Optimal Policy for Forest Tree MDP: Action Value (4)}
\begin{figure}
	\includegraphics[width=11cm]{fig/lec02/Forest_Markov_Decision_Process_Optimal_Action_Value_Gamma09.pdf}
		\caption{Action values under optimal policy (\hl{$\gamma=0.9$},	$\alpha=0.2$)}
	\label{fig:Forest_Markov_Decision_Process_Optimal_Action_Value_Gamma09}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Solving an MDP by Calculating Optimal State and Action-Values%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Direct Numerical State and Action-Value Calculation}
\begin{itemize}
	\item Possible only for \hl{small action and state-space} MDPs
	\begin{itemize}
		\item 'Solving' Backgammon with $\approx 10^{20}$ states?
	\end{itemize}
	\pause
	\item Another issue: initial values?\pause
	\item And yet another issue: \hl{nonlinear system with multiple unknowns}
		\begin{itemize}
			\item Local solvers = local solutions
			\item Initial-point-dependent
		\end{itemize}
		\pause
		\item And finally: total \hl{environment knowledge} required
\end{itemize}
\pause
\begin{block}{Framing the reinforcement learning problem}
Facing the above issues, RL addresses mainly two topics:
\begin{itemize}
	\item Approximate solutions of complex decision problems.
	\item Learning of such approximations based on environment interactions.
\end{itemize}

\end{block}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Summary %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Summary: What You've Learned Today}
\begin{itemize}
	\item Differentiate finite Markov process models with or w/o rewards and actions.\pause
	\item Interpret such stochastic processes as simplified abstractions of real-world problems.\pause
	\item Understand the importance of value functions to describe the agent's performance. \pause
	\item Formulate value-function equation systems by the Bellman principle.\pause
	\item Recognize optimal policies.\pause
	\item Setting up nonlinear equation systems for retrieving optimal policies by the Bellman principle.\pause
	\item Solving for different value functions in MRP/MDP contexts by numerical optimization.
\end{itemize}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Final Slide %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{The End for Today}
\begin{figure}
\includegraphics[width=10cm]{fig/lec02/dilbert.png}
\end{figure}
\vspace{1cm}
\centering
Thanks for your attention and have a nice week!
}


%!TEX root = ../thesis.tex
% ******************************* Thesis Appendix B ********************************

\chapter{Conclusion}
The three splitting settings devised earlier highlight different classifiers.
In the maximum similarity setting, which ensured that observation from all parts of the rivers were included in the train, validation, and test sets, $L_1$ Logistic regression outperformed Random Forest. It achieved accuracy scores above 98\% with five features sets.

In the maximum dissimilarity setting, which kept out whole parts of the rivers for testing and validation, Random Forest proves to be on average the better model. It has an accuracy score of 90.45\% when using the OTU MIN CSS set and 90.24\% with OTU CSS and OTU CSS LOG. Logistic regression has relatively low scores, with most models being close to naive guessing. Using $L_2$ regularisation however, we get an accuracy score of 93.90\% using the OTU CSS LOG set, outperforming Random Forest.

Randomly splitting the data (but ensuring that the balance of classes was constant between the train, test, and validation sets) did not favour any method. Both performed equally well, with scores ranging between the maximum similarity and dissimilarity settings.

In all cases the classifiers (with best features set) produce better results than naive guessing, even though most of the prediction errors are concentrated in identifying black water samples.

Using Ordination methods as features produces only one relatively (to other sets) high accuracy score, for Logistic regression in the maximum dissimilarity setting. This is achieved by selecting the subset of its axes explaining 90\% of the variance in the data. In all other cases, the methods are not useful for dimensionality reduction. Furthermore, using the 20-dimensional NMDS configuration does not produce significant results, and it is highly unlikely that if it had converged better scores would be produced. On the other hand, using the OTU CSS LOG set we obtain the highest average accuracy score across all classifiers and split schemes.

Finally the Bayesian logistic regression framework takes a very long time to sample from posteriors where the parameter's dimension is very large (i.e. all OTU and PCoA sets). Using the much smaller NMDS set however, we get promising result; it is comparable with the cross-validated MLE logistic regression model in the maximally similar setting and outperforms it in the dissimilar.

\section*{Further Work}

This work showcases that further machine learning exploration of this particular data set, but also of any other derived from metabarcoding eDNA methods, can be very fruitful but also challenging. As a next step, species importance could be identified using parametric models often used in ecology (like zero-inflated Gaussian mixture) and check if they agree with those obtained through random forest.

 More classifiers and dimensionality reduction techniques that take into account the spatial distribution of samples could also be employed. Furthermore, other sampling approaches can be developed that take into account correlation structures of river samples, and thus evaluate where each classification method fails.

 Problems were encountered because of the unbalanced class distribution. Collecting more samples from the rivers further to the east is one way to combat this, which will also aid in the evaluation of the classifiers.

 Bayesian methods (or versions of), even linear ones like Logistic regression, might be promising if more time is spend on sampling and choosing the right models for our data set.

 Finally, the scope does not have to be limited to water colour classification. The effect of anthropogenic and environmental factors on the Amazonian community composition could be investigated. The process can begin by additional sampling efforts along and around the rivers, collecting data on other ecological variables, like minerals, pollution levels, river size, water flow, proximity to settlements, and land use, to name a few. Data can also be collected on a temporal basis so that the changes can be better understood. Then, time series analysis together with machine learning could be utilised to uncover the complex interactions between species abundance and their environment, and thus the role humans play in the ecosystem.


\newpage
\section{Arithmetic Operations}

\begin{Introduction}[arithmeticoperations]{ARITHMETIC\_OPERATIONS}
This section considers operations defined in REDUCE that concern numbers,
or operators that can operate on numbers in addition, in most cases, to
more general expressions.
\end{Introduction}

\newpage
\begin{Operator}[abs]{ABS}
\index{absolute value}
The \name{abs} operator returns the absolute value of its argument.

\begin{Syntax}
\name{abs}\(\meta{expression}\)
\end{Syntax}

\meta{expression} can be any REDUCE scalar expression.

\begin{Examples}
abs(-a);                    &        ABS(A) \\
abs(-5);                    &        5 \\
a := -10;                   &        A := -10 \\
abs(a);                     &        10 \\
abs(-a);                    &        10
\end{Examples}
\begin{Comments}
If the argument has had no numeric value assigned to it, such as an
identifier or polynomial, \name{abs} returns an expression involving
\name{abs} of its argument, doing as much simplification of the argument
as it can, such as dropping any preceding minus sign.
\end{Comments}
\end{Operator}

\begin{Switch}[adjprec]{ADJPREC}
\index{input}\index{precision}
When a real number is input, it is normally truncated to the
\nameref{precision} in
effect at the time the number is read.  If it is desired to keep the full
precision of all numbers input, the switch \name{adjprec}
(for \meta{adjust precision}) can be turned on.  While on, \name{adjprec}
will automatically increase the precision, when necessary, to match that
of any integer or real input, and a message printed to inform the user of
the precision increase.

\begin{Examples}
on rounded; \\
1.23456789012345; & 1.23456789012 \\
on adjprec; \\
1.23456789012345; \\
*** precision increased to 15 \\
1.23456789012345
\end{Examples}
\end{Switch}

\begin{Operator}[arg]{ARG}
\index{complex}\index{polar angle}
If \nameref{complex} and \nameref{rounded} are on, and {\em arg}
evaluates to a complex number, \name{arg} returns the polar angle of {\em
arg}, measured in radians.  Otherwise an expression in {\em arg} is
returned.

\begin{Examples}
arg(3+4i)  & ARG(3 + 4*I) \\
on rounded, complex; \\
ws; & 0.927295218002 \\
arg a; & ARG(A)
\end{Examples}

\end{Operator}

\begin{Operator}[ceiling]{CEILING}
\index{integer}

\begin{Syntax}
\name{ceiling}\(\meta{expression}\)
\end{Syntax}

This operator returns the ceiling (i.e., the least integer greater than or
equal to its argument) if its argument has a numerical value.  For
negative numbers, this is equivalent to \nameref{fix}.  For non-numeric
arguments, the value is an expression in the original operator.

\begin{Examples}
ceiling 3.4; & 4 \\
fix 3.4; & 3 \\
ceiling(-5.2); & -5 \\
fix(-5.2); & -5 \\
ceiling a; & CEILING(A)
\end{Examples}

\end{Operator}

\begin{Operator}[choose]{CHOOSE}
\name{choose}(\meta{m},\meta{m}) returns the number of ways of choosing
\meta{m} objects from a collection of \meta{n} distinct objects --- in other
words the binomial coefficient.  If \meta{m} and \meta{n} are not positive
integers, or $m > n$, the expression is returned unchanged.
than or equal to
\begin{Examples}
choose(2,3); & 3 \\
choose(3,2); & CHOOSE(3,2) \\
choose(a,b); & CHOOSE(A,B)
\end{Examples}
\end{Operator}

\begin{Operator}[]deg2dms{DEG2DMS}
\index{degrees}\index{radians}
\begin{Syntax}
\name{deg2dms}\(\meta{expression}\)
\end{Syntax}

In \nameref{rounded} mode, if \meta{expression} is a real number, the
operator \name{deg2dms} will interpret it as degrees, and convert it to a
list containing the equivalent degrees, minutes and seconds.  In all other
cases, an expression in terms of the original operator is returned.

\begin{Examples}
deg2dms 60; & DEG2DMS(60) \\
on rounded; \\
ws; & \{60,0,0\} \\
deg2dms 42.4; & \{42,23,60.0\} \\
deg2dms a; & DEG2DMS(A)
\end{Examples}

\end{Operator}

\begin{Operator}[deg2rad]{DEG2RAD}
\index{degrees}\index{radians}

\begin{Syntax}
\name{deg2rad}\(\meta{expression}\)
\end{Syntax}

In \nameref{rounded} mode, if \meta{expression} is a real number, the
operator \name{deg2rad} will interpret it as degrees, and convert it to
the equivalent radians.  In all other cases, an expression in terms of the
original operator is returned.

\begin{Examples}
deg2rad 60; & DEG2RAD(60) \\
on rounded; \\
ws; & 1.0471975512 \\
deg2rad a; & DEG2RAD(A)
\end{Examples}

\end{Operator}

\begin{Operator}[difference]{DIFFERENCE}
The \name{difference} operator may be used as either an infix or prefix binary
subtraction operator.  It is identical to \name{-} as a binary operator.

\begin{Syntax}
\name{difference}\(\meta{expression},\meta{expression}\) or

\meta{expression} \name{difference} \meta{expression}
      \{\name{difference} \meta{expression}\}\optional
\end{Syntax}

\meta{expression} can be a number or any other valid REDUCE expression.  Matrix
expressions are allowed if they are of the same dimensions.

\begin{Examples}

difference(10,4);                                     &         6 \\

15 difference 5 difference 2;                         &         8 \\

a difference b;                                       &         A - B

\end{Examples}

\begin{Comments}
The \name{difference} operator is left associative, as shown in the second
example above.

\end{Comments}
\end{Operator}

\begin{Operator}[dilog]{DILOG}
\index{dilogarithm function}
The \name{dilog} operator is known to the differentiation and integration
operators, but has numeric value attached only at \name{dilog(0)}.  Dilog is
defined by
\begin{TEX}
\begin{displaymath}
  dilog(x) = -\int{{log(x)\,dx}\over{x-1}}
\end{displaymath}
\end{TEX}
\begin{INFO}
  dilog(x) = -int(log(x),x)/(x-1)
\end{INFO}
\begin{Examples}
df(dilog(x**2),x);          &      - \rfrac{2*LOG(X^{2})*X}{X^{2}  - 1}\\

int(dilog(x),x);            &     DILOG(X)*X - DILOG(X) + LOG(X)*X - X \\

dilog(0);                   &     \rfrac{PI^{2}}{6}
\end{Examples}

\end{Operator}

\begin{Operator}[dms2deg]{DMS2DEG}
\index{degrees}\index{radians}

\begin{Syntax}
\name{dms2deg}\(\meta{list}\)
\end{Syntax}

In \nameref{rounded} mode, if \meta{list} is a list of three real numbers,
the operator \name{dms2deg} will interpret the list as degrees, minutes
and seconds and convert it to the equivalent degrees.  In all other cases,
an expression in terms of the original operator is returned.

\begin{Examples}
dms2deg \{42,3,7\}; & DMS2DEG(\{42,3,7\}) \\
on rounded; \\
ws; & 42.0519444444 \\
dms2deg a; & DMS2DEG(A)
\end{Examples}

\end{Operator}

\begin{Operator}[dms2rad]{DMS2RAD}
\index{degrees}\index{radians}

\begin{Syntax}
\name{dms2rad}\(\meta{list}\)
\end{Syntax}

In \nameref{rounded} mode, if \meta{list} is a list of three real numbers,
the operator \name{dms2rad} will interpret the list as degrees, minutes
and seconds and convert it to the equivalent radians.  In all other cases,
an expression in terms of the original operator is returned.

\begin{Examples}
dms2rad \{42,3,7\}; & DMS2RAD(\{42,3,7\}) \\
on rounded; \\
ws; & 0.733944887421 \\
dms2rad a; & DMS2RAD(A)
\end{Examples}

\end{Operator}

\begin{Operator}[factorial]{FACTORIAL}
\index{gamma}
\begin{Syntax}
\name{factorial}\(\meta{expression}\)
\end{Syntax}

If the argument of \name{factorial} is a positive integer or zero, its
factorial is returned.  Otherwise the result is expressed in terms of the
original operator.  For more general operations, the \nameref{gamma} operator
is available in the \nameref{Special Function Package}.

\begin{Examples}
factorial 4; & 24 \\
factorial 30 ; & 265252859812191058636308480000000 \\
factorial(a) ; FACTORIAL(A)
\end{Examples}

\end{Operator}

\begin{Operator}[fix]{FIX}
\index{integer}
\begin{Syntax}
\name{fix}\(\meta{expression}\)
\end{Syntax}

The operator \name{fix} returns the integer part of its argument, if that
argument has a numerical value.  For positive numbers, this is equivalent
to \nameref{floor}, and, for negative numbers, \nameref{ceiling}.  For
non-numeric arguments, the value is an expression in the original operator.

\begin{Examples}
fix 3.4; & 3 \\
floor 3.4; & 3 \\
ceiling 3.4; & 4 \\
fix(-5.2); & -5 \\
floor(-5.2); & -6 \\
ceiling(-5.2); & -5 \\
fix(a); & FIX(A)
\end{Examples}

\end{Operator}

\begin{Operator}[fixp]{FIXP}
\index{integer}
The \name{fixp} logical operator returns true if its argument is an integer.
\begin{Syntax}
\name{fixp}\(\meta{expression}\) or \name{fixp} \meta{simple\_expression}
\end{Syntax}

\meta{expression} can be any valid REDUCE expression, \meta{simple\_expression}
must be a single identifier or begin with a prefix operator.

\begin{Examples}
if fixp 1.5 then write "ok" else write "not";
			     &             not \\
if fixp(a) then write "ok" else write "not";
			     &             not \\
a := 15;                     &             A := 15 \\
if fixp(a) then write "ok" else write "not";
			     &             ok
\end{Examples}

\begin{Comments}
Logical operators can only be used inside conditional expressions such as
\name{if}\ldots\name{then} or \name{while}\ldots\name{do}.
\end{Comments}
\end{Operator}

\begin{Operator}[floor]{FLOOR}
\index{integer}

\begin{Syntax}
\name{floor}\(\meta{expression}\)
\end{Syntax}

This operator returns the floor (i.e., the greatest integer less than or
equal to its argument) if its argument has a numerical value.  For
positive numbers, this is equivalent to \nameref{fix}.  For non-numeric
arguments, the value is an expression in the original operator.

\begin{Examples}
floor 3.4; & 3 \\
fix 3.4; & 3 \\
floor(-5.2); & -6 \\
fix(-5.2); & -5 \\
floor a; & FLOOR(A)
\end{Examples}

\end{Operator}

\begin{Operator}[expt]{EXPT}
The \name{expt} operator is both an infix and prefix binary exponentiation
operator.  It is identical to \name{^} or \name{**}.
\begin{Syntax}
\name{expt}\(\meta{expression},\meta{expression}\)
 or \meta{expression} \name{expt} \meta{expression}
\end{Syntax}
\begin{Examples}
a expt b;                    &    A^{B} \\
expt(a,b);                   &    A^{B} \\
(x+y) expt 4;                &    X^{4} + 4*X^{3}*Y + 6*X^{2}*Y^{2} + 4*X*Y^{3} + Y^{4}
\end{Examples}
\begin{Comments}
Scalar expressions may be raised to fractional and floating-point powers.
Square matrix expressions may be raised to positive powers, and also to
negative powers if non-singular.

\name{expt} is left associative.  In other words, \name{a expt b expt c} is
equivalent to \name{a expt (b*c)}, not \name{a expt (b expt c)}, which
would be right associative.
\end{Comments}
\end{Operator}

\begin{Operator}[gcd]{GCD}
\index{greatest common divisor}\index{polynomial}
The \name{gcd} operator returns the greatest common divisor of two
polynomials.
\begin{Syntax}
\name{gcd}\(\meta{expression},\meta{expression}\)
\end{Syntax}
\meta{expression} must be a polynomial (or integer), otherwise an error
occurs.

\begin{Examples}
gcd(2*x**2 - 2*y**2,4*x + 4*y);                         &       2*(X + Y) \\
gcd(sin(x),x**2 + 1);                                   &       1  \\
gcd(765,68);                                            &       17
\end{Examples}

\begin{Comments}
The operator \name{gcd} described here provides an explicit means to find the
gcd of two expressions.  The switch \name{gcd} described below simplifies
expressions by finding and canceling gcd's at every opportunity.  When
the switch \nameref{ezgcd} is also on, gcd's are figured using the EZ GCD
algorithm, which is usually faster.
%%% The \nameref{heugcd} switch is also available, providing
%%% gcd's by the Heuristic algorithm.
\end{Comments}
\end{Operator}

\begin{Operator}[ln]{LN}
\index{logarithm}
\begin{Syntax}
\name{ln}\(\meta{expression}\)
\end{Syntax}
\meta{expression} can be any valid scalar REDUCE expression.

The \name{ln} operator returns the natural logarithm of its argument.
However, unlike \nameref{log}, there are no algebraic rules associated
with it; it will only evaluate when \nameref{rounded} is on, and the
argument is a real number.

\begin{Examples}
ln(x);                     &         LN(X) \\
ln 4;                      &         LN(4) \\
ln(e);                     &         LN(E) \\
df(ln(x),x);               &         DF(LN(X),X) \\
on rounded; \\
ln 4;                      &         1.38629436112 \\
ln e;                      &         1
\end{Examples}

\begin{Comments}
Because of the restricted algebraic properties of \name{ln}, users are
advised to use \nameref{log} whenever possible.
\end{Comments}

\end{Operator}

\begin{Operator}[log]{LOG}
\index{logarithm}
The \name{log} operator returns the natural logarithm of its argument.
\begin{Syntax}
\name{log}\(\meta{expression}\) or \name{log} \meta{expression}
\end{Syntax}

\meta{expression} can be any valid scalar REDUCE expression.

\begin{Examples}
log(x);                     &         LOG(X) \\
log 4;                      &         LOG(4) \\
log(e);                     &         1 \\
on rounded; \\
log 4;                      &         1.38629436112
\end{Examples}

\begin{Comments}
\name{log} returns a numeric value only when \nameref{rounded} is on.  In that
case, use of a negative argument for \name{log} results in an error
message.  No error is given on a negative argument when REDUCE is not in
that mode.
\end{Comments}
\end{Operator}

\begin{Operator}[logb]{LOGB}
\index{logarithm}
\begin{Syntax}
\name{logb}\(\meta{expression}\,\meta{integer}\)
\end{Syntax}
\meta{expression} can be any valid scalar REDUCE expression.

The \name{logb} operator returns the logarithm of its first argument using
the second argument as base.  However, unlike \nameref{log}, there are no
algebraic rules associated with it; it will only evaluate when
\nameref{rounded} is on, and the first argument is a real number.

\begin{Examples}
logb(x,2);                 &         LOGB(X,2) \\
logb(4,3);                 &         LOGB(4,3) \\
logb(2,2);                 &         LOGB(2,2) \\
df(logb(x,3),x);           &         DF(LOGB(X,3),X) \\
on rounded; \\
logb(4,3);                 &         1.26185950714 \\
logb(2,2);                 &         1
\end{Examples}

\end{Operator}

\begin{Operator}[max]{MAX}
\index{maximum}
The operator \name{max} is an n-ary prefix operator, which returns the largest
value in its arguments.
\begin{Syntax}
\name{max}\(\meta{expression}\{,\meta{expression}\}\optional\)

\end{Syntax}

\meta{expression} must evaluate to a number.  \name{max} of an empty list
returns 0.

\begin{Examples}
max(4,6,10,-1);              &        10 \\
<<a := 23;b := 2*a;c := 4**2;max(a,b,c)>>;
			     &        46 \\
max(-5,-10,-a);              &        -5
\end{Examples}

\end{Operator}

\begin{Operator}[min]{MIN}
\index{minimum}
The operator \name{min} is an n-ary prefix operator, which returns the
smallest value in its arguments.
\begin{Syntax}
\name{min}\(\meta{expression}\{,\meta{expression}\}\optional\)
\end{Syntax}

\meta{expression} must evaluate to a number. \name{min} of an empty list
returns 0.
\begin{Examples}
min(-3,0,17,2);              &        -3 \\
<<a := 23;b := 2*a;c := 4**2;min(a,b,c)>>;
			     &        16 \\
min(5,10,a);                 &        5
\end{Examples}
\end{Operator}

\begin{Operator}[minus]{MINUS}
The \name{minus} operator is a unary minus, returning the negative of its
argument.  It is equivalent to the unary \name{-}.
\begin{Syntax}
\name{minus}\(\meta{expression}\)

\end{Syntax}

\meta{expression} may be any scalar REDUCE expression.

\begin{Examples}
minus(a);                    &          - A \\
minus(-1);                   &          1 \\
minus((x+1)**4);             &          - (X^{4} + 4*X^{3} + 6*X^{2} + 4*X + 1)
\end{Examples}

\end{Operator}

\begin{Operator}[nextprime]{NEXTPRIME}
\index{prime number}
\begin{Syntax}
\name{nextprime}\(\meta{expression}\)
\end{Syntax}

If the argument of \name{nextprime} is an integer, the least prime greater
than that argument is returned.  Otherwise, a type error results.

\begin{Examples}
nextprime 5001; & 5003  \\
nextprime(10^30); & 1000000000000000000000000000057 \\
nextprime a; & ***** A invalid as integer
\end{Examples}

\end{Operator}

\begin{Switch}[noconvert]{NOCONVERT}
Under normal circumstances when \name{rounded} is on, \REDUCE\ converts the
number 1.0 to the integer 1.  If this is not desired, the switch
\name{noconvert} can be turned on.
\begin{Examples}
on rounded; \\
1.0000000000001; & 1 \\
on noconvert; \\
1.0000000000001; & 1.0 \\
\end{Examples}
\end{Switch}

\begin{Operator}[norm]{NORM}
\index{complex}
\begin{Syntax}
\name{norm}\(\meta{expression}\)
\end{Syntax}

If \name{rounded} is on, and the argument is a real number, \meta{norm}
returns its absolute value.  If \name{complex} is also on, \meta{norm}
returns the square root of the sum of squares of the real and imaginary
parts of the argument.  In all other cases, a result is returned in
terms of the original operator.

\begin{Examples}
norm (-2); & NORM(-2) \\
on rounded;\\
ws; &  2.0 \\
norm(3+4i); & NORM(4*I+3) \\
on complex;\\
ws; &  5.0\\
\end{Examples}

\end{Operator}

\begin{Operator}[perm]{PERM}
\index{permutation}
\begin{Syntax}
perm(\meta{expression1},\meta{expression2})
\end{Syntax}

If \meta{expression1} and \meta{expression2} evaluate to positive integers,
\name{perm} returns the number of permutations possible in selecting
\meta{expression1} objects from \meta{expression2} objects.
In other cases, an expression in the original operator is returned.

\begin{Examples}
perm(1,1); & 1 \\
perm(3,5); & 60 \\
perm(-3,5); & PERM(-3,5) \\
perm(a,b); & PERM(A,B)
\end{Examples}

\end{Operator}

\begin{Operator}[plus]{PLUS}
The \name{plus} operator is both an infix and prefix n-ary addition
operator.  It exists because of the way in which {\REDUCE} handles such
operators internally, and is not recommended for use in algebraic mode
programming. \nameref{plussign}, which has the identical effect, should be
used instead.
\begin{Syntax}
\name{plus}\(\meta{expression},\meta{expression}\{,\meta{expression}\}
\optional\) or \\
        \meta{expression} \name{plus} \meta{expression} \{\name{plus} \meta{expression}\}\optional
\end{Syntax}

\meta{expression} can be any valid REDUCE expression, including matrix
expressions of the same dimensions.

\begin{Examples}
a plus b plus c plus d;      &      A + B + C + D \\
4.5 plus 10;                 & \rfrac{29}{2} \\\\
plus(x**2,y**2);             &      X^{2} + Y^{2}
\end{Examples}
\end{Operator}

\begin{Operator}[quotient]{QUOTIENT}
The \name{quotient} operator is both an infix and prefix binary operator that
returns the quotient of its first argument divided by its second.  It is
also a unary \nameref{recip}rocal operator.  It is identical to \name{/} and
\nameref{slash}.
\begin{Syntax}
\name{quotient}\(\meta{expression},\meta{expression}\) or
\meta{expression} \name{quotient} \meta{expression}  or
\name{quotient}\(\meta{expression}\) or
\name{quotient} \meta{expression}
\end{Syntax}

\meta{expression} can be any valid REDUCE scalar expression.  Matrix
expressions can also be used if the second expression is invertible and the
matrices are of the correct dimensions.
\begin{Examples}
quotient(a,x+1);              &         \rfrac{A}{X + 1} \\
7 quotient 17;                &         \rfrac{7}{17} \\
on rounded; \\
4.5 quotient 2;               &          2.25 \\
quotient(x**2 + 3*x + 2,x+1); &          X + 2 \\
matrix m,inverse; \\
m := mat((a,b),(c,d));        & \begin{multilineoutput}{6cm}
M(1,1) := A;
M(1,2) := B;
M(2,1) := C
M(2,2) := D
\end{multilineoutput}\\

inverse := quotient m;        & \begin{multilineoutput}{8cm}
INVERSE(1,1) := \rfrac{D}{A*D - B*C}
INVERSE(1,2) := - \rfrac{B}{A*D - B*C}
INVERSE(2,1) := - \rfrac{C}{A*D - B*C}
INVERSE(2,2) := \rfrac{A}{A*D - B*C}
\end{multilineoutput}

\end{Examples}

\begin{Comments}
The \name{quotient} operator is left associative: \name{a quotient b quotient c}
is equivalent to \name{(a quotient b) quotient c}.

If a matrix argument to the unary \name{quotient} is not invertible, or if the
second matrix argument to the binary quotient is not invertible, an error
message is given.
\end{Comments}
\end{Operator}

\begin{Operator}[rad2deg]{RAD2DEG}
\index{degrees}\index{radians}
\begin{Syntax}
\name{rad2deg}\(\meta{expression}\)
\end{Syntax}

In \nameref{rounded} mode, if \meta{expression} is a real number, the
operator \name{rad2deg} will interpret it as radians, and convert it to
the equivalent degrees.  In all other cases, an expression in terms of the
original operator is returned.

\begin{Examples}
rad2deg 1; & RAD2DEG(1) \\
on rounded; \\
ws; & 57.2957795131 \\
rad2deg a; & RAD2DEG(A)
\end{Examples}

\end{Operator}

\begin{Operator}[rad2dms]{RAD2DMS}
\index{degrees}\index{radians}

\begin{Syntax}
\name{rad2dms}\(\meta{expression}\)
\end{Syntax}

In \nameref{rounded} mode, if \meta{expression} is a real number, the
operator \name{rad2dms} will interpret it as radians, and convert it to a
list containing the equivalent degrees, minutes and seconds.  In all other
cases, an expression in terms of the original operator is returned.

\begin{Examples}
rad2dms 1; & RAD2DMS(1) \\
on rounded; \\
ws; & \{57,17,44.8062470964\} \\
rad2dms a; & RAD2DMS(A)
\end{Examples}

\end{Operator}

\begin{Operator}[recip]{RECIP}
\name{recip} is the  alphabetical name for the division operator \name{/}
or \nameref{slash} used as a unary operator.  The use of \name{/} is preferred.

\begin{Examples}
recip a; & \rfrac{1}{A} \\
recip 2; & \rfrac{1}{2}
\end{Examples}

\end{Operator}

\begin{Operator}[remainder]{REMAINDER}
\index{polynomial}
The \name{remainder} operator returns the remainder after its first
argument is divided by its second argument.

\begin{Syntax}
\name{remainder}\(\meta{expression},\meta{expression}\)
\end{Syntax}

\meta{expression} can be any valid REDUCE polynomial, and is not limited
to numeric values.

\begin{Examples}
remainder(13,6);                                        &         1 \\
remainder(x**2 + 3*x + 2,x+1);                          &         0  \\
remainder(x**3 + 12*x + 4,x**2 + 1);                    &         11*X + 4 \\
remainder(sin(2*x),x*y);                                &         SIN(2*X)
\end{Examples}

\begin{Comments}
In the default case, remainders are calculated over the integers.  If you
need the remainder with respect to another domain, it must be declared
explicitly.

If the first argument to \name{remainder} contains a denominator not equal to
1, an error occurs.
\end{Comments}
\end{Operator}

\begin{Operator}[round]{ROUND}
\index{integer}

\begin{Syntax}
\name{round}\(\meta{expression}\)
\end{Syntax}

If its argument has a numerical value, \name{round} rounds it to the
nearest integer.  For non-numeric arguments, the value is an expression in
the original operator.

\begin{Examples}
round 3.4; & 3 \\
round 3.5; & 4 \\
round a; & ROUND(A)
\end{Examples}

\end{Operator}

\begin{Command}[setmod]{SETMOD}
\index{modular}
The \name{setmod} command sets the modulus value for subsequent \nameref{modular}
arithmetic.
\begin{Syntax}
\name{setmod} \meta{integer}
\end{Syntax}

\meta{integer} must be positive, and greater than 1.  It need not be a prime
number.

\begin{Examples}
setmod 6;                    &                1 \\
on modular; \\
16;                          &                4 \\
x^2 + 5x + 7;                &                X^{2} + 5*X + 1 \\
x/3;                         &                \rfrac{X}{3} \\
setmod 2;                    &                6 \\
(x+1)^4;                     &                X^{4} + 1 \\
x/3;                         &                X
\end{Examples}
\begin{Comments}
\name{setmod} returns the previous modulus, or 1 if none has been set
before.  \name{setmod} only has effect when \nameref{modular} is on.

Modular operations are done only on numbers such as coefficients of
polynomials, not on the exponents.   The modulus need not be prime.
Attempts to divide by a power of the modulus produces an error message, since the
operation is equivalent to dividing by 0.  However, dividing by a factor
of a non-prime modulus does not produce an error message.
\end{Comments}
\end{Command}

\begin{Operator}[sign]{SIGN}

\begin{Syntax}
\name{sign} \meta{expression}
\end{Syntax}

\name{sign} tries to evaluate the sign of its argument. If this
is possible \name{sign} returns one of 1, 0 or -1.  Otherwise, the result
is the original form or a simplified variant.

\begin{Examples}
        sign(-5)      &  -1\\
        sign(-a^2*b)  &  -SIGN(B)\\
\end{Examples}

\begin{Comments}
Even powers of formal expressions are assumed to be positive only as long
as the switch \nameref{complex} is off.
\end{Comments}
\end{Operator}

\begin{Operator}[sqrt]{SQRT}
\index{square root}
The \name{sqrt} operator returns the square root of its argument.
\begin{Syntax}
\name{sqrt}\(\meta{expression}\)
\end{Syntax}

\meta{expression} can be any REDUCE scalar expression.

\begin{Examples}
sqrt(16*a^3);               &         4*SQRT(A)*A \\
sqrt(17);                    &         SQRT(17) \\
on rounded; \\
sqrt(17);                    &         4.12310562562 \\
off rounded; \\
sqrt(a*b*c^5*d^3*27);      &
                     3*SQRT(D)*SQRT(C)*SQRT(B)*SQRT(A)*SQRT(3)*C^{2}*D
\end{Examples}
\begin{Comments}
\name{sqrt} checks its argument for squared factors and removes them.
% If the \name{reduced} switch is on, square roots of a product become a
% product of square roots.

Numeric values for square roots that are not exact integers are given only
when \nameref{rounded} is on.

Please note that \name{sqrt(a**2)} is given as \name{a}, which may be
incorrect if \name{a} eventually has a negative value.  If you are
programming a calculation in which this is a concern, you can turn on the
\nameref{precise} switch, which causes the absolute value of the square root
to be returned.
\end{Comments}
\end{Operator}

\begin{Operator}[times]{TIMES}
The \name{times} operator is an infix or prefix n-ary multiplication
operator.  It is identical to \name{*}.
\begin{Syntax}
\meta{expression} \name{times} \meta{expression} \{\name{times} \meta{expression}\}\optional

 or \name{times}\(\meta{expression},\meta{expression} \{,\meta{expression}\}\optional\)
\end{Syntax}

\meta{expression} can be any valid REDUCE scalar or matrix expression.
Matrix expressions must be of the correct dimensions.  Compatible scalar
and matrix expressions can be mixed.

\begin{Examples}
var1 times var2;             &         VAR1*VAR2 \\
times(6,5);                  &         30 \\
matrix aa,bb; \\
aa := mat((1),(2),(x))\$ \\
bb := mat((0,3,1))\$ \\
aa times bb times 5;         &
\begin{multilineoutput}{6cm}
[0   15    5 ]
[            ]
[0   30   10 ]
[            ]
[0  15*X  5*X]
\end{multilineoutput}
\end{Examples}
\end{Operator}


%
\section{Demo}%
%
\edef\maxSat{MAX-SAT}%
\edef\maxTSat{MAX-3SAT}%
\def\oFlip{\ensuremath{1}-flip}%
\def\tFlip{\ensuremath{2}-flip}%
\def\mFlip{\ensuremath{m}-flip}%
%
\gdef\maxSatClauses{\textcolor{blue}{\ensuremath{k}}}%
\gdef\maxSatVariables{\textcolor{green}{\ensuremath{n}}}%
\gdef\maxSatVariable{\ensuremath{x}}%
\gdef\maxSatVariablei#1{\ensuremath{\maxSatVariable_{#1}}}%
\gdef\maxSatFormula{\ensuremath{B}}%
%
\begin{frame}[t]%
\frametitle{\maxSat}%
\begin{itemize}%
\item \only<8->{Maximum }Satisfiability Problems (SAT)\only<-7>{\scitep{HS2000SAORFROS}}\only<8->{\scitep{HS2005SLSFAA}}\uncover<2->{:%
\begin{itemize}
\item Given: Formula \maxSatFormula\ in Boolean logic%
\uncover<3->{ with of \maxSatVariables\ Boolean variables $\vec{\maxSatVariable}=(\maxSatVariablei{1}, \maxSatVariablei{2},\dots, \maxSatVariablei{\maxSatVariables})$%
\uncover<4->{, which appear either directly or negated%
\uncover<5->{ in \maxSatClauses\ \inQuotes{\texttt{or}} clauses%
\uncover<6->{, which are all combined with into one \inQuotes{\texttt{and}}}}}}%
\item<7-> \only<-7>{SAT Goal: find a setting for these variables so that $\maxSatFormula$ becomes \texttt{true}}%
\only<8->{\maxSat\ Goal\scitep{HS2005SLSFAA}: minimize objective function $\objectiveFunctionb{\vec{x}} = \textnormal{number of clauses which are \texttt{false}}$.}%
\item<9-> $\objectiveFunctionb{\vec{x}}=0$ $\Longrightarrow$ all clauses are \texttt{true}, SAT problem solved%
\end{itemize}%
}%
%
\item<10-> We want to compare the performance of \textcolor{red}{six different} (trivial) \textcolor{red}{algorithm setups}\uncover<11->{:%
\begin{enumerate}%
\item 1-flip Hill Climber%
\item 1-flip Hill Climber with Restarts%
\item 2-flip Hill Climber%
\item 2-flip Hill Climber with Restarts%
\item $m$-flip Hill Climber%
\item $m$-flip Hill Climber with Restarts%
\end{enumerate}%
}%
%
\item<12-> For the experiment, we want to solve $10\times 10$ (simple) instances from \satLib\expandafter\scitep{\satLibReferences} with \textcolor{red}{10 different values of \maxSatVariables} (and \maxSatClauses)%
%
\end{itemize}%
%
\locateGraphic{-2,7-9}{width=0.55\paperwidth}{graphics/problem_examples/sat/sat}{0.225}{0.54}%
\locateGraphic{3}{width=0.55\paperwidth}{graphics/problem_examples/sat/sat_variables}{0.225}{0.54}%
\locateGraphic{4}{width=0.55\paperwidth}{graphics/problem_examples/sat/sat_negation}{0.225}{0.54}%
\locateGraphic{5}{width=0.55\paperwidth}{graphics/problem_examples/sat/sat_or}{0.225}{0.54}%
\locateGraphic{6}{width=0.55\paperwidth}{graphics/problem_examples/sat/sat_and}{0.225}{0.54}%
%
\strut\\\strut\\\strut\\\strut\\\strut\\\strut\\\strut\\\strut%
%
\end{frame}%
%
\input{part_install.tex}%
%
\begin{frame}[b,label=maxSatInteractiveDemoEnd]%
\frametitle{The Flow / Demo}%
%
\only<-1>{%
\strut\vfill\strut%
\centering{\LARGE{\textbf{\alert{The evaluator works as follows\dots}}}}%
\strut\vfill\strut%
}\only<2->{%
\strut\\\strut\\\strut\\\strut\\\strut\\\strut\\\strut\\\strut%
}%
%
\locate{2-3}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_input_1_results}}{0.05}{0.16}%
\locate{4-5}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_input_2_dimensions}}{0.05}{0.16}%
\locate{6-7}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_input_3_instances}}{0.05}{0.16}%
\locate{8-9}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_input_4_experiment}}{0.05}{0.16}%
\locate{10-11}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_input_5_driver}}{0.05}{0.16}%
\locate{12}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_config_1_config}}{0.05}{0.16}%
\locate{13}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_config_2_evaluation}}{0.05}{0.16}%
\locate{14-15}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_evaluation}}{0.05}{0.16}%
\locate{16}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_output_1_graphic}}{0.05}{0.16}%
\locate{17}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_output_2_document}}{0.05}{0.16}%
\locate{18}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_output_3_latex}}{0.05}{0.16}%
\locate{19-21}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_output_4_pdf}}{0.05}{0.16}%
\locate{21}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow_output_5_xhtml}}{0.05}{0.16}%
\locate{22-}{\includegraphics[width=0.9\paperwidth]{graphics/flow/flow}}{0.05}{0.16}%
%
\only<2->{%
\begin{small}%
\begin{itemize}%
%
\only<-9>{%
\item We got a couple of log files for each experiment\uncover<3->{: 6 experiments in our example, each with $10\times 10\times 20=\numprint{2000}$ log files}%
%}%
%
%\only<4-5>{%
\item<4-> We specify which dimensions we have measured\uncover<5->{: \measureFEs, \measureRuntime, and \measureObjectiveValue\ in our example\uncover<-5>{ (\alert{demo})}}%
%}%
%
%\only<6-7>{%
\item<6-> We specify which benchmark instances we have and what their features are\uncover<7->{: $10\times 10$ instances in our example, with features \maxSatVariables\ and \maxSatClauses\uncover<-7>{ (\alert{demo})}}%
%}%
%
%\only<8-9>{%
\item<8-> For each experiment, we specify the parameters\uncover<9->{: in our example, these are \texttt{algorithm}, \texttt{operator}, \texttt{restart}\uncover<-9>{ (\alert{demo})}}%
}%
%
\only<10-13>{%
\item<10-> An \inQuotes{input driver} loads the data\uncover<11->{: most commonly, the data will be in Comma-, Tab-, or Space-Separated-Values format (\textit{CSV}), but we also support \bbob\expandafter\scitep{\bbobReferences} and \tspSuite\expandafter\scitep{\tspSuiteReferences}}%
%}%
%
%\only<12>{%
\item<12-> Via a configuration file, we choose which input and output formats to use, as well as which file specifies the evaluation process%
%}%
%
%\only<13>{%
\item<13-> The \texttt{evaluation.xml} specifies \emph{how} to evaluate the data, i.e., which evaluation modules to apply%
}%
%
\only<14-15>{%
\item<14-> An evaluation module prints on particular type of information about an experiment or experiment set, such as the ECDF, or a table with final results, etc\dots%
\item<15-> Evaluation modules can be applied multiple times, with different configurations (e.g., we can plot ECDFs for different target solution qualities)%
}%
%
\only<16>{%
\item<16-> We can choose among several different formats to be used for graphics, including EPS\scitep{A1992EPFFS}, PDF\scitep{ISO320002008}, PGF (\LaTeX), SVG(Z), EMF, PNG\scitep{RFC2083}, GIF\scitep{CI1990GIFV8}, BMP, and JPG%
\\\strut%
\vspace{0.19\paperheight}%
\strut\\%
}%
%
%
\only<17->{%
\item<17-> We can also choose among different formats for the report documents, including\only<-17>{\dots}%
%
\uncover<18->{ %
\LaTeX\scitep{MGBCR2004TLC,GMS1994TLC,L1994LADPSUGARM,OPHS2011TNSSITLOLI1M}\only<21->{, XHTML\scitep{W3C2010XHTML,C2009NPOHXADHC}\only<22->{, and export to other programs}}\uncover<19-20>{:%
\begin{itemize}%
\item can automatically be compiled to PDF\scitep{ISO320002008}, if a \LaTeX\ compiler (such as TeXLive\scitep{TEXLIVE} or MiKTeX\scitep{MIKTEX}) is auto-detected%
\item<20-> different document classes, such as IEEEtran\scitep{IEEETRAN}, Springer LLNCS\scitep{SPRINGERLNCS}, ACM sig-alternate\scitep{ACMSIGALTERNATE} can be chosen%
\end{itemize}%
}}%
}%
%
\end{itemize}%
\end{small}}%
%
\end{frame}%
%
%


\subsection{Artistic style transfer for videos}
\label{seq:ruder}
\textit{Artistic style transfer for videos} is a paper written by Manuel Ruder et al. \cite{Ruder:1} And is based on Gatys \cite{Gatys:1} way of implementing style transfer, but introduces a temporal constraint in order to get rid of the randomness that appears when styling each frame individually. \nextline
\subsubsection{Loss function}
The temporal constraint is described in the loss function of the style transfer model. The idea is to penalize the model for deviations between two adjacent frames. So as well as having a content-loss and a style-loss, they introduce a third component for temporal loss:
\begin{equation}
\mathcal{L}_{temporal}(x, \omega, c) = \frac{1}{D}\sum_{k=1} c_k \cdot (x_k - \omega_k)^2
\end{equation}
\nextline This loss component takes three arguments: x, $\omega$ and c. x is the model variable, or more specifically the image we are styling. $\omega$ is the warped previous styled frame. By warped we mean warped using the flow calculated between the current frame and the previous frame. This can be done using a variety of flow calculation algorithms and the article mentions some of these, including DeepFlow and EpicFlow. The c parameter is a mask with the same shape as the styled image, containing integer values 0 or 1. The loss function calculates the squared difference on each pixel (k) of the frame and uses the c-matrix as a weighting for the pixels. For every channel in every pixel where the c-matrix has a value 0 we tell the loss function that this channel in the pixel does not count on the temporal loss. In short, the loss function gives higher loss if the generated image is different from the warped previous stylized frame on every pixel where the c-matrix has value 1. If c has value 0 the difference is not added to the loss.

\subsubsection{Usage of optical flow}
\label{sec:usage_of_optical_flow}
In this implementation the optical flow is used in order to warp previously styled images, detecting disoccluded areas and motion boundaries. These areas are in turn used to create the c-matrix used in the loss function. this is done by giving the disoccluded areas and motion boundaries the value 0 in the c-matrix, and the rest of the image the value 1.\newline\newline To detect the disoccluded areas between two frames, we need the forward flow $(w)$, backward flow$(\hat{w})$, and the forward flow warped to the second image (\~{w}):  \newline \begin{equation}\~{w}(x, y) = w( (x, y) + \hat{w}(x, y))\end{equation}. \newline Using these functions the disoccluded areas are calculated using this inequality:\newline
\begin{equation}
|\~{w} + \hat{w}|^2 > 0.01(|\~{w}|^2 + |\hat{w}|^2) + 0.5
\end{equation}
When it comes to detecting motion boundaries this inequality is used:
\begin{equation}
|\nabla\hat{u}|^2 + |\nabla\hat{v}|^2 > 0.01|\hat{w}|^2 + 0.002
\end{equation}
Where $\hat{u}$ and $\hat{v}$ are the two components of the backward flow function output: $\hat{w}(x, y) = (u, v)$

\subsubsection{Long-term temporal loss}
\label{sec:long_term_temporal_loss}
In the first section, the temporal loss is described using only the previous frame, resulting in the total loss-function:
\begin{equation}
\mathcal{L}_{shortterm}(p^{(i)}, a, x) = \alpha \mathcal{L}_{content} + \beta \mathcal{L}_{style} + \gamma \mathcal{L}_{temporal}(x^{(i)}, \omega_{i-1}^i(x^{i-1}, c^{i-1, i}))
\end{equation}
where the temporal loss is the function we described earlier, $x^{(i)}$ is the currently generated frame,  $\omega_{i-1}^i(x^{i-1})$ is the previous stylized image warped with the flow from previous frame to current frame, and $c^{i-1, i}$ is the weights of disoccluded areas and motion boundaries between last image and current image. However, the longterm loss function is described like this: \newline
\begin{equation}
\mathcal{L}_{longterm}(p^{(i)}, a, x) = \alpha \mathcal{L}_{content} + \beta \\\mathcal{L}_{style} + \gamma \sum_{j\in J: i-j \leq 1}\mathcal{L}_{temporal}(x^{(i)}, \omega_{i-j}^i(x^{i-j}), c_{long}^{i-j, i}))
\end{equation}
where e.g $J = [1, 2, 4]$. the most important part of this long-term loss function are the $c_{long}$ weights. They are calculated like this:\newline
\begin{equation}
c_{long}^{(i-j, i)} = max(c^{(i-j, i)} - \sum_{k\in J: i - k > i - j} c^{(i-k, i)}, 0)
\end{equation}
Here, max is taken element-wise.\newline
\subsubsection{Multi-pass algorithm}
When generating videos with the new longterm temporal loss, the contrast is less diverse near image boundaries, and is especially noticable in videos with a lot of camera motion, mostly because the flow of information to base each frame on only comes from the first frame. This issue was solved by developing a multi-pass algorithm that initially stylizes every frame individually and then blend frames with non-disoccluded parts of previous frames warped according to the optical flow, and then again run some optimization steps for some iterations with the blend as an initialization.\newline
The initialization of frame i for pass j, when processed in forward direction is created like this:\newline
if i = 1:
\begin{equation}
x'^{(i)(j)} = x^{(i)(j-1)}
\end{equation}
else:
\begin{equation}
x'^{(i)(j)} = \delta c^{i-1, i} \circ \omega_{i-1}^i(x^{(i-1)(j)}) + (\bar{\delta}1 + \delta \bar{c}^{(i-1, i)})\circ x^{(i)(j-1)}
\end{equation}\newline
when processed in backward direction, it's created like this:\newline
if i = $N_{frames}$:\newline
\begin{equation}
x'^{(i)(j)} = x^{(i)(j-1)}
\end{equation}
else:\newline
\begin{equation}
x'^{(i)(j)} = \delta c^{i+1, i} \circ \omega_{i+1}^i(x^{(i+1)(j)}) + (\bar{\delta}1 + \delta \bar{c}^{(i+1, i)})\circ x^{(i)(j-1)}
\end{equation} \newline
In these calculations $\circ$ means element-wise vector multiplication. $\delta$ and $1 -\bar{\delta}$ are the blend factors where 1 is a vector of all ones. $\omega$ is the function for warping an image using the flow between two images. And c is the c-matrix we described in section \ref{sec:usage_of_optical_flow}.