\documentclass{article}

\usepackage{fancyhdr}
\usepackage{extramarks}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage[plain]{algorithm}
\usepackage{algpseudocode}
\usepackage{matlab-prettifier}
\usepackage{graphicx}
\usepackage[export]{adjustbox}
%
% Basic Document Settings
%
\lstMakeShortInline[style=Matlab-editor]"

\topmargin=-1in
\evensidemargin=0in
\oddsidemargin=0in
\textwidth=6.5in
\textheight=9.0in
\headsep=0.25in

\linespread{1.1}

\rhead{\firstxmark}
\lfoot{\lastxmark}
\cfoot{\thepage}

\renewcommand\headrulewidth{0.4pt}
\renewcommand\footrulewidth{0.4pt}

%
% Homework Details
%   - Title
%   - Due date
%   - Class
%   - Section/Time
%   - Instructor
%   - Author
%

\newcommand{\hmwkTitle}{AMATH 482 Homework 5 : Background Subtraction in Video Streams}
\newcommand{\hmwkDueDate}{March 15, 2019}
\newcommand{\hmwkClassInstructor}{Professor Nathan Kutz}
\newcommand{\hmwkAuthorName}{\textbf{Skyler Hallinan}}

%
% Title Page
%

\title{
    \textmd{\textbf{\text{ } \hmwkTitle}}\\
}

\author{\hmwkAuthorName}
\date{}

\begin{document}
\maketitle

\section*{\fontsize{19}{15}\selectfont Abstract}
		We recorded several videos with a foreground object and a background object (a student (object) walking in various locations (background)), with subtle variations in each video, such as distance from object to the camera, as well as amount of noise in the data. We used the dynamic mode decomposition to separate the foreground and background from the videos, truncating our matrices by looking at energies captured by singular values. We also looked at the omega values from these decompositions to see how much of the background we were capturing with additional modes. We were able to successfully separate foreground and background from five different videos, even with high noise levels.
\section*{\fontsize{19}{15}\selectfont Introduction and Overview}
	We can think of complex systems as high-dimensional, nonlinear dynamical systems that exhibit many phenomena in both space and time. Even though these systems are complex, they can often by represented by low-dimensional spatio-temporal coherent structures. The dynamic mode decomposition allows us to discover dynamical systems from this otherwise high-dimensional data. It is equation-free and based on the underlying data; it's output, the decomposition of complex data into spatio-temporal coherent structures, is practical and useful. At the same time, it is simple, fast to execute (cost of algorithm is SVD), and does not make underlying assumptions about the data. At the core, the dynamic mode decomposition is a combination of spatial-dimensionality reduction techniques such as singular value decomposition and proper orthogonal decomposition and Fourier transforms in time. \\ \\
	We have recorded a set of five videos to analyze. The dynamic mode decomposition relies on having snapshots of data at $k$ points in time from a dynamical system; each frame in our video represents this data at each $t_k$. We will use the dynamic mode decomposition to separate the foreground from the background in our videos. We used a laptop to record our videos so that we could have a stable camera so that our background could be free of noise, and so that we could easily separate our foreground and background. These five videos have different levels of noise, distance from the foreground to background, and color difference between background and foreground. \\ \\
	The dynamic mode decomposition will return a low-rank approximate reconstruction of our data, which will represent the background of our video. By subtracting this low-rank approximation from the original data, we will be able to get the sparse reconstruction, which will represent the foreground object. Finally, we will use a residual matrix "R" to remove negative pixel values from our sparse matrix and compensate the low-rank matrix and sparse matrix accordingly.
\section*{\fontsize{19}{15}\selectfont Theoretical Background}
	The DMD, given a dynamical system and two sets of data $X$ and $X'$ where $\mathbf{x}_{k}^{\prime}=\mathbf{F}\left(\mathbf{x}_{k}\right)$ such that F is a map corresponding to time evolution, finds the eigendecomposition of the best-fit linear operator $A$ where $\mathbf{X}^{\prime} \approx \mathbf{A X} \Rightarrow \mathbf{A}=\mathbf{X}^{\prime} \mathbf{X}^{\dagger}$ As a result, the DMD modes are the eigenvectors of A, and each DMD mode corresponds to a particular eigenvalue (of A). This solution minimizes the error $\left\|\mathbf{X}^{\prime}-\mathbf{A} \mathbf{X}\right\|_{F}$ where $||\cdot||_F$ is the Frobenius norm denoted by $\|\mathbf{X}\|_{F}=\sqrt{\sum_{j=1}^{n} \sum_{k=1}^{m} X_{j k}^{2}}$. It does this by constructing an approximate locally linear dynamical system $\frac{d \mathbf{x}}{d t}=\mathcal{A} \mathbf{x}$ with initial condition $x(0)$ and solution of $\mathbf{x}(t)=\sum_{k=1}^{n} \phi_{k} \exp \left(\omega_{k} t\right) b_{k}=\Phi \exp (\Omega t) \mathbf{b}$, where $\phi_k$ and $\omega_k$ are the eigenvectors and eigenvalues of the matrix A, and $b_k$ is the coordinates of x(0) in the eigenvector basis. We solve  $\mathrm{A}=\exp (\mathcal{A} \Delta t)$ with $\mathbf{x}_{k}=\sum_{j=1}^{r} \boldsymbol{\phi}_{j} \lambda_{j}^{k} b_{j}=\boldsymbol{\Phi} \Lambda^{k} \mathbf{b}$, minimizing the least-squares trajectory of $x_k$. \\ \\
	The DMD approximates modes of the Koopman operator, a linear, infinite-dimensional operator that is a ''nonlinear dynamical system on the Hilbert space of measurement functions of the states''. This Koopman operator advances measurements along with the flow map $F$ as shown below.
\begin{align*}
K\ g &=g \circ \mathrm{F} \\
\Longrightarrow \quad K\ g\left(\mathbf{x}_{k}\right) &=g\left(\mathbf{F}\left(\mathbf{x}_{k}\right)\right)=g\left(\mathbf{x}_{k+1}\right)
\end{align*}
	The data needed for dynamical mode decomposition (DMD) is that from a dynamical system $\frac{d \mathbf{x}}{d t}=\mathbf{f}(\mathbf{x}, t ; \mu)$ where $\mathbf{x}(t) \in \mathbb{R}^{n}$ is a vector that represents the state of our dynamical system at time $t$, $\mu$ represents parameters of the system, and $f(\cdot)$ represents the dynamics. We can represent the continuous-time dynamics from the previous equation as a corresponding discrete-time representation where we sample at $\Delta t$ and denote the time in the equation as  $\mathbf{x}_{k}=\mathbf{x}(k \Delta t)$. Then, we have a discrete-time flow map described by $\mathbf{x}_{k+1}=\mathbf{F}\left(\mathbf{x}_{k}\right)$ and $\mathrm{y}_{k}=\mathrm{g}\left(\mathrm{x}_{k}\right)$ at $t_k$ values where  $k = 1,2,...n$ for n different times. The measurements equal the states in many applications, that is, $y_k = x_k$. In addition, our initial conditions are $x(t_0) = x_0$. Thus we have a set of governing equations and an initial condition. We represent our data into two large snapshots $X$ and $X'$, where $X$ is the column-wise data from $x_1$ to $x_{m-1}$ and $X'$ is the column-wise data from $x_2$ to $x_m$ such that $\mathrm{X}^{\prime} \approx \mathrm{AX}$ and so that we can calculate the best-fit $A$ matrix via the pseudo inverse  $\mathbf{A}=\mathbf{X}^{\prime} \mathbf{X}^{\dagger}$, which minimizes the error as stated above. We assume this data is high-dimensional so the size of our snapshot is much larger than the number of snapshots. This allows us to project our data onto POD modes to solve for A without explicitly computing it.\\ \\
We calculate the DMD in the following way: We first take the singular value decomposition of $X$ such that $\mathrm{X}=\mathrm{U} \Sigma \mathrm{V}^{*}$, where $U \in \mathbb{C}^{n \times r}, \Sigma \in \mathbb{C}^{r \times r}, \mathbf{V} \in \mathbb{C}^{m \times r}$, and we define $r$ to be the rank of the reduced SVD approximation of X. We then determine the matrix A from earlier  by computing the $r\times r$ projection of $A$ onto POD modes via $\tilde{\mathrm{A}}=\mathrm{U}^{\prime} \mathrm{AU}=\mathrm{U} \mathrm{X}^{\prime} \mathrm{V} \Sigma^{-1}$. We see that this $\tilde{A}$ defines the low-dimensional linear model of our dynamical system on POD coordinates: $\tilde{\mathbf{x}}_{k+1}=\tilde{\mathbf{A}} \tilde{\mathbf{x}}_{k}$. Next, we calculate the eigendecomposition of $\tilde{A}$, with $\tilde{\mathrm{A}} \mathrm{W}=\mathrm{W} \Lambda$, where our $W$ columns represent eigenvectors and $\Lambda$ contains a diagonal matrix of our eigenvalues. Finally, we reconstruct the eigendecomposition by creating a matrix where the columns represent exact DMD modes: $\Phi=\mathbf{X}^{\prime} \mathbf{V} \Sigma^{-1} \mathbf{W}$. We can then, afer declaring  $\omega_{k}=\ln \left(\lambda_{k}\right) / \Delta t$, project the future solution at times in the future using: $\mathbf{x}(t) \approx \sum_{k=1}^{r} \phi_{k} \exp \left(\omega_{k} t\right) b_{k}=\Phi \exp (\Omega t) \mathbf{b}$. This can be interpreted as the least-square fit/regresion of a best-fit linear dynamical system $\mathbf{x}_{k+1}=\mathbf{A} \mathbf{x}_{k}$. Once again, this is constructed so that we minimize the norm $\left\|\mathrm{x}_{k+1}-\mathrm{Ax}_{k}\right\|_{2}$ across all timepoints/snapshots. Finally, we calculate the initial coefficient values $b_k$ by considering our initial snapshot $x_1$ at time $t_1 = 0$. Solving, we find the pseudo inverse (since the eigenvector matrix is not square) to get our $b$:  $\mathbf{b}=\mathbf{\Phi}^{\dagger} \mathbf{x}_{1}$\\ \\
We see that we can use the DMD spectrum of frequencies to subtract background modes. We create two data frames from our initial: One from the beginning to the $last-1$ frame, and one from the 2nd to $last$ frame, where each of these represent a data snapshot: the data matrix $X$, and the shifted data matrix $X'$ . We use the first data frame for our SVD and corresponding analysis. We assume that $\omega_{p},$ where $p \in\{1,2, \ldots, \ell\},$ satisfies $\left\|\omega_{p}\right\| \approx 0,$ and that $\left\|\omega_{j}\right\| \forall j \neq p$ is bounded away from zero, giving us the following:
\begin{align*}
\mathrm{X}_{\mathrm{DMD}}=\underbrace{b_{p} \varphi_{p} e^{\omega_{p} t}}_{\text { Background Video }}+\underbrace{\sum_{j \neq p} b_{j} \varphi_{j} e^{\omega_{j} t}}_{\text { Foreground Video }}
\end{align*}
This will produce a $X_{DMD}$ matrix the same dimensions as our original $X$ matrix, although each term is complex. We will solve for the various parameters using the procedure outlined previously. \\ \\
After we get this DMD matrix $\mathbf{X}_{\mathrm{DMD}}^{\text { Low-Rank }}=b_{p} \varphi_{p} e^{\omega_{p} t}$, since, we know $\mathbf{X}=\mathbf{X}_{\mathrm{DMD}}^{\mathrm{Low}-\mathrm{Rank}}+\mathbf{X}_{\mathrm{DMD}}^{\mathrm{Sparse}}$, we can solve for the sparse matrix $\mathbf{X}_{\mathrm{DMD}}^{\text { Sparse }}=\sum_{i \neq n} b_{j} \varphi_{j} e^{\omega_{j} t}$ via  $\mathbf{X}_{\mathrm{DMD}}^{\mathrm{Sparse}}=\mathbf{X}-\left|\mathbf{X}_{\mathrm{DMD}}^{\mathrm{Low}-\mathrm{Rank}}\right|$.
Finally, with these low-rank and sparse DMD approximations, of the image, since we subtracted the low-rank approximation from our initial data to obtain our sparse approximation reconstruction, we may obtain negative values in our sparse matrix. This does not make sense because pixel values cannot be negative. We put these negative values into a residual $n\times m$ matrix $R$ and then create our low-rank and sparse reconstructions as follows:
\begin{align*}
\mathbf{X}_{\mathrm{DMD}}^{\mathrm{Low}-\mathrm{Rank}} \leftarrow \mathbf{R}+\left|\mathbf{X}_{\mathrm{DMD}}^{\mathrm{Low}-\mathrm{Rank}}\right| \\
\mathbf{X}_{\mathrm{DMD}}^{\mathrm{Sparse}} \leftarrow \mathbf{X}_{\mathrm{DMD}}^{\mathrm{Sparse}}-\mathbf{R}
\end{align*}
This allows us for us to account for the magnitudes of complex values from the DMD reconstruction, while preventing pixel intensities from being below zero, making sure that both approximate low-rank and sparse DMD reconstructions are real-valued, and maintaining that $\mathbf{X}=\mathbf{X}_{\mathrm{DMD}}^{\mathrm{Low}-\mathrm{Rank}}+\mathbf{X}_{\mathrm{DMD}}^{\mathrm{Sparse}}$. Adding these "R" values back will not necessarily make our DMD better at separating foreground and background, but it will make sure that we have the non-negative conditions that we discussed earlier.
\\ \\ The DMD is a way to make a low-rank approximation of linear maping that approximate the nonlinear dynamics of a given system. It is simple and equation-free, and can be used for future state prediction for any time $t$. The DMD does have its limitations. Like PCA, it is based on a singular value decomposition that extracts correlated patterns from the data. Like SVD, it cannot handle efficiently handle invariances in the data. Translational and rotational invariances as well as transient time phenomena are not characterized well by SVD and therefore DMD.
\section*{\fontsize{19}{15}\selectfont Algorithm Development}
We used "VideoReader" to create a "vid" object to load in our mp4 videos. We created a $dt$ using $\frac{1}{\text{Framerate}}$ of the loaded video, and a time vector by spacing from 0 to the "Length" of the video in intervals of "dt". Using a loop similar to the one used in Homework 3, we loaded in each frame of the video. We converted this image matrix to grayscale (to allow us to do computation on it) using "rgb2gray" and converted its values to double (from uint8). To reduce computational time needed, we scaled the image down by a factor of $\frac{1}{4}$ using "imresize" so that the final image was $180\times320$. We used "reshape" to convert this into a column, and added it to our data matrix. We repeated this at all time points, effectively creating a data matrix where each column represented a grayscale image at a specific point in time, in chronological order from left to right. \\ \\
	Next, we split this data matrix into two data matrices, "X1" and "X2" , where X1 represented the snapshots of data from frame 1 to the last -1 frame, while the second matrix represented the data from snapshot frame 2 to the last frame. We created these two vectors this way because our goal was to determine how to get from one snapshot "X1" to the next snapshot "X2" using some linear transformation "A", and these could be approximations of $X$ and $X'$. We then took the singular value decomposition of "X1" using "svd(X1)". We plotted the diagonal values of the "s" matrix using "diag" divided by the sum of the diag values to see how much energy each singular value could capture in this system. We looked at this graph and determined how many significant modes there were, that is, how many modes that captured a significant proportion of the energy of the system, and saved this value as "r". \\ \\
We then truncated our "u", "s", and "v" matrices from our "svd" to contain only 1:"r" of its original row or column values (For example, we set "U = u(:,1:r)"), effectively only pulling out "r" modes from these dataframes. We then transformed to the low-rank domain by constructing a new variable "Atilde" which was equal to "U_r' * X2 * V_r / S_r". We took the eigen value decomposition of "Atilde" using "[W,D] eig" to get its eigen vectors and eigen values. This was essentially taking the approximate low-rank subspace "Atilde" and doing an eigen decomposition to diagonalize frequencies. We created our DMD modes using "Phi = X2 * V_r / S_r * W_r", and looked at the spectra modes by finding the eigen values (by taking the "diag" eigen values "D" matrix). We were able to obtain our "omega" values by taking the log of these eigen values and dividing them by our "dt". We then plotted these (after determining an ideal number of modes to use). \\ \\
We then calculated our DMD solution. We set our initial condition to be the first frame of our "X1" matrix (the first column). We then solved for our "b" value using "b= Phi\ x1". We then created an empty "zeros" matrix of size "(r, length(t))", and iteratively created a time dynamics matrix, which we populated at each time point using "(b.*exp(omega*t(iter)))". We then multiplied "Phi" by this time dynamics matrix to get our low-rank DMD approximation "Xdmd". \\ \\
Our original data matrix "X1" is the sum of the low-rank and the sparse matrix approximations, so we subtracted "Xdmd" from "X1" to get our sparse DMD reconstruction approximation. We then created our R matrix to account for the residual negative values of our sparse matrix. We did this by filtering our sparse matrix so that it only contained the negative values, and we set "R" to this. To make a more accurate low-rank DMD and sparse DMD approximation, we added "R" to the absolute value "abs" of our low rank matrix to get our final low-rank DMD approximation. To get our final sparse DMD approximation, we subtracted "R" from our original sparse matrix. \\ \\
We then arbitrarily picked a frame number, and took one column from the sparse matrix, the low-rank matrix, the sparse - "R" matrix, the "abs("low-rank") + R" matrix, the original data matrix "X1", and the "X" total reconstruction of low-rank + sparse.  We reshaped this column, which corresponded to an image at a time point, into an image, and used "imshow" to visualize how they were different. We also used "implay" to compare the entire matrices by playing them as videos, but we did not include those in this paper. We repeated the above steps for all five videos.
\section*{\fontsize{19}{15}\selectfont Computational Results}
\begin{figure}[H]
\begin{center}
\includegraphics[width = 12cm]{energy1}
\caption{\label{fig:scaled_diss} Plots of (normalized) energy captured by singular values (left) and omega values (right) (Video 1)}
\end{center}
\end{figure}
\begin{figure}[H]
\begin{center}
\includegraphics[width = 16cm]{vid1image}
\caption{\label{fig:scaled_diss} Video separation and reconstruction (Video 1)}
\end{center}
\end{figure}
\textbf{Video 1: Ideal Case} \\
In this video, we recorded from the laptop that was sitting on a laptop, to hold a background constant. The background object was the ladder and wall of a dorm room, while the foreground object was a student walking throughout the room at various distances. \\ \\
We see that our energy diagram of the singular values  (Figure 1) shows that about three modes are significant. We set our "r" value to be 3, so we truncated our "u", "s", and "v" matrices from 1:3 from "svd(X1)" and calculated the low-rank approximation. We also see that the absolute value of our first "omega" value is extremely close to 0, while the next two are a bit higher. The "omega" close to 0 indicates that it is a background mode. We see that although we used an "r = 3", that is, we assumed three significant modes to determine our background DMD low-rank matrix, we could probably achieve similar results using only one mode. This is something that we computationally tested and determined to be true. \\ \\
From Figure 2, we see that our sparse reconstruction, which should show the foreground, looks relatively similar for both cases where we incorporated our "R" matrix and when we did not. In both cases, the foreground (the student) is relatively visible and is clearly separated from the background. In terms of the low-rank reconstruction, the background, adding "R" seems to have made the DMD reconstruction worse. There is a definite presence of the object (the student) in the form of floating clothes, although some of the foreground object is now transparent. We see however that the low-rank reconstruction that does not use "R" does a great job of only showing the background. Finally, our total reconstruction, which is the sum of the low rank and the sparse approximations, gave us an image that looked like the original video. Thus, our DMD separation has worked.
\begin{figure}[H]
\begin{center}
\includegraphics[width = 12cm]{energy2}
\caption{\label{fig:scaled_diss} Plots of  (normalized) energy captured by singular values (left) and omega values (right)(Video 2)}
\end{center}
\end{figure}
\begin{figure}[H]
\begin{center}
\includegraphics[width = 16cm]{vid2image}
\caption{\label{fig:scaled_diss} Video separation and reconstruction (Video 2)}
\end{center}
\end{figure}
\textbf{Video 2: Long Distance} \\
In this video, our object was still the student. The student walked around in a room (with the walls the background) at farther distances away than they did in Video 1. This was to test the effect of increasing distance - Would the student become part of the background because motion is harder to perceive farther away? \\ \\
We see that our singular value energy spectrum (Figure 3) shows that there were two dominant modes that accounted for a large portion of the energy in the system, so once again we set our "r" to this value and truncated accordingly. When we plotted our "omega" values, we saw that once again only one was very low - which showed that we realistically only needed one mode to create the background approximation using DMD (even though we used two here). \\ \\
In terms of accuracy of our DMD reconstructions, once again, the two sparse reconstructions did about the same at extracting the object. In this case, the foreground student object is a little more faint, but the sparse reconstruction did a good job of avoiding to pick up background. We see
that once again the low-rank reconstruction without using "R" created a better background image than did the low-rank approximation that used "R". Once again, there is a small trace of the foreground image in the low-rank reconstruction that has "R" addition. Finally, our total reconstruction looks accurate again.
\begin{figure}[H]
\begin{center}
\includegraphics[width = 12cm]{energy3}
\caption{\label{fig:scaled_diss} Plots of (normalized) energy captured by singular values (left) and omega values (right) (Video 3)}
\end{center}
\end{figure}
\begin{figure}[H]
\begin{center}
\includegraphics[width = 16cm]{vid3image}
\caption{\label{fig:scaled_diss} Video separation and reconstruction (Video 3)}
\end{center}
\end{figure}
\textbf{Video 3: Brighter Background} \\
In this video, we changed the scenario by increasing the brightness of the background to increase visual separation between the foreground and background object. We achieved this by filming towards a light and a window, both of which contributed light. Our foreground object was the same distance relatively as in Video 1 (ideal case). \\ \\
We see via Figure 5 that our energy captured by the singular values makes it harder to distinguish how many modes truly can represent this system, since it goes down like a gradient. However, we chose to use 5 modes for this video as the first 5 modes all captured somewhat significant portions of the energy of the system. The plot of omega values shows how small the first omega is - It is so low that it cannot be visualized on the graph at the same scale that the other four omega values are at! Once again, it seems that we could have done this reconstruction with only one mode instead of five. \\ \\
We see from our reconstructions that the sparse reconstruction in both cases does a good job of clearly showing the moving foreground object, although it does pick up a little bit of the background (to the left). Once again, the low-rank reconstruction without R addition does a great job of showing the background, while the low-rank reconstruction with R addition had trace remnants of the object. Finally, the total reconstruction is good once again.

\begin{figure}[H]
\begin{center}
\includegraphics[width = 12cm]{energy4}
\caption{\label{fig:scaled_diss} Plots of (normalized) energy captured by singular values (left) and omega values (right) (Video 4)}
\end{center}
\end{figure}
\begin{figure}[H]
\begin{center}
\includegraphics[width = 16cm]{vid4image}
\caption{\label{fig:scaled_diss} Video separation and reconstruction (Video 4)}
\end{center}
\end{figure}
\textbf{Video 4: Light Noise} \\ \\
In this video, we changed the scenario by adding noise to the system. Previously, the laptop was placed on a table while recording, and was therefore stationary and had virtually no noise introduced due to shaky camera. However, in this case, we had a person hold the laptop as still as they could. Although humans can hold things pretty steady, there will still be some slight shaking, and there will be much more noise introduced than if the recording device was sitting on a table. We introduced this light noise to see what it's effects on the reconstruction were. \\ \\
From our singular value plot, we saw approximately three significant modes that we used. Once again, our plot of "omega" values showed that only one of these modes was relatively low and contributed to the background approximation. Nevertheless, we continued on with three modes. \\ \\
We see that even with a slight amount of noise introduced, both sparse reconstructions do not pick up the foreground object as well as they did in the previous three scenarios. Both sparse reconstructions show only slight traces of the object, and pick up some of the background to the right. Although both of the sparse reconstructions are similar, we do see major differences in the low-rank reconstructions. In terms of the low-rank reconstruction without "R" addition, we see that the background looks blurry, which is probably a result of the slight noise introduced; however, we see that there is still a complete removal of the foreground object from the background. The low-rank reconstruction with "R" addition is less blurry, but does a poor job of removing the foreground object. Both the foreground and the background are both clearly visible still, so it does not do a good job. Once again, our total reconstruction matches our original video.
\begin{figure}[H]
\begin{center}
\includegraphics[width = 12cm]{energy5}
\caption{\label{fig:scaled_diss} Plots of (normalized) energy captured by singular values (left) and omega values (right) (Video 4)}
\end{center}
\end{figure}
\begin{figure}[H]
\begin{center}
\includegraphics[width = 16cm]{vid5image}
\caption{\label{fig:scaled_diss} Video separation and reconstruction (Video 3)}
\end{center}
\end{figure}
\textbf{Video 5: Heavy Noise} \\
In this video, we introduced heavy noise to our video via intentional, large-scale shaking of the laptop (recording device). This video was recorded in the exact location of Video 4 (Light Noise) using the same foreground object (student walking around), so that we could directly compare how light and heavy noise affect the decompositions. \\ \\
We approximated that there were two significant modes from our singular value decomposition and subsequent energy plot. However, the plot of "omega" values once again confirmed that only one of these modes was important for background extraction; it was so small that it could not be visualized on the same scale as the other mode. \\ \\
In terms of image reconstruction, we see that both of our sparse reconstruction did worse at extracting the foreground, compared to Video 4 (Light Noise). The sparse reconstruction shows more of the buildings to the right, which are part of the background. Once again, although the low-rank approximation without "R" addition is significantly blurry, it successfully removes the object image from it. While the low-rank reconstruction with "R" addition has a more clear image, it leaves a significant part of the foreground object. Finally, we see that the total reconstruction matches the original video well.

\section*{\fontsize{19}{15}\selectfont Summary and Conclusions}
We recorded five sets of videos, with a moving foreground, and a static background. We performed the DMD algorithm to create a low-rank and sparse approximation of this data which represented the foreground and background of the object. We were able to do this successfully in a variety of scenarios, including two where we introduced noise to the videos. \\ \\
Overall, although the energy plots showed that there were some modes that may be significant, our plots of the actual "omega" values showed that only one mode was important to determining the background in every case. These omega values represent the eigen vectors, and a lower value means that it captures more of the background. Therefore we could have theoretically done all of these approximations by taking only the most significant mode from our singular value decomposition. We also see that across videos, it seems that videos with lower omega values had better background approximations; our videos with high noise had blurry backgrounds but still had relatively good foreground object and background separation. In addition, we found that our "R" matrix did not do particularly well - although it did not differ very much in the sparse reconstructions, when we added it to our low-rank reconstructions (represent background), it added parts of the object back, which decreased the accuracy of the foreground and background extraction. \\ \\
We did record a video in which the camera panned to somewhat follow the foreground's movement, effectively changing the background, albeit slightly. However, we did not do analysis on this video as we realized afterwards that this does not create a distinguished foreground and background; in this video, both are moving as time goes on, making them effectively both foreground objects. \\ \\
One item we did not visualize was how changing the number of modes affected the difference between the low-rank and sparse reconstructions with and without R. We found that increasing the number of modes used still lead to better background and foreground separation when we didn't use R, compared to reconstructions where we did use the R matrix. \\ \\
One last thing that we could have changed is the length of the video. It may have been harder to separate the foreground from the background if there was less data, i.e, if there were fewer frames in the data. It would be interesting to experimentally test how length of the video changes the decomposition and reconstruction of the videos. \\ \\
\pagebreak

\section*{\fontsize{19}{15}\selectfont Appendix A}
\subsection*{MATLAB functions used and implementation}
"abs(X)" : Returns the absolute value of every element in "X". We used this when calculating our DMD's approximate sparse reconstruction. \\ \\
"diag(X)" : Returns a column vector of the diagonal values of a matrix. We used this to generate our variance numbers from SVD. \\ \\
"double(X)" : Converts elements of "X" into type "double". We used this to convert "uint8" values into type "double" for analysis. \\ \\
"[W,D] = eig(A)" : Returns diagonal matrix "D" of eigenvalues and matrix "W" whose columns are the corresponding right eigenvectors, so that "A*V" = "V*D". We used this to find the eigen decomposition of our "Atilde" matrix. \\ \\
"imread(file)": Reads in an image file as a 3D or 2D matrix (based on if the images is color or grayscale). We used this to load in the Yale Faces for part 1. \\ \\
"implay(X)" : If "X" is a 3D matrix, plays "X" as a video, where each "(:,:,i)" slice is a timepoint to extract a matrix (therefore a frame) to display. We used this to compare original video to the separated ones. \\ \\
"imshow(X)" : Displays the matrix "X" as an image. We used this to compare frames the original video to the DMD's sparse and low-rank reconstruction frames. \\ \\
"length(A)" : Returns the length of the vector A. We used this as a parameter to reshape the data from columns back to matrices. \\ \\
"reshape(X,sz)" : Reshapes the array or matrix "X" to the size "sz". We used this to resize our image matrices from video frames into column vectors at each timepoint for data storage. We also used this to convert these column vectors back to a matrix, to view the image. \\ \\
"size(X)" : Returns the dimensions of the matrix "X". We used this to calculate the size of our matrix in our DMD algorithm. \\ \\
"[u,s,v] = svd(X)" : Returns the $U$, $S$, and $V$ matrices corresponded with the singular value decomposition of "X". We used this in our initial steps of the DMD algorithm. \\ \\
"uint8(X)" : Converts elements of "X" into type "uint8". We used this to convert our double values to uint8 to view them using "imshow". \\ \\
"VideoReader(mov.mp4)" : Creates a "VideoReader" object, with information such as frame rate and duration. We used this to load in our video files for analysis, and create our "dt" and "t" vectors. \\ \\
"zeros(X,Y)" : Creates a matrix of zeros of a specified size "X" and "Y". We used this to create an empty time dynamics matrix to iteratively fill in through a loop. \\ \\

\section*{\fontsize{19}{15}\selectfont Appendix B}
\subsection*{MATLAB code}
\begin{lstlisting}[style=Matlab-editor]
clear all; close all; clc;

clear all; close all; clc;

%% Read in files (vid 1)
%Replace this with whatever movie to load in
vid1 = VideoReader('mov5.mp4');
dt = 1/vid1.Framerate;
t = 0:dt:vid1.Duration;
vidFrames = read(vid1);
numFrames = get(vid1,'numberOfFrames');
%% Load
for k = 1 : numFrames
    mov(k).cdata = vidFrames(:,:,:,k);
    mov(k).colormap = [];
end

fdata = [];
for j=1:numFrames
    X=frame2im(mov(j));

    fdata = [fdata, reshape(double(rgb2gray(imresize(X, 0.25))), [180*320,1])];
    %imshow(imresize(X, 0.25)); drawnow
end

%%  DMD
%%%%%% body of DMD %%%%%%%%%%
X1 = fdata(:,1:end-1);
X2 = fdata(:,2:end);

[U,S,V] = svd(X1, 'econ');

subplot(1,2,1)
plot(diag(S)/sum(diag(S)), 'bo', 'Linewidth', 2, 'MarkerSize', 10, 'MarkerFaceColor', 'g');
title("Energies of Singular Values", 'Fontsize', 24); ylabel("Energy Captured"); xlabel("Singular Values");
grid on

r = 2;

U_r = U(:, 1:r); % truncate to rank-r
S_r = S(1:r, 1:r);
V_r = V(:, 1:r);
Atilde = U_r' * X2 * V_r / S_r; % low-rank dynamics
[W_r, D] = eig(Atilde);
Phi = X2 * V_r / S_r * W_r; % DMD modes

lambda = diag(D); % discrete-time eigenvalues
omega = log(lambda)/dt; % continuous-time eigenvalues

subplot(1,2,2)
bar(abs(omega), 'm')
title("Omega Values (Absolute Value)",'Fontsize', 24);
set(gca,'XTick',[]); xlabel("Omegas"); ylabel("Absolute Value of Omegas");
grid on
%% Compute DMD mode amplitudes b
x1 = X1(:, 1);
b = Phi\x1;

%% DMD reconstruction
mm1 = size(X1, 2); % mm1 = m - 1
time_dynamics = zeros(r, mm1);
t = (0:mm1-1)*dt; % time vector
for iter = 1:mm1,
time_dynamics(:,iter) = (b.*exp(omega*t(iter)));
end;
Xdmd = Phi * time_dynamics;

%% Create Sparse and Nonsparse

Xsparse = X1 - abs(Xdmd);

R = Xsparse.*(Xsparse<0);

X_bg = R + abs(Xdmd);
X_fg = Xsparse - R;

X_reconstructed =  X_fg + X_bg;

%% Display
temp1 = reshape(Xsparse, [180,320, length(t)]);
temp2 = reshape(Xdmd, [180,320,length(t)]);
temp3 = reshape(X_bg, [180,320,length(t)]);
temp4 =  reshape(X_fg, [180,320,length(t)]);
temp5 =  reshape(X_reconstructed, [180,320,length(t)]);
temp6 =  reshape(X1, [180,320,length(t)]);
temp7 = reshape(R, [180,320,length(t)]);

% implay(uint8(temp3));
% imshow(uint8(temp3(:,:,100)))
% imshow(uint8(temp3(:,:,100)))
% imshow(uint8(temp3(:,:,100)))

%% Plotting
figure()
framenum = 100;

subplot(2,3,1)
imshow(uint8(temp6(:,:,framenum)))
title("Original Video");

subplot(2,3,2)
imshow(uint8(temp1(:,:,framenum)))
title("Sparse Reconstruction (No R subtraction)");

subplot(2,3,3)
imshow(uint8(temp2(:,:,framenum)))
title("Low Rank Reconstruction (No R addition)");

subplot(2,3,6)
imshow(uint8(temp3(:,:,framenum)))
title("Low Rank Reconstruction (R addition)");

subplot(2,3,5)
imshow(uint8(temp4(:,:,framenum)))
title("Sparse Reconstruction (R subtraction)");

subplot(2,3,4)
imshow(uint8(temp5(:,:,framenum)))
title("Total Reconstruction (Sparse + Low Rank)");

sgtitle(strcat("Video 5 (Frame ", int2str(framenum), "): ", int2str(r)," Modes"), 'Fontsize', 20)
\end{lstlisting}
\end{document}


%!TEX program = xelatex
\documentclass[10pt, compress, handout]{beamer}
\usepackage[titleprogressbar]{../../cls/beamerthemem}

\setbeamertemplate{caption}[numbered]
\setbeamertemplate{theorems}[numbered]
\newcounter{example}
\resetcounteronoverlays{example}
\newtheorem{crl}{Corollary}[theorem]
\newtheorem{eg}[example]{Example}
\newtheorem*{solution*}{Solution}

\usepackage{booktabs}
\usepackage[scale=2]{ccicons}
\usepackage{minted}

\usepackage{cleveref}
\crefname{example}{Example}{Examples}

\usepgfplotslibrary{dateplot}

\usemintedstyle{trac}

\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
\resetcounteronoverlays{algorithm}

\usepackage{version}
% \excludeversion{proof}
% \excludeversion{solution*}

\usepackage{mathtools}
\usepackage{multicol}
\usepackage{qtree}

\usepackage{tikz}

\makeatletter
\def\old@comma{,}
\catcode`\,=13
\def,{%
    \ifmmode%
    \old@comma\discretionary{}{}{}%
    \else%
    \old@comma%
    \fi%
}
\makeatother

\title{CENG 4120 Tutorial of Week 02}
\subtitle{Boolean Functions}
\author{LI Haocheng}
\institute{Department of Computer Science and Engineering}

\begin{document}

\maketitle

\subsection{Shannon Expansion}

\begin{frame}[allowframebreaks]
\frametitle{Shannon Expansion}
\begin{theorem}
	\label{thm:shannon}
    If $f: \mathbb{B}^n \rightarrow \mathbb{B}$ is a Boolean function, then
    \begin{equation}
    	\forall (x_1, \ldots) \in \mathbb{B}^n,
    	f(x_1, \ldots) = x_1' f(0, \ldots) + x_1 f(1, \ldots).
    \end{equation}
\end{theorem}
\newpage

\textbf{Proof.} \begin{enumerate}
	\item Suppose $f$ is a \textit{constant function}, defined by $f(X) = b$ for all $X \in \mathbb{B}^n$,
	where $b$ is a fixed element of $\mathbb{B}$. Then \begin{equation}
		f(x_1, \ldots) = b = (x_1' + x_1) b = x_1'b + x_1b = x_1' f(0, \ldots) + x_1 f(1, \ldots).
	\end{equation}
	\item Suppose $f$ is a \textit{projection function}, defined by $f(X) = x_1$ for all $X \in \mathbb{B}^n$. Then \begin{equation}
	f(x_1, \ldots) = x_1 = 0 + x_1 \cdot 1 = x_1' \cdot 0 + x_1 \cdot 1 = x_1' f(0, \ldots) + x_1 f(1, \ldots).
	\end{equation}
	\item Suppose $f$ is a \textit{projection function}, defined by $f(X) = x_i$ for all $X \in \mathbb{B}^n$, where $i \in \{2, \ldots, n\}$. Then \begin{equation}
	f(x_1, \ldots) = x_i = (x_1' + x_1) x_i = x_1' x_i + x_1 x_i = x_1' f(0, \ldots) + x_1 f(1, \ldots).
	\end{equation}
	\newpage

	\item Suppose~\Cref{thm:shannon} to hold for Boolean functions $g$ and $h$, i.e., \begin{align}
		\begin{aligned}
		g(x_1, \ldots) & = x_1' g(0, \ldots) + x_1 g(1, \ldots),\\
		h(x_1, \ldots) & = x_1' h(0, \ldots) + x_1 h(1, \ldots).
		\end{aligned}
	\end{align}
	We show that~\Cref{thm:shannon} then holds
	for the $n$-variable functions $g + h$, $gh$, and $g'$.
	\begin{enumerate}
		\item \begin{align}
			\begin{aligned}
			(g + h)(x_1, \ldots) = & g(x_1, \ldots) + h(x_1, \ldots)\\
			= & (x_1' g(0, \ldots) + x_1g(1, \ldots)) + (x_1'h(0, \ldots) + x_1h(1, \ldots))\\
			= & x_1'(g(0, \ldots) + h(0, \ldots)) + x_1(g(1, \ldots) + h(1, \ldots))\\
			= & x_1'[(g + h)(0, \ldots)] + x_1[(g + h)(1, \ldots)].
			\end{aligned}
		\end{align}
		\item \begin{align}
			\begin{aligned}
			(gh)(x_1, \ldots) = & (g(x_1, \ldots))(h(x_1, \ldots))\\
			= & (x_1'g(0, \ldots) + x_1g(1, \ldots)) (x_1'h(0, \ldots) + x_1h(1, \ldots))\\
			= & x_1' (g(0, \ldots)h(0, \ldots)) + x_1(g(1, \ldots)h(1, \ldots))\\
			= & x_1'[(gh)(0, \ldots)] + x_1[(gh)(1, \ldots)].
			\end{aligned}
		\end{align}
		\item \begin{align}
			\begin{aligned}
			(g')(x_1, \ldots) = & (g(x_1, \ldots))'\\
			= & (x_1'g(0, \ldots) + x_1g(1, \ldots))'\\
			= & (x_1 + (g(0, \ldots))')(x_1' + (g(1, \ldots))')\\
			= & x_1'[(g')(0, \ldots)] + x_1[(g')(1, \ldots)] + [(g')(0, \ldots)][(g')(1, \ldots)]\\
			= & x_1'[(g')(0, \ldots)] + x_1[(g')(1, \ldots)].
			\end{aligned}
		\end{align}
	\end{enumerate}
\end{enumerate}
\end{frame}

\begin{frame}[fragile]
\frametitle{Min-term}
\begin{definition}
	Any function can be expressed as a \textbf{sum of products} of $n$ literals, called the
	\textbf{minterms} of the function, by recursive expansion. Alternatively, it can be represented
	as a \textbf{product of sums} of $n$ literals. called \textbf{maxterms}.
\end{definition}
\begin{columns}
	\column{.8\linewidth}
	\begin{eg}
		\label{eg:cofactor}
		Consider the majority function $f = ab + bc + ac$. \begin{enumerate}
			\item Write out its cofactor w.r.t. $a$ and $a'$.
			\item Re-write the function with the cofactors above.
			\item Write out its cofactor w.r.t. $ab$, $ab'$, $a'b$, and $a'b'$.
			\item Re-write the function with the cofactors above.
		\end{enumerate}
	\end{eg}
	\column{.3\linewidth}
	\begin{solution*}
		\begin{enumerate}
			\item<2-> $f_{a} = b + c, f_{a'} = bc$.
			\item<3-> $f = a(b + c) + a'bc$.
			\item<4-> $f_{ab} = 1, f_{ab'} = c, f_{a'b} = c, f_{a'b'} = 0$.
			\item<5> $f = ab + ab'c + a'bc$.
		\end{enumerate}
	\end{solution*}
\end{columns}
\end{frame}

\begin{frame}[fragile]
\frametitle{Unate}
\begin{definition}
    A function $f(x_1, x_2, \ldots, x_i, \ldots, x_n)$ is (positive/negative) \textbf{unate in variable} $x_i$, if $f_{x_i} \supseteq f_{x_i'} (f_{x_i} \subseteq f_{x_i'})$. Otherwise it is binate (or mixed) in that variable.

    A function is (positive/negative) \textbf{unate} if it is (positive/negative) unate in all
    support variables. Otherwise it is binate (or mixed).

    Note that stating $f_{x_i} \supseteq f_{x_i'}$ is equivalent to say that the set of minterms of $f_{x_i}$ includes the set of minterms of $f_{x_i'}$, or $f_{x_i} \ge f_{x_i'}$, for all possible assignments to the
    other variables $x_j \colon j \ne i, j = 1, 2, \ldots, n$.
\end{definition}
\begin{eg}
	Is $f = a + b + c'$ is positive or negative unate w.r.t. variable $a$, $b$, or $c$?
\end{eg}
\onslide<2>\begin{solution*}
	$f$ is positive unate w.r.t. variable $a$ and $b$.
	$f$ is negative unate w.r.t. variable $c$.
\end{solution*}
\end{frame}

\begin{frame}[fragile]
\frametitle{Quantification}
\begin{eg}
    Consider the majority function $f = ab + bc + ac$ with $f_{a} = b + c$ and $f_{a'} = bc$ in~\Cref{eg:cofactor}. Write out \begin{enumerate}
    	\item the Boolean difference w.r.t. $a$,
    	\item the universal quantification of $f$ w.r.t. $a$, and
    	\item the existential quantification of $f$ w.r.t. $a$.
    \end{enumerate}
\end{eg}
\begin{solution*}
    \begin{enumerate}
        \item<2-> $\frac{\partial f}{\partial a} = bc' + b'c$.
        \item<3-> $(\forall a~f) = bc$.
        \item<4> $(\exists a~f) = b + c$.
    \end{enumerate}
\end{solution*}
\end{frame}

\plain{Questions?}

\end{document}


\chapter{Off-policy methods with approximation}

\section{Summary}

In the tabular case on-policy methods with translate very easily to off-policy methods. This is not the case with the semi-gradient methods, the off-policy methods need \textbf{special care as converge is much harder to reach}.

The challenge of off-policy methods is split into 2 parts:
\begin{enumerate}
	\item update target
	\item change update distribution
\end{enumerate}

Some terminology used in this chapter:
\begin{itemize}
	\item $b$: behavior policy
	\item $\pi$: target policy
	\item $\hat{v} \approx v_{\pi}$
	\item $\hat{q} \approx q_{\pi}$
\end{itemize}

\subsection{Semi-Gradient methods}
Just as with the tabular algorithms we define the per-step importance sampling ratio as equation~\ref{eq:per-step importance sampling ratio}. The TD(0) algorithm then becomes equation~\ref{eq:semi-gradient TD(0) off-policy state-value update} and expected SARSA becomes~\ref{eq:semi-gradient expected sarsa off-policy action-value update}.

\begin{equation}
\rho_t = \rho_{t:t} = \frac{\pi(A_t|S_t)}{b(A_t|S_t)}
\label{eq:per-step importance sampling ratio}
\end{equation}

\begin{equation}
\begin{split}
w_{t+1} & = w_t + \alpha \rho_t \nabla \hat{v}(S_t, w_t) \\
\delta_t & = R_{t+1} + \gamma \hat{v}(S_{t+1}, w_t) - \hat{v}(S_t, w_t) \\
\delta_t & = R_{t+1} + \bar{R}_t + \hat{v}(S_{t+1}, w_t) - \hat{v}(S_t, w_t) \\
\end{split}
\label{eq:semi-gradient TD(0) off-policy state-value update}
\end{equation}

Expected sarsa without approximation doesn't need the importance sampling as it check's all future actions. The \textbf{approximation} however needs to take into account the \textbf{contribution of a certain state-action}. This is not yet handled here.

\begin{equation}
\begin{split}
w_{t+1} & = w_t + \alpha \rho_t \nabla \hat{q}(S_t, A_t, w_t) \\
\delta_t & = R_{t+1} + \gamma \sum_{a} p(a|S_{t+1})\hat{q}(S_{t+1}, a, w_t) - \hat{q}(S_t, w_t) \\
\delta_t & = R_{t+1} + \bar{R}_t + \gamma \sum_{a} p(a|S_{t+1})\hat{q}(S_{t+1}, a, w_t)  - \hat{v}(S_t, w_t)
\end{split}
\label{eq:semi-gradient expected sarsa off-policy action-value update}
\end{equation}

The n-step sarsa can be generalized in the same way as illustrated by equation~\ref{eq:n-step SARSA semi-gradient}.

\begin{equation}
\begin{split}
w_{t+n} & = w_{t+n-1} + \alpha \rho_{t+1} ... \rho_{t+n-1}\Big[ G_{t:t+n}-\hat{q}(S_t, A_t, w_{t+n-1}) \Big] \nabla \hat{q}(S_t, A_t, w_{t+n-1})\\
G_{t:t+n} & = R_{t+1} + ... \gamma^{n-1}R_{t+n} + \gamma^n \hat{q}(S_{t+n}, A_{t+n}, w_{t+n-1}) \\
G_{t:t+n} & = R_{t+1} - \bar{R}_{t} + ... R_{t+n} - \bar{R}_{t+n-1} +  \hat{q}(S_{t+n}, A_{t+n}, w_{t+n-1}) \\
\end{split}
\label{eq:n-step SARSA semi-gradient}
\end{equation}

The tree backup algorithm doesn't need importance sampling and can be converted into a offline version (equation~\ref{eq:semi-gradient n-step tree backup algorithm offline}).

\begin{equation}
\begin{split}
w_{t+n} & = w_{t+n} + \alpha \Big[ G_{t:t+n} - \hat{q}(S_t, A_t, w_{t+n-1}) \nabla \hat{q}(S_t, A_t, w_{t+n-1}) \Big] \\
G_{t:t+n} & = \hat{q}(S_t, A_t, w_{t-1}) + \sum_{k=t}^{t+n-1} \delta_k \prod_{i=t+1}^{k}\gamma\pi(A_i|S_i) \\
\delta_t & = R_{t+1} + \gamma \sum_{a} p(a|S_{t+1})\hat{q}(S_{t+1}, a, w_t) - \hat{q}(S_t, w_t) \\
\delta_t & = R_{t+1} + \bar{R}_t + \gamma \sum_{a} p(a|S_{t+1})\hat{q}(S_{t+1}, a, w_t)  - \hat{v}(S_t, w_t) \\
\end{split}
\label{eq:semi-gradient n-step tree backup algorithm offline}
\end{equation}

\subsection{Off-policy divergence}
Bairds counter example illustrated in figure~\ref{fig:bairds counter example} will diverge on some off-policy methods. Exercise 11.3 uses Q-learning, and the weights diverge. It does converge with on-policy methods, so off-policy methods need a bit of special care.

\begin{itemize}
	\item $b$: dashed lines p=6/7, solid lines p=1/7
	\item $\pi$: always solid lines
	\item reward=0 on all transitions
	\item discount rate is $\gamma=0.99$
\end{itemize}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}

	\node[circle, draw] at (0, 0) (a) {$2w_1 + w_8$};
	\node[circle, draw] at (3, 0) (b) {$2w_2 + w_8$};
	\node[circle, draw] at (6, 0) (c) {$2w_3 + w_8$};
	\node[circle, draw] at (9, 0) (d) {$2w_4 + w_8$};
	\node[circle, draw] at (12, 0) (e) {$2w_5 + w_8$};
	\node[circle, draw] at (15, 0) (f) {$2w_6 + w_8$};

	\node[circle, draw] at (7.5, -4) (g) {$w_7 + 2w_8$};
	\node[] at (7.5, 1) (gup) {};

	\draw[] [->, auto] (a) to node {} (g);
	\draw[] [->, auto] (b) to node {} (g);
	\draw[] [->, auto] (c) to node {} (g);
	\draw[] [->, auto] (d) to node {} (g);
	\draw[] [->, auto] (e) to node {} (g);
	\draw[] [->, auto] (f) to node {} (g);

	\draw[dashed] [<->, auto, bend left] (a) to node {} (gup);
	\draw[dashed] [<->, auto, bend left] (b) to node {} (gup);
	\draw[dashed] [<->, auto] (c) to node {} (gup);
	\draw[dashed] [<->, auto] (d) to node {} (gup);
	\draw[dashed] [<->, auto, bend right] (e) to node {} (gup);
	\draw[dashed] [<->, auto, bend right] (f) to node {} (gup);

	\draw[dashed] [-, auto] (gup) to node {} (g);

	\path (g) edge [loop below] node {} (g);

	\end{tikzpicture}
	\caption{Bairds counter example}
	\label{fig:bairds counter example}
\end{figure}

Some approximation methods do not extrapolate from the observed targets, and so are always stable. These are called \textbf{averagers}, and include weighted regression and nearest neighbors. But not the more popular methods such as tile coding and artificial neural networks. (ANNs)

\subsection{The Deadly Triad}
The dangerous instability of off-policy methods occurs when combining 3 elements, these are called \textbf{The Deadly Triads}.

\begin{itemize}
	\item \textbf{Function approximation}: A way of generalizing from a state-space.
	\item \textbf{Bootstrapping}: Update targets using estimates, and not exclusively using actual rewards.
	\item \textbf{Off-Policy training}: Training on a distribution of transitions other than that produces by the target policy.
\end{itemize}

All tree elements are often required, function approximation is needed to generalize on large problems. Bootstrapping speeds up convergence, and lowers memory usage. For example Monte-Carlo methods use no bootstrapping but require you to put the entire trajectory in memory. (which might be very big) And finally off-policy learning is required to learn specific aspects of a process, just like humans do.

\subsection{Linear Value-function Geometry}
The distance between two value functions $||v_1 - v_2||^2_\mu$ is defined using the norm of equation~\ref{eq:value function norm}, it takes into account the importance of the states using $\mu(s)$.

\begin{equation}
||v||^2_\mu = \sum_{s\in S}\mu(s)v(s^2)
\label{eq:value function norm}
\end{equation}

\begin{equation}
\Pi v = v_w \text{ where } w=\argmin_{w\in \Re^d}|| v-v_w ||^2_\mu
\label{eq:projection operator closes approximated value function}
\end{equation}

In Monte-Carlo estimation the value function is approximated using the operator $\Pi$ defined in equation~\ref{eq:projection operator closes approximated value function}. It minimizes the squared distance between the exact $v_\pi$ and approximated $v_w$. The TD methods however find an other value function using the bellman operator (equation~\ref{eq:bellman operator}).

\begin{equation}
v_\pi(s) = \sum_a \pi(a|s) \sum_{s',r}p(s', r)[r+\gamma v_{\pi}(s')]
\label{eq:chapter11 bellman equation}
\end{equation}
The bellman equation (equation~\ref{eq:chapter11 bellman equation}) leads to the Bellman error equation (equation~\ref{eq:bellman error}). The bellman error is the expected value of the TD-error.
\begin{equation}
\begin{split}
\delta_w(s) & = \Big( \sum_a \pi(a|s) \sum_{s', r} p(s', r | a, s)[r+\gamma v_w(s')] \Big) - v_w(s) \\
& = \EX\big[ R_{t+1} + \gamma v_w(S_{t+1}) - v_w(S_t) | S_t = s, A_t \sim \pi \big]\\
\end{split}
\label{eq:bellman error}
\end{equation}

The Mean Squared Bellman error $\overline{BE}$ of equation~\ref{eq: Mean Squared Bellman error} indicates the overall error in the value function.
\begin{equation}
\overline{\text{BE}}(w) = || \bar{\delta}_w ||^2_\mu
\label{eq: Mean Squared Bellman error}
\end{equation}
By applying the Bellman operator of equation~\ref{eq:bellman operator} to the value function repeatably $v_\pi = B_\pi v_\pi$ to the optimal approximated value function is obtained as the fixed point.
\begin{equation}
(B_\pi v)(s) = \sum_a \pi(a|s) \sum_{s', r} p(s', r | s, a)[r+\gamma v(s')]
\label{eq:bellman operator}
\end{equation}
For any approximated value function $v$, we define the \textbf{Mean Square Projected Bellman Error} as $\overline{PBE}$ (equation~\ref{eq:Mean Squared Projected Bellman Error}). With linear approximation there always exists a vector $w$ where $\overline{PBE}(w_{optimal})=0$. This however does not mean that the point $w_{optimal}$ is always stable under off-policy training.
\begin{equation}
\overline{PBE}(w) = || \Pi\bar{\delta}_w||^2_\mu
\label{eq:Mean Squared Projected Bellman Error}
\end{equation}

\subsection{Gradient Descent in the Bellman error}
Until now only the \textbf{Monte-Carlo methods were true gradient descent methods}, and so they always converge robustly. A naive approach would be to use the \textbf{expected value of the TD error} as an objective function as illustrated by equation~\ref{eq:mean squared TD error}.
\begin{equation}
\delta_t = R_{t+1} + \gamma \hat{v}(S_{t+1}, W_t) - \hat{v}(S_t, W_t)
\end{equation}
\begin{equation}
\begin{split}
\overline{TDE} & = \sum_{s\in S} \mu(s) \EX[\delta_t^2 | S_t, A_t \sim \pi] \\
& = \sum_{s\in S} \mu(s) \EX[\rho_t \delta_t^2 | S_t, A_t \sim b] \\
& = \EX_b[\rho_t \delta^2_t] \text{   (if $\mu$ is the distribution ecncountered under b)}
\end{split}
\label{eq:mean squared TD error}
\end{equation}

The SGD per-step update becomes equation~\ref{eq:SGD per-step update mean squared TD error}. This is the same as the semi-gradient TD update except for the last term $\gamma \nabla\hat{v}(S_{t+1},w_t)$. This algorithm is referred to as \textbf{the naive residual-gradient algorithm}. It converges robustly, but not necessary to an optimal value function.
\begin{equation}
\begin{split}
w_{t+1} & = w_t - \frac{1}{2}\alpha \nabla(\rho_t \delta_t^2) \\
& = w_t - \alpha \rho^t \delta_t \nabla\delta_t \\
& = w_t + \alpha \rho_t\delta_t(\nabla \hat{v}(S_t, w_t) - \gamma \nabla \hat{v}(S_{t+1}, w_t))
\end{split}
\label{eq:SGD per-step update mean squared TD error}
\end{equation}

The minimization of $\overline{TDE}$ doesn't always accurately predict. A minimization of the \textbf{bellman error} however should result in a accurate prediction.

\begin{equation}
\nabla \delta_t = \gamma \nabla\hat{v}(S_{t+1}, W_t) - \nabla\hat{v}(S_t, W_t)
\label{eq:gradient TD error}
\end{equation}

\begin{equation}
\begin{split}
w_{t+1} & = w_t - \frac{1}{2} \alpha \nabla (\EX_{\pi}[{\delta_t^2}]) \\
& = w_t - \frac{1}{2}\alpha\nabla(\EX_b[\rho_t\delta_t]^2) \\
& = w_t - \alpha \EX_b[\rho_t\delta_t] \nabla\EX_b[\rho_t\delta_t] \\
& = w_t - \alpha \EX_b[\rho_t(R_{t+1} + \gamma \hat{v}(S_{t+1}, w) - \hat{v}(S_t, w))] \EX_b[\rho_t\nabla\delta_t] \\
& = w_t - \alpha \Big[\EX_b[\rho_t(R_{t+1} + \gamma \hat{v}(S_{t+1}, w)] - \hat{v}(S_t, w))\Big] \\
& \qquad \Big[ \EX_b[\rho_t (\gamma \nabla\hat{v}(S_{t+1}, W_t) )] - \nabla\hat{v}(S_t, W_t) \Big]\\
& = w_t + \alpha \Big[\EX_b[\rho_t(R_{t+1} + \gamma \hat{v}(S_{t+1}, w)] - \hat{v}(S_t, w))\Big] \\
& \qquad \Big[  \nabla\hat{v}(S_t, W_t) - \gamma\EX_b[\rho_t \nabla\hat{v}(S_{t+1}, W_t) ]\Big]\\
\end{split}
\end{equation}

\textbf{note:} $\rho_t$ stays inside the expected values, as its the importance sampling and depending on $\sim b$.

This algorithm is called \textbf{residual-gradient}, the two expected values need seperate sample's in order for the algorithm not to be biased. There are two way's to make this algorithm work:
\begin{enumerate}
	\item If the next state is deterministic, then 2 next samples will be the same, and the naive approach will work.
	\item If there is a simulator simply roll-back on $S_{t+1}$ and get a second sample.
\end{enumerate}
The residual-gradient algorithm is a true SGD algorithm and converges robustly with both linear and non-linear function approximation. In the linear case the solution is optimal and unique.

However:
\begin{itemize}
	\item The residual-gradient algorithm is much slower then the semi-gradient
	\item If there is true function-approximation the minimum of $\overline{BE}$ is not the optimal value. (see drawing page 267 $\overline{BE}$ is not at same spot as $\Pi v_\pi$)
	\item "The bellman error is no learn-able" (next subsection)
\end{itemize}

\subsection{The Bellman Error is Not Learnable}
Typically the term "learnable" is used to define value's that can be learned in a polynomial amount of data.  It turns out that there are things that can't be "learned", no matter the amount of data. (example lower half of page 274).

So Some properties can be calculated using knowledge of the internal structure but can't be learned from the outside no matter how much data there is. The Bellman Error $\overline{BE}=||\delta_w||^2_\mu$ is such a property, as is the value error $\overline{VE}=||v_w - v_\pi||^2_\mu$, but the parameter $w$ that optimizes $\overline{VE}$ is observable.

\textbf{The mean square return error} defined in equation~\ref{eq:Mean Square Return Error} is the expectation under $\mu$. It is always observable, and it is equal to $\overline{VE}$ plus a variance term that does not depend on $w$. So the \textbf{optimum of $\overline{VE}$ for w must also be observable}.
\begin{equation}
\begin{split}
\overline{RE} & = \EX[(G_t - \hat{v}(S_t, w))^2] \\
& = \overline{VE}(w) + \EX[(G_t-v_{\pi}(S_t))^2] \\
\end{split}
\label{eq:Mean Square Return Error}
\end{equation}

In summary
\begin{itemize}
	\item $\overline{PBE}$ and $\overline{TDE}$ can be learned from the data. However there optimums differ from each other and $\overline{BE}$.
	\item $\overline{BE}$ is not at all learn-able, knowledge of the internal structure of the MDP is required to optimize $\overline{BE}$. The Residual gradient algorithm requires sampling from the same state twice. This obviously requires internal knowledge of the MDP.
\end{itemize}

\subsection{Gadient-TD Methods}

\section{Exercises}

\subsection{Exercise 11.1 page 259}
\textbf{Convert the equation of n-step off-policy TD to semi-gradient form. Give accompanying definitions of the return for both the episodic and continuing case.}

\begin{equation}
V_{t+n}(S_t) = V_{t+n-1}(S_t) + \alpha \rho_{t:t+n+1} \Big[ G_{t:t+n} - V_{t+n+1}(S_t) \Big]
\tag{equation 7.9 from the book page 148}
\end{equation}
Very similar to equation~\ref{eq:n-step SARSA semi-gradient}:
\begin{equation}
\begin{split}
w_{t+n} & = w_{t+n-1} + \alpha \rho_{t+1} ... \rho_{t+n-1}\Big[ G_{t:t+n}-\hat{v}(S_t, w_{t+n-1}) \Big] \nabla \hat{v}(S_t, w_{t+n-1}) \\
G_{t:t+n} & = R_{t+1} + ... \gamma^{n-1}R_{t+n} + \gamma^n \hat{v}(S_{t+n}, w_{t+n-1}) \\
G_{t:t+n} & = R_{t+1} - \bar{R}_{t} + ... R_{t+n} - \bar{R}_{t+n-1} +  \hat{v}(S_{t+n}, w_{t+n-1})
\end{split}
\end{equation}

\subsection{Exercise 11.2 page 259}
\textbf{Convert the equations of n-step $Q(\omega)$ to semi-gradient form. Give definitions that cover both the episodic and continuing case.}

Let $\sigma \in [0, 1]$ be the degree of sampling.
\begin{equation}
Q_{t+n}(S_t, A_t) = Q_{t+n-1}(S_t, A_t) + \alpha \rho_{t+1:t+n} \big[ G_{t:n+n} - Q_{t+n-1}(S_t, A_t) \big]
\tag{equation 7.11 handbook page 149}
\end{equation}
\begin{equation}
G_{t:h} = R_{t+1} + \gamma \Big( \sigma_{t+1}\rho_{t+1} + (1-\sigma_{t+1})\pi(A_{t+1}|S_{t+1}) \Big) \Big( G_{t+1:h} -Q_{h-1}(S_{t+1}, A_{t+1}) \Big) + \gamma \bar{V}_{h-1}(S_{t+1})
\tag{equation 7.17 handbook page 155, $Q(\sigma)$}
\end{equation}

It seems straightforward to do this:
\begin{equation}
\begin{split}
w_{t+n} & = w_{t+n-1} + \alpha \rho_{t+1:t+n}\big[ G_{t:n+n} - \hat{q}(S_t, A_t) \big]\\
G_{t:h} & = R_{t+1} + \gamma \Big( \sigma_{t+1}\rho_{t+1} + (1-\sigma_{t+1})\pi(A_{t+1}|S_{t+1}) \Big) \Big( G_{t+1:h} -Q_{h-1}(S_{t+1}, A_{t+1}) \Big) + \gamma \bar{V}_{h-1}(S_{t+1}) \\
G_{t:h} & = R_{t+1} -\bar{R}_t + \gamma \Big( \sigma_{t+1}\rho_{t+1} + (1-\sigma_{t+1})\pi(A_{t+1}|S_{t+1}) \Big) \Big( G_{t+1:h} -Q_{h-1}(S_{t+1}, A_{t+1}) \Big) + \gamma \bar{V}_{h-1}(S_{t+1}) \\
\end{split}
\end{equation}

\subsection{Exercise 11.3 page 264}
\textbf{Apply one-step semi-gradient Q-learning to Baird's counter example and show empirically that it's weights diverge}

\subsection{Exercise 11.4 prove eq 11.24 handbook}
Prove equation~\ref{eq:exercise 11.4 to prove eq 11.24 handbook}.
\begin{equation}
\begin{split}
\overline{RE} & = \EX[(G_t - \hat{v}(S_t, w))^2] \\
& = \overline{VE}(w) + \EX[(G_t-v_{\pi}(S_t))^2] \\
\end{split}
\label{eq:exercise 11.4 to prove eq 11.24 handbook}
\end{equation}

The value error $\overline{VE}$ is the average error under $\mu$ of the approximated value function and the optimal value function $\overline{VE}(w) = || \hat{v}_w - v_\pi ||_{\mu}^2$. The return error is also the average under $\mu$, so we can replace $\hat{v}(S_t, w)$ with $v_\pi + \overline{VE}(w)$. Or more formally:
\begin{equation}
\begin{split}
\overline{RE} & = \EX[(G_t - \hat{v}(S_t, w))^2] \\
& = \EX[(G_t-v_{\pi}(S_t)+\overline{VE}(w))^2] \\
& = \overline{VE}(w) + \EX[(G_t-v_{\pi}(S_t))^2] \\
\end{split}
\label{eq:exercise 11.4 solution}
\end{equation}


\section{Combining Operators}\label{sec:combining}
% TODO: Add epigraph (if fitting quote exists)
% TODO: Move zero-testing out of the current document part

Two types of operators have been introduced
in \autoref{sec:introduction}:

\begin{itemize}
\item Operators that manipulate bits independently in parallel:
    \lstinline$OR$, \lstinline$AND$
\item Operators that chain through multiple bits of the argument:
    \lstinline$INC$, \lstinline$DEC$
\end{itemize}

Combining pairs from these two sets will allow
to create more complex formulas.
\lstinline$NOT$ (part of the first group) is left out,
since it would simply invert the result
and thereby not contributing to interesting results.
It will be useful as a third operator later on.

\subsection*{Combining two operators}
Working a single bitwords \lstinline$x$, we can
manipulate \lstinline$x$ with \lstinline$INC$, \lstinline$DEC$
and can combine the result with the original argument
to add or remove certain bits of \lstinline$x$
using \lstinline$OR$, \lstinline$AND$.

\lstinline$x AND (DEC x)$ for example
inverts all bits up to the rightmost \lstinline$1$
and then turns off every bit in \lstinline$x$ that hasn't been inverted.
This means every bit up to the rightmost \lstinline$1$ has been turned off.
This is equivalent to turning off the rightmost \lstinline$1$
(on the right are only trailing \lstinline$0$s anyway).

Other combinations follow a similar pattern as demonstrated in
\autoref{table:dec-and}, \autoref{table:inc-or},
\autoref{table:inc-and} and \autoref{table:dec-or}.

\begin{table}[H]
\centering
\begin{tabular}{r|ccc}
\lstinline$x$ & left of rightmost \lstinline$1$
    & rightmost \lstinline$1$ & trailing \lstinline$0$s\\
\hline
\lstinline$(x-1)$ & unchanged & \lstinline$0$ & \lstinline$1...1$\\
\lstinline$x&(x-1)$ & unchanged
    & \lstinline$0$ & unchanged\\
& \multicolumn{3}{c}{
    \fbox{turn off the rightmost \lstinline$1$}}
\end{tabular}
\caption{\lstinline$x AND (DEC x)$}
\label{table:dec-and}
\end{table}

By switching the two operators for their counterparts
\lstinline$INC$ and \lstinline$OR$,
the meaning of the formula gets switched as well.

\begin{table}[H]
\centering
\begin{tabular}{r|ccc}
\lstinline$x$ & left of rightmost \lstinline$0$
    & rightmost \lstinline$0$ & trailing \lstinline$1$s\\
\hline
\lstinline$(x+1)$ & unchanged & \lstinline$1$ & \lstinline$0...0$\\
\lstinline$x|(x+1)$ & unchanged
    & \lstinline$1$ & unchanged\\
& \multicolumn{3}{c}{
    \fbox{turn on the rightmost \lstinline$0$}}
\end{tabular}
\caption{\lstinline$x OR (INC x)$}
\label{table:inc-or}
\end{table}

Using the defined sets of operators, two other combinations are possible,
which handle multiple (trailing) bits instead of a single (rightmost) one.

\begin{table}[H]
\centering
\begin{tabular}{r|ccc}
\lstinline$x$ & left of rightmost \lstinline$0$
    & rightmost \lstinline$0$ & trailing \lstinline$1$s\\
\hline
\lstinline$(x+1)$ & unchanged & \lstinline$1$ & \lstinline$0...0$\\
\lstinline$x&(x+1)$ & unchanged
    & unchanged & \lstinline$0...0$\\
& \multicolumn{3}{c}{
    \fbox{turn off trailing \lstinline$0$s}}
\end{tabular}
\caption{\lstinline$x AND (INC x)$}
\label{table:inc-and}
\end{table}

Again switching the two operators
for their counterparts \lstinline$DEC$ and \lstinline$OR$
leads to the meaning of the formula getting switched as well.

\begin{table}[H]
\centering
\begin{tabular}{r|ccc}
\lstinline$x$ & left of rightmost \lstinline$1$
    & rightmost \lstinline$1$ & trailing \lstinline$0$s\\
\hline
\lstinline$(x-1)$ & unchanged & \lstinline$0$ & \lstinline$1...1$\\
\lstinline$x|(x-1)$ & unchanged
    & unchanged & \lstinline$1...1$\\
& \multicolumn{3}{c}{
    \fbox{turn on trailing \lstinline$0$s}}
\end{tabular}
\caption{\lstinline$x OR (DEC x)$}
\label{table:dec-or}
\end{table}

\subsubsection*{Zero-Testing}
The two formulas build with \lstinline$AND$ turn off bits.
Therefore their result can be zero-tested (in a meaningful way):
\begin{itemize}
\item \lstinline$x AND (DEC x)$:\\
    \lstinline$!(x&(x-1))$$\equiv$``is \lstinline$x$ $0$ or a power of $2$''
\item \lstinline$x AND (INC x)$:\\
    \lstinline$!(x&(x+1))$$\equiv$``is \lstinline$x$ $2^m - 1, m \in [0,n]$''
    (for a bitword of length $n$)
\end{itemize}

\subsection*{Combining Three Operators}

\lstinline$NOT$ has been left out at the beginning.
With the four above introduced formulas however, new possibilities arise.
There are three options to use the negation on them:
% TODO add WHY for 1/2
\begin{enumerate}
\item invert the full formula:\\
    \lstinline$~(x&(x-1))$, \lstinline$~(x|(x+1))$,
    \lstinline$~(x&(x+1))$, \lstinline$~(x|(x-1))$\\
    this just inverts the original result,
    meaning that:\\
    up to the rightmost \lstinline$0$/\lstinline$1$
    the turning on/off gets swapped\\
    and additionally the left part is inverted
\item invert the incremented/decrement argument:\\
    \lstinline$x&(~(x-1))$, \lstinline$x|(~(x+1))$,
    \lstinline$x&(~(x+1))$, \lstinline$x|(~(x-1))$\\
    this simply turns on/off the section
    left of the rightmost \lstinline$0$/\lstinline$1$
\item invert the original argument:\\
    \lstinline$(~x)&(x-1)$, \lstinline$(~x)|(x+1)$,
    \lstinline$(~x)&(x+1)$, \lstinline$(~x)&(x-1)$\\
\end{enumerate}

While the first two options deliver foreseeable results,
the inverting of the unchanged argument is the most complex
and promising option, as demonstrated in
\autoref{table:not-inc-and}, \autoref{table:not-dec-or},
\autoref{table:not-dec-and} and \autoref{table:not-inc-or}.

\begin{table}[H]
\centering
\begin{tabular}{r|ccc}
\lstinline$x$ & left of rightmost \lstinline$0$
    & rightmost \lstinline$0$ & trailing \lstinline$1$s\\
\hline
\lstinline$(~x)$ & inverted & \lstinline$1$ & \lstinline$0...0$\\
\lstinline$(x+1)$ & unchanged & \lstinline$1$ & \lstinline$0...0$\\
\lstinline$(~x)&(x+1)$ & \lstinline$0...0$
    & \lstinline$1$ & \lstinline$0...0$\\
& \multicolumn{3}{c}{
    \fbox{a single \lstinline$1$ at the rightmost \lstinline$0$}}
\end{tabular}
\caption{\lstinline$(NOT x) AND (INC x)$}
\label{table:not-inc-and}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{r|ccc}
\lstinline$x$ & left of rightmost \lstinline$1$
    & rightmost \lstinline$1$ & trailing \lstinline$0$s\\
\hline
\lstinline$(~x)$ & inverted & \lstinline$1$ & \lstinline$0...0$\\
\lstinline$(x-1)$ & unchanged & \lstinline$0$ & \lstinline$1...1$\\
\lstinline$(~x)|(x-1)$ & \lstinline$1...1$
    & \lstinline$0$ & \lstinline$1...1$\\
& \multicolumn{3}{c}{
    \fbox{a single \lstinline$0$ at the rightmost \lstinline$1$}}\\
\end{tabular}
\caption{\lstinline$(NOT x) OR (DEC x)$}
\label{table:not-dec-or}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{r|ccc}
\lstinline$x$ & left of rightmost \lstinline$1$
    & rightmost \lstinline$1$ & trailing \lstinline$0$s\\
\hline
\lstinline$(~x)$ & inverted & \lstinline$1$ & \lstinline$1...1$\\
\lstinline$(x-1)$ & unchanged & \lstinline$0$ & \lstinline$1...1$\\
\lstinline$(~x)&(x-1)$ & \lstinline$0...0$
    & \lstinline$0$ & \lstinline$1...1$\\
& \multicolumn{3}{c}{
    \fbox{\lstinline$1$s at trailing \lstinline$0$s}}\\
\end{tabular}
\caption{\lstinline$(NOT x) AND (DEC x)$}
\label{table:not-dec-and}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{r|ccc}
\lstinline$x$ & left of rightmost \lstinline$0$
    & rightmost \lstinline$0$ & trailing \lstinline$1$s\\
\hline
\lstinline$(~x)$ & inverted & \lstinline$1$ & \lstinline$1...1$\\
\lstinline$(x+1)$ & unchanged & \lstinline$1$ & \lstinline$0...0$\\
\lstinline$(~x)|(x+1)$ & \lstinline$1...1$
    & \lstinline$1$ & \lstinline$0...0$\\
& \multicolumn{3}{c}{
    \fbox{\lstinline$0$s at trailing \lstinline$1$s}}\\
\end{tabular}
\caption{\lstinline$(NOT x) OR (INC x)$}
\label{table:not-inc-or}
\end{table}

\subsubsection*{Inverting}
Additionally, every formula can be inverted in total.
This is especially useful when the result is ``a single \lstinline$0$''
like for \lstinline$(NOT x) OR (DEC x)$.
The result then has the same scheme as \lstinline$(NOT x) AND (INC x)$.


\documentclass[11pt]{article}
\setlength{\parskip}{.08in}
\renewcommand\arraystretch{1.2}
\usepackage{commath}
\usepackage{listings}
\usepackage{hyperref}
\usepackage{float}
\setlength{\parindent}{0in}

\lstdefinestyle{mystyle}{
basicstyle=\fontsize{10}{7}\selectfont\ttfamily
}
\lstset{style=mystyle}

% Added packages
\usepackage{setspace}
\usepackage{enumerate}
\usepackage{bm}
\usepackage{amsmath}
\makeatletter
\renewcommand*\env@matrix[1][\arraystretch]{%
  \edef\arraystretch{#1}%
  \hskip -\arraycolsep
  \let\@ifnextchar\new@ifnextchar
  \array{*\c@MaxMatrixCols c}}
\makeatother
%\usepackage[dvips, bookmarks, colorlinks=true, plainpages = false,
%citecolor = green, urlcolor = blue, filecolor = blue] {hyperref}
%\usepackage[font=small,format=plain,labelfont=bf,up,textfont=it,up]{caption}

% Set the beginning of a LaTeX document
\usepackage{graphicx}
\graphicspath{ {./images/} }

%\usepackage{endnotes}
%\let\footnote=\endnote

\usepackage{titlesec}
\titleformat{\section}
  {\normalfont\fontfamily{phv}\fontsize{14}{17}\selectfont}{\thesection}{1em}{}

\oddsidemargin 0.25in
\evensidemargin 0.25in

\headheight 0.0in
\headsep 0.0in
\topmargin -0.25in

\textwidth 6.0in
\textheight 9.0in

\paperheight 11.0in
\paperwidth 8.5in

\begin{document}

\title{Extrapolation of Climate Data using Numerical Methods}
\author{Philippe Nadon\footnote{Corresponding author: nadon@ualberta.ca}}

\date{ \today}
\maketitle

{\small Faculty of Science, University of Alberta}\\
{\small Camrose, Alberta, Canada}\\[0.5cm]

\thispagestyle{empty}

%\onehalfspacing
\doublespacing

\begin{abstract}
Climate change is currently a major concern for society, due to the irreversible damage predicted by our models if we continue our current trend of excessive emissions and pollution. In this paper various methods of data-fitting and extrapolation are explored, and compared to see how accurately they represent the data provided.

\end{abstract}

\textbf{Keywords:} Climate, Numerical Methods, Extrapolation

\newpage

\setcounter{page}{1}

\section{Introduction}
Data is obtained from Berkeley Earth, the table representing the "Global Average Temperature Anomaly with Sea Ice Temperature Inferred from Air Temperatures". This data spans from 1850 to 2018, and shows us how global temperatures have changed in the last 168. Using this data, we can use the method of least squares to fit various equations onto this data, and then extrapolate the data's trend into the future. These include a linear, quadratic, and cubic fit, which are approximated by solving an $nxn$ matrix, where n is the degree of the polynomial + 1. Afterwards, two more fits are used, an exponential fit and finally 2 different combinations of a polynomial and exponential. These last three are solved using Matlab's \textit{lsqcurvefit} function. Although modern techniques are far more sophisticated, these results will still give us a general understanding of what will happen if we do not change our habits which impact the planet's climate.

\section{Methods}
Multiple equations were used to fit the data so we could compare their respective accuracies and predictions. They are listed below:
\begin{equation}\label{eq:linear}
linear = \alpha + \beta x
\end{equation}
\begin{equation}\label{eq:quadratic}
quadratic = \alpha + \beta x + \gamma x^2
\end{equation}
\begin{equation}\label{eq:cubic}
cubic = \alpha + \beta x + \gamma x^2 + \delta x^3
\end{equation}
\begin{equation}\label{eq:exp}
exponential = \alpha + e^{\beta x}
\end{equation}
\begin{equation}\label{eq:mix1}
mixed_1 = \alpha + (\gamma + \delta x)e^{\beta x}
\end{equation}
\begin{equation}\label{eq:mix2}
mixed_2 = \alpha + (\gamma + \delta x + \epsilon x^2)e^{\beta x}
\end{equation}

\subsection{ Linear, Quadratic, Cubic Fits}
The following code was used to calculate the best fit for the data, using equations \eqref{eq:linear}, \eqref{eq:quadratic}, and \eqref{eq:cubic}. The method of least squares can be solved by forming the problem into a system of linear equations in the form $Ax = B$, and solving for x.

\begin{lstlisting}[language=Matlab, style=myStyle]
function res = extrapolatePolyToYear( t, data, degree)
    N = length(data);
    rows = degree + 1;
    A = zeros( rows, rows);
    B = zeros( rows, 1);

    for n = 1:N
        for i = 1:rows
            % for A:
            % The diagonal entries:
            A(i, i) = A(i, i) + t(n) ^ (2 * ( i - 1));
            % The upper / lower entries (mirror along diagonal)
            for j = 1:(i - 1)
                A( i, j) = A(i, j) + t(n) ^ ( i + j - 2);
                A( j, i) = A(i, j);
            end

            % for B:
            B(i) = B(i) + data(n) * t(n) ^ (i - 1);
        end
    end
    disp(A);
    disp(B);

    res = linsolve( A, B);
end
\end{lstlisting}
This function receives an array containing t values, an array of data, and the degree of polynomial you wish to fit onto the data. It then returns an array containing the coefficients of the polynomial that is to be fit onto the data.\\

\subsection{ Exponential and Mixed Fits}
Equations \eqref{eq:exp}, \eqref{eq:mix1}, and \eqref{eq:mix2} were simpler to solve, but involved modifying the initial guesses until the plot fit well. The plot was done using the Matlab function \textit{lsqcurve}. The initial guesses are as follows:\\
Exponential:
\[ \alpha = -0.4, \beta = 0.02, \gamma = 0.1 \]
Exponential and Linear:
\[ \alpha = -0.5, \beta = 0.02, \gamma = -0.4, \delta = -0.002 \]
Exponential and Quadratic:
\[ \alpha = -0.5, \beta = 0.02, \gamma = -0.4, \delta = -0.002, \epsilon = 0.03 \]

\section{Results}
\subsection{ Linear, Quadratic, and Cubic Fits}
After computing the data for a linear, quadratic, and cubic fit, the following results were obtained:

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{linear.png}
    \caption{Linear extrapolation of climate anomalies. The linear fit predicted an anomaly value of $9.79*10^{-1}$ in year 2100, with a cumulative error of $5.71*10^{1}$ calculated between the years 1850 and 2018.}
    \label{fig:linear}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{quadratic.png}
    \caption{Quadratic extrapolation of climate anomalies. The quadratic fit predicted an anomaly value of $2.26*10^{0}$ in year 2100, with a cumulative error of $3.38*10^{1}$ calculated between the years 1850 and 2018.}
    \label{fig:quadratic}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{cubic.png}
    \caption{Cubic extrapolation of climate anomalies. The cubic fit predicted an anomaly value of $3.63*10^{0}$ in year 2100, with a cumulative error of $3.17*10^{1}$ calculated between the years 1850 and 2018.}
    \label{fig:cubic}
\end{figure}

\subsection{ Exponential and Mixed Fits}
After computing the data for an exponential fit, and two different mixed fits, the following results were obtained:

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{exp.png}
    \caption{Exponential extrapolation of climate anomalies. The exponential fit predicted an anomaly value of $5.04*10^0 $ in year 2100, with a cumulative error of $3.14*10^2 $ calculated between the years 1850 and 2018.}
    \label{fig:exp}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{mix1.png}
    \caption{ Mixed exponential and linear extrapolation of climate anomalies. The mixed fit predicted an anomaly value of $ 1.32*10^0$ in year 2100, with a cumulative error of $4.89*10^1 $ calculated between the years 1850 and 2018.}
    \label{fig:mix1}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{mix2.png}
    \caption{Mixed exponential and quadratic extrapolation of climate anomalies. The mixed fit predicted an anomaly value of $ 2.29*10^0$ in year 2100, with a cumulative error of $ 3.37*10^1$ calculated between the years 1850 and 2018.}
    \label{fig:mix2}
\end{figure}

\section{Analysis}
\subsection{Accuracy of Fit}
We can see that the fit becomes progressively more accurate as the degree of the polynomial increases, for both the mixed and purely polynomial fits. Initially, equation \eqref{eq:mix2} returned a nonsensical plot since it predicted a sharp decline in anomalies just past the last data point, but increasing the maximum iterations for \textit{lsqcurve}, we obtained a much more precise and logical fit. The interesting analysis we can make is that although equations \eqref{eq:mix1} and \eqref{eq:mix2} should have \textit{at least} the same amount of accuracy as \eqref{eq:exp} since we can simply set the coefficients of the polynomial portion to zero, it turns out due to computational limits that more error is introduced than eliminated. Matlab itself limits the number of computations allowed before it detects that the minimum difference of results between iterations has been reached and stops computing. Although the mixed results offer better fits than their purely polynomial counter parts, the cubic and exponential fits offer the most accurate fits according to the sum of the differences between the data points and the equation.\\
The fit with the lowest error appears to be the exponential fit, followed by the cubic fit. It is therefore likely that these two will offer the most accurate interpretation of the data.
\subsection{Interpretation of Results}
Both the exponential and cubic fit predict a far more aggressive increase in the anomaly value than any other fit, which is concerning. The exponential fit predicts an anomaly of $+31.4$ in the year 2100, while the cubic fit predicts an anomaly of $+31.7$. Considering the value has remained around 0.5 to 1 in 2018, for climate anomalies to triple or even quintiple is surely not a good sign. What this value means essentially is that we should expect to see more aggressive, unpredictible weather changes in the future if our impact on the environment does not change. Tornadoes, lightning storms, flash floods, and summer-time blizzards will become more and more common, which will impose a much higher cost in infrastructure maintenance and repair, and drastically increase the rate of injuries and deaths caused by weather.

\section{Conclusion and Future Considerations}
This project has helped demonstrate the utility of numerical analysis for predicting trends in areas such as climate, and also prompts the reader to consider the impact that our society has on the climate of our planet. For future improvements, one would potentially include more than just data from a single source, and improve the fit in areas such as extending computation time, using data types which can hold more precision for calculations, and using different equations altogether, such as a sin wave, as it appears from the graphs that there tends to be a trend where the anomaly value fluctuates in the shape of a wave. Reflecting on the trends at the time, such as vehicle and factory efficiency, and general consumerism would also help to improve accuracy, as those two areas are huge contributors to climate change.
\end{document}