\subsection{Results} \label{subsec:results_linear_regression}

This \href{https://github.com/am-kaiser/CompSci-Project-1/blob/main/regression_analysis/examples/linear_regression_analysis.ipynb}{jupyter notebook} consists of examples to apply different regression techniques. It also consists of widgets where the precalculated statistical performance of different regression methods can be viewed for over $10^5$ parameter combinations. The data for these experiments is included in the repository. At the end of the jupyter notebook, one can find code that will allow for simulating more parameter combinations if needed. This will overwrite the existing data and also would take a significant time to run. For $10^5$ parameter combinations, it took us about $2$ hours to obtain the data.

\subsubsection{Terminology of symbols and variable names}
Here, we list the names of the variables that are the options in the widgets of the jupyter notebook. The corresponding symbols if applicable are also shown. These symbols relate the variable names to their corresponding mathematical notation used in this report. The general template is the following \\
\\
Symbol: variable name → explanation \\
\\
The list of widget variables are:
\begin{itemize}
    \item $p:$  polynomial order → the order of 2D polynomial used for fitting
    \item $var:$  noise\_var → the variance of the zero mean Gaussian noise added to the Franke data
    \item $r:$  ratio → testing ratio. For example, if the ratio is 0.1 then the test set comprises 10 percent of the dataset.
    \item $n:$  num → the number of points taken along each direction of the input vector. If num = 20, then the total number of data points is 20*20 = 400
    \item stat → Displays the chosen statistic
    \begin{itemize}
        \item $MSE_{train}:$ training MSE
        \item $MSE_{test}:$ testing MSE
        \item ${R^2_{train}}:$ training R2 score
        \item ${R^2_{test}}:$ testing R2 score
        \item $bias_{test}:$ bias in test data
        \item $var_{test}:$ variance in test prediction
    \end{itemize}
    \item method → to choose the regression type
    \begin{itemize}
        \item direct solution and stochastic gradient descent solution available
        \begin{itemize}
            \item OLS
            \item OLS with bootstrap resampling
            \item OLS with cross-validation resampling
            \item Ridge
            \item Ridge with bootstrap resampling
            \item Ridge with cross-validation resampling
        \end{itemize}
        \item direct Solution only (using scikit learn)
        \begin{itemize}
            \item Lasso
            \item Lasso with bootstrap resampling
            \item Lasso with cross-validation resampling
        \end{itemize}
    \end{itemize}
    \item $n_b:$  n\_boot → number of times bootstrap sampling is performed in the bootstrap method. Changing this for methods that don't involve bootstrap sampling will not have any effect
    \item $k_f:$  k\_fold → number of folds in the cross-validation method. Changing this for methods that don't involve cross-validation will not have any effect
    \item $\lambda_r:$  ridge\_lambda → regularisation parameter for ridge regression
    \item $\lambda_l:$ lasso\_lambda → regularisation parameter for lasso regression
    \item $\eta:$ learn\_rates → parameter that controls the descent jump size in gradient descent methods
    \item $n_e:$ epoch → parameter that controls the number of times the algorithm works through the entire dataset in gradient descent methods
    \item $n_{batch}:$ number of batches → parameter that controls the mini-batch size in gradient descent methods
\end{itemize}

Here, we present selected results of linear regression that we find particularly interesting.

\subsubsection{Ordinary Least Squares (OLS)}

Foremost, we look at the $MSE_{train}$ of OLS (see figure \ref{fig:ols1}). We see that the $MSE_{train}$ reduces as the model complexity or polynomial order($p$) increases. In other words, with increasing complexity, our model becomes more flexible. As the input data is an exponential function that can be expressed as a series of polynomials, increasing the terms in the fitted polynomial should decrease the $MSE_{train}$. We also see that for large values of test ratio, i.e. $r = 0.4$, and low number of data points, $n = 100$, which amounts to about $60$ data points in the training set, the $MSE_{train}$ is significantly lower for high $p$.

\begin{figure}[htb]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/ols2.png}
  \caption{}
  \label{fig:ols1}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/ols1.png}
  \caption{}
  \label{fig:ols2}
\end{subfigure}
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/ols3.png}
  \caption{}
  \label{fig:ols3}
\end{subfigure}
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/Overfitted_Data.png}
  \caption{}
  \label{fig:overfit}
\end{subfigure}
\caption{OLS for noiseless Franke data: (a) $MSE_{train}$, (b) $MSE_{test}$ and (c) Generalization error plotted as functions of model complexity. (d) Schematic describing overfitting of data due to higher model complexity (Source: Ghiles, Wikipedia). }
\label{fig:OLS1}
\end{figure}

Now, if we shift attention to the $MSE_{test}$, we see that for r=0.4, n=100, the generalization is poor (see figure \ref{fig:ols2}). This is the result of overfitting. This also happens for r=0.1, n=100 albeit not as strongly. This reveals that even for low values of $MSE_{train}$, the model can perform poorly. We recognize that if the number of datapoints is higher in the training set, then there is no overfitting. This is also evident by looking at the generalization error(see figure \ref{fig:ols3}). The model 'sees' more variety of data during the training phase which prevents it from overfitting. Provided we don't overfit, the $MSE_{test}$ reduces as p  increases. It should be noted that irrespective of the number of data points used for training, there exist p's where we will experience overfitting. The figure \ref{fig:overfit} shows a 1D polynomial with order equal to the number of points which will perform much worse than a linear polynomial for unseen data. If $n_{train}$ is the number of training data points then to prevent overfitting in many situations, we need $n_{train} >> p$.
\newline
\newline
Next, we see that the introduction of noise into the dataset increases $MSE_{train}$ (see figure \ref{fig:ols4}). We also see that introducing noise affects the $MSE_{test}$ of a dataset with a larger number of points significantly when model complexity is high (see figure \ref{fig:ols5}). $MSE_{test}$ decreases initially with model complexity but later plateaus. This portrays that the higher degree of freedom, introduced by additional polynomial terms, doesn't sufficiently capture the noise. When the model complexity is low, the prediction is unaffected by the presence of noise.
\newline\newline
\begin{figure}[htb]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/ols5.png}
  \caption{}
  \label{fig:ols4}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/ols4.png}
  \caption{}
  \label{fig:ols5}
\end{subfigure}
\caption{OLS for $r=0.1$ and $n=900$: (a) $MSE_{train}$ and (b) $MSE_{test}$ plotted as functions of model complexity}
\label{fig:OLS2}
\end{figure}

\begin{figure}[htb]
\centering
\includegraphics[width=.7\linewidth]{Images/ols7.png}
\caption{OLS for $r=0.1$, $var=0$ and $n=900$. Bias and variance plotted as functions of model complexity}
\label{fig:OLS3}
\end{figure}

If we now look at the bias and variance, they are both roughly constant as the model complexity increases (see figure \ref{fig:OLS3}). This might be a discrepancy in our calculation and it prevents us from understanding the bias-variance tradeoff. This behavior is observed even when other parameters are changed. As variance measures, the spread of our prediction, and the plots show that our model always has a finite spread. As bias is a measure of the mean deviation of our prediction from the truth, the constant values raise questions on our calculations. We know that parameters like $r$, $var$, $p$ and $n$ have an impact on the $MSE_{test}$, but as the bias seems to be roughly the same then that shows that irrespective of the parameters, our model has approximately the same mean value of the predicted output. Either our model is capturing the mean well almost always or our calculations of bias and variance are incorrect.

\subsubsection{Ridge and lasso Regression}
In ridge regression, we use the $L_2$ norm of the parameter vector $\boldsymbol \beta$ as a penalty term. As we are not tuning the size of the elements of the design matrix, the interrelations between the different columns should remain the same. This is the case when the design matrix is orthogonal. As we use polynomials as basis functions, the columns of the design matrix are likely to be independent. They will not be independent if there is a correlation between $x_{i1}$ and $x_{i2}$ which is not true in our dataset. So, the parameters in ridge regression would be rescaled by a factor of $\frac{1}{1+\lambda_r}$ compared to the OLS parameters \cite{mehta2019high}. However, since that there might be errors in calculating the pseudoinverse, the scaling might not always be by the factor of $\frac{1}{1+\lambda_r}$. So, not all parameters are scaled the same. This would deviate the predictions from that of the OLS. We can see this in the figure \ref{fig:ridge7} and \ref{fig:ridge7b} where the regularisation parameter affects both the training and prediction phase. The training and testing error both worsen as $\lambda_r$ increases. The deviation from the OLS errors kicks in at a lower model complexity for higher values of  $\lambda_r$. For lower model complexity, the ridge regression performs similarly to OLS. But, as the complexity increases, ridge regression has a poorer prediction. It is unclear why this happens, but the most likely reason might be numeric underflow or overflow.

\begin{figure}[htb]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/ridge7b.png}
  \caption{}
  \label{fig:ridge7}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/ridge7b.png}
  \caption{}
  \label{fig:ridge7b}
\end{subfigure}
\caption{Ridge regression for $r=0.1$, $var=0$ and $n=900$: (a) $MSE_{train}$ and (b) $MSE_{test}$ plotted as functions of model complexity for different regularisation strength}
\label{fig:Ridge1}
\end{figure}

Meanwhile, in lasso regression, we use the $L_1$ norm of the parameter vector $\boldsymbol \beta$ as a penalty term. Visually, this is equivalent to optimizing the parameters until we hit the edges of a constraint hypercuboid. While, in ridge regression, we hit the surface of a constraint hyperellipsoid. See chapter 3 of \cite{friedman2001elements} for more details. This means that in lasso regression when the optimum parameters are reached, the components of some of the parameter vector $\boldsymbol \beta$ are 0. These features were not important enough in affecting the combined optimization of the least square error and the $L_1$ norm of the parameter vector $\boldsymbol \beta$. We see in the figure \ref{fig:lasso7} that both the training error and testing error worsen as the $\lambda_l$ increases. But, the performance is much worse than for ridge regression and is several orders of magnitude worse than OLS. As $\lambda_l$ increases, more components of $\boldsymbol \beta$ turn out to be 0.

\begin{figure}[htb]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/lasso7b.png}
  \caption{}
  \label{fig:lasso7}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/lasso7.png}
  \caption{}
  \label{fig:lasso7b}
\end{subfigure}
\caption{lasso regression for $r=0.1$, $var=0$ and $n=900$: (a) $MSE_{train}$ and (b) $MSE_{test}$ plotted as functions of model complexity for different regularisation strength}
\label{fig:Lasso1}
\end{figure}

\begin{figure}[htb]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/orl2.png}
  \caption{}
  \label{fig:orl1}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/orl1.png}
  \caption{}
  \label{fig:orl2}
\end{subfigure}
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/orl3.png}
  \caption{}
  \label{fig:orl3}
\end{subfigure}
\caption{OLS, Ridge($\lambda_r = \num{1e-4}$) and lasso($\lambda_l = \num{1e-6}$) regression for $r=0.1$, $var=0$ and $n=900$: (a) $MSE_{train}$, (b) $MSE_{test}$ and (c) $R^2_{test}$ plotted as functions of model complexity}
\label{fig:ORL1}
\end{figure}

The figure \ref{fig:orl1} and \ref{fig:orl2} show the relative performance in training and testing for OLS, ridge, and lasso regression. Here, we show the best-case scenario for Ridge and lasso among the parameters screened by us. OLS outperforms ridge regression by an order of magnitude while it is 2 orders of magnitude better than lasso. This is explained by the fact that in OLS, the optimization involves lowering the squared error but in the other methods, there is an additional cost incurred for the size of components of parameter vectors. The figure \ref{fig:orl3} shows the $R^2_{test}$ error which is a measure of the variance captured by our model in relation to the actual variance. We see that as the model complexity increases, the $R^2_{test}$ tends to reach 1 for both OLS and ridge regression. While in lasso regression, the variance capture is much poorer. Certainly, the results here indicate that OLS performs much better than Ridge and lasso. Why would these two methods remain relevant? This is because there can be situations where the size of components of $\boldsymbol \beta$ can cause a numerical overflow. Additionally, shrinkage methods like Ridge and lasso reduce the prediction variability in many situations \cite{friedman2001elements}.

\subsubsection{Regression with Bootstrap and Cross-validation}
\begin{figure}[ht]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Images/ols9.png}
  \caption{}
  \label{fig:ols9}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Images/ols8.png}
  \caption{}
  \label{fig:ols8}
\end{subfigure}
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Images/ridge9.png}
  \caption{}
  \label{fig:ridge9}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Images/ridge8.png}
  \caption{}
  \label{fig:ridge8}
\end{subfigure}
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Images/lasso9.png}
  \caption{}
  \label{fig:lasso9}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Images/lasso8.png}
  \caption{}
  \label{fig:lasso8}
\end{subfigure}
\caption{For $r=0.1$, $var=0$ and $n=900$: (a), (c), (e) $MSE_{train}$ and  (b), (d), (f) $MSE_{test}$ plotted as functions of model complexity}
\label{fig:OLS_resample}
\end{figure}

\begin{figure}[h!]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Images/ols15.png}
  \caption{}
  \label{fig:ols15}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{Images/ols14.png}
  \caption{}
  \label{fig:ols14}
\end{subfigure}
\caption{For $r=0.1$, $var=0$ and $n=100$: (a) $MSE_{train}$ and (b) $MSE_{test}$ plotted as functions of model complexity}
\label{fig:OLS_resample2}
\end{figure}

The figure \ref{fig:OLS_resample} shows the comparison between OLS, Ridge, and lasso methods with bootstrap and cross-validation resampling. Resampling is primarily performed to reduce the uncertainty in statistics like the mean squared error. Often, statistics of a single fit might not be the true indicator of the performance of the model due to the inherent stochasticity while splitting data into train and test sets, and in the noise. However, we see that in the case of OLS, the introduction of resampling has almost no effect. This can be attributed to the fact that due to $n = 900$, the model 'sees' a variety of data during the training phase. If the number of data points is reduced for example to $n=100$ (see figure \ref{fig:OLS_resample2}), then the effect of resampling is more prominent. We notice that for $n=100$, the training is equally well for OLS and OLS with cross-validation resampling. But, $MSE_{train}$ is lower for bootstrap for high values of $p$. This is an indicator of overfitting where, in bootstrap resampling, some of the data is 'seen' again due to sampling with replacement. This reduces the effective training set size and makes the model more prone to overfitting. This is also reflected in poor $MSE_{test}$ for OLS with bootstrap resampling. We also think that OLS with cross-validation resampling provides more reliable statistics than OLS without resampling as statistics of OLS with cross-validation can be thought of as a numerical average of statistics of multiple instances of OLS without resampling. The introduction of resampling in Ridge and lasso has no effect on training (figures \ref{fig:ridge9} and \ref{fig:lasso9}) but they do smoothen out the irregularities of the $MSE_{test}$ (figures \ref{fig:ridge8} and \ref{fig:lasso8}).

\subsubsection{OLS with Stochastic Gradient Descent}
For stochastic gradient descent (SGD) we have three additional parameters: the epoch $e$, learning rate $\eta$, and the number of mini-batches $n_{batch}$. See section \ref{subsec:SGD} for a detailed description of the SGD algorithm. The resource  \cite{masters_revisiting_2018} showed that using small batch sizes achieves the best training stability and test performance across a wide range of experiments. The best results were obtained for batch sizes equal to or smaller than 32. Therefore, we focus in this discussion on batch sizes 1, 2, 5, 10, 32. For a batch size of 1, we have a classical stochastic gradient descent. Otherwise, the algorithm is called mini-batch gradient descent. Usually, the number of epochs is large to sufficiently minimize the error by the learning algorithm. Therefore, we compare the output with epochs of size 10, 100, 500, 1000. Lastly, the learning rate is a small positive number and traditionally between 0 and 1. We analyze the output with the following learning rates 0.001, 0.01, 0.1, 1.0. \newline \newline
While the MSE for the training dataset hardly varies for different learn rates, number of epochs, or batch sizes the MSE for the test dataset improves slightly if the parameters are chosen correctly and the order of the fitted polynomial is high (see figure \ref{fig:ols_sgd}). The best results for $MSE_{test}$ were obtained for $n_{batch}=10, n_e=32$ and $\eta=0.1$. But even for the size of the mini-batches, which displays the "biggest" variance in the $MSE_{test}$, the difference is only between roughly $10^{-3}$ and $10^{-4}$ (see figure \ref{fig:olssg_test_batch}). To conclude, the choice of the SGD parameters, which we tested, does not seem to play an important part for the performance of the fitting in this case.

\begin{figure}[H]
\centering
\begin{subfigure}{.49\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/olssgd_train_learn_rate_all_batch_32_epoch_10.png}
  \caption{}
  \label{fig:olssgd_train_learn_rate}
\end{subfigure}%
\begin{subfigure}{.49\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/olssgd_test_learn_rate_all_batch_32_epoch_10.png}
  \caption{}
  \label{fig:olssgd_test_learn_rate}
\end{subfigure}
\begin{subfigure}{.49\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/olssgd_train_learn_rate_1_batch_all_epoch_10.png.png}
  \caption{}
  \label{fig:olssgd_train_batch}
\end{subfigure}
\begin{subfigure}{.49\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/olssgd_test_learn_rate_1_batch_all_epoch_10.png}
  \caption{}
  \label{fig:olssg_test_batch}
\end{subfigure}
\begin{subfigure}{.49\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/olssgd_train_learn_rate_1_batch_32_epoch_all.png}
  \caption{}
  \label{fig:olssgd_train_epoch}
\end{subfigure}
\begin{subfigure}{.49\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{Images/olssgd_test_learn_rate_1_batch_32_epoch_all.png}
  \caption{}
  \label{fig:olssgd_test_epoch}
\end{subfigure}
\caption{For $r=0.1$, $var=0$, $n=900$: $MSE_{train}$ (left column) and $MSE_{test}$ (right column) plotted as functions of model complexity. a), b): $n_{batch}=10, n_e=32$, c), d): $\eta=0.1, n_e=10$, e), f): $\eta=0.1, n_{batch}=32$}
\label{fig:ols_sgd}
\end{figure}
Similar results were also obtained for optimizing the cost function for ridge regression. When we compare the test and train MSE for ridge and OLS regression with and without SGD (see figure \ref{fig:ols_sgd_ridge}) we clearly see that for OLS there is nearly no difference if the mini-batch stochastic gradient descent is applied or not. In contrary, when we look at ridge regression both the train and test MSE are better without applying the mini-batch stochastic gradient descent. For this simulation we used:  $n_{batch}=10, n_e=32, \eta=0.1$ and $\lambda_{ridge}=0.0001$.
\begin{figure}[H]
	\centering
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=.9\linewidth]{Images/sgd_ols_ridge_train.png}
		\caption{}
		\label{fig:ols_sgd_ridge_train}
	\end{subfigure}%
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=.9\linewidth]{Images/sgd_ols_ridge_test.png}
		\caption{}
		\label{fig:ols_sgd_ridge_test}
	\end{subfigure}
	\caption{For $r=0.1$, $var=0$, $n=900$: $MSE_{train}$ and $MSE_{test}$ plotted as functions of model complexity.}
	\label{fig:ols_sgd_ridge}
\end{figure}


\section{Prerequisites}
\label{sec:prerequisites}

  In this section, we will give a brief overview of the terms and definitions
  that we will use throughout this report. These definitions should not be
  considered complete or even absolutely accurate, as they are a personal
  choice of words with the goal of giving the reader an impression of what
  we are talking about.

  \subsection{Neural networks}
  \label{sub:neural_networks}

    A \textit{neuron} is a programmatic structure (or function) that tries to resemble
    neurons inside the brain. In its most basic form, it takes a vector of
    input variables, calculates a weighted sum and applies an activation function
    (typically a sigmoid function like \textit{tanh}).

    A \textit{neural network} is a collection of neurons which are typically
    arranged in layers, where the input of a layer is the output of the
    preceeding one. The output of the last layer is called the output of
    the network.

  \subsection{Recurrent Neural Networks}
  \label{sub:recurrent_neural_networks}

    A \textit{recurrent neural network} is a neural network where the
    output of one or more layers is fed back as input to the network
    or a selection of neurons typically scaled by a factor between 0
    and 1. This allows the network to make decisions based on previous
    results / events.

  \subsection{LSTM and GRU}
  \label{sub:lstm_and_gru}

    The problem with basic RNNs is that they cannot store information for a long time.
    If a recurrent neural network tries to keep a piece of information over a longer
    period of time, it would have to constantly feed it back as input but this would
    suffer from an exponential decay. Therefore they are not suited for language processing
    where we may need to refer to information that was mentioned a long time ago \cite{hrdipl}, \cite{bengio}.

    Fortunately, Hochreiter and Schmidhuber came up with a solution called
    Long Short-Term Memory \cite{lstm}.  This type of cell possesses an
    extra internal layer that allows it to store information, acting as a
    kind of memory. The output of the cell is then made up of the current
    input and the data that is stored inside. They solve the problem of
    exponential decay by giving a network means to set aside important
    pieces of information and use it later on, without the danger of
    losing it due to decay.  See (\ref{fig:lstm}) for a schematic of the
    internal structure. In short, LSTMs possess so called \textit{gates}
    that determine if and what information should be stored. The \textit{forget
    gate} (marked by an '\verb+x+' on the upper line) decides whether or not
    to drop the current memory state and the \textit{input gate} (marked
    by a '\verb|+|') decides which values to add to the current state.

    \begin{figure}
    \begin{center}
        \includegraphics[width=0.9 \linewidth]{lstm.png}
    \end{center}
        \caption{Schematic of an LSTM cell \cite{colah}}
    \label{fig:lstm}
    \end{figure}

	GRUs (Gated Recurrent Units) are a slight simplification of LSTMs \cite{gru}. They
	combine the forget and input gate into a single update gate to reduce the computational workload
	a little bit, but are otherwise essentially the same.

  \subsection{Embeddings}
  \label{sub:embeddings}

    Usually when solving a multi class classification problem the input and output values are
	represented as one hot encoded vectors. This means that each value $i$ is turned into an
	$n$-dimensional vector where the $i$-th row has a $1$ in it and all other rows are $0$ (where
	$n$ is the number of available classes). When feeding such a possibly huge input vector
	into a layer of a network, that is, multiplying said vector with an $n \times m$-dimensional
	matrix where $m$ is the number of neurons in that layer, one quickly realizes that the result
	of such computation is merely the $i$-th row of that matrix. To reduce the computational
	overhead, it is common to skip the idea of having neurons in such case and simply perform
	a lookup in a huge matrix; the result being an $m$-dimensional vector. This is called an
    \textit{embedding} because the input vector is being embedded in an $m$-dimensional vector space (\ref{fig:embed}).

	If trained correctly, the embedding layer can be used to group words with similar semantics closer
	together while keeping words that have different meaning farther away. The Word2Vec-Model \cite{w2v}
	does this so well that it allows calculations like: \verb|Warsaw - Poland + Germany = Berlin|.

    \begin{figure}
    \begin{center}
        \includegraphics[width=0.8 \linewidth]{lookup_matrix.png}
    \end{center}
    \caption{Example of a lookup in an embedding layer}
    \label{fig:embed}
    \end{figure}

  \subsection{Keras}
  \label{sub:keras}

    Keras is a Python module that contains functions and classes to easily create
    and manipulate neural networks. It contains not only basic neurons, but
    also more involved units like LSTM-Cells or Embedding-Layers (plus
     many more).


%
%  Labels
%
%  Created by Mustafa Youldash on 2014-04-25.
%  Copyright (c) 2014 Mustafa YOULDASH. All rights reserved.
%
\documentclass[12pt,a4paper,oneside,onecolumn]{article}

% .......................................... Preamble

% Include images and figures
\usepackage{graphicx}

% Use \usepackage[fleqn]{amsmath} to make the math environments left-aligned instead of centered
\usepackage[fleqn]{amsmath}

% .......................................... End

\begin{document}

	\section{Pictures}
	\label{sec:Pictures}

	\section{Referencing}

	I greeted in section \ref{sec:greetings}.

	\begin{figure}[htp]
		\centering
		\includegraphics[scale=0.6]{Directory}
		\caption{Install Directory}
		\label{fig:directory}
	\end{figure}
	Figure \ref{fig:directory} shows a snapshot of an install phase.

	\section{Greetings}
	\label{sec:greetings}

	Hello!

	\clearpage

	\section{Formulas}
	\label{sec:Formulas}

	\begin{equation} \label{eq:solve}
	x^2 - 5 x + 6 = 0
	\end{equation}

	\begin{equation}
	x_1 = \frac{5 + \sqrt{25 - 4 \times 6}}{2} = 3
	\end{equation}

	\begin{equation}
	x_2 = \frac{5 - \sqrt{25 - 4 \times 6}}{2} = 2
	\end{equation}

	and so we have solved equation \ref{eq:solve}

\end{document}


\chapter{Literature Review}
\label{litreview}

\section{Rasterisation rendering}
Modern graphical hardware is heavily specialised for the tasks involved in rasterisation. Rasterisation refers to the conversion of mathematically defined vector geometry to a raster image on screen.

There are two main operations involved in rendering using rasterisation: transformation, the act of converting geometry defined in 3D space to coordinates on the screen, taking into account perspective; and rasterisation itself, the act of converting this transformed vector geometry into a raster image for display. These operations, and the operations required to complete them, are the purpose for which modern graphics hardware is specialised.

\subsection{Transformation}
In graphics, the term transformation refers to the mathematical mapping of one set of coordinates to another. The main transformations involved in 3D rendering include translation, rotation, scaling, and projection. Translation is defined as a transformation which offsets a point by a certain amount in each axis, whereas rotation is a transformation which rotates a point around the origin by some angle in some axis. Scaling is a transformation which increases the distance from 0 in each axis by a multiplier.

\begin{figure}
	\centering
	\subfloat[Translation]{\includegraphics[width=2in]{translation.png}}
	~
	\subfloat[Rotation]{\includegraphics[width=2in]{rotation.png}}
	~
	\subfloat[Scaling]{\includegraphics[width=2in]{scaling.png}}

	\caption{Geometric transformations}
	\label{fig:transformations}
\end{figure}

Projection is a transformation which describes the mapping of three-dimensional points onto a two-dimensional plane. In order to give the rendered image perspective, we use a perspective projection. A perspective transformation takes into account the field of view (FOV) of the viewer, as shown in figure \ref{fig:perspective-projection}, in order to provide a perspective-correct representation of the scene.

\begin{figure}
\centering
	\includegraphics[width=14cm]{frustum.png}
	\caption{A perspective projection transformation}
	\label{fig:perspective-projection}
\end{figure}

In rasterisation rendering, we aim to take vector values defining the "world space" (position within the scene) coordinates of the geometry of the surfaces we are attempting to render and map them to the screen such that it produces a 2D image of the 3D scene.

In order to accomplish this, we consider three main transformations: perspective projection, which is defined by a projection transformation; the model transformation, which uses translation, rotation, and scale transformations in order to represent the object's position in the scene; and the view transformation, which uses a translation and a rotation transformation in order to represent the camera's position and rotation in the scene.

We use 4x4 matrices to represent these transformations, as this allows us to represent perspective projection and translation transformations (as well as the other required transformations.) As the matrices are square, an added advantage that they can be multiplied together to result in a combined transformation known as a model-view-projection (MVP) matrix.

This overall transformation defines the transformation of points from world space to a position on the viewport, in "viewport space". This normalised coordinate space, often from -1 to 1 on both axes, where -1 and 1 are the extremities of the field of view, must then be scaled to their final coordinates on the screen, in "screen space", based on the dimensions of the viewport. Vertices outside of this range are outside of the field of view of the projection.

This scaling can be described very simply. For a viewport space defined from -1 to 1 on both axes, where -1 represents the left and bottom of the viewport, and 1 represents the top and right of the viewport, this transformation can be done by adding 1 to both coordinates to get them in the range 0..2, dividing both coordinates by 2 to get them in the range 0..1, and multiplying by the dimensions of the viewport, to get the coordinates in screen space. This can be written algebraically as:

\begin{align}
	x_{screen} &= (1 + x_{viewport}) \cdot \frac{width_{viewport}}{2}
	\\
	y_{screen} &= (1 + y_{viewport}) \cdot \frac{height_{viewport}}{2}
\end{align}

In different rendering frameworks the definition of the viewport can vary, but the principles are the same. The basic transformations required to make this conversion are simply a 2D translation and scale.

\subsection{Rasterisation}
Once geometry has been transformed into screen space, the process of rasterising triangles in order to display them on the screen is incredibly simple (see section \ref{pct} on polygon scan conversion for one possible method.) However, this is not the only step which is required to produce a 2D representation of a 3D scene. The next problem is how we should colour each pixel in the rasterised triangle in order to make it look 3D. Once a pixel is determined to be inside the triangle, the pixel is then shaded in order to determine its final colour on the screen.

\subsubsection{Shading}
Shading refers the process of determining a colour for a pixel on the screen. With rasterisation-based rendering techniques, this colour is usually determined using attributes from the triangle. Estimations of a number of lighting effects are used in order to determine this colour. There are three main contributions normally taken into account with rasterisation-based rendering techniques: ambient reflection, diffuse reflection, and specular reflection.

\begin{figure}
	\centering
	\subfloat[Flat shading]{\includegraphics[width=3in]{flat_shading.png}}
	~
	\subfloat[Smooth shading]{\includegraphics[width=3in]{smooth_shading.png}}

	\caption{The difference between flat and smooth shading}
	\label{fig:flat-vs-smooth-shading}
\end{figure}

In order to calculate the way a surface is lit, a normal vector describing the normal of the plane on which a point on the surface sits. One approach is to use the normal of the plane on which the triangle sits, which produces results that are not smooth as shown in figure \ref{fig:flat-vs-smooth-shading}. On the other hand, an approach called Gouraud shading \parencite{gouraud71shading}, or smooth shading, aims to simulate the lighting of a smooth surface by defining a normal for each vertex, typically an average normal for all faces sharing that vertex, and then shade each vertex of the triangle, finally interpolating the colour in between to produce a smooth fade. This effect is demonstrated in \ref{fig:colour-interpolation} by defining each vertex to have one of the primary colours and then fading in between them, creating a smooth fade.

\begin{figure}
\centering
	\includegraphics[width=14cm]{interpolation.png}
	\caption{Colour interpolated across a triangle}
	\label{fig:colour-interpolation}
\end{figure}

\paragraph{Ambient reflection}
Ambient reflection refers to a small amount of light which is scattered around the scene. As rasterisation-based approaches often do not consider global effects, the ambient contribution to a surface's reflection intensity, $I_{ambient}$, is typically modelled as a constant for a given scene, depending on the desired effect.

\paragraph{Diffuse reflection}

\begin{figure}
\centering
	\includegraphics[width=8cm]{diffuse.png}
	\caption{An example of diffuse reflection}
	\label{fig:diffuse-reflection}
\end{figure}

Diffuse reflection refers to the scattering of light by a matte surface. Diffuse reflection is usually modelled using Lambert's cosine law, which describes the way light is scattered by a perfectly matte, or lambertian, surface.

Using Lambert's cosine law, the intensity of reflected light due to lambertian reflectance, $I_{diffuse}$, can be calculated by taking the dot product of the normalised light-direction vector $\vec{L}$ with the surface normal $\vec{N}$, as below:

\[
	I_{diffuse} = \vec{L} \cdot \vec{N}
\]

This value can then be multiplied by the colour of the surface and the intensity of the incoming light in order to determine the overall diffuse component of the surface's reflection.

\paragraph{Specular reflection}

\begin{figure}
\centering
	\includegraphics[width=8cm]{specular.png}
	\caption{The shiny look of this sphere is caused by specular reflection}
	\label{fig:specular-reflection}
\end{figure}

Specular reflection refers to the bright spots of light that are reflected by shiny objects. Specular highlights are visible where the surface normal lies half way between the light direction and the view direction.

Two main models are used to estimate the contribution of specular reflection to the overall reflection of a surface: the Phong shading model \parencite{phong75shading}, and a computationally lighter modification known as the Blinn-Phong shading model \parencite{blinn77blinnphong}.

These approaches improve upon Gouraud shading by interpolating the normal across the triangle, rather than just the shaded colour, increasing the computational intensity of shading, but allowing smooth specular highlights to be calculated.

\begin{figure}
\centering
	\includegraphics[width=14cm]{light_relationships.png}
	\caption{The relationship between the direction of viewing, light direction, the half-angle in between the direction of viewing and the light direction, and the normal of the surface being viewed. $\vec{L}$ is the normalised light vector which points from the surface being shaded to the light source. $\vec{N}$ is the normal of the surface being shaded. $\vec{H}$ is the direction vector half way between the angle of viewing, $\vec{V}$, and the angle of the light direction, $\vec{L}$}
	\label{fig:specular-direction-relationships}
\end{figure}

The specular component in Blinn-Phong shading is calculated by computing a half-way vector $\vec{H}$ between the view and light directions \parencite{blinn77blinnphong}:

\[
	\vec{H} = \frac{\vec{L} + \vec{V}}{|\vec{L} + \vec{V}|}
\]

Or, more simply:

\[
	\vec{H} = normalise(\vec{L} + \vec{V})
\]

This vector is shown in figure \ref{fig:specular-direction-relationships}.

The specular contribution to the overall reflection can then be calculated as:

\[
	I_{specular} = (\vec{H} \cdot \vec{N})^s
\]

Where s is the specular power, which determines the intensity of specular highlights produced by the surface. How varying the specular power affects the intensity of the specular reflections produced is shown in figure \ref{fig:specular-power}.

\begin{figure}
	\centering
	\subfloat[s = 16]{\includegraphics[width=2in]{specular_s16.png}}
	~
	\subfloat[s = 32]{\includegraphics[width=2in]{specular_s32.png}}
	~
	\subfloat[s = 64]{\includegraphics[width=2in]{specular_s64.png}}

	\caption{The effect of varying specular power}
	\label{fig:specular-power}
\end{figure}

\paragraph{Overall reflection}
The overall reflection of these surfaces can then be calculated additively:

\[
	I_{overall} = I_{ambient} + I_{diffuse} + I_{specular}
\]

\subsubsection{Blending}
Blending is another aspect of shading, which allows rasterised surfaces to give the appearance of partial or full transparency. By adding a fourth component to colours, an alpha component, we can define how opaque an object is, where a surface with an alpha value of 1 is fully opaque, and a surface with an alpha value of 0 is fully transparent.

By considering the alpha component of surfaces, and applying blending to additively combine the surface's colour with the background colour, the background colour is allowed to show through, as shown in figure \ref{fig:alpha-blending}.

\begin{figure}
\centering
	\includegraphics[width=14cm]{alpha_blending.png}
	\caption{Demonstration of alpha blending - a cube is alpha blended with the sphere in the background to give the appearance of transparency}
	\label{fig:alpha-blending}
\end{figure}

\section{Rasterisation-based volume rendering}
Researchers have used many approaches to map volume rendering operations directly to rasterisation hardware, as modern graphics hardware is extremely specialised for the purposes of rasterisation. As this sort of hardware is ubiquitous in modern computers, methods of rendering volumes in real-time using this hardware are highly sought-after. In order to understand why these approaches are not suitable for achieving our objectives, it is important to understand these approaches in order and appreciate their limitations.

\subsection{Billboards}
Billboards are texture-mapped planes which typically have no rotation component. In other words, they are always facing directly at the camera. Billboards have long since been used to render complex 3D objects in real time. By reducing complex geometry to relatively few billboards, it is possible to create a convincing enough looking representation of a 3D object. By utilising transparency and alpha blending capabilities of rasterisation hardware, it is possible to allow multiple layers of billboards to show through.

A common example as used in games is for tree rendering. A technique demonstrated by \citeauthor{garcia05tree} utilises a set of billboards, referred to in the paper as a cloud of billboards, in order to vastly simplify complicated tree models defined by hundreds of thousands of polygons into a few fixed billboards, each of which can approximate many leaves. This technique is demonstrated in figure \ref{fig:billboard-cloud-trees}.

\begin{figure}
\centering
	\includegraphics[width=14cm]{tree_billboards.png}
	\caption{Billboard cloud tree rendering}
	\label{fig:billboard-cloud-trees}
\end{figure}

Similar techniques have also been successfully developed for volume rendering. \cite{harris02real} combined the simulation of multiple forward scattering within a cloud volume with billboards used to accelerate rendering. By utilising impostors, billboards subtly intended to replace 3D objects, this approach accelerates rendering by rendering to the billboards and reusing the results between frames. This method allows for realistic real-time rendering of clouds.

The limitation of this approach is that it does not simulate global effects, and as such, the rest of the scene does not have an effect on the rendering of an individual cloud volume. In fact, the cloud rendering approach utilised in this paper is very specific in the effects which it simulates, simulating very homogeneous looking clouds that are not affected very much by the scene or surrounding clouds. Although this may be a suitable compromise for some games, it falls short of our objectives to simulate true volumetric effects in real-time. Nonetheless, the impostor technique may be suitable in some cases in order to exploit frame-to-frame coherence.

\subsection{Texture-based approaches}
\cite{engel02interactivehigh-quality} proposes an approach that utilises a 3D texture as proxy geometry, exploiting hardware alpha blending as a means to composite overlapping samples in order to create a 3D image of a volume.

The advantage of this approach is that it maps onto existing rasterisation hardware very well, allowing real-time volume rendering on relatively cheap, ubiquitous graphics hardware. The major limitation of this method, however, is that it only considers overlapping parts of the volume. If one wishes to calculate lighting effects which involve light being reflected or refracted in different directions as we do, this approach is not suitable. In addition, \citeauthor{engel02interactivehigh-quality} found the precision of rasterisation hardware limited when it came to texture representations, which resulted in artifacts in the rendered image.

\begin{algorithm}
\caption{Per-pixel rendering}
\label{alg:render}
\begin{algorithmic}[1]
\Procedure{render\_pixel}{$x, y, tree, projection$}
	\State $out\_colour \gets (0, 0, 0, 0)$
	\State $ray \gets calculate\_ray(x,~y,~projection)$
	\State $intersection \gets \Call{raycast}{tree, ray}$
	\If{$intersection.hit$}
		\State $out\_colour \gets \Call{shade}{tree,~ ray,~ intersection}$
	\EndIf
	\State $\textbf{return} out_colour$
\EndProcedure
\\
\Procedure{shade}{$tree,~ ray,~ intersection$}
	\State $out\_colour \gets (0, 0, 0, 0)$
	\For{$each~light$} \Comment{Shade for each light}
		\State $light\_dir \gets light\_pos - intersection.position$ \Comment{Check shadow}
		\State $shadow\_ray \gets (intersection.position, light_dir)$
		\State $shadow\_intersection \gets \Call{raycast}{tree, shadow\_ray}$
		\If{$shadow\_intersection.hit$}
			\State $\textbf{continue}$
		\EndIf
		\State $out\_colour ~+=~ \Call{diffuse\_reflection}{intersection,~light}$
		\State $out\_colour ~+=~ \Call{specular\_reflection}{intersection,~light}$
	\EndFor
	\\
	\State $reflection\_ray \gets \Call{reflect}{ray,~intersection}$ \Comment{Reflection}
	\State $reflection\_intersection \gets \Call{raycast}{tree, reflection\_ray}$
	\If{$reflection\_intersection.hit$}
		\State $out\_colour ~+=~ \Call{shade}{tree,~ reflection\_ray,~ reflection\_intersection}$
	\EndIf
	\\
	\State $refraction\_ray \gets \Call{refract}{ray,~intersection}$ \Comment{Refraction}
	\State $refraction\_intersection \gets \Call{raycast\_empty}{tree, refraction\_ray}$
	\If{$refraction\_intersection.hit$}
		\State $out\_colour ~+=~ \Call{shade}{tree,~ refraction\_ray,~ refraction\_intersection}$
	\EndIf
\EndProcedure
\end{algorithmic}
\end{algorithm}

\section{Ray tracing}
A more generic approach called ray tracing, similar to the approach in \cite{engel02interactivehigh-quality}, aims to simulate the path of light in reverse. Ray tracing starts by firing rays outwards into the scene from the camera \parencite{appel68raytracing}, traditionally firing one ray per pixel and finding the first intersection of the scene with the ray.

\begin{figure}
\centering
	\includegraphics[width=14cm]{primaryray.png}
	\caption{In ray tracing, primary rays are cast through each pixel on the viewport}
	\label{fig:primaryray}
\end{figure}

Once an intersection is found, additional rays can be spawned in order to take account of the effects of reflection and refraction. Unlike \citeauthor{engel02interactivehigh-quality}'s approach, this allows the ray of light to follow a path that isn't just straight, allowing rendering to account for the contribution to lighting of other parts of the scene that aren't directly overlapping a particular pixel.

\begin{figure}
\centering
	\includegraphics[width=14cm]{secondaryrays.png}
	\caption{Additional rays can then be spawned for each intersection to calculate shadow coverage, reflection, and refraction}
	\label{fig:secondaryrays}
\end{figure}

In addition to allowing light reflected from the rest of the scene to be taken into account, rays can also peer directly into volumes in order to determine the effects of lighting within them. This allows us to consider volumes that are not just homogeneous in nature, as the ray can be affected within the volume by the internal structure of the volume.

\subsection{Direct volume rendering}
Ray tracing allows the direct rendering of a volume, without having to first sample the underlying volume into another format. Any volume (or structure) for which an intersection test can be defined can be rendered by means of ray tracing.

\cite{roth82roughimprove} presented an algorithm for directly rendering constructive solid geometry (CSG), a method for representing 3D shapes which involves combining various geometric shapes using boolean operations. This work described the calculation of eye rays for orthographic as well as perspective projections, as well as a hierarchical data structure for storing and rendering CSG directly. This work also described a number of primitives as well as the relevant necessary intersection tests. Additionally, \citeauthor{roth82roughimprove} demonstrated the application of this algorithm for producing realistically shaded solids for use in computer aided design, and also used shadow rays in order to produce ray traced shadows, and even described a method for anti-aliasing of the resulting image.

The direct rendering of volumes simplifies computation, removing the need for us to rely on intermediate representations of volumes, and allowing us to consider the volume itself as all information may be relevant to the rendering of the current pixel.

Although this allows broad coverage of geometric shapes, it is difficult to represent highly heterogeneous volumes with CSG, as it would require a very large number of separate geometric shapes to model changes in heterogeneous attributes. In addition, it would be impossible to render smoothly changing heterogeneous attributes without a very large number of geometric shapes, at which point it would be more efficient to use a grid-based sampling approach. For this reasons, it is unsuitable for our objectives.

\subsection{Space subdivision}
The next important advancement in ray casting technology was presented by~\cite{glassner84space}. By using an octree, a hierarchical volume representation, to partition space, and storing a list of objects in each node of the tree, he greatly reduced the number of comparisons required for each ray. This allowed the rendering of hundreds, or even thousands of objects in a scene without greatly increasing the computation time for each object. Despite this, an object format which is capable of storing heterogeneous volumes is still required.

On the other hand, \cite{wilhelms00octreesfor} presented a space-efficient octree representation that is designed for representing voxel data directly. By not storing empty space within the tree, a technique referred to as sparse voxel octrees (SVOs), a considerable amount of space is saved, reducing the memory footprint of the stored voxel data. Additionally, as the voxel data is stored directly using this kind of approach, it is possible to represent heterogeneous volumes with this structure.

\cite{amanatides87afast} presented a significant optimisation to the traditional ray tracing algorithm, by presenting a method of mathematically determining the next voxel in a traversal and automatically stepping directly to its boundary. Using this new algorithm, traversing from one voxel to its neighbour requires only two floating point comparisons and one floating point addition. This greatly reduces computation time for ray casts, and is the basis for a large amount of derivative work on efficient ray casting.

\cite{arvo88octreewalking} extends this technique to the octree, presenting an efficient linear-time voxel walking algorithm for octrees with a best case complexity of O(log N). \citeauthor{arvo88octreewalking}'s approach is a top-down approach which ensures that nodes are only considered once per ray and visits the voxels in the correct order, resulting in the O(log N) best case complexity. \citeauthor{arvo88octreewalking}'s algorithm involves checking the intersection of the ray with the root node of the octree, and recursively shortening the spans of the ray that are considered for intersection.

\subsection{GPU implementation}
Due to the fact that ray tracing utilises one primary ray per pixel, a fact that both limits it and works to its advantage in its basic implementation, the process is highly parallelisable. Additionally over the last decade, a breakthrough occurred in graphics hardware, which saw GPUs becoming massively parallel general-purpose stream processors, as opposed to the original fixed-function format. \cite{purcell02gpuraytracing} investigates this trend and explains how ray tracing can be mapped to graphics hardware. \citeauthor{purcell02gpuraytracing} demonstrate how to reformulate ray tracing as a streaming computation, making it appropriate for GPU implementation. The process is as follows:

% Hide number from itemize list item
\let\oldlabelitemi\labelitemi
\renewcommand{\labelitemi}{}

\begin{itemize}
	\item For each pixel of the screen:

	\begin{enumerate}
		\item Generate eye rays
		\item Traverse acceleration structure

			\begin{enumerate}
				\item Do intersection tests
			\end{enumerate}

		\item Shade hit

			\begin{enumerate}
				\item Generate shading rays (repeating 2-3 as necessary)
			\end{enumerate}

	\end{enumerate}
\end{itemize}

\citeauthor{purcell02gpuraytracing} were the first to demonstrate that efficient real-time ray tracing in the GPU is possible without any changes in architecture.

% Restore old itemize list item style
\let\labelitemi\oldlabelitemi

\subsection{Efficient GPU traversal}
\cite{aila2009hpg} discusses the mapping of elementary ray tracing operations such as traversal and intersection onto GPUs. This work focuses on understanding the efficiency of GPU ray traversal rather than directly presenting an efficient ray traversal implementation, highlighting the fact that very little is understood about the performance of fast ray traversal algorithms. By comparing the performance of these algorithms against a theoretical upper bound, \citeauthor{aila2009hpg} demonstrate that previous methods are off by a factor of 1.5x - 2.5x theoretical optimal performance, and highlight previously unidentified inefficiencies in hardware work distribution.

Their work demonstrates that reliance on persistent threads rather than hardware work distribution mechanisms can improve the performance of the fastest GPU trace() kernels significantly.

\subsection{Efficient octree storage}
\cite{wilhelms00octreesfor} present a space-efficient octree representation that does not allocate memory for empty space - what has become known as a sparse voxel octree (SVO). \citeauthor{wilhelms00octreesfor}'s structure utilises a pointer-less "linear octree" structure which avoids storing pointers between every node.

\cite{laine10efficientsvos} expand on these ideas by storing pointers between blocks of octrees and otherwise determining a particular child's address using an index and a parent's 8-bit child mask. As the child's pointer can be obtained from a lookup table using this index and child mask, it is unnecessary to store a pointer for every voxel, making the structure far more efficient.

As \citeauthor{laine10efficientsvos} demonstrate, the SVO structure can be efficiently ray traced with an implicit level of depth (LOD) mechanism, as traversal can be terminated as soon as a found voxel is smaller than the current pixel on the screen. At high resolutions, however, even with the efficient SVO storage structure presented by \citeauthor{laine10efficientsvos}, mid-sized scenes still take up an excessive amount of storage, with \citeauthor{laine10efficientsvos}'s test scenes utilising up to 4GBs of memory, the upper bounds of memory on today's GPUs.

However, while extremely efficient to ray cast, and fairly compact when compared to other structures, \citeauthor{laine10efficientsvos}'s structure is still fairly heavy on memory usage. In their paper, \citeauthor{laine10efficientsvos} address this by only encoding surfaces, an approach that is obviously not possible if volumetric effects are desired.

To combat memory usage, they additionally implement a concept called contours, in which surface voxels are restrained to a non-cubic shape using a pair of parallel planes, and once this shape provides a good enough approximation of the surface, the tree does not need to be generated any further. \citeauthor{laine10efficientsvos}'s implementation for this is very efficient, and automatically takes into account multiple levels of contours in order to produce a shape extremely close to the source data. A similar process could also be used to prevent unnecessary depth in homogeneous regions, where the current approximation of the volume is sufficiently good.

Additionally, more recent work by \citeauthor{kampe2013dags} has greatly reduced the memory usage of SVOs by allowing identical regions of the tree to share pointers, allowing the reuse of data between regions. This new structure is referred to as a direct analytic graph, or sparse voxel DAG. \citeauthor{kampe2013dags} have found that, using this technique, the memory usage of even highly irregular scenes can be reduced by 1 to 3 orders of magnitude. Despite this being most effective for more homogeneous scenes, the gains shown for highly irregular scenes are also very promising.

\subsection{Animation}
Due to the nature of deformation, animation has previously not been possible with voxel structures because it would leave gaps between voxels. Additionally, deforming a hierarchical structure means that there is no longer a guarantee that all child nodes lie inside their parent nodes. \cite{bautembach2011animated} presented an approach to the animation of voxel octrees by expanding the parent voxels so that they still encompass all child voxels. \citeauthor{bautembach2011animated}'s approach demonstrates an efficient and effective approach capable of character animation with sparse voxel octrees, something which had previously been considered infeasible.

\subsection{Caching}
\cite{ruff13dynamiccaching} presented a caching-like strategy which is capable of storing data generated in previous frames which can then be used to accelerate rendering in later frames. \citeauthor{ruff13dynamiccaching} claim that their approach can be integrated to any existing ray tracing solution, and will allow data to be reused between frames. A significant limitation of this approach, however, is that it is tailored to static scenes. \citeauthor{ruff13dynamiccaching} point out that any movement of the scene objects would generate inconsistencies in cached data, although they do provide some potential solutions to this problem.

\subsection{Limitations of ray tracing}

\begin{figure}
\centering
	\centering
	\subfloat[hard shadows]{\includegraphics[width=3in]{hard_shadows.png}}
	~
	\subfloat[soft shadows]{\includegraphics[width=3in]{soft_shadows.png}}

	\caption{The hard edges of shadows generated by ray tracing, next to the soft shadows created by path tracing}
	\label{fig:shadow-hard-edge}
\end{figure}

Despite the numerous advantages of ray tracing, it has one major disadvantage: ray tracing can not simulate fuzzy phenomena, such as soft shadows. Figure \ref{fig:shadow-hard-edge} demonstrates shadows as simulated by ray tracing. Shadows produced by ray tracing are defined by explicit boundaries, where in real life, the edges of shadows are not well defined. In order to understand why ray tracing has this limitation, it is important to understand exactly what it is trying to approximate.

\begin{figure}
\centering
	\includegraphics[width=14cm]{tree_shadow.jpg}
	\caption{The soft edges of a real shadow}
	\label{fig:shadow-soft-edge}
\end{figure}

\subsubsection{The Rendering Equation}
\cite{kajiya86therendering} presented a single integral equation which generalises a variety of rendering techniques. This equation has become known as "The Rendering Equation", the title of his 1986 paper. Solving it produces an accurate simulation of many effects, including soft shadows, depth of field, ambient occlusion, global illumination and motion blur. The physical basis to this equation is as an approximation to Maxwell's equation for electromagnetism. Solving this equation is the main challenge in realistic rendering~\parencite{dimov07numericalmethods}.

Ray tracing evaluates this integral at a single point, the point of the intersection of a ray with the scene, and as such, it is not a good approximation. For this reason, ray tracing is known as a point-sampling approach. A number of approaches have been developed to address this limitation, such as cone tracing, distributed ray tracing, and path tracing.

\begin{figure}
\centering
	\includegraphics[width=14cm]{aliasing.png}
	\caption{The aliased edges caused by point sampling approaches}
	\label{fig:ray-tracing-aliases}
\end{figure}

\subsubsection{Cone tracing}
\citeauthor{amanatides84conetracing} presented a new method of ray tracing using cones rather than half-line rays in order to sample each pixel. The spread angle of the cone is chosen such that the radius of the cone's base is the size of the pixel according to the projection. The difficulty in using this method is the complexity of the intersection tests of a cone with the scene. \citeauthor{amanatides84conetracing} describes the necessary intersection calculations between a cone and various objects.

As cone tracing is no longer a point-sampling method, anti-aliasing is no longer required, and yet still only one sample per pixel is required. \citeauthor{amanatides84conetracing}' approach also allows for built in level of detail, soft shadows and blurred reflections. Given efficient intersection tests, this allows cone tracing to achieve similar performance to ray tracing with numerous advantages.

\subsubsection{Distributed ray tracing}
\cite{cook84distributed} also provides a solution to these problems with traditional ray tracing. By distributing the directions of the rays according to the analytic functions they sample, \citeauthor{cook84distributed}'s approach allows ray tracing to render fuzzy phenomena. By utilising this technique, \citeauthor{cook84distributed} enable ray tracing to produce blurred reflections, translucency and soft shadows, motion blur and depth of field.

The advantage of this approach over \citeauthor{amanatides84conetracing}' cone tracing approach is that the ray intersection calculations do not change, merely the ray directions are distributed~\parencite{cook84distributed}. Similarly to \cite{amanatides84conetracing}, the approach presented in this paper also does not require any more rays than vanilla ray tracing. The disadvantage to this approach is that the rays are still point-sampled, and therefore, anti-aliasing techniques are still required in order to reduce the effects of aliasing on the edges of objects.

\subsubsection{Path tracing}
\citeauthor{kajiya86therendering}'s own approach involves a Monte Carlo solution of the rendering equation. This Monte Carlo solution involves randomly distributing rays over the integral's domain, and then averaging them. This produces an extremely accurate and unbiased image, which naturally simulates many visual effects, including soft shadows, global illumination, and depth of field. Unfortunately, in order to get high quality images from this approach, a large number of rays must be traced in order to avoid very visible noisy artifacts.

Despite this, \citeauthor{bikker13brigade} have managed to get a path tracer running in real-time by utilising the persistent threads technique demonstrated by \cite{aila2009hpg}. This approach still displays noisy artifacts at first however, with the approximation improving over time as the camera is stationary and more samples are averaged. Despite this, the noise is still being improved, and with more computing power, the ability to average more samples per pixel faster will naturally reduce the amount of noise generated over time.


% -*- TeX-master: "master.tex" -*-

\setcounter{section}{9}
\setcounter{subsection}{0}
\subsection{Natural Transformations}

Natural transformation is the third thing in the most important foundations of
category theory. The first is the definition of category, and the second is
functor, natural transformation is the third.

Category is about structure. Functors are mappings between categories that
preserves the structure. The intuition about functor is that they take a
category and embed it into another category. It's like searching a pattern in a
category. We would also want to compare two functors, or said differently, images
given by functors. So compare things by mapping one into another, that's a
natural way of modeling comparsion.

Natural transformations are defined as structure-preserving mappings between
functors. Given two categories, $C$ and $D$, and two functors $F$ and $G$ which
$F,G: C\to D$, let $a$ be an object in $C$, which $F$ maps $a$ to $F a$, $G$
maps $a$ to $G a$. If we want to map one functor to another functor, we want to
map $F a$ to $G a$. We already have morphisms in $D$ for mapping $F a$ to $G a$.
To define mapping between functors on $a$ is the same as picking a morphism in
$D$ that maps from $F a$ to $G a$. So a natural transformation will, for every
object $a$ in $C$, pick a morphism in $\Hom_D(F a, G a)$. This way a natural
transformation is creating a family of morphisms in $D$. These individual
morphisms are called \emph{components} of the natural transformation. Denoted as
$\alpha_a$ where $\alpha$ is the natural transformation.

So far, we only talked about what a natural transformation does on objects, we
also need to take a look on morphisms. Given objects $a, b$ in $C$ and morphism
$f: a \to b$, we have the functors $F, G$ which map $a$ to $F a$, $G a$, and map
$b$ to $F b$, $G b$, respectively. Let natural transform $\alpha$ on $a$ and
$b$ be $\alpha_a: F a\to G a$ and $\alpha_b: F b \to G b$.

\begin{figure}[h]
\centering
\begin{tikzcd}[math mode=true, sep=large]
  a \ar{rrr}{F}\ar{rrrdd}[near start]{G}\ar[green]{d}{f} & & & F
  a\ar[red]{dd}[near start]{\alpha_a}\ar[green]{rd}{F f} & \\
  b \ar[crossing over]{rrrr}{F}\ar{rrrrdd}{G} & & & & F b\ar[red]{dd}{\alpha_b} \\
  & & & G a\ar[green]{rd}{G f} & \\
  & & & & G b
\end{tikzcd}
\caption{natural transofrm}
\end{figure}

We need to have the natural transform preserve structures of $F$ and $G$. That
means these two paths from $F a$ to $G b$ must be equal. In other words, the
diagram must commute.

Naturality condition is very strong

Example in set. Natruality is almost determined.


\chapter{Background}\label{chap:background}
In this chapter we will briefly introduce some basic model definitions and notation conventions for the following chapters.

\section{Basic Model Definitions and Notations}\label{sec:backgroundBasic}
A Setup $A$ is defined by
\begin{itemize}
    \item a set $S$ of $M$ Sources: $S=\{S_1,S_2,...,S_M\}$,
    \item a set $R$ of $N$ Receivers: $R=\{R_1,R_2,...,S_N\}$,
    \item a set $Q$ of $K$ Obstacles: $Q=\{Q_1,Q_2,...,Q_K\}$ and
    \item a tuple of physical properties: $(\vartheta,\phi, p)$
\end{itemize}
where
\begin{conditions}
    M           & number of source nodes, $M \in \mathbb{N}$ \\
    N           & number of receiver nodes, $N \in \mathbb{N}$ \\
    K           & number of obstacles, $K \in \mathbb{N}$ \\
    \vartheta   & temperature in $[^{\circ}\textrm{C}]$ \\
    \phi        & relative humidity, $\phi \rightarrow [0, 1]$ \\
    p           & atmospheric pressure in $[\textrm{Pa}]$
\end{conditions}

An Environment $E$ is defined by
\begin{itemize}
    \item a set $Q$ of $K$ Obstacles: $Q=\{Q_1,Q_2,...,Q_K\}$ and
    \item a tuple of physical properties: $(\vartheta,\phi, p)$
\end{itemize}

Therefore a Setup $A$ can also be described by
\begin{itemize}
    \item a set of $M$ Sources: $\{S_1,S_2,...,S_M\}$,
    \item a set of $N$ Receivers: $\{R_1,R_2,...,S_N\}$ and
    \item an Environment $E$
\end{itemize}

\section{Spherical Coordinates}
For the spherical coordinates, in particular the spherical angles, the \emph{mathematical} convention as shown in \figref{fig:spherical} will be used.
\input{figures/background/figSpherical}
where
\begin{conditions}
    r       & radial distance, $r \rightarrow \mathbb{R}$ \\
    \theta  & azimuthal angle, $\theta \rightarrow (-\pi, \pi]$ \\
    \varphi & polar angle, $\varphi \rightarrow [0, \pi]$
\end{conditions}


\section{Direct Derivation for Expected 2-call@k}

\label{appendix.2call}

The following derivation serves as a specific example to help understanding the derivation of optimizing expected $n$-call@$k$ for general $n \geq 1$. This is a straightforward extension to optimizing expected 1-call@$k$, utilizing a similar derivational approach based on a logical expansion of the optimization objective:
\begin{align*}
  s_k^* & = \argmax_{s_k} \mathbb{E} \!\left[ R_k \geq 2 \left| S_{k-1}^*, s_k, \vec{q} \right.\right] \\
%%
  & = \argmax_{s_k} \mathbb{E} \!\Bigg[ (r_1=1 \wedge r_2=1) \vee (r_1=0 \wedge r_2=1 \wedge r_3=1) \vee (r_1=1 \wedge r_2=0 \wedge r_3=1) \,\vee \\
  & \hspace{21.7mm} (r_1=0 \wedge r_2=0 \wedge r_3=1 \wedge r_4=1) \vee (r_1=0 \wedge r_2=1 \wedge r_3=0 \wedge r_4=1) \,\vee \\
  & \hspace{21.7mm} (r_1=1 \wedge r_2=0 \wedge r_3=0 \wedge r_4=1) \vee \dots \vee \\
  & \hspace{21.7mm} (r_1=0 \wedge \dots \wedge r_{k-2}=0 \wedge r_{k-1}=1 \wedge r_k=1) \,\vee \\
  & \hspace{21.7mm} (r_1=0 \wedge \dots \wedge r_{k-3}=0 \wedge r_{k-2}=1 \wedge r_{k-1} = 0 \wedge r_k=1) \vee \dots \vee \\
  & \hspace{21.7mm} (r_1=1 \wedge r_2=0 \wedge \dots \wedge r_{k-1}=0 \wedge r_k=1) \Bigg| S_{k-1}^*, s_k, \vec{q} \,\Bigg] \\
%%
\end{align*}

Next we reorganize our formula to isolate those terms mentioning $r_k$:
\begin{align*}
 s_k^* & = \argmax_{s_k} \mathbb{E} \!\Bigg[ (r_1=1 \wedge r_2=1) \vee (r_1=0 \wedge r_2=1 \wedge r_3=1) \vee (r_1=1 \wedge r_2=0 \wedge r_3=1) \,\vee \\
  & \hspace{21.7mm} (r_1=0 \wedge r_2=0 \wedge r_3=1 \wedge r_4=1) \vee (r_1=0 \wedge r_2=1 \wedge r_3=0 \wedge r_4=1) \,\vee \\
  & \hspace{21.7mm} (r_1=1 \wedge r_2=0 \wedge r_3=0 \wedge r_4=1) \vee \dots \vee \\
  & \hspace{21.7mm} \left.\left. \bigvee_{j=1}^{k-1} \left( r_k=1 \wedge \bigwedge_{\substack{i=1 \\ i \neq j}}^{k-1} r_i=0 \wedge r_j=1 \right) \right| S_{k-1}^*, s_k, \vec{q} \,\right]
\end{align*}

Now we rewrite disjoint events as a sum of probabilities, ignoring constant probabilities independent of $r_k$ that do not change the $\argmax_{s_k}$ results:
\begin{align*}
 s_k^* & = \argmax_{s_k} \sum_{j=1}^{k-1} P \!\!\left(\left. r_k=1 \wedge \bigwedge_{\substack{i=1 \\ i \neq j}}^{k-1} r_i=0 \wedge r_j=1 \right| S_{k-1}^*, s_k, \vec{q} \,\right)
\end{align*}

Next we formally evaluate the above query w.r.t.\ our latent subtopic binary relevance model in Figure~\ref{fig:gm} and simplify:
\begin{align*}
 s_k^* & = \argmax_{s_k} \sum_{j=1}^{k-1} \left( \sum_{t_1, \dots, t_k, t} P(t|\vec{q}) \, P(t_k|s_k) \mathbb{I} [t_k=t] \, P(t_j|s_j^*) \mathbb{I} [t_j=t] \prod_{\substack{i=1 \\ i \neq j}}^{k-1} P(t_i|s_i^*) \mathbb{I} [t_i \neq t] \right) \\
  & = \argmax_{s_k} \sum_t P(t|\vec{q}) \, P(t_k=t|s_k) \sum_{j=1}^{k-1} \left( P(t_j=t|s_j^*) \prod_{\substack{i=1 \\ i \neq j}}^{k-1} \left( 1 - P \! \left( t_i=t|s_i^* \right) \right) \right)
\end{align*}

Assuming that $\forall i \; P(t_i|s_i) \in \{0,1\}$ and $P(t|\vec{q}) \in \{0,1\}$, the objective becomes:
\begin{align}
  s_k^* & = \argmax_{s_k} \sum_t P(t|\vec{q}) \, P(t_k = t | s_k) \sum_{j=1}^{k-1} \left( P(t_j=t|s_j^*) \prod_{\substack{i=1 \\ i \neq j}}^{k-1} \left( 1 - P \! \left( t_i=t|s_i^* \right) \right) \right) \nonumber
\end{align}

Exploiting the same inversion idea used to replace $\Box$ in~\eqref{eq:partial_simp}:
\begin{align}
  s_k^* & = \argmax_{s_k} \sum_t P(t|\vec{q}) \, P(t_k = t | s_k) \sum_{j=1}^{k-1} \left( \!\! P(t_j=t|s_j^*) \! \left[ 1 - \left( \! 1 - \prod_{\substack{i=1 \\ i \neq j}}^{k-1} \left( 1 - P \! \left( t_i=t|s_i^* \right) \right) \!\! \right) \right] \right) \nonumber
\end{align}

Distributing $P(t_j=t|s_j^*)$, the relevance and diversity components begin to emerge:
\begin{align}
 s_k^* & = \argmax_{s_k} \sum_t P(t|\vec{q}) \, P(t_k = t | s_k) \nonumber \\
  & \hspace{30mm} \sum_{j=1}^{k-1} \left[ P \! \left( t_j=t|s_j^* \right) - P \! \left( t_j=t|s_j^* \right) \left( 1 - \prod_{\substack{i=1 \\ i \neq j}}^{k-1} \left( 1 - P \! \left( t_i=t|s_i^* \right) \right) \right) \right] \nonumber
\end{align}

Finally, making the deterministic subtopic probability assumption that $P(t_k = t | s_k) \in \{ 0,1 \}$ and $P(t_j=t|s_j^*) \in \{ 0, 1 \} $, we obtain the final result:
\begin{align}
  &s_k^* = \argmax_{s_k} \sum_t P(t|\vec{q}) \, P(t_k = t | s_k) \sum_{j=1}^{k-1} P(t_j=t|s_j^*) \nonumber \\
  & \hspace{16mm} - \sum_t P(t|\vec{q}) \, P(t_k = t | s_k) \sum_{j=1}^{k-1} P(t_j=t|s_j^*) \max_{\substack{ i \in [1, k-1] \\ i \neq j}} P(t_i=t|s_i^*) \nonumber
\end{align}

We remark that this is of the same form as~\eqref{eq.ncall.alt2} in
Section~\ref{subsec:ncall} when $n=2$, albeit derived in a much
simpler fashion under this more restrictive case.  The derivation for
2-call@$k$ from this point proceeds identically to that already
provided following~\eqref{eq.ncall.alt2}.


%% SECTION HEADER /////////////////////////////////////////////////////////////////////////////////////
\section{Structural Damping}
\label{sec:damping}

%% SECTION CONTENT ////////////////////////////////////////////////////////////////////////////////////
Propagating waves in the structure attenuate due to many factors, including geometric spreading, material damping, and dissipation into the adjacent domain.
This study adopted the Rayleigh damping model for the \ac{cfrp} skin and adhesive layer, while the damping for the aluminium core and \ac{pzt} was neglected.
Rayleigh damping model is defined as \cite{wandowski2017guided}
\begin{eqnarray}
	\textbf{D}_{dd}^e = \alpha_M \textbf{M}_{dd}^e + \beta_K \textbf{K}_{dd}^e.
	\label{eq:damping}
\end{eqnarray}
where \(\alpha_M\) and \(\beta_K\) are the mass- and stiffness- proportionality coefficients. However, \(\beta_K\) is equal to zero to ensure that matrix \(\textbf{D}_{dd}\) remains diagonal.