\documentclass[11pt, oneside]{article}   	% use "amsart" instead of "article" for AMSLaTeX format
\usepackage{geometry}                		% See geometry.pdf to learn the layout options. There are lots.
\geometry{letterpaper}                   		% ... or a4paper or a5paper or ...
%\geometry{landscape}                		% Activate for rotated page geometry
%\usepackage[parfill]{parskip}    		% Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}				% Use pdf, png, jpg, or eps§ with pdflatex; use eps in DVI mode
								% TeX will automatically convert eps --> pdf in pdflatex
\usepackage{amssymb}

%SetFonts

%SetFonts

\setlength\parindent{0pt}

\title{Why Use Logarithmic Returns?}
\author{Lucas Louca}
%\date{}							% Activate to display a given date or no date
\usepackage{amsmath}

\begin{document}
\maketitle
%\section{}
%\subsection{}

\section{Preliminaries}
Calculating simple returns is done using:
\begin{equation}
R=\frac{P_i - P_j}{P_j}
\end{equation}

To find the total return over a period $T$ with $n$ sub-periods, we need to compound the growth for each sub-period, as shown here:
\begin{equation}
	P_f = P_0(1+R_1)(1+R_2)...(1+R_n)
\end{equation}
with:
\begin{center}
	\begin{itemize}
		\item $P_f$ - final Price
		\item $P_0$ - initial Price
		\item $R_x$ - Return for each sub-period.
	\end{itemize}
\end{center}

If each return $R_x$ were identical, we could write this in a simplified exponential form:
\begin{equation}
	P_f =P_0(1+R_n)^n
\end{equation}
But, the return in each sub-period is rarely identical, so compounding in this manner is impossible. For this reason, we turn our attention to logarithmic returns. Logarithmic returns measure the rate of exponential growth. Instead of measuring the percent of price change for each sub-period, we measure the exponent of its natural growth during that time. Later, we can add each sub-period's exponential growth to get the total growth for the period $T$.
\\
\\
If you invest an amount  $P_1$ for one year at an $r\%$ interest rate, at the end of the year you'll have an amount:
\begin{equation}
	P_2 = P_1(1+r)
\end{equation}
For example, with a $4\%$ return, $P_2 = 1.04P_1$. Sometimes your interest might be compounded, for example, you get your money back plus $2\%$ at the end of six months, and then can earn $2\%$  interest on both your original principal as well as on the  first six month's interest:
\begin{equation}
	P_2 = P_1(1+\frac{r}{2})(1+\frac{r}{2}) = P_1(1+\frac{r}{2})^2
\end{equation}
Similarly, compounded quarterly you will get:
\begin{equation}
	P_2 = P_1(1+\frac{r}{4})^4
\end{equation}
It turns out the formula converges to a particular function as the frequency of compounding $n$ gets arbitrarily large, this limit being:
\begin{equation}
	\displaystyle{\lim_{x \to \infty}}(1 + \frac{r}{n})^n = e^r
\end{equation}
Thus if you earn $r\%$ interest that is \emph{compounded continuously}, at the end of the year your money will be:
\begin{equation}
	P_2 = P_1\displaystyle{\lim_{x \to \infty}}(1 + \frac{r}{n})^n = P_1e^r
\end{equation}
or grown by a factor:
\begin{equation}
	\frac{P_2}{P_1} = e^r
\end{equation}
where $e$ is a special number (approximately equal to $2.72$). Taking the natural logarithm is just the inverse of the above operation:
\begin{equation}
	r = ln(\frac{P_2}{P_1})
\end{equation}
$r = ln(\frac{P_i}{P_j})$ is referred to as the \emph{logarithmic return}. Often times you will see the following equation:
\begin{equation}
	ln(R_i + 1) = ln(\frac{P_i}{P_j})
\end{equation}
This is easy to derive:
\begin{align}
	\begin{split}
		R_i & = \frac{P_i - P_j}{P_j} \\
		R_i & = \frac{P_i}{P_j} - 1\\
		R_i + 1 &  = \frac{P_i}{P_j}\\
		\rightarrow\\
		ln(R_i + 1) & = ln(\frac{P_i}{P_j}) \\
		&  = ln(P_i) - ln(P_j)\\
	\end{split}
\end{align}

\section{Property 1:  Symmetry}
Simple returns are not \emph{symmetric}: positive and negative percent ordinary returns of equal magnitude do not cancel each other out and result in a net change. For example, if your portfolio goes up by $50\%$ (say from $\$100$ to $\$150$) and then declines by $50\%$ (say from $\$150$ to $\$75$), you're not back where you started. If you calculate your average percentage return (in this case, $0\%$), that's not a particularly useful summary of the fact that you actually ended up $25\%$ below where you started. That is, if you just calculate the average like so:
\begin{equation}
	\frac{(\frac{150-100}{100} + \frac{75-150}{150})}{2} = 0\%
\end{equation}
the arithmetic average suggests that you have earned an average of $0\%$. But in reality you lost $25\%$. Do you see the problem here? Arithmetic return has a positive bias.
\\
\\
Logarithmic returns are useful for mathematical finance. One of the advantages is that the logarithmic returns are symmetric. While ordinary returns are not, logarithmic returns of equal magnitude but opposite signs will cancel each other out. This means that an investment of $\$100$ that yields a simple return of $50\%$ followed by a simple return of $-50\%$ will result in $\$75$, while an investment of $\$100$ that yields a logarithmic return of $50\%$ followed by a logarithmic return of $-50\%$ will come back to $\$100$.
\\
\\
Consider this example. You have invested $\$100$ in a stock in year 1, it grows to  $\$200$ in year 2 giving you a return of $100\%$. In year 3 the stock price comes back to $\$100$ from  $\$200$ giving you a negative return of $50\%$. What do you think is the average return? If you calculate the average like so:
\begin{equation}
	\frac{(100\% - 50\%)}{2} = 25\%
\end{equation}
the arithmetic average is suggesting that you have earned an average of $25\%$. But in reality you have earned $0\%$. That's not a particularly useful summary. On the other hand, the average log return is indeed $0\%$ which is more informative:
\begin{center}
  \begin{tabular}{ | c | c | c | c | }
    \hline
    Period & Price & Simple Return & Log Return \\ \hline
    Year 1 & $100$ & - & - \\ \hline
    Year 2 & $200$ & $100\%$ & $69\%$ \\ \hline
    Year 3 & $100$ & $-50\%$ & $-69\%$ \\
    \hline
  \end{tabular}
\end{center}
See the symmetry of the log returns in the above table?

\section{Property 2:  Time-additivity}
Simple returns are not \emph{additive} over time. Adding returns for multiple periods does not yield the total return over the total length of time. In other words, simple returns don't satisfy the property of \emph{time consistency}. For example, if your portfolio goes from $\$10$ to $\$11$ in one year and then to $\$12$ the next year, your annual simple returns would be $10.0\%$ and $9.09\%$ respectively. Adding those returns up would sum up to $19.09\%$, which is not your actual return of $20\%$.
\\
\\
On the other hand, logarithmic returns are \emph{additive} over time. Consider an ordered sequence of $n$ trades. Calculating the compounding return, which is the running return of this sequence of trades over time:
\begin{equation}
	(1+R_1)(1+R_2)...(1+R_n) = \displaystyle{\prod_{i}^{n}} (1 + R_i)
\end{equation}
where $R_i$ is the return for each period. Taking the natural logarithm of this would give us:
\begin{align*}
	\begin{split}
	ln(\displaystyle{\prod_{i}^{n}} (1 + R_i))
	& = \displaystyle{\sum_{i}^{n}} ln(1 + R_i) \\
	& = ln(1 + R_1) + ln(1 + R_2) + \dots + ln(1 + R_n) \\
	& = ln(\frac{P_1}{P_0}) + ln(\frac{P_2}{P_1}) + ln(\frac{P_3}{P_2}) + \dots + ln(\frac{P_{n-1}}{P_{n-2}}) + ln(\frac{P_n}{P_{n-1}}) \\
	& = ln(P_1) - ln(P_0) + ln(P_2) - ln(P_1) + \dots + ln(P_n) - ln(P_{n-1}) \\
	& = ln(P_n) - ln(P_0)
	\end{split}
\end{align*}
As we can see, if we sum up the logarithmic returns $ln(\frac{P_i}{P_j})$ for each period, we end up with $ln(P_n) - ln(P_0) = ln(\frac{P_n}{P_0})$. In other words, the compound return over $n$ periods is merely the difference in log between initial and final periods.
\\
\\
Back to our portfolio example, if your portfolio goes from $\$10$ to $\$11$ in one year and then to $\$12$ the next year, your logarithmic return $r$ over the period of two years would be:
\begin{equation}
	r = ln(P_2) - ln(P_0) = ln(12) - ln(10) = 0.18
\end{equation}
which is the same as summing the logarithmic returns of each individual period:
\begin{equation}
ln(\frac{P_1}{P_0}) + ln(\frac{P_2}{P_1})  = ln(\frac{11}{10}) + ln(\frac{12}{11}) = 0.18
\end{equation}
In terms of algorithmic complexity, this simplification reduces $\mathcal{O}(n)$ multiplications to $\mathcal{O}(1)$ additions. This is a huge win for moderate to large $n$.
\\
\\
$P_2$ can also be expressed as $e$ raised to the logarithmic return $r$:
\begin{align*}
	\begin{split}
		P_2 & = P_0 (1+R_1)(1+R_2) \\
		& = P_0  \displaystyle{\prod_{0}^{2}} (1 + R_i) \\
		& = P_0 e^{ln( \displaystyle{\prod_{0}^{2}} (1+R_i))} \\
		& = P_0 e^{ln(\frac{P_2}{P_0})} \\
		& = P_0 e^{ln(\frac{12}{10})} \\
		& = 12
	\end{split}
\end{align*}

And, because the logarithmic returns are additive, the arithmetic mean of log returns is much more informative: the average log return, times the number of sub-periods, tells us the exact total compounded return over the period of study.

\section{Property 3:  Viewing Economic Data}
To refresh your memory of school math, logs are just another way of writing exponential equations, one that allows you to separate the exponent on one side of the equation. The equation $2^4 = 16$ can be rewritten as $log_2 16 = 4$ and pronounced "log to the base 2 of 16 is 4." The equation $y = log_b (x)$ means that $y$ is the power or exponent that $b$ is raised to in order to get $x$.
\\
\\
On a linear scale, a change between two values is perceived on the basis of the \emph{difference} between the values. Thus, for example, a change from $1$ to $2$ would be perceived as the same amount of increase as from $4$ to $5$. On a logarithmic scale, a change between two values is perceived on the basis of the \emph{ratio} of the two values. That is, a change from $1$ to $2$ (ratio of $1:2$) would be perceived as the same amount of increase as a change from $4000$ to $8000$ (also a ratio of $1:2$). You can see that log scales allow a large range to be displayed without small values being compressed down into bottom of the graph. Log scales respond to skewness towards large values.
\\
\\
Consider this example. The revenue for Boeing in 2010 was about $2^6$ billion dollars while the revenue for Ford Motor was about $2^7$. In a linear scale, the revenue for Ford is the revenue for Boeing plus the difference between these two revenues. We call this additive. Since $2^6 = 64$ and $2^7 = 128$, we see that the difference is about $64$ billion dollars. In a log scale, the difference is multiplicative. Since $2^7 = 2^6 \cdot 2$, we see that the revenues for Ford Motor are about double those for Boeing. This is what we mean by saying that we use logarithmic scales to show multiplicative factors.
\\
\\
Logarithms are often a much more useful way to look at economic data. For example, here is a graph of the SP500 index going back to 1871. Plotted on arithmetic scale, one can see nothing in the  first century, whereas the most recent decade appears insanely volatile.
\begin{center}
	\includegraphics[scale=0.4]{sp500_arithmetic.png}
\end{center}
On the other hand, if you plot these same data on a log scale, you do no longer consider absolute change, but relative change. That is, $log_2(16) - log_2(8) = log_2(2000) - log_2(1000)$. Plotted this way, it's clear that, in percentage terms, the recent volatility of stock prices is actually modest relative to what happened in the Great Depression in the 1930's.
\begin{center}
	\includegraphics[scale=0.4]{sp500_log.png}
\end{center}

\end{document}


%
% body.tex
%
% Copyright © 2020 Libao Jin <jinlibao@outlook.com>
% Distributed under terms of the MIT license.
%
\ifx\undefined\sol
Submit online; make sure to sign the clause. All your work, including the signed page, should be together as one PDF file.
\vspace{3em}
\else
{ \large
\textbf{Honesty Clause: {\color{red}I hereby testify that all the work on this project is \underline{solely my own}. I have complied with the academic honesty guidelines as stated in the University regulations.}}

% \special{pdf:ann width 3in height 1in
%   <<
%     /T (My Signature)
%     /Subtype /Widget
%     /FT /Sig
%     /F 4
%     /Q 1
%     /MK << /BC [] >>
%   >>
% }

\begin{Form}
\TextField[width=21em,
           name = print.name,
           format = {
               var f = this.getField('print.name');
               % f.textFont = 'Verdana';
               % f.strokeColor = ['T'];
               % f.fillColor = ['T'];
               f.userName = 'Print Name'
               },
           value = ,
           charsize = 12pt]
          {Name: }
\end{Form}
\begin{Form}
\TextField[width=10em,
           format   ={AFDate_FormatEx("mm/dd/yyyy");},
           keystroke={AFDate_KeystrokeEx("mm.dd.yyyy");},
           value = ,
           charsize = 12pt]
           {Date: }
\end{Form}
}
\vspace{5em}
\fi

\textbf{Instruction}

\begin{enumerate}[label={\arabic*.}]
  \item Go to \url{https://www.overleaf.com} and sign in (required).
  \item Click \emph{Menu} (up left corner), then \emph{Copy Project}.
  \item Go to \verb|LaTeX/meta.tex| (the file \verb|meta.tex| under the folder \verb|LaTeX|) to change the section and your name, e.g.,
    \begin{itemize}
      % \item change title to \verb|\title{MATH 3340-01 Scientific Computing Homework 6}|
      \item change author to \verb|\author{Albert Einstein \& Carl F. Gauss}|
    \end{itemize}
  \item You need to write function/script files, store results to output files. Here are suggested names for function files, script files, output files, and figure files:
    \begin{table}[!hbtp]
      \centering
      % \caption{caption}
      % \label{tab:label}
      \begin{tabular}{cllll}
        \toprule
        Problem & Function File                              & Script File       & Output File         \\
        \midrule
        1       & \verb|simpson.m| \&  \verb|gaussSimpson.m| & \verb|final_p1.m| & \verb|final_p1.txt| \\
        2       & \verb|steepestDescent.m|                   & \verb|final_p2.m| & \verb|final_p2.txt| \\
        \bottomrule
      \end{tabular}
    \end{table}

    Once finished, you need to upload these files to the folder \verb|src| on Overleaf. If you have different filenames, please update the filenames in \verb|\lstinputlisting{../src/your_script_name.m}| accordingly. You can code in the provided files in \href{https://libaoj.in/courses/2020s/MATH3340/Homework/9/final.zip}{final.zip}, and use the MATLAB script \verb|save_results.m| to generate the output files and store the graphs to \verb|.pdf| files automatically (the script filenames should be exactly same as listed above).
  \item Recompile, and download the generated \verb|.pdf| file.
  \item \textbf{\color{red}{Important}}: \underline{Enter your name and the date} \emph{before} you submit it on WyoCourses.
\end{enumerate}

\ifx\undefined\sol
\newpage
\fi
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Problem 1
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 1}%
\label{sec:problem_1}
Write a MATLAB code that uses the parameterization method to calculate the value of the area integral
\begin{equation*}
I = \int_{-1}^{1} \int_{x^{2}}^{4 - x^{2}} [x^{2} y + e^{-(x^{2} + y^{2})}] \, dy dx.
\end{equation*}
Your code should use Gauss quadrature in the $x$-direction (the outer integral) and Simpson's rule in the $y$-direction (the inner integral). For the Gauss integral, the code should allow you to choose up to six integration points (i.e., you should have the option $N =  1, N = 2, \ldots, N = 6$). Similarly, for the inner integral, you should be able to choose an arbitrary number of panels, $K$; the number of patches will then be $n = 2K$ with the evaluation points numbered $y_{0}, y_{1}, y_{2}, \ldots, y_{n}$ as in the notes. Run the code with $N = 2$ and $K = 2$, then with $N = 6$ and $K = 4$. Print the results you obtained for these two cases. Do not round off your results to four decimal places, the MATLAB default; instead, print all available decimals. Indicate clearly the values of $N$ and $K$ used for each calculation and submit all your code used for this problem. While this is your choice, it is probably best to organize it as a script calling the pertaining function (or functions), as usual.
\begin{solution}
  \quad
  \begin{itemize}
  \item Output file \verb|final_p1.txt|:
    \lstinputlisting[style=Plain]{../src/final_p1.txt}
  \item Function file \verb|simpson.m|:
    \lstinputlisting[style=MATLAB]{../src/simpson.m}
  \item Function file \verb|gaussSimpson.m|:
    \lstinputlisting[style=MATLAB]{../src/gaussSimpson.m}
  \item Script file \verb|final_p1.m|:
    \lstinputlisting[style=MATLAB]{../src/final_p1.m}
  \end{itemize}
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Problem 2
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 2}%
\label{sec:problem_2}
Solve the linear system of equations:
\begin{equation*}
  \begin{cases}
    3x + y - z = 4 \\
    x + 5y + 2z = -1 \\
    -x + 2y + 5z = 1
  \end{cases}
\end{equation*}
using the steepest descent method starting from the usual initial guess $\mathbf{x}^{0} = [0, 0, 0]^{T}$. For each iteration, print the iteration number $k$, the current approximation $\mathbf{x}^{k}$, and the norm of the residual at this current location. Start with $k = 0$ and produce a table with this information. Convergence should be declared when the norm of the residual is below $10^{-7}$. Submit all pieces of code that you used to solve the problem, together with the aforementioned results.
\begin{solution}
  \quad
  \begin{itemize}
  \item Output file \verb|final_p2.txt|:
    \lstinputlisting[style=Plain]{../src/final_p2.txt}
  \item Function file \verb|steepestDescent.m|:
    \lstinputlisting[style=MATLAB]{../src/steepestDescent.m}
  \item Function file \verb|steepestDescent2.m|:
    \lstinputlisting[style=MATLAB]{../src/steepestDescent2.m}
  \item Function file \verb|goldenSection.m|:
    \lstinputlisting[style=MATLAB]{../src/goldenSection.m}
  \item Script file \verb|final_p2.m|:
    \lstinputlisting[style=MATLAB]{../src/final_p2.m}
  \end{itemize}
\end{solution}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2003-2018 by The University of Queensland
% http://www.uq.edu.au
%
% Primary Business: Queensland, Australia
% Licensed under the Apache License, version 2.0
% http://www.apache.org/licenses/LICENSE-2.0
%
% Development until 2012 by Earth Systems Science Computational Center (ESSCC)
% Development 2012-2013 by School of Earth Sciences
% Development from 2014 by Centre for Geoscience Computing (GeoComp)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{figure}[t]
\centerline{\includegraphics[width=4.in]{figures/twodheatdiff}}
\caption{Example 3: 2D model: granitic intrusion of sandstone country rock}
\label{fig:twodhdmodel}
\end{figure}

\sslist{example03a.py and cblib.py}

Building upon our success from the 1D models, it is now prudent to expand our
domain by another dimension.
For this example we use a very simple magmatic intrusion as the basis for our
model. The simulation will be a single event where some molten granite has
formed a cylindrical dome at the base of some cold sandstone country rock.
Assuming that the cylinder is very long
we model a cross-section as shown in \reffig{fig:twodhdmodel}. We will implement
the same
diffusion model as we have used for the granite blocks in \refSec{Sec:1DHDv00}
but will add the second spatial dimension and show how to define
variables depending on the location of the domain.
We use \file{onedheatdiff001b.py} as the starting point to develop this model.

\section{Example 3: Two Dimensional Heat Diffusion for a basic Magmatic
Intrusion}
\label{Sec:2DHD}

To expand upon our 1D problem, the domain must first be expanded. In fact, we
have solved a two dimensional problem already but essentially ignored the second
dimension. In our definition phase we create a square domain in $x$ and
$y$\footnote{In \esc the notation
$x_{0}$ and $x_{1}$ is used for $x$ and $y$, respectively.}
that is $600$ meters along each side \reffig{fig:twodhdmodel}. Now we set the
number of discrete spatial cells to $150$ in both directions and the radius of
the intrusion to $200$ meters with the centre located at the $300$ meter mark
on the $x$-axis. Thus, the domain variables are;
\begin{python}
mx = 600*m #meters - model length
my = 600*m #meters - model width
ndx = 150 #mesh steps in x direction
ndy = 150 #mesh steps in y direction
r = 200*m #meters - radius of intrusion
ic = [300*m, 0] #coordinates of the centre of intrusion (meters)
qH=0.*J/(sec*m**3) #our heat source temperature is zero
\end{python}
As before we use
\begin{python}
model = Rectangle(l0=mx,l1=my,n0=ndx, n1=ndy)
\end{python}
to generate the domain.

There are two fundamental changes to the PDE that we have discussed in
\refSec{Sec:1DHDv00}. Firstly,
because the material coefficients for granite and sandstone are different, we
need to deal with
PDE coefficients which vary with their location in the domain. Secondly, we
need to deal with the second spatial dimension. We can investigate these two
modifications at the same time.
In fact, the temperature model \refEq{eqn:hd} we utilised in
\refSec{Sec:1DHDv00} applied for the
1D case with a constant material parameter only. For the more general case
examined in this chapter, the correct model equation is
\begin{equation}
\rho c_p \frac{\partial T}{\partial t} -  \frac{\partial }{\partial x}
\kappa \frac{\partial T}{\partial x} -  \frac{\partial }{\partial y} \kappa
\frac{\partial T}{\partial y} = q_H
\label{eqn:hd2}
\end{equation}
Notice that for the 1D case we have $\frac{\partial T}{\partial y}=0$ and
for the case of constant material parameters $\frac{\partial }{\partial x}
\kappa = \kappa  \frac{\partial }{\partial x}$ thus this new equation coincides
with a simplified model equation for this case. It is more convenient
to write this equation using the $\nabla$ notation as we have already seen in
\refEq{eqn:commonform nabla};
\begin{equation}\label{eqn:Tform nabla}
\rho c_p \frac{\partial T}{\partial t}
-\nabla \cdot \kappa \nabla T = q_H
\end{equation}
We can easily apply the backward Euler scheme as before to end up with
\begin{equation}
\frac{\rho c_p}{h} T^{(n)} -\nabla \cdot \kappa \nabla T^{(n)}  =
q_H +  \frac{\rho c_p}{h} T^{(n-1)}
\label{eqn:hdgenf2}
\end{equation}
which is very similar to \refEq{eqn:hdgenf} used to define the temperature
update in the simple 1D case.
The difference is in the second order derivative term
$\nabla \cdot \kappa \nabla T^{(n)}$. Under the light of the more general case
we need to revisit the \esc PDE form as shown in \ref{eqn:commonform2D}.
For the 2D case with variable PDE coefficients the form needs to be read as
\begin{equation}\label{eqn:commonform2D 2}
-\frac{\partial }{\partial x} A_{00}\frac{\partial u}{\partial x}
-\frac{\partial }{\partial x} A_{01}\frac{\partial u}{\partial y}
-\frac{\partial }{\partial y} A_{10}\frac{\partial u}{\partial x}
-\frac{\partial }{\partial x} A_{11}\frac{\partial u}{\partial y}
+ Du = f
\end{equation}
So besides the settings $u=T^{(n)}$, $D = \frac{\rho c _{p}}{h}$ and
$f = q _{H} + \frac{\rho c_p}{h} T^{(n-1)}$ as we have used
before (see \refEq{ESCRIPT SET}) we need to set
\begin{equation}\label{eqn: kappa general}
A_{00}=A_{11}=\kappa; A_{01}=A_{10}=0
\end{equation}
The fundamental difference to the 1D case is that $A_{11}$ is not set
to zero but $\kappa$,
which brings in the second dimension. It is important to note that the
coefficients of the PDE may depend on their location in the domain which does
not influence the usage of the PDE form. This is very convenient as we can
introduce spatial dependence to the PDE coefficients without modification to
the way we call the PDE solver.

A very convenient way to define the matrix $A$ in \refEq{eqn: kappa general}
can be carried out using the Kronecker $\delta$ symbol\footnote{see
\url{http://en.wikipedia.org/wiki/Kronecker_delta}}. The
\esc function \verb|kronecker| returns this matrix;
\begin{equation}
\verb|kronecker(model)| = \left[
\begin{array}{cc}
 1 & 0 \\
 0 & 1 \\
\end{array}
\right]
\end{equation}
As the argument \verb|model| represents a two dimensional domain the matrix is
returned as a $2 \times 2$ matrix
(in the case of a three-dimensional domain a $3 \times 3$ matrix is returned).
The call
\begin{python}
mypde.setValue(A=kappa*kronecker(model),D=rhocp/h)
\end{python}
sets the PDE coefficients according to \refEq{eqn: kappa general}.

We need to check the boundary conditions before we turn to the question: how do
we set $\kappa$. As pointed out \refEq{NEUMAN 2} makes certain assumptions on
the boundary conditions. In our case these assumptions translate to;
\begin{equation}
-n \cdot \kappa \nabla T^{(n)} =
-n_{0} \cdot \kappa \frac{\partial T^{(n)}}{\partial x} -
n_{1} \cdot  \kappa \frac{\partial T^{(n)}}{\partial y} = 0
\label{eq:hom flux}
\end{equation}
which sets the normal component of the heat flux $- \kappa \cdot (\frac{\partial
T^{(n)}}{\partial x}, \frac{\partial T^{(n)}}{\partial y})$ to zero. This means
that the region is insulated which is what we want.
On the left and right face of the domain where we have
$(n_{0},n_{1} ) = (\mp 1,0)$
this means $\frac{\partial T^{(n)}}{\partial x}=0$ and on the top and bottom on
the domain
where we have  $(n_{0},n_{1} ) = (\pm 1,0)$ this is
$\frac{\partial T^{(n)}}{\partial y}=0$.

\section{Setting variable PDE Coefficients}
Now we need to look into the problem of how we define the material coefficients
$\kappa$ and $\rho c_p$ depending on their location in the domain.
We can make use of the technique used in the granite block example in
\refSec{Sec:1DHDv00}
to set up the initial temperature. However,
the situation is more complicated here as we have to deal with a
curved interface between the two sub-domains.

Prior to setting up the PDE, the interface between the two materials must be
established.
The distance $s\ge 0$ between two points $[x,y]$ and
$[x_{0},y_{0}]$ in Cartesian coordinates is defined as:
\begin{equation}
 (x-x_{0})^{2}+(y-y_{0})^{2} = s^{2}
\end{equation}
If we define the point $[x_{0},y_{0}]$ as $ic$ which denotes
the centre of the semi-circle of our intrusion, then for all the points $[x,y]$
in our model we can calculate a distance to $ic$.
All the points that fall within the given radius $r$ of our intrusion will have
a corresponding
value $s < r$ and all those belonging to the country rock will have a value $s >
r$. By subtracting $r$ from both of these conditions we find $s-r < 0$ for all
intrusion points and $s-r > 0$ for all country rock points.
Defining these conditions within the script is quite simple and is done using
the following command:
\begin{python}
bound = length(x-ic)-r #where the boundary will be located
\end{python}
This definition of the boundary can now be used with the \verb|whereNegative|
command again to help define the material constants and temperatures in our
domain.
If \verb|kappai| and \verb|kappac| are the
thermal conductivities for the intrusion material granite and for the
surrounding sandstone, then we set;
\begin{python}
x=Function(model).getX()
bound = length(x-ic)-r
kappa = kappai * whereNegative(bound) + kappac * (1-whereNegative(bound))
mypde.setValue(A=kappa*kronecker(model))
\end{python}
Notice that we are using the sample points of the \verb|Function| function space
as expected for the PDE coefficient \verb|A|\footnote{For the experienced user: use
\texttt{x=mypde.getFunctionSpace("A").getX()}.}.
The corresponding statements are used to set $\rho c_p$.

Our PDE has now been properly established. The initial conditions for
temperature are set out in a similar manner:
\begin{python}
#defining the initial temperatures.
x=Solution(model).getX()
bound = length(x-ic)-r
T= Ti*whereNegative(bound)+Tc*(1-whereNegative(bound))
\end{python}
where \verb|Ti| and \verb|Tc| are the initial temperature in the regions of the
granite and surrounding sandstone, respectively. It is important to
notice that we reset \verb|x| and \verb|bound| to refer to the appropriate
sample points of a PDE solution\footnote{For the experienced user: use
\texttt{x=mypde.getFunctionSpace("r").getX()}.}.

\begin{figure}[ht]
\centerline{\includegraphics[width=4.in]{figures/heatrefraction001.png}}
\centerline{\includegraphics[width=4.in]{figures/heatrefraction030.png}}
\centerline{\includegraphics[width=4.in]{figures/heatrefraction200.png}}
\caption{Example 3a: 2D model: Total temperature distribution ($T$) at time step
$1$, $20$ and $200$. Contour lines show temperature.}
\label{fig:twodhdans}
\end{figure}

\section{Contouring \esc Data using \modmpl.}
\label{Sec:2DHD plot}
To complete our transition from a 1D to a 2D model we also need to modify the
plotting procedure. As before we use \modmpl to do the plotting in this case a
contour plot. For 2D contour plots \modmpl expects that the data are regularly
gridded. We have no control over the location and ordering of the sample points
used to represent the solution. Therefore it is necessary to interpolate our
solution onto a regular grid.

In \refSec{sec: plot T} we have already learned how to extract the $x$
coordinates of sample points as
\verb|numpy| array to hand the values to \modmpl. This can easily be extended
to extract both the $x$ and the $y$ coordinates;
\begin{python}
import numpy as np
def toXYTuple(coords):
    coords = np.array(coords.toListOfTuples()) #convert to Tuple
    coordX = coords[:,0] #X components.
    coordY = coords[:,1] #Y components.
    return coordX,coordY
\end{python}
For convenience we have put this function into \file{clib.py} so we can use
this function in other scripts more easily.

We now generate a regular $100 \times 100$ grid over the domain ($mx$ and $my$
are the dimensions in the $x$ and $y$ directions) which is done using the
\modnumpy function \verb|linspace|.
\begin{python}
from clib import toXYTuple
# get sample points for temperature as  for contouring
coordX, coordY = toXYTuple(T.getFunctionSpace().getX())
# create regular grid
xi = np.linspace(0.0,mx,75)
yi = np.linspace(0.0,my,75)
\end{python}
The values \verb|[xi[k], yi[l]]| are the grid points.

The remainder of our contouring commands resides within a \verb|while| loop so
that a new contour is generated for each time step. For each time step the
solution must be re-gridded for \modmpl using the \verb|griddata| function. This
function interprets irregularly located values \verb|tempT| at locations defined
by \verb|coordX| and \verb|coordY| as values at the new coordinates of a
rectangular grid defined by
\verb|xi| and \verb|yi|. The output is \verb|zi|. It is now possible to use the
\verb|contourf| function which generates colour filled contours. The colour
gradient of our plots is set via the command
\verb|pl.matplotlib.pyplot.autumn()|, other colours are listed on the \modmpl web page\footnote{see
\url{http://matplotlib.sourceforge.net/api/}}. Our results are then contoured,
visually adjusted using the \modmpl functions and then saved to a file.
\verb|pl.clf()| clears the figure in readiness for the next time iteration.
\begin{python}
#grid the data.
zi = pl.matplotlib.mlab.griddata(coordX,coordY,tempT,xi,yi)
# contour the gridded data, plotting dots at the randomly spaced data points.
pl.matplotlib.pyplot.autumn()
pl.contourf(xi,yi,zi,10)
CS = pl.contour(xi,yi,zi,5,linewidths=0.5,colors='k')
pl.clabel(CS, inline=1, fontsize=8)
pl.axis([0,600,0,600])
pl.title("Heat diffusion from an intrusion.")
pl.xlabel("Horizontal Displacement (m)")
pl.ylabel("Depth (m)")
pl.savefig(os.path.join(save_path,"Tcontour%03d.png") %i)
pl.clf()
\end{python}
The function \verb|pl.contour| is used to add labelled contour lines to the
plot.
The results for selected time steps are shown in \reffig{fig:twodhdans}.

\section{Advanced Visualisation using VTK}

\sslist{example03b.py}
An alternative approach to \modmpl for visualisation is the usage of a package
which is based on the Visualization Toolkit (VTK) library\footnote{see \url{http://www.vtk.org/}}.
There is a variety of packages available. Here we use the package \mayavi\footnote{see
\url{http://code.enthought.com/projects/mayavi/}} as an example.

\mayavi is an interactive, GUI driven tool which is
really designed to visualise large three dimensional data sets where \modmpl
is not suitable. But it is also very useful when it comes to two dimensional
problems.
The decision of which tool is the best can be subjective and users should
determine which package they require and are most comfortable with. The main
difference between using \mayavi (or other VTK based tools) and \modmpl is that
the actual visualisation is detached from the calculation by writing the
results to external files and importing them into \mayavi. In 3D the best
camera position for rendering a scene is not obvious before the results are
available. Therefore the user may need to try different settings before the
best is found. Moreover, in many cases a 3D interactive visualisation is the
only way to really understand the results (e.g. using stereographic projection).

To write the temperatures at each time step to data files in the VTK file format
one needs to import \verb|saveVTK| from the \weipa module and call it:
\begin{python}
from esys.weipa import saveVTK
while t<=tend:
      i+=1 #counter
      t+=h #current time
      mypde.setValue(Y=qH+T*rhocp/h)
      T=mypde.getSolution()
      saveVTK(os.path.join(save_path,"data.%03d.vtu"%i, T=T)
\end{python}
The data files, e.g. \file{data.001.vtu}, contain all necessary information to
visualise the temperature and can be directly processed by \mayavi. Note that
there is no re-gridding required. The file extension \file{.vtu} is
automatically added if not supplied to \verb|saveVTK|.

\begin{figure}[ht]
\centerline{\includegraphics[width=4.in]{figures/ScreeshotMayavi2n1}}
\caption{Example 3b: \mayavi start up window}
\label{fig:mayavi window}
\end{figure}

\begin{figure}[ht]
\centerline{\includegraphics[width=4.in]{figures/ScreeshotMayavi2n2}}
\caption{Example 3b: \mayavi data control window}
\label{fig:mayavi window2}
\end{figure}
After you run the script you will find the
result files \file{data.*.vtu} in the result directory \file{data/example03}.
Run the command
\begin{python}
>> mayavi2 -d data.001.vtu -m Surface &
\end{python}
from the result directory. \mayavi will start up a window similar to
\reffig{fig:mayavi window}.
The right hand side shows the temperature at the first time step. To show
the results at other time steps you can click at the item \texttt{VTK XML file
(data.001.vtu) (timeseries)}
at the top left hand side. This will bring up a new window similar to the window
shown in \reffig{fig:mayavi window2}. By clicking at the arrows in the top
right corner you can move forwards and backwards in time.
You will also notice the text \textbf{T} next to the item \texttt{Point scalars
name}. The
name \textbf{T} corresponds to the keyword argument name \texttt{T} we have
used in the \verb|saveVTK| call. In this menu item you can select other results
you may have written to the output file for visualisation.

\textbf{For the advanced user}: Using the \modmpl to visualise spatially
distributed data is not MPI compatible. However, the \verb|saveVTK| function
can be used with MPI. In fact, the function collects the values of the sample
points spread across processor ranks into a single file.
For more details on writing scripts for parallel computing please consult the
\emph{user's guide}.


\documentclass[14pt,a4paper]{article}
\setlength\parindent{0pt}
\setlength{\parskip}{1em}
\usepackage{array}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{mathrsfs}
\usepackage{graphicx}
%\usepackage[toc,page]{appendix}

\begin{document}
\title{Deep Quaternion Networks}
\author{Chase Gaudet \\  cjg7182@louisiana.edu
   \and Anthony Maida \\  maida@louisiana.edu}
\maketitle

\begin{abstract}
The field of deep learning has seen significant advancement in recent years.
However, much of the existing work has been focused on real-valued numbers.
Recent work has shown that a deep learning system using the complex numbers can be deeper for a fixed parameter budget compared to its real-valued counterpart.
In this work, we explore the benefits of generalizing one step further into the hyper-complex numbers, quaternions specifically, and provide the architecture components needed to build deep quaternion networks.
We go over quaternion convolutions, present a quaternion weight initialization scheme, and present algorithms for quaternion batch-normalization.
These pieces are tested in a classification model by end-to-end training on the CIFAR-10 and CIFAR-100 data sets and a segmentation model by end-to-end training on the Kitti Road Segmentation data set.
The quaternion networks show improved convergence to a real-valued and complex-valued networks, especially on the segmentation task.
\end{abstract}

\section{Introduction}
There have been many advances in deep neural network architectures in the past few years.
One such improvement is a normalization technique called batch normalization \cite{ioffe2015batch} that standardizes the activations of layers inside a network using minibatch statistics.
It has been shown to regularize the network as well as provide faster and more stable training.
Another improvement comes from architectures that add so called shortcut paths to the network.
These shortcut paths connect later layers to earlier layers typically, which allows for the stronger gradients to propagate to the earlier layers.
This method can be seen in Highway Networks \cite{srivastava2015training} and Residual Networks  \cite{he2016deep}.
Other work has been done to find new activation functions with more desirable properties.
One example is the exponential linear unit (ELU) \cite{clevert2015fast}, which attempts to keep activations standardized.
All of the above methods are combating the vanishing gradient problem \cite{hochreiter1991untersuchungen} that plagues deep architectures.
With solutions to this problem appearing it is only natural to move to a system that will allow one to construct deeper architectures with as low a parameter cost as possible.

Other work in this area has explored the use of complex and hyper-complex numbers, which are a generalization of the complex, such as quaternions.
Using complex numbers in recurrent neural networks (RNNs) has been shown to increase learning speed and provide a more noise robust memory retrieval mechanism \cite{arjovsky2016unitary, danihelka2016associative, wisdom2016full}.
The first formulation of complex batch normalization and complex weight initialization is presented by \cite{trabelsi2017deep} where they achieve some state of the art results on the MusicNet data set.
Hyper-complex numbers are less explored in neural networks, but have seen use in manual image and signal processing techniques \cite{bulow1999hypercomplex, sangwine2000colour, bulow2001hypercomplex}.
Examples of using quaternion values in networks is mostly limited to architectures that take in quaternion inputs or predict quaternion outputs, but do not have quaternion weight values \cite{rishiyur2006neural, kendall2015posenet}.
There are some more recent examples of building models that use quaternions represented as real-values.
In \cite{parcollet2016quaternion} they used a quaternion multi-layer perceptron (QMLP) for document understanding and \cite{minemoto2017feed} uses a similar approach in processing multi-dimensional signals.

Building on \cite{trabelsi2017deep} our contribution in this paper is to formulate quaternion convolution, batch normalization, and weight initialization.
There arises some difficulty over complex batch normalization that we had to overcome as their is no analytic form for our inverse square root matrix.

\section{Motivation and Related Work}
The ability of quaternions to effectively represent spatial transformations and analyze multi-dimensional signals makes them promising for applications in artificial intelligence.

One common use of quaternions is for representing rotation into a more compact form.
PoseNet \cite{kendall2015posenet} used a quaternion as the target output in their model where the goal was to recover the $6-$DOF camera pose from a single RGB image.
The ability to encode rotations may make a quaternion network more robust to rotational variance.

Quaternion representation has also been used in signal processing.
The amount of information in the phase of an image has been shown to be sufficient to recover the majority of information encoded in its magnitude by Oppenheim and Lin \cite{oppenheim1981importance}.
The phase also encodes information such as shapes, edges, and orientations.
Quaternions can be represented as a 2~x~2 matrix of complex numbers, which gives them a group of phases potentially holding more information compared to a single phase.

Bulow and Sommer \cite{bulow2001hypercomplex} used the higher complexity representation of quaternions by extending Gabor's complex signal to a quaternion one which was then used for texture segmentation.
Another use of quaternion filters is shown in \cite{sangwine2000colour} where they introduce a new class of filter based on convolution with hyper-complex masks, and present three color edge detecting filters.
These filters rely on a three-space rotation about the grey line of RGB space and when applied to a color image produce an almost greyscale image with color edges where the original image had a sharp change of color.
More quaternion filter use is shown in \cite{shi2007quaternion} where they show that it is effective in the context of segmenting color images into regions of similar color texture.
They state the advantage of using quaternion arithmetic is that a color can be represented and analyzed as a single entity, which we will see holds for quaternion convolution in a convolutional neural network architecture as well in Section \ref{s:qc}.

A quaternionic extension of a feed forward neural network, for processing multi-dimensional signals, is shown in \cite{minemoto2017feed}.
They expect that quaternion neurons operate on multi-dimensional signals as single entities, rather than real-valued neurons that deal with each element of signals independently.
A convolutional neural network (CNN) should be able to learn a powerful set of quaternion filters for more impressive tasks.

Another large motivation is discussed in \cite{trabelsi2017deep}, which is that complex numbers are more efficient and provide more robust memory mechanisms compared to the reals \cite{bulow1999hypercomplex, sangwine2000colour, bulow2001hypercomplex}.
They continue that residual networks have a similar architecture to associative memories since the residual shortcut paths compute their residual and then sum it into the memory provided by the identity connection.
Again, given that quaternions can be represented as a complex group, they may provide an even more efficient and robust memory mechanisms.

\section{Quaternion Network Components}
This section will include the work done to obtain a working deep quaternion network.
Some of the longer derivations are shown in the appendix.

\subsection{Quaternion Representation}
In 1833 Hamilton proposed complex numbers $\mathbb{C}$ be defined as the set $\mathbb{R}^2$ of ordered pairs $(a, b)$ of real numbers.
He then began working to see if triplets $(a,b,c)$ could extend multiplication of complex numbers.
In 1843 he discovered a way to multiply in four dimensions instead of three, but the multiplication lost commutativity.
This construction is now known as quaternions.
Quaternions are composed of four components, one real part, and three imaginary parts.
Typically denoted as
\begin{equation}
\mathbb{H} = \{a + b\textit{i} + c\textit{j} + d\textit{k}~:~a,b,c,d \in \mathbb{R}\}
\label{eq:quaternion1}
\end{equation}
where $a$ is the real part, $(i,j,k)$ denotes the three imaginary axis, and $(b,c,d)$ denotes the three imaginary components.
Quaternions are governed by the following arithmetic:
\begin{equation}
i^2=j^2=k^2=ijk=-1
\label{eq:quarternion2}
\end{equation}
which, by enforcing distributivity, leads to the noncommutative multiplication rules
\begin{equation}
ij=k,~jk=i,~ki=j,~ji=-k,~kj=-i,~ik=-j
\label{eq:quarternion3}
\end{equation}

Since we will be performing quaternion arithmetic using reals it is useful to embed $\mathbb{H}$ into a real-valued representation.
There exists an injective homomorphism from $\mathbb{H}$ to the matrix ring $M(4,\mathbb{R})$ where $M(4,\mathbb{R})$ is a 4x4 real matrix.
The 4~x~4 matrix can be written as
\begin{align}
\begin{bmatrix}
 a & -b & -c & -d \\
 b & a & -d & c \\
 c & d & a & -b \\
 d & -c & b & a
\end{bmatrix}= &a
\begin{bmatrix}
 1 & 0 & 0 & 0 \\
 0 & 1 & 0 & 0 \\
 0 & 0 & 1 & 0 \\
 0 & 0 & 0 & 1
\end{bmatrix}
+ b
\begin{bmatrix}
 0 & -1 & 0 & 0 \\
 1 & 0 & 0 & 0 \\
 0 & 0 & 0 & -1 \\
 0 & 0 & 1 & 0
\end{bmatrix}
+ \nonumber \\
&c
\begin{bmatrix}
 0 & 0 & -1 & 0 \\
 0 & 0 & 0 & 1 \\
 1 & 0 & 0 & 0 \\
 0 & -1 & 0 & 0
\end{bmatrix}
+ d
\begin{bmatrix}
 0 & 0 & 0 & -1 \\
 0 & 0 & -1 & 0 \\
 0 & 1 & 0 & 0 \\
 1 & 0 & 0 & 0
\end{bmatrix}.
\label{eq:m4r}
\end{align}
This representation of quaternions is not unique, but we will stick to the above in this paper.
It is also possible to represent $\mathbb{H}$ as $M(2,\mathbb{C})$ where $M(2,\mathbb{C})$ is a 2~x~2 complex matrix.

With our real-valued representation a quaternion real-valued $2D$ convolution layer can be expressed as follows.
Say that the layer has $N$ feature maps such that $N$ is divisible by 4.
We let the first $N/4$ feature maps represent the real components, the second $N/4$ represent the $i$ imaginary components, the third $N/4$ represent the $j$ imaginary components, and the last $N/4$ represent the $k$ imaginary components.

\subsection{Quaternion Differentiability}
In order for the network to perform backpropagation the cost function and activation functions used must be differentiable with respect to the real, $i$, $j$, and $k$ components of each quaternion parameter of the network.
As the complex chain rule is shown in \cite{trabelsi2017deep}, we provide the quaternion chain rule which can be seen in \ref{a:diff} of the Appendix.

\subsection{Quaternion Convolution}\label{s:qc}
Convolution in the quaternion domain is done by convolving a quaternion filter matrix $\textbf{W}=\textbf{A}+\textit{i}\textbf{B}+\textit{j}\textbf{C}+\textit{k}\textbf{D}$ by a quaternion vector $\textbf{h}=\textbf{w}+\textit{i}\textbf{x}+\textit{j}\textbf{y}+\textit{k}\textbf{z}$.
Performing the convolution by using the distributive property and grouping terms one gets
\begin{align}
\textbf{W}\ast \textbf{h} = &~(\textbf{A}\ast\textbf{w}-\textbf{B}\ast\textbf{x}-\textbf{C}\ast\textbf{y}-\textbf{D}\ast\textbf{z}) + \nonumber \\
&\textit{i}(\textbf{A}\ast\textbf{x}+\textbf{B}\ast\textbf{w}+\textbf{C}\ast\textbf{z}-\textbf{D}\ast\textbf{y}) + \nonumber \\
&\textit{j}(\textbf{A}\ast\textbf{y}-\textbf{B}\ast\textbf{z}+\textbf{C}\ast\textbf{w}+\textbf{D}\ast\textbf{x}) + \nonumber \\
&\textit{k}(\textbf{A}\ast\textbf{z}+\textbf{B}\ast\textbf{y}-\textbf{C}\ast\textbf{x}+\textbf{D}\ast\textbf{w}).
\label{eq:convolve1}
\end{align}
Using a matrix to represent the components of the convolution we have:
\begin{equation}
\begin{bmatrix}
 \mathscr{R}(\textbf{W}\ast \textbf{h}) \\
 \mathscr{I}(\textbf{W}\ast \textbf{h}) \\
 \mathscr{J}(\textbf{W}\ast \textbf{h}) \\
 \mathscr{K}(\textbf{W}\ast \textbf{h})
\end{bmatrix}
=
\begin{bmatrix}
 \textbf{A} & -\textbf{B} & -\textbf{C} & -\textbf{D}\\
 \textbf{B} & \textbf{A} & -\textbf{D} & \textbf{C} \\
 \textbf{C} & \textbf{D} & \textbf{A} & -\textbf{B} \\
 \textbf{D} & -\textbf{C} & \textbf{B} & \textbf{A} \\
\end{bmatrix}
\ast
\begin{bmatrix}
 \textbf{w} \\
 \textbf{x} \\
 \textbf{y} \\
 \textbf{z}
\end{bmatrix}
\label{eq:}
\end{equation}
An example is shown in Fig.~\ref{f:quatconv} where one can see how quaternion convolution forces a linear depthwise mixture of the channels. This is similar to a mixture of standard convolution and depthwise separable convolution from \cite{chollet2016xception}. This reuse of filters on every layer and combination may help extract texture information across channels as seen in \cite{shi2007quaternion}.

\begin{figure}[h]
	\centering
		\includegraphics[width=1.0\textwidth]{figures/quatconv.png}
	\caption{An illustration of quaternion convolution.}
	\label{f:quatconv}
\end{figure}

\subsection{Quaternion Batch-Normalization}
Batch-normalization \cite{ioffe2015batch} is used by the vast majority of all deep networks to stabilize and speed up training.
It works by keeping the activations of the network at zero mean and unit variance.
The original formulation of batch-normalization only works for real-values.
Applying batch normalization to complex or hyper-complex numbers is more difficult, one can not simply translate and scale them such that their mean is 0 and their variance is 1.
This would not give equal variance in the multiple components of a complex or hyper-complex number.
To overcome this for complex numbers a whitening approach is used \cite{trabelsi2017deep}, which scales the data by the square root of their variances along each of the two principle components.
We use the same approach, but must whiten 4D vectors.

However, an issue arises in that there is no nice way to calculate the inverse square root of a 4~x~4 matrix.
It turns out that the square root is not necessary and we can instead use the Cholesky decomposition on our covariance matrix.
The details of why this works for whitening can be found in \ref{a:whitening} of the Appendix.
Now our whitening is accomplished by multiplying the \textbf{0}-centered data ($\textbf{x} - \mathbb{E}[\textbf{x}]$) by \textbf{W}:
\begin{equation}
\tilde{x} = \textbf{W}(\textbf{x} - \mathbb{E}[\textbf{x}])
\label{eq:white4d}
\end{equation}
where \textbf{W} is one of the matrices from the Cholesky decomposition of $\textbf{V}^{-1}$ where \textbf{V} is the covariance matrix given by:
\begin{align}
\textbf{V}
=&
\begin{bmatrix}
 V_{rr} & V_{ri} & V_{rj} & V_{rk} \\
 V_{ir} & V_{ii} & V_{ij} & V_{ik} \\
 V_{jr} & V_{ji} & V_{jj} & V_{jk} \\
 V_{kr} & V_{ki} & V_{kj} & V_{kk}
\end{bmatrix} \nonumber \\
=& \small
\begin{bmatrix}
\mbox{Cov}(\mathscr{R}\{\textbf{x}\}, \mathscr{R}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{R}\{\textbf{x}\}, \mathscr{I}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{R}\{\textbf{x}\}, \mathscr{J}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{R}\{\textbf{x}\}, \mathscr{K}\{\textbf{x}\}) \\
\mbox{Cov}(\mathscr{I}\{\textbf{x}\}, \mathscr{R}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{I}\{\textbf{x}\}, \mathscr{I}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{I}\{\textbf{x}\}, \mathscr{J}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{I}\{\textbf{x}\}, \mathscr{K}\{\textbf{x}\}) \\
\mbox{Cov}(\mathscr{J}\{\textbf{x}\}, \mathscr{R}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{J}\{\textbf{x}\}, \mathscr{I}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{J}\{\textbf{x}\}, \mathscr{J}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{J}\{\textbf{x}\}, \mathscr{K}\{\textbf{x}\}) \\
\mbox{Cov}(\mathscr{K}\{\textbf{x}\}, \mathscr{R}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{K}\{\textbf{x}\}, \mathscr{I}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{K}\{\textbf{x}\}, \mathscr{J}\{\textbf{x}\}) & \mbox{Cov}(\mathscr{K}\{\textbf{x}\}, \mathscr{K}\{\textbf{x}\}) \\
\end{bmatrix}
\label{eq:V4d}
\end{align}
where Cov$(\cdot)$ is the covariance and $\mathscr{R}\{x\}$, $\mathscr{I}\{x\}$, $\mathscr{J}\{x\}$, and $\mathscr{K}\{x\}$ are the real, $i$, $j$, and $k$ components of $\textbf{x}$ respectively.

Real-valued batch normalization also uses two learned parameters, $\beta$ and $\gamma$.
Our shift parameter {\boldmath$\beta$} must shift a quaternion value so it is a quaternion value itself with real, $i$, $j$, and $k$ as learnable components.
The scaling parameter {\boldmath$\gamma$} is a symmetric matrix of size matching $\textbf{V}$ given by:
\begin{equation}
\mathbf{\gamma}
=
\left(
\begin{array}{cccc}
V_{rr} & V_{ri} & V_{rj} & V_{rk} \\
V_{ri} & V_{ii} & V_{ij} & V_{ik} \\
V_{rj} & V_{ij} & V_{jj} & V_{jk} \\
V_{rk} & V_{ik} & V_{jk} & V_{kk}
\end{array}
\right)
\label{eq:gamma}
\end{equation}
Because of its symmetry it has only ten learnable parameters.
The variance of the components of input $\tilde{\textbf{x}}$ are variance 1 so the diagonal of {\boldmath$\gamma$} is initialized to $1/\sqrt{16}$ in order to obtain a modulus of 1 for the variance of the normalized value.
The off diagonal terms of {\boldmath$\gamma$} and all components of {\boldmath$\beta$} are initialized to 0.
The quaternion batch normalization is defined as:
\begin{equation}
\mbox{BN}(\tilde{\textbf{x}}) = \mathbf{\gamma}\tilde{\textbf{x}} + \mathbf{\beta}
\label{eq:qbn}
\end{equation}

\subsection{Quaternion Weight Initialization}
The proper initialization of weights is vital to convergence of deep networks.
In this work we derive our quaternion weight initialization using the same procedure as Glorot and Bengio \cite{glorot2010understanding} and He et al. \cite{he2015delving}.

To begin we find the variance of a quaternion weight:
\begin{align}
W = &~|W|e^{(\mbox{cos}\phi_1 \textit{i} + \mbox{cos}\phi_2 \textit{j} + \mbox{cos}\phi_3 \textit{k})\theta} \nonumber \\
= &~\mathscr{R}\{W\} + \mathscr{I}\{W\} + \mathscr{J}\{W\} + \mathscr{K}\{W\}.
\label{eq:quaternion_weight}
\end{align}
where $|W|$ is the magnitude, $\theta$ and $\phi$ are angle arguments, and $\mbox{cos}^2\phi_1 + \mbox{cos}^2\phi_2 + \mbox{cos}^2\phi_3 = 1$ \cite{turner2002}.

Variance is defined as
\begin{equation}
\mbox{Var}(W) = \mathbb{E}[|W|^2] - (\mathbb{E}[W])^2,
\label{eq:variance}
\end{equation}
but since $W$ is symmetric around 0 the term $(\mathbb{E}[W])^2$ is 0.
We do not have a way to calculate $\mbox{Var}(W) = \mathbb{E}[|W|^2]$ so we make use of the magnitude of quaternion normal values $|W|$, which follows an independent normal distribution with four degrees of freedom (DOFs).
We can then calculate the expected value of $|W|^2$ to find our variance
\begin{equation}
\mathbb{E}[|W|^2] = \int_\infty^\infty x^2 f(x) ~dx = 4\sigma^2
\label{eq:expected}
\end{equation}
where $f(x)$ is the four DOF distribution shown in the Appendix.

And since $\mbox{Var}(W) = \mathbb{E}[|W|^2]$, we now have the variance of $W$ expressed in terms of a single parameter $\sigma$:
\begin{equation}
\mbox{Var}(W) = 4\sigma^2.
\label{eq:variance_sigma}
\end{equation}

%Taking the variance of this term gives
%\begin{equation}
%\mbox{Var}(|W|) = \mathbb{E}[|W|^2] - (\mathbb{E}[|W|])^2
%\label{eq:absolute_variance}
%\end{equation}
%Inserting \eqref{eq:variance} into \eqref{eq:absolute_variance} and rearranging:
%\begin{equation}
%\mbox{Var}(W) = \mbox{Var}(|W|) + (\mathbb{E}[|W|])^2.
%\label{eq:combined_variance}
%\end{equation}
%This form allows us to calculate the variance using the variance and expectation of $|W|$, which is shown in the Appendix, and given by:
%\begin{equation}
%\mathbb{E}[|W|] = , ~\mbox{Var}(|W|) = .
%\label{eq:variance_expectation}
%\end{equation}
%Finally, plugging \eqref{eq:variance_expectation} into \eqref{eq:combined_variance} allows us to express $W$ in terms of a single parameter $\sigma$:
%\begin{equation}
%\mbox{Var}(W) =  = 4\sigma^2.
%\label{eq:variance_sigma}
%\end{equation}

To follow the Glorot and Bengio \cite{glorot2010understanding} initialization we have $\mbox{Var}(W) = 2/(n_{in}+n_{out})$, where $n_{in}$ and $n_{out}$ are the number of input and output units respectivly.
Setting this equal to \eqref{eq:variance_sigma} and solving for $\sigma$ gives $\sigma = 1/\sqrt{2(n_{in}+n_{out})}$.
To follow He et al. \cite{he2015delving} initialization that is specialized for rectified linear units (ReLUs) \cite{nair2010rectified}, then we have $\mbox{Var}(W) = 2/n_{in}$, which again setting equal to \eqref{eq:variance_sigma} and solving for $\sigma$ gives $\sigma = 1/\sqrt{2n_{in}}$.

As shown in \eqref{eq:quaternion_weight} the weight has components $|W|$, $\theta$, and $\phi$.
We can initialize the magnitude $|W|$ using our four DOF distribution defined with the appropriate $\sigma$ based on which initialization scheme we are following.
The angle components are initialized using the uniform distribution between $-\pi$ and $\pi$ where we ensure the constraint on $\phi$.

\section{Experimental Results}
Our experiments covered image classification using both the CIFAR-10 and CIFAR-100 benchmarks and image segmentation using the KITTI Road Estimation benchmark.

\subsection{Classification}
We use the same architecture as the large model in \cite{trabelsi2017deep}, which is a 110 layer Residual model similar to the one in \cite{he2016deep}.
There is one difference between the real-valued network and the ones used for both the complex and hyper-complex valued networks.
Because the datasets are all real-valued the network must learn the imaginary or quaternion components.
We use the same technique as \cite{trabelsi2017deep} where there is an additional block immediately after the input which will learn the hyper-complex components
\begin{equation*}
BN \rightarrow ReLU \rightarrow Conv \rightarrow BN \rightarrow ReLU \rightarrow Conv.
\end{equation*}

Since all datasets that we work with are real-valued,
we present a way to learn their imaginary components to let the rest of the network operate in the
complex plane.
We learn the initial imaginary component of our input by performing the operations
present within a single real-valued residual block

This means that to maintain the same approximate parameter count the number of convolution kernels for the complex network was increased.
We however did not increase the number of convolution kernels for the quaternion trials so any increase in model performance comes from the quaternion filters and at a lower hardware budget.

The architecture consists of 3 stages of repeating residual blocks where at the end of each stage the images are downsized by strided convolutions.
Each stage also doubles the previous stage's number of convolution kernels.
The last layers are a global average pooling layer followed by a single fully connected layer with a softmax function is used to classify the input as either one of the 10 classes in CIFAR-10 or one of the 100 classes in CIFAR-100.

We also followed their training procedure of using the backpropagation algorithm with Stochastic Gradient Descent with Nesterov momentum \cite{nesterov1983method} set at 0.9.
The norm of the gradients are clipped to 1 and a custom learning rate scheduler is used.
The learning scheduler is the same used in \cite{trabelsi2017deep} for a direct comparison in performance.
The learning rate is initially set to 0.01 for the first 10 epochs and then set it to 0.1 from epoch 10-100 and then cut it by a factor of 10 at epochs 120 and 150.
Table~\ref{t:results1} presents our results along side the real and complex valued networks.
Our quaternion model outperforms the real and complex networks on both datasets on a smaller parameter budget.

\begin{table}[h]
	\centering
		\begin{tabular}{l c c}
			\hline
			Architecture & CIFAR-10 & CIFAR-100 \\
			\hline
			\cite{he2016deep} & 6.37 & - \\
			\cite{trabelsi2017deep} Complex & 5.60 & 27.09 \\
			Quaternion & \textbf{5.44} & \textbf{26.01}
		\end{tabular}
	\caption{Classification error on CIFAR-10 and CIFAR-100. Note that \cite{he2016deep} is a 110 layer residual network, \cite{trabelsi2017deep} is 118 layer complex network with the same design as the prior except additional initial layers to extract complex mappings.}
	\label{t:results1}
\end{table}

\subsection{Segmentation}
For this experiment we used the same model as the above, but cut the number of residual blocks out of the model for memory reasons given that the KITTI data is large color images.
The 110 layer model has 10, 9, and 9 residual blocks in the 3 stages talked about above, while this model has 2, 1, and 1 and does not perform any strided convolutions.
This gives a total of 38 layers.

This means that to maintain the same approximate parameter count the number of convolution kernels for the complex network was increased.
We however did not increase the number of convolution kernels for the quaternion trials so any increase in model performance comes from the quaternion filters and at a lower hardware budget.

The last layer is a $1 \times 1$ convolution with a sigmoid output so we are getting a heatmap prediction the same size as the input.
The training procedure is also as above, but the learning rate is scheduled differently.
Here we begin at 0.01 for the first 10 epochs and then set it to 0.1 from epoch 10-50 and then cut it by a factor of 10 at 100 and 150.
Table~\ref{t:results2} presents our results along side the real and complex valued networks.
Quaternion outperformed the other two by a larger margin compared to the classification tasks.

\begin{table}[h]
	\centering
		\begin{tabular}{l c c}
			\hline
			Architecture & KITTI \\
			\hline
			Real & 0.747 \\
			Complex & 0.769 \\
			Quaternion & \textbf{0.827}
		\end{tabular}
	\caption{IOU on KITTI Road Estimation benchmark.}
	\label{t:results2}
\end{table}

\section{Conclusions}
We have extended upon work looking into complex valued networks by exploring quaternion values.
We presented the building blocks required to build and train deep quaternion networks and used them to test residual architectures on two common image classification benchmarks.
We show that they have competitive performance by beating both the real and complex valued networks with less parameters.
Future work will be needed to test quaternion networks for more segmentation datasets and for audio processing tasks.

\section{Acknowledgments}
We would like to thank James Dent of the University of Louisiana at Lafayette Physics Department for helpful discussions.
We also thank Fugro for research time on this project.

\clearpage
\bibliography{bib}{}
\bibliographystyle{alpha}

\clearpage
\section{Appendix}
\subsection{The Generalized Quaternion Chain Rule for a Real-Valued Function}\label{a:diff}
We start by specifying the Jacobian.
Let $L$ be a real valued loss function and $q$ be a quaternion variable such that $q = a+\textit{i}~b+\textit{j}~c+\textit{k}~d$ where $a,b,c,d \in \mathbb{R}$ then,
\begin{align}
\nabla_L(q) &= \frac{\partial L}{\partial q} = \frac{\partial L}{\partial a} + \textit{i}~\frac{\partial L}{\partial b} + \textit{j}~\frac{\partial L}{\partial c} + \textit{k}~\frac{\partial L}{\partial d} \\ \nonumber
&= \frac{\partial L}{\partial \mathbb{R}(q)} + \textit{i}~\frac{\partial L}{\partial \mathbb{I}(q)} + \textit{j}~\frac{\partial L}{\partial \mathbb{J}(q)} + \textit{k}~\frac{\partial L}{\partial \mathbb{K}(q)} \\ \nonumber
&= \mathbb{R}(\nabla_L(q)) + \mathbb{I}(\nabla_L(q)) + \mathbb{J}(\nabla_L(q)) + \mathbb{K}(\nabla_L(q))
\label{eq:diff1}
\end{align}
Now let $g = m+\textit{i}~n+\textit{j}~o+\textit{k}~p$ be another quaternion variable where $q$ can be expressed in terms of $g$ and $m,n,o,p \in \mathbb{R}$ we then have,
\begin{align}
\nabla_L(q) &= \frac{\partial L}{\partial g} = \frac{\partial L}{\partial m} + \textit{i}~\frac{\partial L}{\partial n} + \textit{j}~\frac{\partial L}{\partial o} + \textit{k}~\frac{\partial L}{\partial p} \\ \nonumber
&= \frac{\partial L}{\partial a}\frac{\partial a}{\partial m} + \frac{\partial L}{\partial b}\frac{\partial b}{\partial m} + \frac{\partial L}{\partial c}\frac{\partial c}{\partial m} + \frac{\partial L}{\partial d}\frac{\partial d}{\partial m} \\ \nonumber
&~~+ \textit{i}~\left( \frac{\partial L}{\partial a}\frac{\partial a}{\partial n} + \frac{\partial L}{\partial b}\frac{\partial b}{\partial n} + \frac{\partial L}{\partial c}\frac{\partial c}{\partial n} + \frac{\partial L}{\partial d}\frac{\partial d}{\partial n} \right) \\ \nonumber
&~~+ \textit{j}~\left( \frac{\partial L}{\partial a}\frac{\partial a}{\partial o} + \frac{\partial L}{\partial b}\frac{\partial b}{\partial o} + \frac{\partial L}{\partial c}\frac{\partial c}{\partial o} + \frac{\partial L}{\partial d}\frac{\partial d}{\partial o} \right) \\ \nonumber
&~~+ \textit{k}~\left( \frac{\partial L}{\partial a}\frac{\partial a}{\partial p} + \frac{\partial L}{\partial b}\frac{\partial b}{\partial p} + \frac{\partial L}{\partial c}\frac{\partial c}{\partial p} + \frac{\partial L}{\partial d}\frac{\partial d}{\partial p} \right) \\ \nonumber
&= \frac{\partial L}{\partial a} \left( \frac{\partial a}{\partial m} + \textit{i}~\frac{\partial a}{\partial n} + \textit{i}~\frac{\partial a}{\partial o} + \textit{i}~\frac{\partial a}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial b} \left( \frac{\partial b}{\partial m} + \textit{i}~\frac{\partial b}{\partial n} + \textit{i}~\frac{\partial b}{\partial o} + \textit{i}~\frac{\partial b}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial c} \left( \frac{\partial c}{\partial m} + \textit{i}~\frac{\partial c}{\partial n} + \textit{i}~\frac{\partial c}{\partial o} + \textit{i}~\frac{\partial c}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial d} \left( \frac{\partial d}{\partial m} + \textit{i}~\frac{\partial d}{\partial n} + \textit{i}~\frac{\partial d}{\partial o} + \textit{i}~\frac{\partial d}{\partial p} \right) \\ \nonumber
&= \frac{\partial L}{\partial \mathbb{R}(q)} \left( \frac{\partial a}{\partial m} + \textit{i}~\frac{\partial a}{\partial n} + \textit{i}~\frac{\partial a}{\partial o} + \textit{i}~\frac{\partial a}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial \mathbb{I}(q)} \left( \frac{\partial b}{\partial m} + \textit{i}~\frac{\partial b}{\partial n} + \textit{i}~\frac{\partial b}{\partial o} + \textit{i}~\frac{\partial b}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial \mathbb{J}(q)} \left( \frac{\partial c}{\partial m} + \textit{i}~\frac{\partial c}{\partial n} + \textit{i}~\frac{\partial c}{\partial o} + \textit{i}~\frac{\partial c}{\partial p} \right) \\ \nonumber
&~~+ \frac{\partial L}{\partial \mathbb{K}(q)} \left( \frac{\partial d}{\partial m} + \textit{i}~\frac{\partial d}{\partial n} + \textit{i}~\frac{\partial d}{\partial o} + \textit{i}~\frac{\partial d}{\partial p} \right) \\ \nonumber
&= \mathbb{R}(\nabla_L(q)) \left( \frac{\partial a}{\partial m} + \textit{i}~\frac{\partial a}{\partial n} + \textit{i}~\frac{\partial a}{\partial o} + \textit{i}~\frac{\partial a}{\partial p} \right) \\ \nonumber
&~~+ \mathbb{I}(\nabla_L(q)) \left( \frac{\partial b}{\partial m} + \textit{i}~\frac{\partial b}{\partial n} + \textit{i}~\frac{\partial b}{\partial o} + \textit{i}~\frac{\partial b}{\partial p} \right) \\ \nonumber
&~~+ \mathbb{J}(\nabla_L(q)) \left( \frac{\partial c}{\partial m} + \textit{i}~\frac{\partial c}{\partial n} + \textit{i}~\frac{\partial c}{\partial o} + \textit{i}~\frac{\partial c}{\partial p} \right) \\ \nonumber
&~~+ \mathbb{K}(\nabla_L(q)) \left( \frac{\partial d}{\partial m} + \textit{i}~\frac{\partial d}{\partial n} + \textit{i}~\frac{\partial d}{\partial o} + \textit{i}~\frac{\partial d}{\partial p} \right)
\label{eq:diff2}
\end{align}

\subsection{Whitening a Matrix}\label{a:whitening}
Let $\textbf{X}$ be an $n$~x~$n$ matrix and $\mbox{cov}(\textbf{X}) = \mathbf{\Sigma}$ is the symmetric covariance matrix of the same size.
Whitening a matrix linearly decorrelates the input dimensions, meaning that whitening transforms $\textbf{X}$ into $\textbf{Z}$ such that $\mbox{cov}(\textbf{Z}) = \textbf{I}$ where $\textbf{I}$ is the identity matrix \cite{kessy2017optimal}.
The matrix $\textbf{Z}$ can be written as:
\begin{equation}
\textbf{Z} = \textbf{W}(\textbf{X} - \mu)
\label{eq:white1}
\end{equation}
where $\textbf{W}$ is an $n$~x~$n$ `whitening' matrix. Since $\mbox{cov}(\textbf{Z}) = \textbf{I}$ it follows that:
\begin{align}
&\mathbb{E}[\mathbf{Z}\mathbf{Z}^T] = \mathbf{I} \nonumber \\
&\mathbb{E}[\mathbf{W}(\mathbf{X - \mu})(\mathbf{W}(\mathbf{X} - \mu))^T] = \mathbf{I} \nonumber \\
&\mathbb{E}[\mathbf{W}(\mathbf{X - \mu})(\mathbf{X} - \mu)^T\mathbf{W}^T] = \mathbf{I} \nonumber \\
&\mathbf{W}\Sigma\mathbf{W}^T = \mathbf{I} \nonumber \\
&\mathbf{W}\Sigma\mathbf{W}^T\mathbf{W} = \mathbf{W} \nonumber \\
&\mathbf{W}^T \mathbf{W} = \mathbf{\Sigma}^{-1} \label{eq:white2}
\end{align}
From \eqref{eq:white2} it is clear that the Cholesky decomposition provides a suitable (but not unique) method of finding $\textbf{W}$.

\subsection{Cholesky Decomposition}
Cholesky decomposition is an efficient way to implement LU decomposition for symmetric matrices, which allows us to find the square root.
Consider $\textbf{A}\textbf{X} = \textbf{b}$, $\textbf{A}=[a_{ij}]_{n\times n}$, and $a_{ij} = a_{ji}$, then the Cholesky decomposition of $\textbf{A}$ is given by $\textbf{A} = \textbf{L}\textbf{L}'$ where
\begin{equation}
\textbf{L}=
\begin{bmatrix}
 l_{11} & 0 & \ldots & 0 \\
 l_{21} & l_{22} & \ldots & \vdots \\
 \vdots & \vdots & \ddots & 0 \\
 l_{n1} & l_{n2} & ... & l_{nn} \\
\end{bmatrix}
\label{eq:cholesky1}
\end{equation}
Let $l_{ki}$ be the $k^{th}$ row and $i^{th}$ column entry of $\textbf{L}$, then

\[
   l_{ki} =
	 \begin{cases}
      0, & k < i \\
      \sqrt{a_{ii} - \sum_{j=1}^{i-1}l^2_{kj}}, & k=i \\
      \frac{1}{l_{ii}} (a_{ki} - \sum_{j=1}^{i-1}l_{ij}l_{kj}), & i < k
   \end{cases}
\]

\subsection{4 DOF Independent Normal Distribution}
Consider the four-dimensional vector $\textbf{Y} = (S,T,U,V)$ which has components that are normally distributed, centered at zero, and independent.
Then $S$, $T$, $U$, and $V$ all have density functions
\begin{equation}
f_S(x;\sigma) = f_T(x;\sigma) = f_U(x;\sigma) = f_V(x;\sigma) = \frac{e^{-x^2/(2\sigma^2)}}{\sqrt{2\pi\sigma^2}}.
\label{eq:single_dists}
\end{equation}
Let $\textbf{X}$ be the length of $\textbf{Y}$, which means $\textbf{X} = \sqrt{S^2+T^2+U^2+V^2}$.
Then $\textbf{X}$ has the cumulative distribution function
\begin{equation}
F_X(x;\sigma) = \int \int \int \int_{H_x} f_S(\mu;\sigma)f_T(\mu;\sigma)f_U(\mu;\sigma)f_V(\mu;\sigma) ~dA,
\label{eq:cumdist}
\end{equation}
where $H_x$ is the four-dimensional sphere
\begin{equation}
H_x = \left\{(s,t,u,v)~:~\sqrt{s^2+t^2+u^2+v^2} < x \right\}.
\label{eq:4dsphere}
\end{equation}
We then can write the integral in polar representation
\begin{align}
F_X(x;\sigma) = & ~\frac{1}{4\pi^2\sigma^4} \int_0^\pi \int_0^\pi \int_0^{2\pi} \int_0^x r^3e^{-r^2/(2\sigma^2)} \mbox{sin}(\theta) \mbox{sin}(\phi) \mbox{cos}(\psi) ~dr~d\theta~d\phi~d\psi \nonumber \\
= & ~\frac{1}{2\sigma^4} \int_0^x r^3e^{-r^2/(2\sigma^2)} ~dr.
\label{eq:polarint}
\end{align}
The probability density function of $\textbf{X}$ is the derivative of its cumulative distribution function so we use the funamental theorem of calculus on \eqref{eq:polarint} to finally arrive at
\begin{equation}
f_X(x;\sigma) = \frac{d}{dx}F_X(x;\sigma) =  ~\frac{1}{2\sigma^4} x^3e^{-x^2/(2\sigma^2)}.
\label{eq:finaldist}
\end{equation}

%\clearpage
%\begin{appendices}
%\section{Quaternion Weight Initialization}
%\end{appendices}

\end{document}


\section{Analyzing real functions}


\section{Multigrid Background\label{sec:background}}
\markboth{\textsc{MLD2P4 User's and Reference Guide}}
         {\textsc{\ref{sec:background} Multigrid Background}}

Multigrid preconditioners, coupled with Krylov iterative
solvers, are widely used in the parallel solution of large and sparse linear systems,
because of their optimality in the solution of linear systems arising from the
discretization of scalar elliptic Partial Differential Equations (PDEs) on regular grids.
Optimality, also known as algorithmic scalability, is the property
of having a computational cost per iteration that depends linearly on
the problem size, and a convergence rate that is independent of the problem size.

Multigrid preconditioners are based on a recursive application of a two-grid process
consisting of smoother iterations and a coarse-space (or coarse-level) correction.
The smoothers may be either basic iterative methods, such as the Jacobi and Gauss-Seidel ones,
or more complex subspace-correction methods, such as the Schwarz ones.
The coarse-space correction consists of solving, in an appropriately chosen
coarse space, the residual equation associated with the approximate solution computed
by the smoother, and of using the solution of this equation to correct the
previous approximation. The transfer of information between the original
(fine) space and the coarse one is performed by using suitable restriction and
prolongation operators. The construction of the coarse space and the corresponding
transfer operators is carried out by applying a so-called coarsening algorithm to the system
matrix. Two main approaches can be used to perform coarsening: the geometric approach,
which exploits the knowledge of some physical grid associated with the matrix
and requires the user to define transfer operators from the fine
to the coarse level and vice versa, and the algebraic approach, which builds
the coarse-space correction and the associate transfer operators using only matrix
information. The first approach may be difficult when the system comes from
discretizations on complex geometries;
furthermore, ad hoc one-level smoothers may be required to get an efficient
interplay between fine and coarse levels, e.g., when matrices with highly varying coefficients
are considered. The second approach performs a fully automatic coarsening and enforces the
interplay between fine and coarse level by suitably choosing the coarse space and
the coarse-to-fine interpolation (see, e.g., \cite{Briggs2000,Stuben_01,dd2_96} for details.)

MLD2P4 uses a pure algebraic approach, based on the smoothed
aggregation algorithm \cite{BREZINA_VANEK,VANEK_MANDEL_BREZINA},
for building the sequence of coarse matrices and transfer operators,
starting from the original one.
A decoupled version of this algorithm is implemented, where the smoothed
aggregation is applied locally to each submatrix \cite{TUMINARO_TONG}.
A brief description of the AMG preconditioners implemented in MLD2P4 is given in
Sections~\ref{sec:multilevel}-\ref{sec:smoothers}. For further details the reader
is referred to \cite{para_04,aaecc_07,apnum_07,MLD2P4_TOMS}.

We note that optimal multigrid preconditioners do not necessarily correspond
to minimum execution times in a parallel setting. Indeed, to obtain effective parallel
multigrid preconditioners, a tradeoff between the optimality and the cost of building and
applying the smoothers and the coarse-space corrections must be achieved. Effective
parallel preconditioners require algorithmic scalability to be coupled with implementation
scalability, i.e., a computational cost per iteration which remains (almost) constant as
the number of parallel processors increases.

\subsection{AMG preconditioners\label{sec:multilevel}}

In order to describe the AMG preconditioners available in MLD2P4, we consider a
linear system
\begin{equation}
Ax=b, \label{eq:system}
\end{equation}
where $A=(a_{ij}) \in \mathbb{R}^{n \times n}$ is a nonsingular sparse matrix;
for ease of presentation we assume $A$ has a symmetric sparsity
pattern.

Let us consider as finest index space the set of row (column) indices of $A$, i.e.,
$\Omega = \{1, 2, \ldots, n\}$.
Any algebraic multilevel preconditioners implemented in MLD2P4 generates
a hierarchy of index spaces and a corresponding hierarchy of matrices,
\[ \Omega^1 \equiv \Omega \supset \Omega^2 \supset \ldots \supset \Omega^{nlev},
\quad A^1 \equiv A, A^2, \ldots, A^{nlev}, \]
by using the information contained in $A$, without assuming any
knowledge of the geometry of the problem from which $A$ originates.
A vector space $\mathbb{R}^{n_{k}}$ is associated with $\Omega^k$,
where $n_k$ is the size of $\Omega^k$.
For all $k < nlev$, a restriction operator and a prolongation one are built,
which connect two levels $k$ and $k+1$:
\[
    P^k \in \mathbb{R}^{n_k \times n_{k+1}}, \quad
    R^k \in \mathbb{R}^{n_{k+1}\times n_k};
\]
the matrix $A^{k+1}$ is computed by using the previous operators according
to the Galerkin approach, i.e.,
\[
  A^{k+1}=R^kA^kP^k.
\]
In the current implementation of MLD2P4 we have $R^k=(P^k)^T$
A smoother with iteration matrix $M^k$ is set up at each level $k < nlev$, and a solver
is set up at the coarsest level, so that they are ready for application
(for example, setting up a solver based on the $LU$ factorization means computing
and storing the $L$ and $U$ factors). The construction of the hierarchy of AMG components
described so far corresponds to the so-called build phase of the preconditioner.

\begin{figure}[t]
\begin{center}
\framebox{
\begin{minipage}{.85\textwidth}
\begin{tabbing}
\quad \=\quad \=\quad \=\quad \\[-3mm]
procedure V-cycle$\left(k,A^k,b^k,u^k\right)$ \\[2mm]
\>if $\left(k \ne nlev \right)$ then \\[1mm]
\>\> $u^k = u^k + M^k \left(b^k - A^k u^k\right)$ \\[1mm]
\>\> $b^{k+1} = R^{k+1}\left(b^k - A^k u^k\right)$ \\[1mm]
\>\> $u^{k+1} =$ V-cycle$\left(k+1,A^{k+1},b^{k+1},0\right)$ \\[1mm]
\>\> $u^k = u^k + P^{k+1} u^{k+1}$ \\[1mm]
\>\> $u^k = u^k + M^k \left(b^k - A^k u^k\right)$ \\[1mm]
\>else \\[1mm]
\>\> $u^k = \left(A^k\right)^{-1} b^k$\\[1mm]
\>endif \\[1mm]
\>return $u^k$ \\[1mm]
end
\end{tabbing}
\end{minipage}
}
\caption{Application phase of a V-cycle preconditioner.\label{fig:application_alg}}
\end{center}
\end{figure}

The components produced in the build phase may be combined in several ways
to obtain different multilevel preconditioners;
this is  done in the application phase, i.e., in the computation of a vector
of type $w=B^{-1}v$, where $B$ denotes the preconditioner, usually within an iteration
of a Krylov solver \cite{Saad_book}. An example of such a combination, known as
V-cycle, is given in Figure~\ref{fig:application_alg}. In this case, a single iteration
of the same smoother is used before and after the the recursive call to the V-cycle (i.e.,
in the pre-smoothing and post-smoothing phases); however, different choices can be
performed. Other cycles can be defined; in MLD2P4, we implemented the standard V-cycle
and W-cycle~\cite{Briggs2000}, and a version of the K-cycle described
in~\cite{Notay2008}.

\subsection{Smoothed Aggregation\label{sec:aggregation}}

In order to define the prolongator $P^k$, used to compute
the coarse-level matrix $A^{k+1}$, MLD2P4 uses the smoothed aggregation
algorithm described in \cite{BREZINA_VANEK,VANEK_MANDEL_BREZINA}.
The basic idea of this algorithm is to build a coarse set of indices
$\Omega^{k+1}$ by suitably grouping the indices of $\Omega^k$ into disjoint
subsets (aggregates), and to define the coarse-to-fine space transfer operator
$P^k$ by applying a suitable smoother to a simple piecewise constant
prolongation operator, with the aim of improving the quality of the coarse-space correction.

Three main steps can be identified in the smoothed aggregation procedure:
\begin{enumerate}
        \item aggregation of the indices of $\Omega^k$ to obtain $\Omega^{k+1}$;
        \item construction of the prolongator $P^k$;
        \item application of $P^k$ and $R^k=(P^k)^T$ to build $A^{k+1}$.
\end{enumerate}

In order to perform the coarsening step, the smoothed aggregation algorithm
described in~\cite{VANEK_MANDEL_BREZINA} is used. In this algorithm,
each index $j \in \Omega^{k+1}$ corresponds to an aggregate $\Omega^k_j$ of $\Omega^k$,
consisting of a suitably chosen index $i \in \Omega^k$ and indices that are (usually) contained in a
strongly-coupled neighborood of $i$, i.e.,
\begin{equation}
\label{eq:strongly_coup}
   \Omega^k_j \subset \mathcal{N}_i^k(\theta) =
   \left\{ r \in \Omega^k: |a_{ir}^k| > \theta \sqrt{|a_{ii}^ka_{rr}^k|} \right \} \cup \left\{ i \right\},
\end{equation}
for a given threshold $\theta \in [0,1]$ (see~\cite{VANEK_MANDEL_BREZINA} for the details).
Since this algorithm has a sequential nature, a decoupled
version of it is applied, where each processor independently executes
the algorithm on the set of indices assigned to it in the initial data
distribution. This version is embarrassingly parallel, since it does not require any data
communication. On the other hand, it may produce some nonuniform aggregates
and is strongly dependent on the number of processors and on the initial partitioning
of the matrix $A$. Nevertheless, this parallel algorithm has been chosen for
MLD2P4, since it has been shown to produce good results in practice
\cite{aaecc_07,apnum_07,TUMINARO_TONG}.

The prolongator $P^k$ is built starting from a tentative prolongator
$\bar{P}^k \in \mathbb{R}^{n_k \times n_{k+1}}$, defined as
\begin{equation}
\bar{P}^k =(\bar{p}_{ij}^k), \quad  \bar{p}_{ij}^k =
\left\{ \begin{array}{ll}
1 & \quad \mbox{if} \; i \in \Omega^k_j, \\
0 & \quad \mbox{otherwise},
\end{array} \right.
\label{eq:tent_prol}
\end{equation}
where $\Omega^k_j$ is the aggregate of $\Omega^k$
corresponding to the index $j \in \Omega^{k+1}$.
$P^k$ is obtained by applying to $\bar{P}^k$ a smoother
$S^k \in \mathbb{R}^{n_k \times n_k}$:
$$
P^k = S^k \bar{P}^k,
$$
in order to remove nonsmooth components from the range of the prolongator,
and hence to improve the convergence properties of the multilevel
method~\cite{BREZINA_VANEK,Stuben_01}.
A simple choice for $S^k$ is the damped Jacobi smoother:
\[
S^k = I - \omega^k (D^k)^{-1} A^k_F ,
\]
where $D^k$ is the diagonal matrix with the same diagonal entries as $A^k$,
$A^k_F = (\bar{a}_{ij}^k)$ is the filtered matrix defined as
\begin{equation}
\label{eq:filtered}
  \bar{a}_{ij}^k =
   \left \{ \begin{array}{ll}
   a_{ij}^k & \mbox{if } j \in \mathcal{N}_i^k(\theta), \\
   0            & \mbox{otherwise},
   \end{array} \right.
   \; (j \ne i),
   \qquad
   \bar{a}_{ii}^k = a_{ii}^k - \sum_{j \ne i} (a_{ij}^k - \bar{a}_{ij}^k),
\end{equation}
and $\omega^k$ is an approximation of $4/(3\rho^k)$, where
$\rho^k$ is the spectral radius of $(D^k)^{-1}A^k_F$ \cite{BREZINA_VANEK}.
In MLD2P4 this approximation is obtained by using $\| A^k_F \|_\infty$ as an estimate
of $\rho^k$. Note that for systems coming from uniformly elliptic
problems, filtering the matrix $A^k$ has little or no effect, and
$A^k$ can be used instead of $A^k_F$. The latter choice is the default in MLD2P4.

\subsection{Smoothers and coarsest-level solvers\label{sec:smoothers}}

The smoothers implemented in MLD2P4 include the Jacobi and block-Jacobi methods,
a hybrid version of the forward and backward Gauss-Seidel methods, and the
additive Schwarz (AS) ones (see, e.g., \cite{Saad_book,dd2_96}).

The hybrid Gauss-Seidel
version is considered because the original Gauss-Seidel method is inherently sequential.
At each iteration of the hybrid version, each parallel process uses the most recent values
of its own local variables and the values of the non-local variables computed at the
previous iteration, obtained by exchanging data with other processes before
the beginning of the current iteration.

In the AS methods, the index space $\Omega^k$ is divided into $m_k$
subsets $\Omega^k_i$ of size $n_{k,i}$,  possibly
overlapping. For each $i$ we consider the restriction
operator $R_i^k \in \mathbb{R}^{n_{k,i} \times n_k}$
that maps a vector $x^k$ to the vector $x_i^k$ made of the components of $x^k$
with indices in $\Omega^k_i$, and the prolongation operator
$P^k_i = (R_i^k)^T$. These operators are then  used to build
$A_i^k=R_i^kA^kP_i^k$, which is the restriction of $A^k$ to the index
space $\Omega^k_i$.
The classical AS preconditioner $M^k_{AS}$ is defined as
\[
    ( M^k_{AS} )^{-1} = \sum_{i=1}^{m_k} P_i^k (A_i^k)^{-1} R_i^{k},
\]
where $A_i^k$ is supposed to be nonsingular. We observe that an approximate
inverse of $A_i^k$ is usually considered instead of $(A_i^k)^{-1}$.
The setup of $M^k_{AS}$ during the multilevel build phase
involves
\begin{itemize}
  \item the definition of the index subspaces $\Omega_i^k$ and of the corresponding
  operators $R_i^k$ (and $P_i^k$);
  \item the computation of the submatrices $A_i^k$;
  \item the computation of their inverses (usually approximated
    through  some form  of incomplete factorization).
\end{itemize}
The computation of $z^k=M^k_{AS}w^k$, with $w^k \in \mathbb{R}^{n_k}$, during the
multilevel application phase, requires
\begin{itemize}
	\item the restriction of $w^k$ to the subspaces $\mathbb{R}^{n_{k,i}}$,
	  i.e.\ $w_i^k = R_i^{k} w^k$;
	\item the computation of the vectors $z_i^k=(A_i^k)^{-1} w_i^k$;
	\item the prolongation and the sum of the previous vectors,
    i.e.\ $z^k = \sum_{i=1}^{m_k} P_i^k z_i^k$.
\end{itemize}
Variants of the classical AS method, which use modifications of the
restriction and prolongation operators, are also implemented in MLD2P4.
Among them, the Restricted AS (RAS) preconditioner usually
outperforms the classical AS preconditioner in terms of convergence
rate and of computation and communication time on parallel distributed-memory
computers, and is therefore the most widely used among the AS
preconditioners~\cite{CAI_SARKIS}.

Direct solvers based on sparse LU factorizations, implemented in the
third-party libraries reported in Section~\ref{sec:third-party}, can be applied
as coarsest-level solvers by MLD2P4. Native inexact solvers based on
incomplete LU factorizations, as well as Jacobi, hybrid (forward) Gauss-Seidel,
and block Jacobi preconditioners are also available. Direct solvers usually
lead to more effective preconditioners in terms of algorithmic scalability;
however, this does not guarantee parallel efficiency.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "userguide"
%%% End:


\section{Invariants on Data Structure}
\label{invariants}
Data Structure from {\tt .h} file shown below.
\begin{verbatim}
typedef struct {
	KEYTYPE  key;
	VALTYPE val;
	uint64_t	hash	: 32;
	uint64_t	psl	: 16;
} q_rh_bucket_t;

typedef struct {
	uint32_t	size;
	uint32_t	nitems;
	uint64_t	divinfo;
	q_rh_bucket_t *	buckets;
	uint64_t	hashkey;
	uint32_t	minsize;
} q_rhashmap_t;

\end{verbatim}

\begin{invariant}
divinfo \(\neq 0\)
\end{invariant}

\begin{invariant}
nitems \(<\) size
\end{invariant}

\begin{invariant}
size \(>\) 0
\end{invariant}

\begin{invariant}
hashkley \(\neq\) 0
\end{invariant}

\begin{invariant}
minsize \(>\) 0
\end{invariant}

\begin{invariant}
minsize \(\leq\) size
\end{invariant}

\begin{invariant}
buckets[i].key \(= 0 \Rightarrow\)
(i) buckets[i].hash = 0
(ii) buckets[i].val = 0
(ii) buckets[i].psl = 0
\end{invariant}

\begin{invariant}
buckets[i].key \(\neq 0 \Rightarrow\)
(i) buckets[i].hash \(\neq 0\),
(ii) buckets[i].val = 0,
(ii) buckets[i].psl = 0
\end{invariant}

\begin{invariant}
Keys are unique.
  \(\exists i:\)  buckets[i].key = k \(\Rightarrow\)
  \(\not \exists j:\)  \( j \neq i \) {\bf and}  buckets[j].key = k
\end{invariant}


\chapter{Reference Orientations} \label{chap:orientations}

The order of the vertices of a $n$-cell implicitly determines the boundary $k$-cells,
thus it is crucial to provide the vertices in the correct order.
While one- and two-dimensional objects provide little space for variations of reference orientations,
the situation changes in higher dimensions. The reference orientations
of the boundary $k$-cells provides ample of variations, while only a few choices
satisfy requirements such as consistent cell normals.

The reference orientations in {\ViennaGrid} are chosen such that the tuples
of vertex IDs for all boundary cells are in ascending order. This is accomplished
in a generic way for a unit-$n$-cell from the simplex and hypercube families in the $n$-dimensional space as follows:
\begin{enumerate}
 \item Start with a $1$-cell at the points $(1, 0, \ldots)$ and $(0, 1, 0, \ldots)$, enumerate the points and set $k=2$.
 \item Prolongate the $k-1$-cell to a $k$-cell in the $k$-dimensional subspace induced by the first $k$ unit vectors.
 \item Enumerate the new vertices.
 \item If $k=n$, stop. Otherwise, go back to $2$.
\end{enumerate}

The families of simplices and hypercubes differ in the way the the prolongation is carried out.
Details are given in the following subsections.

Boundary $k$-cells are ordered with respect to the less-than operator acting on the tuple of vertices in ascending order.
The first entry has precedence over the second entry, etc.

\section{Simplices}
\begin{figure}[tb]
\centering
 \includegraphics[width=0.7\textwidth]{figures/orientation-simplex.eps}
 \caption{Reference orientations of a triangle (left) and a tetrahedron (right).}
 \label{fig:orientation-simplex}
\end{figure}

The prolongation from a $k-1$-simplex to a $k$-simplex is straightforward, since only one vertex is added.
Resulting reference orientations for a triangle and a tetrahedron are given in Fig.~\ref{fig:orientation-simplex}.

Boundary $k$-cell orientations are chosen such that the tuple of vertices is sorted in increasing order.
Note that this does not lead to a consistent orientation of $n$-cell normals. In particular,
the boundary $k$-cells of a triangle and a tetrahedron are thus ordered as follows:
\begin{center}
 \begin{tabular}{|l|c|c|}
  \hline
              & Triangle                  & Tetrahedron   \\
  \hline
   $1$-cells  & $[0,1]$, $[0,2]$, $[1,2]$ & $[0, 1]$, $[0, 2]$, $[0,3]$, \\
              &                           & $[1, 2]$, $[1, 3]$, $[2,3]$ \\
  \hline
   $2$-cells  & $[0,1,2]$                 & $[0,1,2]$, $[0,1,3]$, \\
              &                           & $[0,2,3]$, $[1,2,3]$ \\
  \hline
 \end{tabular}

\end{center}

\section{Hypercube}
\begin{figure}[tb]
\centering
 \includegraphics[width=0.7\textwidth]{figures/orientation-hypercube.eps}
 \caption{Reference orientations of a quadrilateral (left) and a hexahedron (right).}
 \label{fig:orientation-hypercube}
\end{figure}

For the prolongation from a unit-$k-1$-hypercube to a unit-$k$-hypercube, the standard tensor
construction is used. This results in a second $k-1$-hypercube shifted along the $k$-th unit vector,
with the new vertices enumerated in the same orientation as the initial $k-1$-hypercube.
This procedure can also be seen in Fig.~\ref{fig:orientation-hypercube}, where
the two $1$-cells used for the prolongation to the hexahedron are $[0,1,2,3]$ and $[4,5,6,7]$.

For reference, the edges and faces of a quadrilateral and a hexahedron are ordered as follows:
\begin{center}
 \begin{tabular}{|l|c|c|}
  \hline
              & Quadrilateral                      & Hexahedron   \\
  \hline
   $1$-cells  & $[0,1]$, $[0,2]$, $[1,3]$, $[2,3]$ & $[0,1]$, $[0,2]$, $[0,4]$, $[1,3]$, $[1,5]$, $[2,3]$, \\
              &                                    & $[2,6]$, $[3,7]$, $[4,5]$, $[4,6]$, $[5,7]$, $[6,7]$ \\
  \hline
   $2$-cells  & $[0,1,2,3]$                        & $[0,1,2,3]$, $[0,1,4,5]$, $[0,2,4,6]$, \\
              &                                    & $[1,3,5,7]$, $[2,3,6,7]$, $[4,5,6,7]$ \\
  \hline
 \end{tabular}

\end{center}

\NOTE{Mind that the the reference orientations of a {\ViennaGrid} quadrilateral and a {\ViennaGrid} hexahedron coincide with those of the VTK types \lstinline|VTK_PIXEL| and \lstinline|VTK_VOXEL|, but differ from the orientations of the VTK types \lstinline|VTK_QUAD| and \lstinline|VTK_HEXAHEDRON|.}


\section*{Standard Extensions}

\subsection*{RV32M Multiply Extension}
\begin{center}
\begin{tabular}
{T | l | c | T | T | T | C } \hline
\rm Inst & Name              & FMT & \rm Opcode & \rm F3 & \rm F7 & \rm Description (C)     \\ \hline
mul      & MUL               & R   & 0110011    & 0x0    & 0x01   & rd = (rs1 * rs2)[31:0]  \\
mulh     & MUL High          & R   & 0110011    & 0x1    & 0x01   & rd = (rs1 * rs2)[63:32] \\
mulsu    & MUL High (S) (U)  & R   & 0110011    & 0x2    & 0x01   & rd = (rs1 * rs2)[63:32] \\
mulu     & MUL High (U)      & R   & 0110011    & 0x3    & 0x01   & rd = (rs1 * rs2)[63:32] \\
div      & DIV               & R   & 0110011    & 0x4    & 0x01   & rd = rs1 / rs2          \\
divu     & DIV (U)           & R   & 0110011    & 0x5    & 0x01   & rd = rs1 / rs2          \\
rem      & Remainder         & R   & 0110011    & 0x6    & 0x01   & rd = rs1 \% rs2         \\
remu     & Remainder (U)     & R   & 0110011    & 0x7    & 0x01   & rd = rs1 \% rs2         \\
\hline
\end{tabular}
\end{center}

\subsection*{RV32A Atomic Extension}

\begin{itemize}
    \item \texttt{aq}: acquire access bit -- this operation must occur before later memory ops
    \item \texttt{rl}: release access bit -- this operation must occur after earlier memory ops
\end{itemize}

\begin{center}
\begin{tabular}{R@{}W@{}W@{}R@{}R@{}F@{}R@{}O}
\instbitrange{31}{27} &
\instbit{26} &
\instbit{25} &
\instbitrange{24}{20} &
\instbitrange{19}{15} &
\instbitrange{14}{12} &
\instbitrange{11}{7} &
\instbitrange{6}{0} \\
\hline
\multicolumn{1}{|c|}{funct5} &
\multicolumn{1}{c|}{aq} &
\multicolumn{1}{c|}{rl} &
\multicolumn{1}{c|}{rs2} &
\multicolumn{1}{c|}{rs1} &
\multicolumn{1}{c|}{funct3} &
\multicolumn{1}{c|}{rd} &
\multicolumn{1}{c|}{opcode} \\
\hline
5 & 1 & 1 & 5 & 5 & 3 & 5 & 7 \\
\end{tabular}

\begin{tabular}
{T | l | c | T | T | T | T } \hline
\rm Inst  & Name              & FMT   & \rm Opcode & \rm F3 & \rm F5 & \rm Description (C)         \\ \hline
lr.w      & Load Reserved     & R     & 0101111    & 0x2    & 0x02   & rd = M[rs1], reserve M[rs1] \\
sc.w      & Store Conditional & R     & 0101111    & 0x2    & 0x03   & \parbox[t]{2.5in}{ if (reserved) \{ M[rs1] = rs2; rd = 0 \} \\
                                                                        else \{ rd = 1 \}}   \\
amoswap.w & Atomic Swap       & R     & 0101111    & 0x2    & 0x01   & rd = M[rs1]; swap(rd, rs2); M[rs1] = rd \\
amoadd.w  & Atomic ADD        & R     & 0101111    & 0x2    & 0x00   & rd = M[rs1] + rs2; M[rs1] = rd \\
amoand.w  & Atomic AND        & R     & 0101111    & 0x2    & 0x0C   & rd = M[rs1] \& rs2; M[rs1] = rd \\
amoor.w   & Atomic OR         & R     & 0101111    & 0x2    & 0x0A   & rd = M[rs1] | rs2; M[rs1] = rd \\
amoxor.w  & Atomix XOR        & R     & 0101111    & 0x2    & 0x04   & rd = M[rs1] \^{} rs2; M[rs1] = rd \\
amomax.w  & Atomic MAX        & R     & 0101111    & 0x2    & 0x14   & rd = max(M[rs1], rs2); M[rs1] = rd \\
amomin.w  & Atomic MIN        & R     & 0101111    & 0x2    & 0x10   & rd = min(M[rs1], rs2); M[rs1] = rd \\
\hline
\end{tabular}
\end{center}

% \subsection*{RV32F / D Floating-Point Extensions}


\section{Part 4}

The code to run the parts is here:

\lstinputlisting{four.py}

\section{Part 4.a Calculating Linear Growth actor to z = 50}

The file of the functions used for this exercise is:

\lstinputlisting{four_a.py}

The result of this function is given by:

\lstinputlisting{4a.txt}

\section{Part 4.b Calculating Derivative at z = 50}

The file of the functions used for this exercise is:

\lstinputlisting{four_b.py}

The result of this function is given by:

\lstinputlisting{4b.txt}

As can be seen, both the numerical and analytical solutions agree within incredible precision.

\section{Part 4.c Movie}

I ran out of time to do this part of 4.

\section{Part 4.d Second Movie}

I ran out of time to do this part of 4.


\documentclass[11pt]{article}

\usepackage[english, hungarian]{babel}
\usepackage{graphicx}
\usepackage[left=1.5in, right=1.5in, top=1.2in, bottom=1.2in]{geometry}
%\usepackage[left=1in, right=2.3cm, top=1cm, bottom=1cm]{geometry}
\usepackage{layout}
\usepackage{setspace}
%\onehalfspace
\usepackage[hidelinks]{hyperref}
\usepackage{url}
\usepackage{amsmath}

%\usepackage{lmodern}
%\renewcommand*\familydefault{\sfdefault} %% Only if the base font of the document is to be sans serif
%\usepackage[T1]{fontenc}

\usepackage{xcolor}
\definecolor{mycolor}{RGB}{100,0,0}
\definecolor{mycolor2}{RGB}{100,0,0}
%\definecolor{mycolor2}{RGB}{0,0,0}

\usepackage{blindtext}
\usepackage{lipsum}

\parindent = 0pt
\begin{document}
\thispagestyle{empty}

\section{Shock Decomposition}
Calculate the growth rate of the compound shocks as
$$\hat{Z}^j_{n,t} = \log{Z^j_{n,t}} - \log{Z^j_{n,t - 1}} \quad \mbox{for} \quad t\geq2. $$

Calculate the time average of the growth rate for each $(n,j)$ pair and subtract it from the growth rate to get the object to be decomposed, $\tilde{Z}^j_{nt}$.

$$\tilde{Z}^j_{nt} = \hat{Z}^j_{n,t} - (T - 1)^{-1} \sum_{t=2}^T \hat{Z}^j_{n,t}$$

The decomposition to sector and country specific parts is done in a way described in Koren and Tenreyro (2007).
In general $\tilde{Z}$ is decomposed as

$$\tilde{Z}^j_{nt} = \lambda^j_t + \mu_{nt} + \epsilon^j_{nt},$$

where $\lambda, \mu$ and $\epsilon$ are estimated as
\begin{align*}
\hat{\lambda}^j_t &= N^{-1} \sum_{n=1}^N  \tilde{Z}^j_{nt} \\
\hat{\mu}_{nt} &= J^{-1} \sum_{j=1}^J \left(\tilde{Z}^j_{nt} - \hat{\lambda}^j_t\right)\\
\hat{\epsilon}^j_{nt} &= \tilde{Z}^j_{nt} - \hat{\lambda}^j_t - \hat{\mu}_{nt}.
\end{align*}
\end{document}