\chapter*{Part One: Hidden Markov Models}
\addcontentsline{toc}{chapter}{Part One}

Note: I collaborated with Vivienne Zhang and Justin Yao.

The first part of the assignment was to implement a Hidden Markov Model (HMM). HMMs model

\subsection*{Computation Problem A}

The most obvious way to represent the four example observations is a series of disconnected two state Moore state machines (see Figure \ref{moore}). There are four patterns alternating between emitted words, thus four state machines. Each state has a 100\% probability of emitting a particular word. The separate machines are only connected by the initial state distribution $\pi$. I chose to represent my machine with a uniform initial distribution, but any $\pi$ where all $\pi_i$ are non-zero probabilities that sum to $1$ works.

\begin{figure}[h]

    \centering
    \begin{tikzpicture}[->, >=stealth', auto, semithick, node distance=3cm]
        \tikzstyle{every state}=[fill=white,draw=black,thick,text=black,scale=1]
        \node[state]    (A)                     {Word $0$};
        \node[state]    (B)[right of=A]   {Word $2$};

        \path
        (A) edge[bend left]			node{$1$}	(B)
        (B) edge[bend left,above]	node{$1$}	(A);
    \end{tikzpicture}
    \caption{A two state Moore machine (or Markov chain) where the emission probability is 1 for every state. The states are named after the emitted word. }
    \label{moore}
\end{figure}

My state transition matrix $\bm{A}$, emission matrix $\bm{B}$, and initial state distribution $\pi$ are as follows:

\noindent
\begin{multicols}{3}

    \begin{equation*}
        A = \begin{bmatrix}
            0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\
            1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
            0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 \\
            0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\
            0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\
            0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\
            0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\
            0 & 0 & 0 & 0 & 1 & 0 & 0 & 0
        \end{bmatrix}
    \end{equation*}
    \break
    \noindent
    \begin{equation*}
        B = \begin{bmatrix}
            1 & 0 & 0 & 0 \\
            0 & 0 & 1 & 0 \\
            1 & 0 & 0 & 0 \\
            0 & 0 & 0 & 1 \\
            0 & 1 & 0 & 0 \\
            0 & 0 & 1 & 0 \\
            0 & 1 & 0 & 0 \\
            0 & 0 & 0 & 1
        \end{bmatrix}
    \end{equation*}
    \break
    \noindent
    \begin{equation*}
        \pi = \begin{bmatrix}
            0.25 & 0.25 & 0.25 & 0.25
        \end{bmatrix}
    \end{equation*}
\end{multicols}

\subsection*{Experimental Problem B}

It's difficult to exactly reason about what the hidden states directly encode. A naturally intuitive way to approach representing a sentence is to represent word 0 as state 0, word 1 as state 1 and so on, however we can start in any state depending on the initial distribution, and middle states may be difficult to distinguish given the vast number of words that can appear in the middle of a sentence.

Another approach would be to model parts of speech as a state, and develop probabilities of transitioning such as from an adjective to the noun it modifies. Unfortunately it's not evident in my results this kind of relationship is modelled.

\begin{center}

    \begin{tabular}{ |r|r| }

        \hline
        \multicolumn{2}{|c|}{Two state HMM word probabilities} \\
        \hline
        State 0     & State 1                                  \\
        \hline
        <unk> 0.112 & i     0.061                              \\
        .     0.085 & ,     0.039                              \\
        the   0.044 & <unk> 0.038                              \\
        a     0.034 & to    0.036                              \\
        ,     0.030 & .     0.035                              \\
        and   0.027 & the   0.029                              \\
        br    0.025 & it    0.025                              \\
        of    0.025 & and   0.020                              \\
        in    0.015 & not   0.014                              \\
        is    0.013 & but   0.014                              \\
        \hline
    \end{tabular}
\end{center}

\begin{center}

    \begin{tabular}{ |r|r|r|r| }

        \hline
        \multicolumn{4}{|c|}{Ten state HMM word probabilities} \\
        \hline
        State 0     & State 1     & State 8     & State 9      \\
        \hline
        <unk> 0.352 & . 0.118     & <unk> 0.101 & <unk> 0.102  \\
        a 0.067     & and 0.107   & in 0.057    & to 0.058     \\
        in 0.0293   & the 0.104   & the 0.054   & . 0.036      \\
        and 0.027   & , 0.103     & a 0.041     & not 0.030    \\
        or 0.024    & a 0.037     & it 0.033    & for 0.024    \\
        to 0.016    & <unk> 0.022 & at 0.024    & they 0.023   \\
        , 0.015     & of 0.020    & that 0.023  & the 0.019    \\
        of 0.008    & that 0.017  & for 0.019   & good 0.018   \\
        all 0.006   & my 0.014    & my 0.017    & them 0.017   \\
        but 0.006   & when 0.014  & and 0.016   & br 0.017     \\
        \hline
    \end{tabular}
\end{center}

\subsection*{Experimental Problem C}

\begin{center}
    \begin{tabular}{|l|}
        \hline
        we one guess flaxseed because to customer ever flat the \\ \hline
        my figured a <unk> snack . sold , bit pieces            \\ \hline
        if has to the <unk> term <unk> it wouldn't the          \\ \hline
        i do if better buy the <unk> . , and                    \\ \hline
        i used best such beat to said there was <unk>           \\ \hline
        i believe my as <unk> a . as amazon and                 \\ \hline
        it money . the make on and so only happen               \\ \hline
        this the regularly them the by ! <unk> a not            \\ \hline
        <unk> out dog ! no <unk> about time a chance            \\ \hline
        i bought not in to better reviewer it of a              \\ \hline
    \end{tabular}
\end{center}

\subsection*{Theoretical Problem D}

\chapter*{Part Two: Trees}
\addcontentsline{toc}{chapter}{Part Two}

\subsection*{Theoretical Problem A}

\subsection*{Part A}
\subsubsection*{Experimental Problem A}
\subsubsection*{Experimental Problem B}
\subsubsection*{Experimental Problem C}

\subsection*{Part B}
\subsubsection*{Experimental Problem A}
\subsubsection*{Experimental Problem B}
\subsubsection*{Experimental Problem C}
\subsubsection*{Theoretical Problem D}


\section{Variants of MST}

%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Feedback edge set}
  \begin{exampleblock}{Feedback edge set (Problem 6.5)}
    \begin{enumerate}
      \item Max-ST
      \item (minimum) feedback edge set $F$:
		\[
		  G' \triangleq G \setminus F \text{ is acyclic.}
		\]
    \end{enumerate}
  \end{exampleblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{MST with specific leaves}
  \begin{exampleblock}{MST with specific leaves (Problem 6.8)}
    \begin{itemize}
      \item $G = (V, E), U \subset V$
      \item finding an MST with $U$ as leaves
    \end{itemize}
  \end{exampleblock}

  \begin{columns}
	\column{0.55\textwidth}
	  \begin{itemize}
		\item MST $T'$ of $G' \triangleq G \setminus U$
		\item attach $\forall u \in U$ to $T'$ (lightest edge)
	  \end{itemize}
	\column{0.40\textwidth}
  	  \fignocaption{width = 0.50\textwidth}{figs/mst-specified-leaves.pdf}
  \end{columns}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{MST with specific edges}
  \begin{exampleblock}{MST with specific edges (Problem 6.9)}
    \begin{itemize}
      \item $G = (V,E), S \subset E$ (no cycle in S)
      \item finding an MST with $E$ as edges
    \end{itemize}
  \end{exampleblock}

  \begin{enumerate}
	\item Run Kruskal
	\item Computing MST on graph of CCs
  \end{enumerate}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Edge weights}
  \begin{exampleblock}{Edge weights (Problem 6.11)}
	\[
	  w(e) > 0, w'(e) = \left(w(e)\right)^2
	\]
  \end{exampleblock}

  \vspace{0.50cm}
  \begin{exampleblock}{Edge weights (Problem 6.10--7)}
	\[
	  \exists e: w(e) < 0
	\]
  \end{exampleblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Linear MST algorithms on special graphs}
  \begin{exampleblock}{Linear MST algorithms on special graphs (Problem 5.25)}
	\begin{enumerate}
	  \item $\forall e \in E: w(e) = 1$
	  \item $m = n + 10$
	  \item $\forall e \in E: w(e) = 1 \lor w(e) = 2$
	\end{enumerate}
  \end{exampleblock}

  \begin{enumerate}
	\item BFS
	\item Cycle-breaking $\times 11$
	\item BFS $\times 2$ ($\equiv$ Kruskal)
  \end{enumerate}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Updating MST}
  \begin{exampleblock}{Updating MST (Problem 6.7)}
	$T = (V, E')$ is an MST of $G = (V, E)$:
	\begin{columns}
	  \column{0.45\textwidth}
		\begin{enumerate}
		  \item $w'(e) < w(e)$
			\begin{enumerate}
			  \item $e \in E'$
			  \item $e \notin E'$
			\end{enumerate}
		\end{enumerate}
	  \column{0.45\textwidth}
		\begin{enumerate}
		  \setcounter{enumi}{1}
		  \item $w'(e) > w(e)$
			\begin{enumerate}
			  \item $e \in E'$
			  \item $e \notin E'$
			\end{enumerate}
		\end{enumerate}
	\end{columns}
  \end{exampleblock}

  \fignocaption{width = 0.15\textwidth}{figs/qrcode-update-mst.png}{\centerline{Let me know if you fully understand this answer.}}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Second-best MST}
  \begin{exampleblock}{Second-best MST (Problem 6.15)}
	Find a second-best (or, generally $k$-th) MST.
  \end{exampleblock}

  \begin{theorem}{Second-best MST}
	A second-best MST $T'$ is of the form ($T$: an MST)
	\[
	  T' = T - e + e'.
	\]
  \end{theorem}

  \centerline{By contradiction (assume distinct edge weights).}

  \[
	w(T) < w(T'') < w(T')
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%


\documentclass[acmsmall, 9pt]{article}

\usepackage[a4paper, total={6in, 10in}]{geometry}
\input{../tex/packages}
\input{../tex/layout-tweaks}
\input{../tex/macros}
\input{../tex/listings}

\setlist[itemize]{leftmargin=4mm}
\setlist[enumerate]{leftmargin=4mm}
\addbibresource{../bibliography.bib}
\begin{document}
\pagestyle{empty}

\title{Higher-Order Polymorphic Typed Lambda Calculus: Approaches to Formalising Grammars with Rich Kinds}

\maketitle
\noindent
System F$_\omega$ \cite{cambridge-lambda-calc, pierce2002types}, also known as higher-order polymorphic lambda calculus, extends System F with richer kinds $K$, namely the kind $K_1 \rightarrow K_2$ of type constructors, and adds type-level lambda-abstraction $\lambda \alpha^K. \, A$ and application $A\,B$.

\begin{align*}
  \text{Kinds} \quad K &::= * \; | \; K_1 \rightarrow K_2\\
  \text{Types} \quad  A, B &::= \iota \; | \;  A \rightarrow B \; | \; \forall \alpha^K . \, A\; | \; \alpha \; | \; \lambda \alpha^K. \, A \; | \; A \, B
\end{align*}

We are also free to extend the set of kinds $K$ with other arbitrary kind constants, which means our set of types we range over with $A, B$ includes non-value types of kind other than $*$. We must therefore consider various details when organizing the type calculus, such as what syntactic classes of type metavariables are there, what are the kinding rules for our types, which kinds of types can be polymorphic over, and what kinds of types can we use in type application. There are a number of options in how to formalize a grammar with rich kinds.

Let's consider adding row types $R$ to our grammar, which is an unordered collection of labels $\ell$. We say that rows have kind $\mathsf{Row}$ and labels have kind $\mathsf{Label}$. From rows, we can then form variants (sums) $\tyAngle{R}$ and records (products) $\tyBrace{R}$ which are value types of kind $*$.

\subsubsection{Type Indiscriminative Method}
\label{sssec:tau-method}
The most general way is to use a single type metavariable $T$ which captures all types of different kinds $K$. We note that this relies on the kinding rules to delineate which types are well-formed. For example:
\begin{align*}
  &\text{Kinds} \quad &K &::= * \; | \; K_1 \rightarrow K_2 \; | \; \mathsf{Row} \; | \; \mathsf{Label} \span\span \\
  &\text{Types} \quad &T &::= c \; | \; \tyFun{T_1}{T_2} \; | \; \forall \alpha^K . \, T\; | \; \alpha \; | \; \lambda \alpha^K. \, T \;| \; T_1 \; T_2 \;    \span\span\\
  &&&\quad \; | \; l \; | \; l ; T \; | \; \cdot \; | \;\tyBrace{T} \; | \; \tyAngle{T} \\
\end{align*}
Here we have type constants $c$ which capture value types such as $\texttt{Bool}$. Functions are written $\tyFun{T_1}{T_2}$. Universal quantification $\forall \alpha^K. \, T$ can quantify over types of any kind $K$ to produce some type $T$ of kind $K'$. Type variables $\alpha$ can be inhabited by types of any kind which $T$ ranges over. Type abstraction $\lambda \alpha^K. \, T$ has higher-kind $K_1 \rightarrow K_2$. Type application $T_1 \, T_2$ can then allow types of kinds other than $*$ and $* \rightarrow *$ to be applied to each other. Lastly, we have labels $l$, rows extended with labels $l; T$, empty rows $\cdot$, records $\tyBrace{T}$, and variants $\tyAngle{T}$.

% but for this to be well-formed under the kinding rules, $T_1$, $T_2$ and $T_1 \rightarrow T_2$ must be of kind $*$.
This grammar permits functions, type application, type abstraction, and universal quantification to work over types of any kind. We note that this grammar on its own cannot dictate what types are well-formed or ill-formed, therefore it is important to have kinding rules to express what is allowed. For example, this type syntax says that a function $l \rightarrow l$ between two types of kind $\mathsf{Label}$ is possible, however this isn't well-formed as we cannot pass or return labels as values.

\begin{figure}[H]
\flushleft \shadebox{$\Delta \vdash T : K$}
\begin{smathpar}
   \inferrule*[lab={\ruleName{constant}}]
   {
   }
   {
      \Delta \vdash c : *
   }
   \and
  \inferrule*[lab={\ruleName{function}}]
  {
     \Delta \vdash T_1 : *
     \\
     \Delta \vdash T_2 : *
  }
  {
     \Delta \vdash \tyFun{T_1}{T_2} : *
  }
  \and
  \inferrule*[lab={\ruleName{forall}}]
  {
     \Delta \concat (\alpha : K) \vdash T : *
  }
  {
     \Delta \vdash \forall \alpha^K. \, T : *
  }
  \and
  \inferrule*[lab={\ruleName{type variable}}]
  {
     \alpha : K \in \Delta
  }
  {
     \Delta \vdash \alpha : K
  }
  \and
  \inferrule*[lab={\ruleName{type constructor}}]
  {
     \Delta \concat (\alpha : K_1) \vdash T : K_2
  }
  {
     \Delta \vdash \lambda \alpha^{K_1} . \, T : \tyFun{K_1}{K_2}
  }
  \and
  \inferrule*[lab={\ruleName{type constructor application}}]
  {
     \Delta \vdash T_1 : \tyFun{K_1}{K_2}
     \\
     \Delta \vdash T_2 : K_1
  }
  {
     \Delta \vdash T_1 \, T_2 : K_2
  }
  \and
  \inferrule*[lab={\ruleName{label}}]
  {
  }
  {
     \Delta \vdash l  : \mathsf{Label}
  }
  \and
  \inferrule*[lab={\ruleName{row-extend}}]
  {
     \Delta \vdash l : \mathsf{Label}
     \\
     \Delta \vdash T : \mathsf{Row}
  }
  {
     \Delta \vdash l; T  : \mathsf{Row}
  }
  \and
  \inferrule*[lab={\ruleName{row-empty}}]
  {
  }
  {
     \Delta \vdash \cdot  : \mathsf{Row}
  }
  \and
  \inferrule*[lab={\ruleName{record}}]
  {
     \Delta \vdash T : \mathsf{Row}
  }
  {
     \Delta \vdash \tyBrace{T} : *
  }
  \and
  \inferrule*[lab={\ruleName{variant}}]
  {
     \Delta \vdash T : \mathsf{Row}
  }
  {
     \Delta \vdash \tyAngle{T} : *
  }
\end{smathpar}
\caption{Kinding Rules}
\end{figure}

\subsubsection{Type Categorization Method \cite{hillerstrom2016liberating}}
\label{sssec:type-categorization-method}
Another method is to distinguish between type metavariables which produce types of different kinds. For example:
\begin{align*}
&\text{Kinds} \quad &K &::= * \; | \; K_1 \rightarrow K_2 \; | \; \mathsf{Row} \; | \; \mathsf{Label}\\
&\text{Value Types} \quad  &A, B &::= c \; | \; A \rightarrow B \; | \; \forall \alpha^K . \, A\; | \; \alpha \; | \; \lambda \alpha^K. \, A \; | \; A \, B \; | \; \{ R \} \; | \; \tyAngle{R}\\
&\text{Row Types} \quad &R &::= l; R \; | \; \cdot \; | \; \rho \\
&\text{Label Types} \quad &l &::= l_1 \; | \; l_2 \; | \; \ldots
\end{align*}
\textbf{Value types} $A,B$ are any types which produce a kind $*$, except for type variables $\alpha$. This includes type constants $c$ and functions $A \rightarrow B$. Universally quantified types, $\forall \alpha^K . \; A$, are able to quantify over types with any kind $K$ in $\forall \alpha^K. \, A$ but the type variable $\alpha^K$ must always be used to return a type $A$ of kind $*$. We also consider type constructors $\lambda \alpha^K. \; A$ of kind $K_1 \rightarrow K_2$ as value types, which take as input a type of rich kind $K_1$ but must eventually produce a type of kind $*$ in $K_2$. Type application $A\;B$ applies a type constructor $A$ to a value type $B$. Lastly, record types $\langle R \rangle$, functions $A \rightarrow B$, universally quantified types $\forall \alpha^K . \; A$, and type application $A\;B$, which are all of kind $*$.

Note that although we can abstract over types of kind $K$ in type constructors $\lambda \alpha^K. \, A$, type constructor application $A\,B$ does not allow us to apply type constructors to types of kind other than $*$; this means types such as record constructors $\{ \, \_ \, \}$ cannot exist on their own, only records $\{ R \}$ which are already applied to a row type $R$ to yield a kind $*$.

\lbreak
\textbf{Non-value} types are types which produce kinds other than $*$. This includes row types $R$ and label types $l$. We note that row types $R$ also have their own row type variable $\rho$, which allows $\rho$ to be used in place of where $R$ can occur, and hence row types can be defined polymorphically. The fact that universal quantification is only defined in value types $A, B$ means that row types must be used in the context of a value type where $\rho$ is quantified over by $\forall \alpha^\mathsf{Row} . \; A$ where we can unify $\rho$ and $\alpha$.

\lbreak
This approach more clearly delineates types of different kinds, and restricts type application, type abstraction, and universal quantification to types which produce a kind $*$. This is generally desirable to enforce a stronger well-formed type system within the grammar itself, e.g. that values can only possibly have types $A, B$ with output kinds $*$ at the term-level, and that types which use type constructors to take a richly-kinded type as input must already be fully applied to have a kind $*$. A disadvantage of this is that type constructor application $A\,B$ does not allow us to apply type constructors to types of kind other than $*$.

\subsubsection{\textbullet \; Explicitly Kinded Type Indiscriminative Method \cite{leijen2005extensible}}
When we have a system with rich kinds, we can refine the notion of using a single metavariable $T$ by annotating it with a kind $K$, written $T^K$. This lets us capture types with kinds other than $*$ whilst being explicit about which types are well-formed.

\lbreak
For each kind $K$ we have a collection of types $T^K$; this includes type constants $c^K$, polymorphic types $\forall \alpha^K . \, T^{K'}$, type variables $\alpha^K$, and type application $T_1^{K_2 \rightarrow K} \; T_2^{K_2}$. The kind signatures for type constants $c^K$ are given explicitly in the grammar, where we use wildcards ``$\_$'' to represent arguments of type constants.

\begin{align*}
  &\text{Kinds} \quad &K &::= * \; | \; K_1 \rightarrow K_2 \; | \; \mathsf{Row} \; | \; \mathsf{Label} \\
  &\text{Types} \quad &T^K &::= c^K \; | \; \forall \alpha^{K_1} . \, T^{K_2}\; | \; \alpha^K \; | \; \lambda \alpha^{K_1}. \, T^{K_2} \; |  \; T_1^{K_2 \rightarrow K} \; T_2^{K_2} \span\span\\
  &\text{Type constants} \quad &c^K &::= \texttt{()}, \texttt{bool}, \texttt{int} &::& \; *\\
  &&&| \quad \_ \rightarrow \_ &::& \; * \rightarrow * \rightarrow * \\
  &&&| \quad l &::& \; \mathsf{Label} \\
  &&&| \quad \_ ; \_ &::& \; \mathsf{Label} \rightarrow \mathsf{Row} \rightarrow \mathsf{Row}\\
  &&&| \quad \cdot &::& \; \mathsf{Row}\\
  &&&| \quad \tyBrace{\_} &::& \; \mathsf{Row} \rightarrow \mathsf{Type}\\
  &&&| \quad \tyAngle{\_} &::& \; \mathsf{Row} \rightarrow \mathsf{Type}\\
\end{align*}

\noindent
We can then let a choice of metavariables range over types over different kinds, e.g. let $\rho \doteq T^{\mathsf{Row}}$, and similarly with type variables, e.g. let $\beta \doteq \alpha^{\mathsf{Row}}$.
\printbibliography
\end{document}


\documentclass[epsfig,10pt,fullpage]{article}

\newcommand{\LabNum}{5}
\newcommand{\CommonDocsPath}{../../common/docs}
\input{\CommonDocsPath/preamble.tex}

\begin{document}

\centerline{\huge OpenCL}
~\\
\centerline{\huge Laboratory Exercise \LabNum}
~\\
\centerline{\large Classification of Handwritten Digits using Machine Learning: CNN-Based Classifier}
~\\

In this exercise you will implement a convolutional neural network (CNN) based classifier of handwritten digits.
You will create software and hardware implementations of a neural network classifier that accepts images of handwritten digits ranging 0 to 9 and determines which digit is present in each image.

\section*{The MNIST Handwritten Digits Database}

In this exercise you will use the MNIST handwritten digits database,
which is widely used by scientists around the world to train and test handwritten digit recognition machines.
The database provides 28x28-pixel 8-bit grayscale images, each containing a single handwritten digit ranging 0 to 9. An example of
an image from the MNIST database is shown in Figure~\ref{fig:mnist_image}.

\begin{figure}[H]
   \begin{center}
       \hspace*{1.5cm}\includegraphics[scale=0.7]{figures/fig_mnist_image}
   \end{center}
   \caption{An example handwritten digit image from the MNIST database}
	\label{fig:mnist_image}
\end{figure}

The database contains a training set of 60000 images which are used to train the classifier, and a test set of 10000 images which are used to test the classifier's accuracy.
The 60000 training images have already been used to train the weights for the classifier.
Your task is to use the provided weights to implement the classifier and test its classification accuracy on the 10000 test images.

\section*{A Simple CNN-Based Classifier for MNIST Digits}

\begin{figure}[H]
   \begin{center}
       \hspace*{0.1cm}\includegraphics[scale=0.6]{figures/fig_cnn_classifier}
   \end{center}
   \caption{The layers of the CNN-based classifier}
	\label{fig:cnn_classifier}
\end{figure}

You will implement the CNN-based Classifier shown in Figure~\ref{fig:cnn_classifier}.
The classifier accepts an MNIST digit image as input which has the dimensions 28x28.
The classifier provides as output the classification result, ranging 0 to 9.
Between the input and output layers, the classifier contains three hidden layers which are described in the following sections.

\subsection*{The Convolution Layer}

\begin{figure}[H]
   \begin{center}
       \hspace*{0.1cm}\includegraphics[scale=0.6]{figures/fig_cnn_layer}
   \end{center}
   \caption{Applying a convolutional matrix to the input image with calculations shown for two of the output values}
	\label{fig:cnn_layer}
\end{figure}

The convolution layer applies 32 5x5 convolutional matrices on the input image to yield 32 feature maps.
Before convolving, the layer adds 2-pixel zero padding to all edges of the input image so that the output map retains the same dimensions as the input image.
Each convolutional matrix is applied to 5x5 patches of the padded image using a stride of 1 pixel.
Each convolutional matrix has an associated trained bias value which is added to each convolution result.
The ReLU activation function (defined as ReLU(x) = x > 0 ? x : 0) is then applied to the biased convolution result.
The ReLU-activated value is then written to the output feature map.
Figure~\ref{fig:cnn_layer} shows a convolutional matrix being applied to the input image to yield one feature map.
The figure shows the calculations involved in producing two of the values in the output feature map.
In the end, the 32 convolutional matrices yield 32 28x28 feature maps for a total output with dimensions 28x28x32.

\subsection*{The Max Pooling Layer}

\begin{figure}[H]
   \begin{center}
       \hspace*{0.1cm}\includegraphics[scale=0.7]{figures/fig_pooling_layer}
   \end{center}
   \caption{Applying the 2x2 max pooling on a 28x28 input with calculations shown for two of the output values}
	\label{fig:pooling_layer}
\end{figure}

The max pooling layer downsamples the feature maps produced by the CNN layer by retaining only the maximum value in each 2x2 block
of values. Downsampling in this layer reduces the number of features in flight  and thus the number of calculations required in the subsequent layers.
It also reduces the amount of memory required to store subsequent  features.
By applying 2x2 max pooling with a stride of 2 pixels on a 28x28x32 input, the layer produces an output with dimensions 14x14x32.

\subsection*{The Fully Connected Layer}

\begin{figure}[H]
   \begin{center}
       \hspace*{0.1cm}\includegraphics[scale=1.0]{figures/fig_fc_layer}
   \end{center}
   \caption{The fully connected layer with calculations shown for two of the output neurons}
	\label{fig:fc_layer}
\end{figure}

The fully connected layer yields 10 output neurons, each of which is calculated as the weighted sum of the 14x14x32 features output from the
previous layer. This means that in total there are 14x14x32x10 = 62720 weights.
In essence, this fully-connected layer is a linear classifier whose input are the features outputted by the max pooling layer,
and output are the likelihoods of the input image belonging to each of the 10 digit classes.
The digit with the highest output value is chosen as the classification result.

\section*{Part I}
\noindent
Implement a program in the C++ language that performs CNN-based classification of MNIST handwritten digits as described in the previous section.
Use the trained weights provided in the \textit{/design\_files/weights\_fxp/} directory.
Test your program by classifying the 10000 MNIST test images stored in the file \textit{/design\_files/t10k-images-idx3-ubyte}.
Compare the results of your classifier to the correct labels stored in the file \textit{/design\_files/t10k-labels-idx1-ubyte} to calculate
your classifier's accuracy as: \[Accuracy = (\#\ of\ correct\ results )/ 10000 * 100\% \]
Compile and execute your program on a DE-Series Linux platform.
Upon completion, your program should output the classification accuracy and the runtime in milliseconds.

\subsubsection*{Parsing the MNIST Database Files}
The MNIST database is provided in .idx files. The file formats of the MNIST images and labels files are shown in Figure~\ref{fig:mnist_file_format}.
Your program must open and parse these files to extract the image pixels and labels. Each pixel is 8 bits and represents a grayscale value.
Each label is also 8 bits, and its value can range from 0 to 9 to indicate the correct classification of the corresponding image.
The 32-bit integers stored in these files are stored in most-significant byte (MSB)-first format, which
must first be converted to least significant byte (LSB)-first format before being used on the ARM processors of the DE-series boards.

\lstset{}
\begin{figure}[H]
\begin{center}
\begin{minipage}[t]{12.7 cm}
\begin{lstlisting}
TEST SET IMAGE FILE (t10k-images-idx3-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000803(2051) magic number
0004     32 bit integer  10000            number of images
0008     32 bit integer  28               number of rows
0012     32 bit integer  28               number of columns
0016     unsigned byte   ??               pixel
0017     unsigned byte   ??               pixel
........
xxxx     unsigned byte   ??               pixel

TEST SET LABEL FILE (t10k-labels-idx1-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  10000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label
The labels values are 0 to 9.
\end{lstlisting}
\end{minipage}
\end{center}
\vspace{-.23in}\caption{File formats of the MNIST files. Source: http://yann.lecun.com/exdb/mnist/}
\label{fig:mnist_file_format}
\end{figure}

\subsubsection*{Parsing the Weights Files}

Trained neural network weights are provided in the directory \textit{/design\_files/weights\_fxp/}. The weights are in 16-bit (8.8) fixed-point format.
In the directory are 10 files named
\textit{fc\_weights\_0}, \textit{fc\_weights\_1}, ... , \textit{fc\_weights\_9} that provide the weights for the 10 neurons of the fully-connected layer.
%Each of these files provides the weights for one of the digits from 0 to 9 as suffixed to the file names.
The digit for which the file provides the weights is suffixed to the file name.
The weights in these files are arranged row-first, starting with weight corresponding to coordinate 0,0 (top-left) of feature map 0 (the output map of convolutional matrix 0).
Finally there is one weights file named \textit{cnn\_weights} which provide the weights for the 32 5x5 convolutional matrices of the CNN layer.
The \textit{cnn\_weights} file contains 32 sets of 26 weights, where the first 25 of each set of 26 correspond to the 5x5 matrix weights, and the 26th
is the bias. The file stores the sets starting from convolutional matrix 0. The 5x5 matrix weights are arranged row-first starting from coordinate 0,0 (top-left) of the 5x5 matrix.
The files can be parsed using the code shown in Figure~\ref{fig:parse_weights_code}.

\lstset{language=C++}
\begin{figure}[H]
\begin{center}
\begin{minipage}[t]{16 cm}
\begin{lstlisting}
// weights must be an array of sufficient size (>= num_weights)
bool read_weights_file(char *filename, short *weights, int num_weights) {
	FILE *f = fopen(filename, "rb");
	if (f == NULL){
		printf("ERROR: could not open %s\n",filename);
		return false;
	}
	int read_elements = fread(weights, sizeof(short), num_weights, f);
	fclose(f);

	if (read_elements != num_weights){
		printf("ERROR: read incorrect number of weights from %s\n", filename);
		return false;
	}
	return true;
}
\end{lstlisting}
\end{minipage}
\end{center}
\vspace{-0.33in}\caption{Function that parses a weights file}
\label{fig:parse_weights_code}
\end{figure}

\section*{Part II}

Create an OpenCL implementation of the CNN-based classifier you created in Part I. You may wish to reuse code from your linear classifier in Exercise 4 to implement the fully-connected layer. Consider whether you will implement a single monolithic OpenCL kernel or divide the work into multiple smaller kernels. If implementing multiple kernels, decide whether to use global memory or pipes to share the data. Carefully consider how you will utilize the limited local memory resources to maximize performance.
Finally, output the accuracy and runtime of your implementation and compare them to your results in Part I.

\section*{Part III (Bonus)}

Devise a neural network topology for a classifier that improves on the accuracy you achieved in Part II.
Create an OpenCL implementation of your classifier. As you will be limited by the amount of resources available to you,
carefully consider the tradeoffs between accuracy, runtime, and FPGA resource usage.
The parameters that you can adjust include:

\begin{itemize}
\item Number of CNN layers, convolutional matrix size(s), number of matrices per layer, stride size
\item Number of pooling layers, size, and stride
\item Number of FC layers, and number of intermediate neurons
\item Precision (8, 16, 32 bit fixed-point, decimal position, mixed precision)
\end{itemize}

To train weights you can use a machine learning framework of your choice. Some common frameworks include Tensorflow and Caffe.

\input{\CommonDocsPath/copyright.tex}
\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2003-2018 by The University of Queensland
% http://www.uq.edu.au
%
% Primary Business: Queensland, Australia
% Licensed under the Apache License, version 2.0
% http://www.apache.org/licenses/LICENSE-2.0
%
% Development until 2012 by Earth Systems Science Computational Center (ESSCC)
% Development 2012-2013 by School of Earth Sciences
% Development from 2014 by Centre for Geoscience Computing (GeoComp)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Newtonian Potential}

In this chapter the gravitational potential field is developed for \esc.
Gravitational fields are present in many modelling scenarios, including
geophysical investigations, planetary motion and attraction and micro-particle
interactions. Gravitational fields also present an opportunity to demonstrate
the saving and visualisation of vector data for \mayavi, and the construction of
variable sized meshes.

The gravitational potential $U$ at a point $P$ due to a region with a mass
distribution of density $\rho(P)$, is given by Poisson's equation
\citep{Blakely1995}
\begin{equation} \label{eqn:poisson}
\nabla^2 U(P) = -4\pi\gamma\rho(P)
\end{equation}
where $\gamma$ is the gravitational constant.
Consider now the \esc general form,
\autoref{eqn:poisson} requires only two coefficients,
$A$ and $Y$, thus the relevant terms of the general form are reduced to;
\begin{equation}
-\left(A_{jl} u_{,l} \right)_{,j} = Y
\end{equation}
One recognises that the LHS side is equivalent to
\begin{equation} \label{eqn:ex10a}
-\nabla A \nabla u
\end{equation}
and when $A=\delta_{jl}$, \autoref{eqn:ex10a} is equivalent to
\begin{equation*}
-\nabla^2 u
\end{equation*}
Thus Poisson's \autoref{eqn:poisson} satisfies the general form when
\begin{equation}
A=\delta_{jl} \text{ and } Y= 4\pi\gamma\rho
\end{equation}
The boundary condition is the last parameter that requires consideration. The
potential $U$ is related to the mass of a sphere by
\begin{equation}
U(P)=-\gamma \frac{m}{r^2}
\end{equation} where $m$ is the mass of the sphere and $r$ is the distance from
the center of the mass to the observation point $P$. Plainly, the magnitude
of the potential is governed by an inverse-square distance law. Thus, in the
limit as $r$ increases;
\begin{equation}
\lim_{r\to\infty} U(P) = 0
\end{equation}
Provided that the domain being solved is large enough, and the source mass is
contained within a central region of the domain, the potential will decay to
zero. This is a dirichlet boundary condition where $U=0$.

This boundary condition can be satisfied when there is some mass suspended in a
free-space. For geophysical models where the mass of interest may be an anomaly
inside a host rock, the anomaly can be isolated by subtracting the density of the
host rock from the model. This creates a fictitious free space model that will
satisfy the analytic boundary conditions. The result is that
$Y=4\pi\gamma\Delta\rho$, where $\Delta\rho=\rho-\rho_0$ and $\rho_0$ is the
baseline or host density. This of course means that the true gravity response is
not being modelled, but the response due to an anomalous mass with a
density contrast $\Delta\rho$.

For this example we have set all of the boundaries to zero but only one boundary
point needs to be set for the problem to be solvable. The normal flux condition
is also zero by default. Note, that for a more realistic and complicated models
it may be necessary to give careful consideration to the boundary conditions of the model,
which can have an influence upon the solution.

Setting the boundary condition is relatively simple using the \verb!q! and
\verb!r! variables of the general form. First \verb!q! is defined as a masking
function on the boundary using
\begin{python}
q=whereZero(x[1]-my)+whereZero(x[1])+whereZero(x[0])+whereZero(x[0]-mx)
mypde.setValue(q=q,r=0)
\end{python}
This identifies the points on the boundary and \verb!r! is simply
ser to \verb!r=0.0!. This is a Dirichlet boundary condition.

\clearpage
\section{Gravity Pole}
\sslist{example10a.py}
A gravity pole is used in this example to demonstrate the vector characteristics
of gravity, and also to demonstrate how this information can be exported for
visualisation to \mayavi or an equivalent using the VTK data format.

The solution script for this section is very simple. First the domain is
constructed, then the parameters of the model are set, and finally the steady
state solution is found. There are quite a few values that can now be derived
from the solution and saved to file for visualisation.

The potential $U$ is related to the gravitational response $\vec{g}$ via
\begin{equation}
\vec{g} = \nabla U
\end{equation}
$\vec{g}$ is a vector and thus, has a a vertical component $g_{z}$ where
\begin{equation}
g_{z}=\vec{g}\cdot\hat{z}
\end{equation}
Finally, there is the magnitude of the vertical component $g$ of
$g_{z}$
\begin{equation}
g=|g_{z}|
\end{equation}
These values are derived from the \esc solution \verb!sol! to the potential $U$
using the following commands
\begin{python}
g_field=grad(sol) #The gravitational acceleration g.
g_fieldz=g_field*[0,1] #The vertical component of the g field.
gz=length(g_fieldz) #The magnitude of the vertical component.
\end{python}
This data can now be simply exported to a VTK file via
\begin{python}
# Save the output to file.
saveVTK(os.path.join(save_path,"ex10a.vtu"),\
        grav_pot=sol,g_field=g_field,g_fieldz=g_fieldz,gz=gz)
\end{python}

It is quite simple to visualise the data from the gravity solution in \mayavi.
With \mayavi open go to File, Load data, Open file \ldots as in
\autoref{fig:mayavi2openfile} and select the saved data file. The data will
have then been loaded and is ready for visualisation. Notice that under the data
object in the \mayavi navigation tree the 4 values saved to the VTK file are
available (\autoref{fig:mayavi2data}). There are two vector values,
\verb|gfield| and \verb|gfieldz|. Note that to plot both of these on the same
chart requires that the data object be imported twice.

The point scalar data \verb|grav_pot| is the gravitational potential and it is
easily plotted using a surface module. To visualise the cell data a filter is
required that converts to point data. This is done by right clicking on the data
object in the explorer tree and selecting the cell to point filter as in
\autoref{fig:mayavi2cell2point}.

The settings can then be modified to suit personal taste. An example of the
potential and gravitational field vectors is illustrated in
\autoref{fig:ex10pot}.

\begin{figure}[ht]
\centering
\includegraphics[width=0.75\textwidth]{figures/mayavi2_openfile.png}
\caption{Open a file in \mayavi}
\label{fig:mayavi2openfile}
\end{figure}

\begin{figure}[ht]
\centering
\includegraphics[width=0.75\textwidth]{figures/mayavi2_data.png}
\caption{The 4 types of data in the imported VTK file.}
\label{fig:mayavi2data}
\end{figure}

\begin{figure}[ht]
\centering
\includegraphics[width=0.75\textwidth]{figures/mayavi2_cell2point.png}
\caption{Converting cell data to point data.}
\label{fig:mayavi2cell2point}
\end{figure}

\begin{figure}[ht]
\centering
\includegraphics[width=0.75\textwidth]{figures/ex10apot.png}
\caption{Newtonian potential with $\vec{g}$ field directionality. The magnitude
of the field is reflected in the size of the vector arrows.}
\label{fig:ex10pot}
\end{figure}
\clearpage

\section{Gravity Well}
\sslist{example10b.py}
Let us now investigate the effect of gravity in three dimensions. Consider a
volume which contains a spherical mass anomaly and a gravitational potential
which decays to zero at the base of the model.

The script used to solve this model is very similar to that used for the gravity
pole in the previous section, but with an extra spatial dimension. As for all
the 3D problems examined in this cookbook, the extra dimension is easily
integrated into the \esc solution script.

The domain is redefined from a rectangle to a Brick;
\begin{python}
domain = Brick(l0=mx,l1=my,n0=ndx, n1=ndy,l2=mz,n2=ndz)
\end{python}
the source is relocated along $z$;
\begin{python}
x=x-[mx/2,my/2,mz/2]
\end{python}
and, the boundary conditions are updated.
\begin{python}
q=whereZero(x[2]-inf(x[2]))
\end{python}
No modifications to the PDE solution section are required. Thus the migration
from a 2D to a 3D problem is almost trivial.

\autoref{fig:ex10bpot} illustrates the strength of a PDE solution. Three
different visualisation types help define and illuminate properties of the data.
The cut surfaces of the potential are similar to a 2D section for a given x or y
and z. The iso-surfaces illuminate the 3D shape of the gravity field, as well as
its strength which is illustrated by the colour. Finally, the streamlines
highlight the directional flow of the gravity field in this example.

The boundary conditions were discussed previously, but not thoroughly
investigated. It was stated, that in the limit as the boundary becomes more
remote from the source, the potential will reduce to zero.
\autoref{fig:ex10bpot2} is the solution to the same gravity problem
but with a slightly larger domain. It is obvious in this case that
the previous domain size was too small to accurately represent the
solution. The profiles in \autoref{fig:ex10p} further demonstrates the affect
the domain size has upon the solution. As the domain size increases, the
profiles begin to converge. This convergence is a good indicator of an
appropriately dimensioned model for the problem, and and sampling location.

\begin{figure}[htp]
\centering
\includegraphics[width=0.75\textwidth]{figures/ex10bpot.png}
\caption{Gravity well with iso surfaces and streamlines of the vector
gravitational potential \textemdash small model.}
\label{fig:ex10bpot}
\end{figure}

\begin{figure}[htp]
\centering
\includegraphics[width=0.75\textwidth]{figures/ex10bpot2.png}
\caption{Gravity well with iso surfaces and streamlines of the vector
gravitational potential \textemdash large model.}
\label{fig:ex10bpot2}
\end{figure}

\begin{figure}[htp]
\centering
\includegraphics[width=0.85\textwidth]{figures/ex10p_boundeff.pdf}
\caption{Profile of the gravitational profile along x where $y=0,z=250$ for
various sized domains.}
\label{fig:ex10p}
\end{figure}
\clearpage

% \section{Gravity Surface over a fault model.}
% \sslist{example10c.py,example10m.py}
% This model demonstrates the gravity result for a more complicated domain which
% contains a fault. Additional information will be added when geophysical boundary
% conditions for a gravity scenario have been established.
%
% \begin{figure}[htp]
% \centering
% \subfigure[The geometry of the fault model in example10c.py.]
% 	{\label{fig:ex10cgeo}
% 	 \includegraphics[width=0.8\textwidth]{figures/ex10potfaultgeo.png}} \\
% \subfigure[The fault of interest from the fault model in
% 	example10c.py.]
% 	{\label{fig:ex10cmsh}
% 	 \includegraphics[width=0.8\textwidth]{figures/ex10potfaultmsh.png}}
% \end{figure}
%
% \begin{figure}[htp]
% \centering
% 	\includegraphics[width=0.8\textwidth]{figures/ex10cpot.png}
% 	\caption{Gravitational potential of the fault model with primary layers and
% 	faults identified as isosurfaces.}
% 	\label{fig:ex10cpot}
% \end{figure}
% \clearpage

\section{Variable mesh-element sizes}
\sslist{example10m.py}
We saw in a previous section that the domain needed to be sufficiently
large for the boundary conditions to be satisfied. This can be troublesome when
trying to solve problems that require a dense mesh, either for solution
resolution of stability reasons. The computational cost of solving a large
number of elements can be prohibitive.S

With the help of Gmsh, it is possible to create a mesh for \esc, which has a
variable element size. Such an approach can significantly reduce the number of
elements that need to be solved, and a large domain can be created that has
sufficient resolution close to the source and extends to distances large enough
that the infinity clause is satisfied.

To create a variable size mesh, multiple meshing domains are required. The
domains must share points, boundaries and surfaces so that they are joined; and
no sub-domains are allowed to overlap. Whilst this initially seems complicated,
it is quite simple to implement.

This example creates a mesh which contains a high resolution sub-domain at its
center. We begin by defining two curve loops which describe the large or
big sub-domain and the smaller sub-domain which is to contain the high
resolution portion of the mesh.
\begin{python}
################################################BIG DOMAIN
#ESTABLISHING PARAMETERS
width=10000.    #width of model
depth=10000.    #depth of model
bele_size=500.  #big element size
#DOMAIN CONSTRUCTION
p0=Point(0.0,    0.0)
p1=Point(width, 0.0)
p2=Point(width, depth)
p3=Point(0.0,   depth)
# Join corners in anti-clockwise manner.
l01=Line(p0, p1)
l12=Line(p1, p2)
l23=Line(p2, p3)
l30=Line(p3, p0)

cbig=CurveLoop(l01,l12,l23,l30)

################################################SMALL DOMAIN
#ESTABLISHING PARAMETERS
xwidth=2000.0   #x width of model
zdepth=2000.0   #y width of model
sele_size=10.   #small element size
#TRANSFORM
xshift=width/2-xwidth/2
zshift=depth/2-zdepth/2
#DOMAIN CONSTRUCTION
p4=Point(xshift, zshift)
p5=Point(xwidth+xshift, zshift)
p6=Point(xwidth+xshift, zdepth+zshift)
p7=Point(xshift,    zdepth+zshift)
# Join corners in anti-clockwise manner.
l45=Line(p4, p5)
l56=Line(p5, p6)
l67=Line(p6, p7)
l74=Line(p7, p4)

csmall=CurveLoop(l45,l56,l67,l74)
\end{python}
The small sub-domain curve can then be used to create a surface.
\begin{python}
ssmall=PlaneSurface(csmall)
\end{python}
However, so that the two domains do not overlap, when the big sub-domain
curveloop is used to create a surface it must contain a hole. The hole is
defined by the small sub-domain curveloop.
\begin{python}
sbig=PlaneSurface(cbig,holes=[csmall])
\end{python}
The two sub-domains now have a common geometry and no over-laping features as
per \autoref{fig:ex10mgeo}. Notice, that both domains have a normal in the
same direction.

The next step, is exporting each sub-domain individually, with an appropriate
element size. This is carried out using the \pycad Design command.
\begin{python}
# Design the geometry for the big mesh.
d1=Design(dim=2, element_size=bele_size, order=1)
d1.addItems(sbig)
d1.addItems(PropertySet(l01,l12,l23,l30))
d1.setScriptFileName(os.path.join(save_path,"example10m_big.geo"))
MakeDomain(d1)

# Design the geometry for the small mesh.
d2=Design(dim=2, element_size=sele_size, order=1)
d2.addItems(ssmall)
d2.setScriptFileName(os.path.join(save_path,"example10m_small.geo"))
MakeDomain(d2)
\end{python}
Finally, a system call to Gmsh is required to merge and then appropriately
mesh the two sub-domains together.
\begin{python}
# Join the two meshes using Gmsh and then apply a 2D meshing algorithm.
# The small mesh must come before the big mesh in the merging call!!@!!@!
sp.call("gmsh -2 "+
        os.path.join(save_path,"example10m_small.geo")+" "+
        os.path.join(save_path,"example10m_big.geo")+" -o "+
        os.path.join(save_path,"example10m.msh"),shell=True)
\end{python}
The ``-2'' option is responsible for the 2D meshing, and the ``-o'' option
provides the output path. The resulting mesh is depicted in
\autoref{fig:ex10mmsh}

To use the Gmsh ``*.msh'' file in the solution script, the mesh reading function
``ReadGmsh'' is required. It can be imported via;
\begin{python}
from esys.finley import ReadGmsh
\end{python}
To read in the file the function is called
\begin{python}
domain=ReadGmsh(os.path.join(mesh_path,'example10m.msh'),2) # create the domain
\end{python}
where the integer argument is the number of domain dimensions.
%
 \begin{figure}[ht]
\centering
\includegraphics[width=0.8\textwidth]{figures/ex10m_geo.png}
\caption{Geometry of two surfaces for a single domain.}
\label{fig:ex10mgeo}
\end{figure}

\begin{figure}[ht]
\centering
\includegraphics[width=0.8\textwidth]{figures/ex10m_msh.png}
\caption{Mesh of merged surfaces, showing variable element size. Elements
range from 10m in the centroid to 500m at the boundary.}
\label{fig:ex10mmsh}
\end{figure}
\clearpage

\section{Unbounded problems}
With a variable element-size, it is now possible to solve the potential problem
over a very large mesh. To test the accuracy of the solution, we will compare
the \esc result with the analytic solution for the vertical gravitational
acceleration $g_z$ of an infinite horizontal cylinder.

For a horizontal cylinder with a circular cross-section with infinite strike,
the analytic solution is give by
\begin{equation}
g_z = 2\gamma\pi R^2 \Delta\rho \frac{z}{(x^2+z^2)}
\end{equation}
where $\gamma$ is the gravitational constant (as defined previously), $R$ is the
radius of the cylinder, $\Delta\rho$ is the density contrast and $x$ and $z$ are
geometric factors, relating the observation point to the center of the source
via the horizontal and vertical displacements respectively.

The accuracy of the solution was tested using a square domain. For each test the
dimensions of the domain were modified, being set to 5, 10, 20 and 40 Km. The
results are compared with the analytic solution and are depicted in
\autoref{fig:ex10q boundeff} and \autoref{fig:ex10q boundeff zoom}. Clearly, as
the domain size increases, the \esc approximation becomes more accurate at
greater distances from the source. The same is true at the anomaly peak, where
the variation around the source diminishes with an increasing domain size.

\begin{figure}[ht]
\centering
\includegraphics[width=0.8\textwidth]{figures/ex10q_boundeff.pdf}
\caption{Solution profile 1000.0 meters from the source as the domain size
increases.}
\label{fig:ex10q boundeff}
\end{figure}

\begin{figure}[ht]
\centering
\includegraphics[width=0.8\textwidth]{figures/ex10q_boundeff_zoom.pdf}
\caption{Magnification of \autoref{fig:ex10q boundeff}.}
\label{fig:ex10q boundeff zoom}
\end{figure}

There is a methodology which can help establish an appropriate zero mass region
to a domain.
\clearpage


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% PROBLEM 1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Problem 1}

A reactor is operating for a long time at some known power density $P_0$.
Then, it instantaneously changes power to some power density $P_1$.
One fission product of interest is $^{135}$Xe, though it has a neglible yield from the initial fission reaction.
$^{135}$Xe precursors $^{135}$Te and $^{135}$I are produced with a combined yield, $y$, of approximately 6\%, before decaying via $\beta^{-}$ decay to $^{135}$I and $^{135}$Xe respectively.
Find the number density of $^{135}$Xe as a function of time after the power change.
(For convenience, let $Q_f$ be the energy produced in a fission reaction, and let $\phi$ be the flux in the reactor. Your answer may be left in terms of these variables.)

\begin{table}[htbp]
	\centering
	\begin{tabular}{|c|c|c|}
			\hline
			Nucleus		&	Half-life 	& Thermal $\sigma_{\text{a}}$ \\
			\hline
			$^{135}$Te	&  $19.0$ s 	& $\sim 0$\\
			$^{135}$I	&  $6.6$ hr 	& $\sim 0$\\
			$^{135}$Xe	&  $9.2$ hr 	& $2.6 \times 10^6$ barns \\
			\hline
	\end{tabular}
	\label{tab:design-specs}
\end{table}


\documentclass[t,usenames,dvipsnames]{beamer}
\usetheme{Copenhagen}
\setbeamertemplate{headline}{} % remove toc from headers
\beamertemplatenavigationsymbolsempty

\usepackage{amsmath, tkz-euclide, tikz, xcolor, pgfplots, array}
\usetkzobj{all}
\pgfplotsset{compat = 1.16}
\usetikzlibrary{arrows.meta, calc, decorations.pathreplacing}
\pgfplotsset{every axis/.append style = {axis lines = middle, axis line style = {<->}}}
\pgfplotsset{every tick label/.append style={font=\tiny}}
\everymath{\displaystyle}

\title{Sum to Product and Product to Sum Identities}
\author{}
\date{}

\AtBeginSection[]
{
  \begin{frame}
    \frametitle{Objectives}
    \tableofcontents[currentsection]
  \end{frame}
}

\begin{document}

\begin{frame}
    \maketitle
\end{frame}

\section{Write a product of trig functions as a sum of trig functions.}

\begin{frame}{Product to Sum Identities}
    \begin{align*}
          & \cos(A + B) &= \cos A \cos B - \sin A \sin B \\
        + & \cos(A - B) &= \cos A \cos B + \sin A \sin B \\ \hline
    \end{align*}
    \begin{align*}
          \cos(A+B)+\cos(A-B) &= 2\cos A \cos B  \\[8pt]
          \frac{1}{2}\left(\cos(A+B) + \cos(A-B)\right) &= \cos A \cos B
    \end{align*}
\end{frame}

\begin{frame}{Other Product to Sum Identities}
    \begin{align*}
        \sin A \sin B &= \frac{1}{2}\left(\cos(A-B) - \cos(A+B)\right) \\[18pt]
        \sin A \cos B &= \frac{1}{2}\left(\sin(A-B) + \sin(A+B)\right) \\
    \end{align*}
\end{frame}

\begin{frame}{Example 1}
Write $\cos(2\theta)\cos(6\theta)$ as a sum
\begin{align*}
    \onslide<2->{\cos(2\theta)\cos(6\theta) &= \frac{1}{2}\left(\cos(2\theta+6\theta) + \cos(2\theta-6\theta)\right)}    \\[15pt]
    \onslide<3->{&=\frac{1}{2}\left(\cos(8\theta) + \cos(-4\theta)\right)}  \\[15pt]
    \onslide<4->{&= \frac{1}{2}\left(\cos(8\theta) + \cos(4\theta)\right)}
\end{align*}
\end{frame}

\section{Write a sum of trig functions as a product of trig functions.}

\begin{frame}{Sum to Product Identities}
    For $\cos(A+B) + \cos(A-B) = 2\cos A \cos B$,  if we let $x = A + B$ and $y = A - B$, we get
    \[
    x + y = 2A \quad \text{and} \quad x - y = 2B
    \]

    from which
    \[
    A = \frac{x+y}{2} \quad \text{and} \quad B = \frac{x-y}{2}
    \]
    \begin{align*}
        \cos x + \cos y &= 2\cos\left(\frac{x+y}{2}\right)\cos\left(\frac{x-y}{2}\right)
    \end{align*}
\end{frame}

\begin{frame}{Other Sum to Product Identities}
    \begin{align*}
        \cos x - \cos y &= -2\sin\left(\frac{x+y}{2}\right) \sin\left(\frac{x-y}{2}\right) \\[15pt]
        \sin x \pm \sin y &= 2 \sin\left(\frac{x \pm y}{2}\right)\cos\left(\frac{x \mp y}{2}\right)
    \end{align*}
\end{frame}

\begin{frame}{Example 2}
Write $\sin\theta - \sin(3\theta)$ as a product.
\begin{align*}
    \onslide<2->{\sin\theta - \sin(3\theta) &= 2\sin\left(\frac{\theta - 3\theta}{2}\right)\cos\left(\frac{\theta + 3\theta}{2}\right)} \\[10pt]
    \onslide<3->{&= 2\sin\left(\frac{-2\theta}{2}\right)\cos\left(\frac{4\theta}{2}\right)} \\[10pt]
    \onslide<4->{&= 2\sin(-\theta)\cos(2\theta)} \\[10pt]
    \onslide<5->{&= -2\sin\theta\cos(2\theta)}
\end{align*}
\end{frame}

\end{document}


\section{Introduction}
%
Equalizers (EQs) \cite{Valimaki2016} are fundamental tools for audio signal
processing, more precisely for filtering, where spectra of audio signals
are intentionally reshaped.
%
For traditional shelving filters
\cite{Kimball1938, Zolzer2011, Reiss2015, Jot2015}, realized as infinite
impulse response (IIR) filters, cutoff frequency, shelving level and $Q$-factor
are design specifications and determine the placement of $M$ poles and $M$ zeros.
%
The steepness of the slope is typically linked to level.
%
For practical levels, a somewhat steeper or flatter slope can be
realized at the cost of frequency responses that are
underdamped (large $Q$) or overdamped (small $Q$), respectively.
%
For the mentioned filter designs of order $M$, the transition slope and bandwidth
are not independently adjustable and for large levels the slope
is always restricted to $\pm 6 \cdot M$ dB/oct, cf. \cite{Holters2006a}.
%
However, the slope and bandwidth of the transition band can be considered
as design specifications, cf. \cite{McGrath2004}.
%
Fractional-order system theory allows to design filters with arbitrary
slope $\nu$~dB/oct ($\nu\in\mathbb{R}$) \cite{Helie2013, Acharya2014}.
%
The intended accuracy of the slope determines the required IIR filter order $M$.
%
A suitable filter design determines their specific alignment.
%
By that shelving filters could be enhanced for separate adjustment of
shelving level and arbitrary slope or lower/upper cutoff frequencies
(determining the bandwidth of the slope).
%
A finite impulse response (FIR) filter design is straightforward.
%
However, standard FIR designs exhibit a linear frequency scale.
%
Thus, control of low frequencies requires a rather long impulse response and
comparably high computational complexity.
%
This might be lowered by a cascade of IIR filters with
inherent minimum phase.
%
The slope of a fractional-order filter and particular its minimum phase
characteristic is a desired property in certain audio applications
\cite{McGrath2004,Spors2010a}.
%
%
%
\NewL In \cite{Schultz2016} a cascade design of second-order shelving
filters with $\pm 3$~dB/oct slope was introduced, to
approximate the frequency response
of the so called half-differentiator/-integrator ($\pm 45$ deg minimum phase)
within certain bandwidth.
%
H{\'e}lie~\cite{Helie2013} recently introduced
a fractional-order lowpass filter design from parallel connection of poles.
%
The treatise gives the theoretical link to the so called diffusive
representation, which represents the transfer function of
a fractional-order lowpass as an integral over infinite,
infinitesimally shifted poles.
%
Approximating this integral by a finite sum leads to a parallel filter
structure.
%
This can be converted to a cascade \cite{Liski2019IEEE}.
%
The usage of a finite number of poles introduces ripples to the ideal
fractional-order slope.
%
In the present contribution, a shelving filter cascade upon this
fundamental concept is designed as generalization of \cite{Schultz2016}
towards a shelving filter with arbitrary fractional-order
slope and adjustable bandwidth.
%
The approach thus explicitly controls the transition band
and could not be traced in literature, but naturally there
are strong links to other filter design concepts.
%
These will be briefly outlined in the following.
%
%
%
\NewL Research on (digital) fractional-order filter design is particularly
pursued in control engineering, cf.
\cite{Chen2003,Barbosa2005,Barbosa2006,Tseng2017}.
%
These IIR filter designs approximate a fractional-order system and pay
special attention to the optimum slope and phase in a specified bandwidth.
%
This often comes at the cost of disregarded shelving bands, which might exhibit
arbitrary magnitude, phase and/or ripple.
%
The low-order -3~dB/oct fractional-order slope lowpass IIR filters
collected in \cite{Whittle2011} are popular in audio engineering,
e.g. to create pink from white noise.
%
%
%
\NewL Furthermore in audio, a cascade of parametric EQs (PEQs) or
shelving filters is often pursued, typically to obtain a
graphical EQ (GEQ) \cite{Holters2006b,Valimaki2017} or a
parametric multiband EQ \cite{McGrath2004,Eastty2008,Eastty2015,Lorente2017}.
%
%
In \cite{McGrath2004} adjustable fractional-order slope and bandwidth
was introduced to shelving filters for loudspeaker
controllers.
%
Using a raised-cosine slope---approximated by a PEQ cascade---%
enhanced the capabilities to equalize sound reinforcement systems.
%
Higher-order shelving slopes without excessive under-/overdamping can be
achieved by higher-order shelving filters \cite{Holters2006a}.
%
This does not allow to design the slope explicitly, but it is linked
to the level and the filter order, as stated before.
%
An example for this characteristic is depicted in
\fig{fig:holters2006-higher-order-shelving} in the present paper.
%
The design in \cite{Holters2006a} allows for
band-shelving filters with steep transition between the pass band and the
shelving band.
%
In \cite{Holters2006b} this was used to design a GEQ
with a cascade of sharp band-shelving filters.
%
With \cite{Holters2006a, Holters2006b} the multiband EQ
described in \cite{Lorente2017} can be straightforwardly designed:
%
A customized cascade is used
to realize multiple band-shelving filters, where asymmetric slopes are linked
to level and filter order.
%
Similar to \cite{McGrath2004} this design enhances capabilities for
sound reinforcement system equalization.
%
A further design \cite{Eastty2008, Eastty2015} uses
cascaded fractional-order shelving filters to obtain a shelving filter
with adjustable lower and upper cutoff frequency and corresponding levels.
%
By that a transition bandwidth is explicitly defined.
%
The slope is optimized such that an intended sharpness in the transition band
is suitable for a target application.
%
For their fractional-order shelving prototypes, the works \cite{Eastty2008, Eastty2015}
use the fact, that
a certain integer order shelving filter with a certain chosen level
yields a specific fractional-order slope in a limited bandwidth.
%
%
%
\NewL In comparison to the mentioned multiband EQs, our paper aims at shelving
filters with adjustable fractional-order slopes over a larger bandwidth,
for which the resulting minimum phase can be useful in certain applications.
%
The paper is thus organized as follows:
%
In Sec.~\ref{sec:filter_design} the filter design method and parametrization
is discussed.
%
In Sec.~\ref{sec:analog_filter_properties} the tuning and limitation properties
of the analog filter type are explained in detail.
%
In Sec.~\ref{sec:discussion} digital filter design, comparison to a digital GEQ design
and potential applications are
discussed.
%
Sec.~\ref{sec:summary} briefly summarizes the outcome.
%
All source
code\footnote{\url{https://github.com/spatialaudio/aes148-shelving-filter}}
to this paper is available.
%, supporting open science paradigms.

\section{Filter Design Method}
\label{sec:filter_design}
The analog filter design found e.g. in \cite[Sec. 3.2]{Valimaki2016},
\cite{Spors2019} is used to realize shelving filters with mid-level cutoff
frequency definition.
%
This is known as \textit{one-half-pad loss} filter characteristic
\cite{Kimball1938}. % since early days of electro-acoustics.
%
For convenience the resulting filter coefficients for 1st/2nd low-/high-shelving
filters are reviewed in the Appendix.
%
The discussion of the filter design method concentrates on the cascade of
2nd order low-shelving filters in the remainder.
%
Cascading high-shelving filters is straightforward.
%
%##############################################################################

\subsection{Cascade of 2nd Order Low-Shelving Filters}
%
The cascade (i.e. the series connection) of $N$  2nd order filters (biquads)
is given as transfer function
%
\begin{align}
\label{eq:cascade}
H(s) = \frac{Y(s)}{X(s)} = \prod\limits_{\mu=0}^{N-1} H_\mu(s)
\end{align}
in Laplace domain, being sketched in \fig{fig:tikz-cascade-biquads}.
%
The $\mu$-th biquad of the cascade is defined as, cf. \eq{eq:generalHbiquad}
%
\begin{align}
H_\mu(s) = \frac
{b_{2,\mu}\,s^2 + b_{1,\mu}\,s + b_{0,\mu}}
{a_{2,\mu}\,s^2 + a_{1,\mu}\,s + a_{0,\mu}},
\end{align}
with the corresponding coefficients, cf. \eq{eq:LS2nd}
%
\begin{align}
b_{2,\mu} = \frac{1}{\omega_{\mathrm{c},\mu}^2},\,\,\,
b_{1,\mu} = \frac{g_\mathrm{B}^{\pm\frac{1}{4}}}{Q \, \omega_{\mathrm{c},\mu}},\,\,\,
b_{0,\mu} = g_\mathrm{B}^{\pm\frac{1}{2}},\\\nonumber
a_{2,\mu} = \frac{1}{\omega_{\mathrm{c},\mu}^2},\,\,\,
a_{1,\mu} = \frac{g_\mathrm{B}^{\mp\frac{1}{4}}}{Q \, \omega_{\mathrm{c},\mu}},\,\,\,
a_{0,\mu} = g_\mathrm{B}^{\mp\frac{1}{2}},
\end{align}
%
with mid-level cutoff frequency $\omega_{\mathrm{c},\mu}$ in rad/s and
the pole/zero $Q$-factor.
%
The required gain parameter $g_\mathrm{B} = 10^{|G_\mathrm{B}|/20}>0$
is derived from the biquad shelving level $G_\mathrm{B}$ in dB.
%
The $\pm$ exponent corresponds to positive/negative $G_\mathrm{B}$.
%
The biquad shelving level $G_\mathrm{B}$ is determined by the
filter design algorithm in \eq{eq:Gb}.
%
While $G_\mathrm{B}$ and the user definable $Q$-factor are held constant for all $\mu$,
the frequency $\omega_{\mathrm{c},\mu}$ is varied over $\mu$ by the filter design
algorithm.
%
In \fig{fig:schematic_magres_2nd_ls} the level response of a single
low shelving biquad is schematically shown.
%

%

%

\input{schematic_magnitude_responses}

\begin{figure*}[t]
\centering
\includegraphics[width=0.8\textwidth]{../graphics/biquad-and-pzmap.pdf}
\caption{Left: Level responses of the cascade of three biquads $H_\mu(s)$
(gray) building $H(s)$ (blue)
with $G = - 30 \lg(2) \approx - 9.03$ dB, $\chi = 10 \lg(2) \approx +3.01$ dB/oct,
$\beta=3$ oct. Right: Corresponding poles/zeros in upper/left $s$-plane with
linear axes scaling. Note that the conjugate-complex poles and zeros are not
shown, cf. \fig{fig:pzmap}. Logarithmic equidistance of the poles and zeros
and alignment along the line which is determined by the $Q$-factor is fundamental for
the filter design.}
\label{fig:biquad-and-pzmap}
\end{figure*}

%
\subsection{Parameters}
%
Based on the fundamental diffusive representation
\cite[Sec. 3.2]{Helie2013},
the cascade \eq{eq:cascade} allows to build shelving filters with fractional-order slopes.
%
Particularly, the design of a shelving filter is possible, where two of the
three intended parameters (cf. \fig{fig:schematic_magres_2nd_arb})
%
\begin{itemize}
\setlength\itemsep{-0.1em}
\item $\chi\neq0$: transition slope in dB/oct
\item $\beta>0$: transition bandwidth in octaves
\item $G$: shelving level in dB
\end{itemize}
%
can be defined by the user, determining the third one from (signs: $-$ for low
shelving filter, $+$ for high shelving filter)
\begin{align}
\label{eq:parameter_triplet}
G = \mp\,\beta \cdot \chi,\qquad
\beta = \bigg|\frac{G}{\chi}\bigg|,\qquad
\chi = \mp\,\frac{G}{\beta}.
\end{align}
%
%
%
For the log-log human hearing characteristics (i.e. amplitude and
frequency resolution) \cite{Fastl2007},
it appears reasonable to equally space the individual mid-level cutoff
frequencies $\omega_{\mathrm{c},\mu}$ on a logarithmic frequency axis.
%
Thus, a user definable parameter is the number of cascaded biquads per
octave $N_{O}$.
%
%
%
%
%
%
\NewL The mid-level cutoff frequencies of the biquads in the cascade \eq{eq:cascade}
may then be stated as
%
\begin{align}
\omega_{\mathrm{c},\mu} = 2^{-(\mu+\frac{1}{2})/N_{O}} \cdot \omega_\mathrm{u}
\end{align}
%
with a user definable upper cutoff frequency $\omega_\mathrm{u}$ in rad/s of the
shelving filter cascade $H(s)$.
%
From the intended bandwidth $\beta$ the lower cutoff frequency of $H(s)$ reads
\begin{align}
\omega_\mathrm{l} = 2^{-\beta} \cdot \omega_\mathrm{u},
\end{align}
%
which can be rewritten to
\begin{align}
\beta = \log_2 \frac{\omega_\mathrm{u}}{\omega_\mathrm{l}}.
\end{align}
%
In \fig{fig:schematic_magres_2nd_arb} the level response
is schematically shown with the introduced variables.
%
The shelving level is equal for each biquad $H_\mu(s)$ and is given by
\begin{align}
\label{eq:Gb}
G_\mathrm{B} = - \frac{\chi}{N_{O}}
\end{align}
in dB.
%
For the user defineable number of biquads $N_O$ per octave,
the total number of required biquads
$N$ for the whole cascade is derived as
\begin{align}
\label{eq:ceil_function}
N = \lceil \beta \cdot N_{O} \rceil
\end{align}
with the ceiling operation $\lceil \cdot \rceil$ yielding nearest larger integer.
%
%
%
%
%
%
\section{Analog Filter Properties}
\label{sec:analog_filter_properties}
%
Other than typical shelving filters,
where shelving level determines the slope characteristics,
the proposed filter cascade allows for adjustable slope and bandwidth.
%
This is exemplarily demonstrated in the following.
%
All plots use the normalized frequency $\omega/\omega_\mathrm{u}$,
therefore the upper cutoff frequency of the shelving filter cascade
$\omega_\mathrm{u}$ needs no explicit definition.
%
%
%Note that increased $Q$ also increases
%
%Tuning of $Q$ is straightforward and behaves similar to traditional shelving
%filters.
%
%Thus, no further details are presented for this issue.
%
\subsection{Working Principle}
%
In \fig{fig:biquad-and-pzmap} the working principle of the filter is shown
with a cascade of $N=3$ biquads and $N_O=1$ biquad per octave.
%
The parameter triplet is chosen to
$G = - 30 \lg(2) \approx - 9.03$ dB,
$\chi = 10 \lg(2) \approx +3.01$ dB/oct
and $\beta=3$ oct.
%
According to \eq{eq:Gb}, the biquad shelving level is
$G_\mathrm{B} = -10 \lg(2) \approx -3.01$ dB.
%
Butterworth factor $Q=\nicefrac{1}{\sqrt{2}}$ \cite[p.792]{Ballou2008} is used.
%Butterworth factor $Q=\nicefrac{1}{\sqrt{2}}$ is used.
%
The Laplace transfer functions can be calculated with the formulas given in
Sec. \ref{sec:filter_design}.
%
The left plot of \fig{fig:biquad-and-pzmap} depicts the level responses of
the individual biquads (gray, cf. \fig{fig:schematic_magres_2nd_ls}),
of the shelving filter cascade (blue, cf. \fig{fig:schematic_magres_2nd_arb}),
and the ideal piecewise linear level response (dotted black line).
%
Due to chosen $N_O=1$, the mid-level cutoff frequencies $\omega_{\mathrm{c},\mu=0,1,2}$
are distributed by integer octaves.
%
The right plot in \fig{fig:biquad-and-pzmap} shows the poles and zeros
that lie in the upper-left $s$-plane.
%
In total, six poles and six zeros (three 2nd order sections each with a
conjugate-complex pole and zero pair) build the shelving filter cascade.
%
The gray dots indicate the mid-level frequencies
$|s_{\mathrm{c},\mu}|=\omega_{\mathrm{c},\mu}$ for $\mu=0,1,2$.
%
With help of the circles,
the values for $\omega_{\mathrm{c},\mu}$ can be read at $\Im(s)$-axis.
%
Furthermore, along the line intersecting all gray dots
(i.e. $\angle s = \nicefrac{3}{4}\pi$ for the chosen $Q=\nicefrac{1}{\sqrt{2}}$),
also all poles and zeros are aligned, cf.
\fig{fig:pzmap}.
%
This pole/zero alignment is typical and fundamental \cite{Helie2013}
for the proposed filter design, which is in contrast to the equiangular
alignment used in
\cite{Holters2006a}, see \fig{fig:holters2006-higher-order-shelving}.
%
%
%
\NewL The chosen $Q$-factor has impact on the achieved frequency response.
%
Around the cutoff frequencies, varying $Q$ has about the same impact as for a
single 2nd order shelving filter.
%
However, for the shelving filter cascade, varying $Q$ has an impact
on the slope and the bandwidth as well.
%
Generally, increasing $Q$ increases ripples along the slope in level and
phase.
%
Decreasing $Q$ decreases the bandwidth, where the intended slope holds.
%
Butterworth factor $Q=\nicefrac{1}{\sqrt{2}}$ and
$Q=\nicefrac{5}{6}$ \cite[p.261]{LangeSigSys1} are good
trade-off choices to obtain moderately damped responses at lower/upper
cutoff frequencies as well as small slope ripple and valid slope bandwidth.
%
To keep focus on other details, this issue is not further
discussed here and in the remainder all filters use Butterworth factor
$Q=\nicefrac{1}{\sqrt{2}}$.

\input{figure_resp}

\subsection{Frequency Responses}
%
In \fig{fig:shelving_filter_setup_cases} different variations of the
parameter triplet shall illustrate the capabilities of the shelving filter
cascade.
%
Each of the left plots shows the resulting level response in dB and each of
the right plots depicts the phase response in degree.
%
All filter cascades exhibit $N_O = 3$ biquads per octave.
All chosen bandwidths are integer values.
Then \eq{eq:ceil_function} yields the number of required biquads
$N = 3\cdot\beta$ and thus filter order $M=2 \cdot N=6\cdot\beta$,
which is convenient for mental calculation.
%
%
%
\NewL In \fig{fig:low-shelving-filter-varying-slope} the shelving level
$G=\pm 20 \lg (4) \approx \pm 12$ dB is fixed.
%
Then either slope $\chi$ or bandwidth $\beta$ can be additionally adjusted.
%
The slopes $\chi = \mp 10 \lg (\sqrt{2}, 2, 4) \approx \mp 1.5, 3, 6$
dB/oct are adjusted
yielding the bandwidths $\beta = 8,4,2$ octaves, respectively using \eq{eq:parameter_triplet}.
%
The somewhat laborious values for $\chi$ were used
because these represent exact
multiples of a first-order system slope of $\pm 20 \lg(2)$ dB/oct.
With chosen $G$ this then results in integer values
for $\beta$.
%
The depicted level responses in
\fig{fig:low-shelving-filter-varying-slope} verify the intended shelving
characteristics.
%
The phase responses indicate minimum-phase characteristics, with
the generally known observations, that phase excess becomes larger the steeper
the slope and the larger the shelving level.
%
For the particular filter with 8 octave bandwidth,
minimum-phase characteristics of constant $\pm22.5$ degrees for
$\chi = \pm 10 \lg\sqrt{2} \approx \pm 1.5$ dB/oct is approximated in a large
frequency band.
%
This is expected for a quarter-differentiator/-integrator.
%
%
%
\NewL In \fig{fig:low-shelving-filter-varying-bandwidth},
$\chi = 10\lg(2)\approx +3.01$ dB/oct is fixed.
%
A positive slope requires a negative shelving level, which can be chosen as
further adjustable parameter.
%
Alternatively, the bandwidth as free parameter can be adjusted, which is pursued
here with $\beta = 1,2,3,6,8$ octaves.
%
With \eq{eq:parameter_triplet} this yields shelving levels
$G = - 10\lg(2)\cdot (1,2,3,6,8)\approx -(3,6,9,18,24)$ dB.
%
The level responses clearly show the intended characteristics.
%
The phase response shows the increasing phase excess in value and frequency
bandwidth with increasing transition bandwidth.
%
The filter with 8 octaves bandwidth approximates $+45$ deg constant
minimum-phase (dotted line) for a large frequency band, resembling half-differentiator
filter characteristics (ideally: +3.01 dB/oct slope and 45 deg phase).
%
%
%
\NewL In \fig{fig:low-shelving-filter-varying-gain} the bandwidth $\beta = 6$
octaves is fixed.
%
Either the slope or the shelving level can be additionally adjusted.
%
Here the chosen integer levels $G = 7,3, -4,-8$ dB yield slopes
$\chi = -\nicefrac{7}{6}, -\nicefrac{3}{6}, +\nicefrac{4}{6}, +\nicefrac{8}{6}$
dB/oct, respectively.
%
The level responses depict the expected characteristics.
%
The phase responses exhibit similar characteristics varying in their
maximum phase excess, which is linked to the shelving level and the achieved
slope.
%
Thus, again: a certain fractional-order slope involves a certain minimum phase
(offset).
%
In the case of an ideal slope over the whole frequency band,
a constant phase
$\angle H(\omega) = \frac{\pi}{2} \frac{\chi}{20\lg(2) \mathrm{dB}/\mathrm{oct}}$
in radian results.
%

\subsection{Constraints and Limitations}
%
Naturally for approximations---in the present case a fractional-order
slope is approximated with a finite number of biquads and a certain octave
resolution---deviations to the ideal case are to be expected.
%
This subsection exemplarily discusses the general limitation patterns of the
proposed filter design method.
%
In \fig{fig:shelving_filter_error_cases} these limitations are
visualized.
%
In \fig{fig:gain-error} the effect of shelving level quantization is depicted.
%
Since the parameter triplet $G, \beta, \chi$ influences $N$ (which
needs to be integer) and $N_O$, only certain $\beta$ and $G$ can be exactly realized with
a chosen parameter set.
%
Note that $\omega_\mathrm{u}$ can be completely freely adjusted though.
%
From \eq{eq:Gb}, \eq{eq:ceil_function}, the resulting shelving
level $G_\mathrm{r}$ thus reads
\begin{align}
G_\mathrm{r} = G_\mathrm{B} \cdot N  = - \frac{\chi}{N_{O}} \cdot
\lceil \beta \cdot N_{O} \rceil
\end{align}
for chosen $\chi$, $\beta$ and $N_O$.
%
Only for $\beta \cdot N_{O}\in\mathbb{N}$ the intended shelving level is equal to
the resulting level $G_\mathrm{r} = G$.
%
The larger the deviation $\beta \cdot N_{O}$ towards the next larger integer,
the larger is the quantization error for the shelving level.
%
This can be reproduced with the chosen parametrization in \fig{fig:gain-error}
for slope
$\chi = +10\lg(2) \approx +3.01$ dB/oct,
bandwidth $\beta=\nicefrac{19}{6}$ octaves and shelving level
$G= - \chi \cdot \beta \approx - 9.53$ dB.
%
Only $N_O = 6$ biquads per octave yield integer $\beta \cdot N_{O} = 19$
and thus $G_\mathrm{r} = G$.
%
This quantization effect must be considered in practical designs and
a tolerable, application dependent quantization step size must be defined by the
user.
%
\NewL In \fig{fig:slope-error} it is demonstrated that the desired slope and
bandwidth is not always being achieved when the chosen parameters are set
unfavorable.
%
The intended bandwidth and slope is well approached if both rule-of-thumb
conditions
\begin{align}
\label{eq:bchiNO_cond}
&\beta \cdot N_O \in \mathbb{N}\\\nonumber
&N_O \geq
\begin{cases}
\frac{\chi}{12\,\mathrm{dB}} &\,\,\,\mathrm{if}\,\,\,\chi \geq 12 \, \mathrm{dB/oct}\\
1 / \mathrm{oct} &\,\,\,\mathrm{if}\,\,\,\chi < 12 \, \mathrm{dB/oct}
\end{cases}
\end{align}
are met at the same time, following from
maximum slope of 12 dB/oct for a 2nd order filter.
%
For \fig{fig:slope-error} the parameter triplet
$\chi = +120 \lg(2) \approx +36$ dB/oct, $\beta = 3$ oct,
$G = -360 \lg(2) \approx -108$ dB was set up.
%
For the chosen $N_O = \nicefrac{1}{3}, \nicefrac{2}{3}, 1, 3$
the first condition of \eq{eq:bchiNO_cond}
always holds, which also yields $G_\mathrm{r}=G$, see discussion for \fig{fig:gain-error}.
%
However, only $N_O=3$ fulfills the second condition of \eq{eq:bchiNO_cond},
then the resulting slope and bandwidth converge well to the intended one.
%
For the other cases the bandwidth appears to be larger.
%
In the extreme case of $N_O=1/3$ only one biquad is utilized, due to
$N = \lceil \beta \cdot N_{O} \rceil$.
%
This single biquad obviously cannot render the
intended frequency response, but rather exhibits that of a
2nd order shelving filter.
%
\input{figure_lim}
%
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{../graphics/digital-3db-per-octave-shelving-filter.pdf}
\caption{
Digital shelving filter cascade with bilinear $z$-transform for
$\chi = 10\lg(2) \approx 3.01$ dB/oct and bandwidth $\beta=6$ octaves, upper
cutoff frequency 2 kHz, sampling frequency 48 kHz, $N_O=1$ biquad per oct.
%
Left: level response, middle: phase response, right: poles/zeros in
$z$-plane.
}
\label{fig:digital-3db-per-octave-shelving-filter}
\end{figure*}
%
\NewL In \fig{fig:ripple-and-zp-frequency} the case of slope ripple as a
function of biquads per octave is discussed.
%
As stated above, the fractional slope is approximated by biquads that are
spaced with certain fractional-octave resolution.
%
Thus it is expected, additionally to the already mentioned limitations,
that the slope exhibits ripples in the level response.
%
For \fig{fig:ripple-and-zp-frequency} the parameter triplet
$\beta=9$ oct,
$G=-10\lg(2)\approx -3$ dB,
$\chi =  \frac{10}{9}\lg(2) \approx + 0.3345$ dB/oct
is set up, realized with
$N_O = \nicefrac{2}{3}, 1, 3$.
%
The condition $\beta \cdot N_O\in\mathbb{N}$ is always fulfilled, thus $G_\mathrm{r}=G$.
%
The choice $N_O=\nicefrac{2}{3}$ violates the 2nd condition of \eq{eq:bchiNO_cond},
whereas $N_O=1,3$ fulfill it.
%
The deviation with respect
to the ideal piecewise linear slope (cf. \fig{fig:schematic_magres_2nd_arb})
is visualized in terms of the error
$20\lg|H(\omega)|-20\lg|H_\mathrm{ideal\,slope}(\omega)|$.
%
It can be clearly observed as a rippled slope response.
%
For $N_O = \nicefrac{2}{3}$ biquads per octave the ripple amounts $\pm 0.1$ dB.
%
For $N_O=3$ biquads per octave the ripple becomes irrelevantly small for
practical audio applications.
%
Note that the error steeply increases at the edges of the transition band,
since the resulting filter naturally deviates from the ideal piecewise linear
slope.
%
Again, one can state that the suitable amount of biquads per octave is highly
dependent on the tolerated ripple.
%
One biquad per octave resolution seems to be a good trade-off for most
audio applications that ask for slopes $|\chi|<12$ dB/oct.
%
%
\NewL In summary, all three limitations and generally $\beta\gg 1$ octave (which in
practice can be relaxed to $\beta>1$),
must be taken into account by the filter designer to customize towards a target
application.

%The proposed design requires more biquads than traditional designs and thus
%more computational load, which nowadays should be acceptable.
%
%Rippe increase for Q incresae!!

\begin{figure}[h!]
\centering
\includegraphics[width=0.68\columnwidth]{../graphics/liski2019-geq-3db-per-oct.pdf}
\caption{Digital GEQ \cite{Liski2019} with $f_\mathrm{s}=48$ kHz. The same shelving slope and bandwidth as in
\fig{fig:digital-3db-per-octave-shelving-filter} is intended by setting the
faders (red dots) accordingly.
GEQ has one octave resolution to be comparable with
\fig{fig:digital-3db-per-octave-shelving-filter}}
\label{fig:liski2019-geq-3db-per-oct}
\end{figure}

\section{Discussion}
\label{sec:discussion}
%
In this section some notes on digital filter design are given.
%
Furthermore, a comparison to a recent cascade GEQ design is presented
and potential meaningful applications are briefly outlined.

\subsection{Digital Filter Design}
\label{sec:digital_filter}
%
So far, analog filter responses were discussed.
%
For practical applications digital filters might be conveniently deployed
instead.
%
Digital filter design is straightforward with either the bilinear or the
matched $z$-transform of the individual 2nd order
analog filters \cite{Ifeachor2002,Berners2003,Nielsen2016}
or as Regalia-Mitra structure with a digital allpass \cite{Jot2015}.
%
If the upper cutoff frequency $f_\mathrm{u} = \frac{\omega_\mathrm{u}}{2\pi}$ is low
compared to the sampling frequency $f_\mathrm{s}$, the introduced artifacts are negligible.
%
In fact, for $f_\mathrm{u} / f_\mathrm{s} < \frac{1}{3}$ appropriate results are yielded,
spanning almost the whole listening range for typical audio sampling frequencies.
%
In \fig{fig:digital-3db-per-octave-shelving-filter} a digital shelving filter
with 6 octave bandwidth and 3 dB/oct slope is depicted for
$f_\mathrm{u}=2$ kHz and $f_\mathrm{s}=48$ kHz using bilinear transform without
frequency pre-warping.
%
For this example the matched z-transform practically yields the same results.
%
Both digital filters closely match the analog target response.
%
%
%
%\NewL Note that (standard) finite impulse response (FIR) designs exhibit a linear
%frequency scale.
%
%A similar shelving filter designed as FIR would require a rather long impulse
%response to achieve the frequency resolution in the low end.
%
%Thus, the number of biquads in IIR design and the number of taps in FIR design
%should be thoroughly evaluated for practical realizations.

\subsection{Comparison to Graphical Equalizer Design}
%
GEQs can be used to approach a shelving filter characteristic with
certain slope over certain bandwidth.
%
In \fig{fig:liski2019-geq-3db-per-oct} this is exemplarily shown, deploying
a very recently proposed GEQ design by Liski \cite{Liski2019}.
%
The filter shall have +3.01 dB/oct slope within 6 octave bandwidth with upper
cutoff frequency of 2 kHz, i.e. the same as
\fig{fig:digital-3db-per-octave-shelving-filter}.
%
The GEQ is designed from a PEQ cascade, which inherently
makes constant level over a larger bandwidth an ambitious task.
%
Thus, compared to the shelving filter cascade, the ripple at the 9 dB constant
level band is larger and the -9 dB level band is not realized at all.
%
Furthermore, a slightly larger ripple along the slope can be observed for the
GEQ by close inspection.
%
Although, the resulting filter might be acceptable, setup of different slopes
and bandwidth in a GEQ might be tedious.
%
Thus, if explicitly requiring shelving filter characteristics,
the proposed shelving filter
cascade could be preferred with its simpler parameter set and more accurate response.

\subsection{Potential Applications}
%
The presented filter could be used in audio mixing and production,
where transition bandwidth might be an additional
parameter for shelving filters.
%
The proposed filter design might improve the loudness filter of \cite{Nielsen2016},
where a cascade of two 1st order shelving filters was designed to compensate
for equal loudness level contours.
%
The dual-band shelving filter
approach \cite{Audfray2018} could make use of the proposed filter.
%
%
%
\NewL Usually bass boosted target transfer functions are desired for large-scale
sound reinforcement, which can be conveniently realized with the proposed
filter design.
%
Furthermore, also so called line \textit{array morphing}
\cite{LAcoustics2013} can be approached with either varying the
shelving level and the slope (\fig{fig:low-shelving-filter-varying-slope})
or varying the level and the bandwidth (\fig{fig:low-shelving-filter-varying-gain}).
%
%
%
\NewL As another example, for wave field synthesis (WFS) \cite{Firtha2017} and
line source array (LSA) applications, a shelving filter
with a +3 dB/oct rising slope, ideally with +45 degree phase,
for the mid-frequency band is required
in order to equalize the inherent acoustic lowpass characteristic of the
deployed line source.
%
The proposed filter design can be straightforwardly parametrized to obtain this
commonly known WFS pre-filter and LSA coupling filter.
%
The frequency responses shown in
\fig{fig:low-shelving-filter-varying-bandwidth} and
\fig{fig:digital-3db-per-octave-shelving-filter}
are typical examples for it.
%
The lower shelf compensates for the loudspeaker array's omni-directional
characteristic at lower frequencies, where array length determines the lower
cutoff frequency \cite{Schultz2016}.
%
The upper shelf compensates for spatial aliasing energy \cite{Spors2010a}, here
upper cutoff frequency is varied to match the different spatial aliasing cutoff
frequencies of sound field synthesis (SFS) methods \cite{Winter2018}.
%
Furthermore, dynamically changing the filter parameters is straightforward and
could be advantageously deployed for dynamic SFS, such as WFS of moving
sources \cite{Firtha2019}.
%
For SFS of focused point sources, filter responses with two different
(fractional-order) slopes are required \cite{Spors2010,Spors2011}.
%
This could be realized with an optimized shelving filter cascade that exhibits
multiple slopes.
%
%
%
\NewL Multiple slope filters, or more generally multiband EQ design is out of
scope of this paper.
%
However, the band-shelving filter concept of \cite{Holters2006a} can be
straightforwardly adopted for our filter design, which then could enhance the
control of the transition bands with adjustable slope and bandwidth.

\section{Summary}
\label{sec:summary}
%
This paper proposes a cascade of second-order shelving filters to realize a
higher-order shelving filter with adjustable transition slope and adjustable
transition bandwidth additionally to the common parameters cutoff frequency
and shelving level.
%
As a natural choice regarding human hearing, the mid-level frequencies of the
biquads are equidistantly aligned with respect to logarithmic frequency axis.
%
Thus, a critical parameter is the number of biquads per octave, which must be
sufficiently high to achieve the intended frequency response, most important
a ripple free shelving slope.
%
Digital filter design for audio applications can be straightforwardly realized
with bilinear or matched-z transform of individual biquads.
%
This IIR filter based design might be preferably used for specific
applications rather than specialized FIR designs or GEQs.
%
With its simple parametrization, the filter could be deployed in various
audio applications. %and to design parametric multiband EQs.

\begin{figure*}[h]
\centering
\includegraphics[width=\textwidth]{../graphics/holters2006-higher-order-shelving.pdf}
\caption{Higher order low-shelving filters from Holters \cite{Holters2006a}
adapted to $\omega_\mathrm{c}$ as mid-level cutoff frequency.
%
Then, poles and zeros are equiangularly aligned along the circle with radius
$|s| = \omega_\mathrm{c}$.
%
The larger the filter order the closer the poles and zeros are aligned to that
circle, cf. 2nd order filter pole/zero-map in \fig{fig:pzmap}.
%
An increasing slope and phase excess can be observed for increased filter order,
however no explicit control of transition bandwidth is possible.}
\label{fig:holters2006-higher-order-shelving}
\end{figure*}

\section*{Appendix: Laplace Transfer Function of Traditional Shelving Filters}
%
The transfer function of a general 2nd order filter for real signals
reads
%
\begin{align}
\label{eq:generalHbiquad}
\tilde{H}(s) = \frac{b_2 s^2 + b_1 s + b_0}{a_2 s^2 + a_1 s + a_0}
\end{align}
for Laplace domain $s = \sigma + \im \omega$ with $b, a\in\mathbb{R}$.
%
For $s = \im \omega$ the frequency response
$\tilde{H}(\omega)=|\tilde{H}(\omega)|\e^{\im \angle \tilde{H}(\omega)}$ over angular
frequency $\omega$ is evaluated.
%
%
%
\NewL
The shelving filter is parametrized with
shelving level $G_\mathrm{B}$ in dB,
mid-level cutoff frequency $\omega_c$ in rad/s
and---for the 2nd order filter---the pole/zero $Q\geq\frac{1}{2}$ factor
(the same $Q$ for pole and zero is applied here for convenience).
%
With $g_\mathrm{B} = 10^{|G_\mathrm{B}| / 20}>0$ the coefficients for 1st/2nd
order low-/high-shelving
filters can be given in the following listing, cf. \cite{Spors2019}.
%
The $\pm$ exponent for $g_\mathrm{B}$ corresponds to the sign of $G_\mathrm{B}$,
i.e. positive or negative shelving level $G_\mathrm{B}$.
%
%
%
\NewL
%
The 1st order low-shelving filter is given by the coefficients
for \eq{eq:generalHbiquad}
\begin{align}
b_2 = 0,\quad b_1 = \frac{1}{\omega_\mathrm{c}},\quad b_0 = g_\mathrm{B}^{\pm\frac{1}{2}},\\\nonumber
a_2 = 0,\quad a_1 = \frac{1}{\omega_\mathrm{c}},\quad a_0 = g_\mathrm{B}^{\mp\frac{1}{2}}.
\end{align}
%
The 1st order high-shelving filter is given by
\begin{align}
b_2 = 0,\quad b_1 = \frac{g_\mathrm{B}^{\pm\frac{1}{2}}}{\omega_\mathrm{c}},\quad b_0 = 1,\\\nonumber
a_2 = 0,\quad a_1 = \frac{g_\mathrm{B}^{\mp\frac{1}{2}}}{\omega_\mathrm{c}},\quad a_0 = 1.
\end{align}
%
The 2nd order low-shelving filter is given by the
coefficients for \eq{eq:generalHbiquad}, cf. \cite[(15)]{Valimaki2016}
\begin{align}
\label{eq:LS2nd}
b_2 = \frac{1}{\omega_\mathrm{c}^2},\quad b_1 = \frac{g_\mathrm{B}^{\pm\frac{1}{4}}}{Q\,\omega_\mathrm{c}},\quad b_0 = g_\mathrm{B}^{\pm\frac{1}{2}},\\\nonumber
a_2 = \frac{1}{\omega_\mathrm{c}^2},\quad a_1 = \frac{g_\mathrm{B}^{\mp\frac{1}{4}}}{Q\,\omega_\mathrm{c}},\quad a_0 = g_\mathrm{B}^{\mp\frac{1}{2}}.
\end{align}
%
The 2nd order high-shelving filter is given by
\begin{align}
b_2 = \frac{g_\mathrm{B}^{\pm\frac{1}{2}}}{\omega_\mathrm{c}^2},\quad b_1 = \frac{g_\mathrm{B}^{\pm\frac{1}{4}}}{Q\,\omega_\mathrm{c}},\quad b_0 = 1,\\\nonumber
a_2 = \frac{g_\mathrm{B}^{\mp\frac{1}{2}}}{\omega_\mathrm{c}^2},\quad a_1 = \frac{g_\mathrm{B}^{\mp\frac{1}{4}}}{Q\,\omega_\mathrm{c}},\quad a_0 = 1.
\end{align}
%
\input{pzmap_ls}
%
The 2nd order filter can be further discussed with the pole-zero map shown
in \fig{fig:pzmap}.
%
%\section*{Appendix II: Zeros and Poles}
%
The zeros and poles of a 2nd order low-shelving filter are
\begin{align}
s_{0} = \omega_\mathrm{c} \:g_\mathrm{B}^{\pm\frac{1}{4}} \: \e^{\im\alpha}\qquad
s_{\infty} = \omega_\mathrm{c} \: g_\mathrm{B}^{\mp\frac{1}{4}} \: \e^{\im\alpha},
\end{align}
and the respective complex conjugates $s_{0}^{\ast}$ and $s_{\infty}^{\ast}$.
The sign in the exponents is again determined
by the shelving level $G_\mathrm{B}$.
The polar angle $\alpha$ in the complex $s$-plane (Laplace domain)
relates to the $Q$-factor
\begin{align}
Q = \frac{-1}{2\cos\alpha}.
\end{align}
For stable and causal filters,
the poles must be in the left-half $s$-plane $\Re(s_{\infty}) < 0$,
hence requiring $\frac{\pi}{2} < \alpha < \frac{3\pi}{2}$.
The corresponding $Q$-factor is always positive
and varies within $[\frac{1}{2}, \infty)$.
This paper mainly considers Butterworth \cite{Ballou2008} $Q=\frac{1}{\sqrt{2}}$,
which yields $\alpha=\frac{3}{4} \pi$
as illustrated in Fig.~\ref{fig:pzmap}.
If $Q=\frac{1}{2}$, i.e. $\alpha=\pi$,
the poles and zeros lie on the real axis,
thus double real poles $s_{0} = s_{0}^{\ast}$ and
$s_{\infty} = s_{\infty}^{\ast}$ are obtained.
%
In this case the 2nd order frequency
response is identical to that of the 1st order shelving filter, although
obtained by different pole/zero-configurations.
%
Thus, it is sufficient to discuss the present filter design based on
2nd order filters.
%
\NewL In the case of high-shelving filters, the poles and zeros are exchanged. %,
%\begin{align}
%s_{0} &= \omega_{c,\mu} \: g_\mathrm{B}^{\mp\frac{1}{4}} \: e^{i\alpha}\\
%s_{\infty} &= \omega_{c,\mu} \: g_\mathrm{B}^{\pm\frac{1}{4}} \: e^{i\alpha}.
%\end{align}
Notice that a high-shelving filter with a shelving gain $\pm g_\mathrm{B}$
has the same poles and zeros as the low-shelving filter with $\mp g_\mathrm{B}$.
The system functions only differ by a scaling factor of $\pm g_\mathrm{B}$
due to the leading coefficients $b_{2}$ and $a_{2}$.
%
The scaling factor is lost information in a pole-zero map, unless explicitly
stated.
%
%
%\input{poles-and-zeros}


% !TEX root = main.tex

\section{Theorems}

\begin{defn}
This is a definition
\end{defn}

\begin{lem}[Zorn's lemma]
This is a lemma
\end{lem}

\begin{theo}
This is a theorem.
\begin{proof}
This is a proof
\end{proof}
\end{theo}

\begin{rmk}
This is a remark.
\end{rmk}

\begin{exc}
This is an exercise.
\end{exc}