\documentclass[a4paper, 11pt]{article}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{url}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{hyperref}
\usepackage[english]{babel}
\usepackage{cleveref}

\begin{document}

\title{Contagion of an exogenous failure in a bank network}
\date{\today}
\author{Marco Tinacci}
\maketitle

All the material of this work, including the up-to-date version of this document, can be found in the public repository \url{https://github.com/marcotinacci/interbank-lending-systemic-risk}.

\section{Introduction} % (fold)
\label{sec:introduction}
We take under consideration large banking systems. In \cite{KrauseGiansante} many real examples of systems are considered and they propose a taxonomy in which tiering and size are the two orthogonal dimensions. We can model this kind of system as a network where connections represent interbank loans. We have a high level of tiering when a few banks dominate the majority of connections, like the UK and Italian market.

To generate realistic bank networks with respect to this taxonomy we will use a Chung-Lu model \cite{ChungLu} instead of a Barabási-Albert \cite{barabasi1999emergence}, as described in \cite{KrauseGiansante}. We adopt this different model because it is not clear how they can conciliate two power law distribution, assets and degrees, with the standard generation algorithm. The structure of the obtained networks result to be similar to their examples. Main concepts about random graphs and centrality measures that has been employed in this work can be found in \cite{Newman}.

We want to analyze what happens when the an exogenous shock make the biggest bank fail generating a chain of losses to other connected banks. To do that we implement a generator of realistic banking systems and the propagation mechanism, then we run some simulation to check how the failure spreads if we change some system parameters. We will describe decisions that has not been specified in \cite{KrauseGiansante} and we will focus on how the contagion mechanism changes when banks adopt heterogenous model of loans (to customers or to other banks) and when they adopt an homogenous model, in our case, a model focused on loans to customers.

% section introduction (end)

\section{Model} % (fold)
\label{sec:model}

\paragraph{Bank network} % definizione grafo orientato pesato
To build the bank network we use the model adopted in \cite{KrauseGiansante}. The single bank is represented by a node of the network and contains some information about the internal balance. Formally we define a node $B$ as follows
$$ B = \langle A,\rho,\beta,\gamma,\alpha \rangle $$
such that $A \in \mathbb{R}^+$, $\rho,\beta,\gamma,\alpha \in [0,1]$, $\rho + \beta \leq 1$ and $\gamma + \alpha \leq 1$. $A$ is the \emph{asset} of $B$, $\rho \cdot A$ are \emph{cash reserves}, $\beta \cdot A$ are \emph{loans to customers} and $(1-\rho-\beta) \cdot A$ are \emph{loans to other banks}. Other fractions are liabilities: $\gamma \cdot A$ are \emph{deposits by customers}, $\alpha \cdot A$ is the \emph{equity} and $(1-\gamma-\alpha) \cdot A$ are \emph{loans received} from other banks. We will consider two main failure condition of a bank: $\rho = 0$ and $\alpha = 0$.

To denote different nodes we use a subscript notation $B_i$ and the same for its internal data $A_i,\rho_i,\beta_i,\gamma_i,\alpha_i$. We denote with $\mathcal{B}$ the set of all banks taken into account in our system.
Interbank loans are represented by the relation $L \subseteq \mathcal{B} \times \mathbb{R^+} \times \mathcal{B}$ and we say that $(B_1,l,B_2) \in L$ when bank $B_1$ lend $l$ to $B_2$, we can also write $B_1 \xrightarrow{l} B_2$.
Now we can define the banking system network as a directed weighted graph $\mathcal{G} = \langle \mathcal{B}, L \rangle$ where $\mathcal{B}$ is the set of bank nodes and $L$ is the relation that contains all the oriented and weighted edges between nodes.

\paragraph{Asset and degree distribution} % asset e gradi come power law
Since we want a random network as close to a realistic scenario as possible, we need to generate bank assets as a power law distribution, to be precise we use the following probability density function
$$ f_{pdf}(x;a) = C x^{-a} $$
where $C \in \mathbb{R}$ is a scaling factor computed such that $f_{pdf}$ is a proper probability density function in the desired range. $a$ is a simulation parameter and is decided depending on the kind of network we want to generate, $C$ will depend on $a$. Internal information $\rho,\beta,\gamma,\alpha$ are chosen uniformly at random in given ranges. Let us denote with $\mathcal{B}^a$ the set of bank nodes generated at random as just described. Also the degree distribution must follow a power law distribution and, to be realistic, we need it to be strongly correlated to the assets distribution. To obtain this we use the algorithm to generate the Chung Lu model \cite{ChungLu} that returns a random (selfloop-free) graph with given expected degrees. Every edge between $B_1$ and $B_2$ is generated with probability
$$ p_{B_1B_2} = \frac{w_{B_1}w_{B_2}}{\sum_B w_B} $$
where $B_1,B_2 \in \mathcal{B}$ and $w_{B_1}, w_{B_2}$ are respectively their expected degree. We use normalized assets of nodes as expected degrees, in this way we obtain a degree distribution strongly correlated to the assets distribution (Figure~\ref{fig:corr}).

\begin{figure}[htbp]
    \centering
    \begin{subfigure}[b]{0.6\textwidth}
		\includegraphics[width=\textwidth]{images/scatter2.png}
        \caption{$a = 2, \rho=0.96$}
	\end{subfigure}
    \begin{subfigure}[b]{0.6\textwidth}
		\includegraphics[width=\textwidth]{images/scatter3.png}
        \caption{$a = 3, \rho=0.96$}
	\end{subfigure}
    \begin{subfigure}[b]{0.6\textwidth}
		\includegraphics[width=\textwidth]{images/scatter4.png}
        \caption{$a = 4, \rho = 0.93$}
	\end{subfigure}
	\caption{Scatter plots between degrees and assets generated trough power laws with different exponent $a$. The minimum correlation $\rho$ is $0.93$}\label{fig:corr}
\end{figure}

Since a loan can be naturally modeled as an \emph{oriented} edge, we need to modify the graph structure in order to obtain such feature, preserving the described properties. We use the following procedure: every undirected edge $(B_i, B_j)$ is replaced by a directed one, with equal probability by $(B_i,B_j)$ or $(B_j,B_i)$. This trivially guarantees that the degree (input and output) of each node in the directed graph is still following a power law distribution.

Finally we need to transform the just described directed graph to a \emph{weighted} directed graph whose weights are interbank loans' amounts. If $(B_i,B_j)$ is an edge of the directed graph then $(B_i,l,B_j)$, $B_i$ lend $l$ to $B_j$, is a weighted edge of the weighted directed graph such that
$$l = \frac{A_j(1-\gamma_j-\alpha_j) \cdot A_i(1-\rho_i-\beta_i)}{\sum_{k \in cred(i)} A_k(1-\gamma_k-\alpha_k)}$$
where $cred(i)$ is the index set of all the creditors of bank $B_i$. Using the previous formula is easy to compute the weighted adjacency matrix. To maintain consistency with every bank balance an asset update is necessary to move the lent money from a bank to another, this however is a minor adjustment that preserves the power law distribution of assets and the correlation with the degree distribution.

\paragraph{Contagion} % meccanismi di contagio
The event we are going to simulate is the contagion of a failure after the bank with the highest asset $B_0$ fails by an exogenous event. Then $B_0$ is forced to repay its debts and, to do so, it also has to call back its loans to other banks. Calling loans in and repaying debts with respect to other banks are two mechanisms that can propagate the failure to other banks and they are called respectively \emph{failure mechanism} and \emph{default mechanism}.

In the failure mechanism every loan to another bank, say $B_c$, is called in and if the creditor bank has not enough cash to restore the debt (condition $\rho = 0$) then it goes bankrupt in turn and the initially failing bank needs to wait for the entire failing procedure of $B_c$ to recover its money. This mechanism can lead to a chain of failures and, in some cases, also in a cyclic chain. When the latter happens it means that a failing bank is calling in a loan from another failing bank, in this case we assume that this bank lose the entire loan. Another likely case is the bankrupt by combined call in of loans: this happens when $B_c$ would be able to repay debts to more than one bank individually, but if more failure happens $B_c$ does not have enough cash to repay the sum of recalled loans. Since the purpose of this procedure is transforming the whole bank asset into cash (basically $\rho = 1$) we also need to call loans to customers back, but only a part of this loans can be recovered, in our case we assume a recovery rate $k$ that is the same for every bank.

After a failing bank terminate to call in its credits it must start to repay its debts. Deposits to customers are the first to be restored, then, if some cash remains, borrowing with respect to other banks must be restored. If the failing bank is able to restore every debt to other banks then the failure cannot propagate to them. Otherwise the bank equally split its remaining cash among all the debtor banks (except for smaller debts, those below the split fraction, that are paid individually) and the difference between the actual debt and the restored part results as a loss for the debtor. This loss is recorded as a decrement of the equity in the liabilities and when equity goes to zero ($\alpha = 0$) the bank goes bankrupt and so the failure propagate. Again, it may happen that a bank would be able to sustain individual losses deriving from a partially repaid debt but it actually goes bankrupt for their summed effect.

The exogenous failure is triggered by setting the biggest bank's equity to zero, this will force it to go bankrupt and to start the contagion. In Figures~\ref{fig:network2}, \ref{fig:network3} and \ref{fig:network4} we show some examples of randomly generated network, before and after the trigger of the exogenous failure, respectively with $a=2$, $a=3$ and $a=4$. Green nodes are active banks, yellow nodes are banks failed by failure mechanism, red nodes are banks failed by default mechanism and the blue node is the trigger. Edges are colored with shades of blues, a darker edge corresponds to an higher loan.

\begin{figure}[htbp]
    \centering
    \begin{subfigure}[b]{0.9\textwidth}
		\includegraphics[width=\textwidth]{images/network2.png}
	\end{subfigure}
    \begin{subfigure}[b]{0.9\textwidth}
		\includegraphics[width=\textwidth]{images/contagion2.png}
	\end{subfigure}
	\caption{Graphical representation of a network generated with $N = 100$ and power law exponent $a = 2$, before and after the trigger of the exogenous failure.}\label{fig:network2}
\end{figure}

\begin{figure}[htbp]
    \centering
    \begin{subfigure}[b]{0.9\textwidth}
		\includegraphics[width=\textwidth]{images/network3.png}
	\end{subfigure}
    \begin{subfigure}[b]{0.9\textwidth}
		\includegraphics[width=\textwidth]{images/contagion3.png}
	\end{subfigure}
	\caption{Graphical representation of a network generated with $N = 100$ and power law exponent $a = 3$, before and after the trigger of the exogenous failure.}\label{fig:network3}
\end{figure}

\begin{figure}[htbp]
    \centering
    \begin{subfigure}[b]{0.9\textwidth}
		\includegraphics[width=\textwidth]{images/network4.png}
	\end{subfigure}
    \begin{subfigure}[b]{0.9\textwidth}
		\includegraphics[width=\textwidth]{images/contagion4.png}
	\end{subfigure}
	\caption{Graphical representation of a network generated with $N = 100$ and power law exponent $a = 4$, before and after the trigger of the exogenous failure.}\label{fig:network4}
\end{figure}

\paragraph{Analysis}

We want to analyze how different kind of networks react to the exogenous failure of the biggest bank of the system. To do that we consider the \emph{fraction failing} index that is trivially the number of banks failed during the chain reaction divided by the initial number of banks. More specifically we want to analyze how this index changes in bank networks where nodes have higher (or lower) fractions of loans to customers or to other banks, tuning the range in which $\beta$ is drawn. We expect to see that the higher is $\beta$ the lower is the amount of infected nodes.

% section model (end)

\section{Experiments} % (fold)
\label{sec:experiments}

We consider the following set up: bank assets $A_i \in [100;10^{10}]$ are drawn from a power law distribution with exponent $a \in \{2,3,4\}$ and recovery rate $r = 0.5$ for every bank. Bank specific fractions are drawn from uniform distribution over the following ranges: $\rho_i \in [0;0.25]$, $\beta_i \in [0;1-\rho_i]$, $\alpha_i \in [0;0.25]$ and $\gamma_i \in [0;1-\alpha_i]$. For this experiment we adopt the following generalized notion of range for the customer loans fraction
\begin{equation}\label{eq:exp1}
	\beta_i^k \in \left[ k \cdot s\ ; 1-\rho_i \right]
\end{equation}
for $k \in [0..niter-1]$ and the step size $s = (1-\rho_i) / niter$. $niter$ is an arbitrary number of iterations, the higher it is the finer is the analysis, we used $niter = 20$. Note that for $k = 0$ it boils down to the range defined initially $\beta_i^0 \equiv \beta_i$. Using this kind of variation we can observe how the fraction failing changes in a bank system where loans to customers are more or less common than interbank loans.

From simulations we obtain experimental results shown in \Cref{fig:exp1,fig:exp2,fig:exp3}. We considered respectively network of dimension $50$, $100$ and $200$ and the plots represent how the fraction failing decrease as $k$ grows, in an average of $20$ simulations for each step. We can see that the fraction failing is always decreasing if we increase the probability of the banking system to put more effort into loans to customers instead of other banks. This is expected since bank connections will not be strong enough to reach the failure conditions when $k$ is high. Another observation to be made is that fraction failing decrease is slower when $a$ is low, moreover the fraction failing seems to be lower for initial values of $k$ but at some point the trend is always inverted. From this we can infer that a core-periphery structure is safer when loaning attitudes are heterogeneous, but it is unsafe when the general approach is based on loans to customers that will generate large losses due to the recovery rate. We can also say that a core-periphery model is harder to make safe if compared to a more homogeneous one. All these observations hold for every dimension of network analyzed in our experiments, in fact dynamics highlighted by the experiments remain evident. When we increase $N$ the only notable difference is a transversal decrease of the fraction failing, from this we can say that larger systems are more resistant to failure contagion.

\begin{figure}[htbp]
    \centering
	\includegraphics[width=.8\textwidth]{images/ff50.png}
	\caption{Plots of failing fraction when ranges change as in~(\ref{eq:exp1}), in a bank network of size $N=50$}\label{fig:exp4}
\end{figure}
\begin{figure}[htbp]
    \centering
	\includegraphics[width=.8\textwidth]{images/ff100.png}
	\caption{Plots of failing fraction when ranges change as in~(\ref{eq:exp1}), in a bank network of size $N=100$}\label{fig:exp5}
\end{figure}
\begin{figure}[htbp]
    \centering
	\includegraphics[width=.8\textwidth]{images/ff200.png}
	\caption{Plots of failing fraction when ranges change as in~(\ref{eq:exp1}), in a bank network of size $N=200$}\label{fig:exp6}
\end{figure}

We also run another experiment where we consider different intervals. This time we consider more homogeneous bank systems choosing the following ranges
\begin{equation}\label{eq:exp2}
	\beta_i^k \in \left[ k \cdot s\ ; (k+1) \cdot s \right]
\end{equation}
again for $k \in [0..niter-1]$ and the step size $s = (1-\rho_i) / niter$. The main difference from the previous experiment consists in analyzing specific intervals of the uniform partition, instead of cumulating intervals. Results depicted in \Cref{fig:exp4,fig:exp5,fig:exp6} show that central intervals easily lead to an higher failing fraction, while we have safer structures with extremely low or high ranges. This means that using mid intervals banks are more sensible to both kind of losses, due to the recovery rate and to not repaid loans. If we consider a system where we isolate one of these two causes we have a lower failing fraction, we can see this in the extreme left and right of every plot. When we change $a$ we notice that core-periphery structures suffer from less fluctuations with respect with more distributed model, then we can say that centric structures are more resistant to interval variations but in the extremes they do not achieve the best results, indeed they suffer recovery rate losses more than structures generated with high $a$. Results do not change when we change the size of the network.

\begin{figure}[htbp]
    \centering
	\includegraphics[width=.8\textwidth]{images/ff50-2.png}
	\caption{Plots of failing fraction when ranges change as in~(\ref{eq:exp2}), in a bank network of size $N=50$}\label{fig:exp1}
\end{figure}
\begin{figure}[htbp]
    \centering
	\includegraphics[width=.8\textwidth]{images/ff100-2.png}
	\caption{Plots of failing fraction when ranges change as in~(\ref{eq:exp2}), in a bank network of size $N=100$}\label{fig:exp2}
\end{figure}
\begin{figure}[htbp]
    \centering
	\includegraphics[width=.8\textwidth]{images/ff200-2.png}
	\caption{Plots of failing fraction when ranges change as in~(\ref{eq:exp2}), in a bank network of size $N=200$}\label{fig:exp3}
\end{figure}

% section experiments (end)

\section{Implementation} % (fold)
\label{sec:implementation}

The source code used for running simulations is completely available on \url{https://github.com/marcotinacci/interbank-lending-systemic-risk}. The code is completely written in python $2.7$, using the \texttt{Spyder} IDE for Mac. The mainly used package is NetworkX~\cite{networkx}, PyGraphviz~\cite{pygraphviz} has also been used for some graphical plot.

\texttt{Network.py} is the module dedicated to the construction of the network. \texttt{Graph} and \texttt{DiGraph}, respectively undirected and directed graph, are the main data structures employed in this code and they are provided by \texttt{networkx}. Method \texttt{initGraph(N,a)} returns an undirected graph whose nodes' information are generated randomly as described in Section~\ref{sec:experiments}, its parameters are the network dimension $N$ and the power law exponent $a$. \texttt{initGraphMod(N,a,k,niter)} is the generalized version of the previous method in which we allow the proposed range refinement. These two functions rely on the \texttt{expected\_degree\_graph(exp\_degree,selfloops} method provided by \texttt{networkx} that implements the Chung Lu model mentioned in Section~\ref{sec:model}. Method \texttt{Graph2DiGraph(g)} is used to convert the undirected graph to a directed one, then, with \texttt{WeightedEdges(g)} it is also possible to compute the amount of every interbank loan and to assign it to the respective edge as its weight. \texttt{UpdatedAssets(g)} is used to implement the initial adjustment of assets due to generated interbank loans. Used in sequence, these four methods allow the generation of the desired network.

Module \texttt{Contagion.py} is dedicated to triggering and spreading the failure reaction. \texttt{contagion(init,dg)} is the initial method that exogenously set the equity of the $init$ node to zero and start the failing procedure, $dg$ is the directed graph representing the bank network. \texttt{windup(b,dg)} is the call in procedure applied to node $b$, firstly it recovers loans from customers applying the recovery rate, then it calls interbank loans in and apply eventual losses. If the conditions are met it also triggers the failure procedure to other nodes. \texttt{repay(b,dg)} is the dual function that handle the second phase in which debts of bank $b$ are restored. Again, if the condition for the failure of a creditor are met according to the default mechanism, then the method can call the failure procedure on it. \texttt{failure(b,dg)} gather the last two methods to provide a standard failure procedure on node $b$.

We have then other secondary modules: \texttt{Plot.py} gather customized plot functions, \texttt{Measures.py} contains some measure of interest, \texttt{Statistics.py} contains the implementation of a power law distribution as a customization of the continuous random variable from package \texttt{scipy.stats}, \texttt{Test.py} contains checks and finally \texttt{main.py} and \texttt{Experiments.py} are the scripts used to glue everything together and to obtain final results.

% section implementation (end)

\bibliographystyle{plain}
\bibliography{biblio}

\end{document}


\section{Uncertainties in $\chi(k)$}
\begin{frame}\frametitle{Propagation of uncertainties in $\chi(k)$}

\begin{cenpage}{95mm}

  Estimating uncertainties in  $\chi(k)$ has always been a challenge.

  \vmm

  We have (by default) estimated the uncertainty in $\chi(k)$ as
  {\BlueEmph{white noise}} {\tiny{(Newville, Boyanov, and Sayers, {\emph{J
          Synch Rad}}, 1999)}}, using $\chi(R)$ between [15, 25] \AA.

\end{cenpage}

\begin{columns}
  \begin{column}[T]{60mm}

    {\onslide+<1->  \includegraphics[width=60mm]{figs/errors/feo_chir}  }

  \end{column}
  \begin{column}[T]{60mm}

    {\onslide+<2-> \includegraphics[width=60mm]{figs/errors/feo_chir_logscale} }

  \end{column}
\end{columns}

{\onslide+<3->

  \begin{cenpage}{105mm}

    The ``high-R'' portion of $\chi(R)$ can estimate the
    ``white noise'' in the data pretty well.

    \vmm
    This is easy to do, but we know it misses an important component:
  \end{cenpage}

  \begin{postitbox}{53mm}
      \RedEmph{uncertainties from  background subtraction}
  \end{postitbox}

}

\end{frame}

\begin{frame}\frametitle{ Uncertainties in $\chi(k)$ from background subtraction}

\vmm
\begin{cenpage}{105mm}

  We can propagate the uncertainties from the fit of the background spline
  to estimate the uncertainty in $\chi(k)$ from the background subtraction.

  \vmm \vmm

  This is {\BlueEmph{not white noise}}.   In fact, it tends to have a peak
  somewhat above $2 R_{\rm bkg}$
\end{cenpage}

\begin{columns}
  \begin{column}[T]{60mm}

    {\only<1> { \includegraphics[width=60mm]{figs/errors/feo_chik_deltachi}}}

    {\only<2,3> { \includegraphics[width=60mm]{figs/errors/feo_deltachik_rbkg} }}

    \end{column}

    \begin{column}[T]{60mm}

      {\only<1>{ \includegraphics[width=60mm]{figs/errors/feo_chir_deltachi} }}
      {\only<2,3>{ \includegraphics[width=60mm]{figs/errors/feo_deltachir_rbkg} }}

    \end{column}

\end{columns}

\vmm

{\onslide+<3-> {

\begin{cenpage}{105mm}

    Using this $\delta\chi(k)$ array typically reduces the fit $\chi^2$
    statistic by $\sim 3\times$ or even more.

    \vmm

    I think this might need more investigation\ldots

\end{cenpage}

}}

\end{frame}


\subsection{Temporal processing}
In addition to the harmoniser stages, the temporal processing includes a multi-tap delay, low-pass filters, random amplitude ramping and a diffusion stage. These can be seen in fig~\ref{fig:struct}.

The multi-tap delay line simulates the delays between the players caused by the finite speed of sound \cite{dudas}. We are assuming 340$\frac{m}{s}$. We calculate the delay for the distances of the players, with the distances varying from just over a metre to several metres. The more delayed channels correspond to players at the back of the section. There are infact two delay stages with equal delays. The first simulates the delay for the hearing the section leader, and the second simulates the delay for the sound traveling from the back of the section to the front.

The low-pass filters simulate absorption in the air in a slightly exaggerated manner \cite{dudas}. They are implemented as simple first-order finite impulse response filters. The more delayed paths are filtered more, since they correspond to sounds which travel over a larger distance.

The random amplitude ramping simulates the dynamic variation which human players do naturally. The randomness helps remove audible periodicity \cite{dudas}. The amplitude ramping is done with simple gains which go back-and-forth linearly between two values during randomised time intervals. The more delayed paths, which correspond to worse players, have a higher difference between the maximum and minimum gains, but lower overall volume.

The diffusion stage is used to get a fuller sound. It is based on a hadamard matrix, which is often used in feedback delay networks for the same purpose. \cite{Jot} Here, however, there is no feedback.

Dudas does not relate the diffusion stage to orchestral section modeling. However, we propose the following: since the players hear eachother's pitches all at the same time, they cannot distinguish the one perfectly in-tune reference. The first stage of harmonisers which is then diffused provides the reference pitch, which is already a little ambiguous. Then the second stage of harmonisers is the players' attempt to follow this reference.


\chapter{Polyhedral Model}
\label{polyhedric:polyhedric}

	This chapter reffers to the project PolyhedralModel.

	\section{What can I find in this project ?}
	\label{polyhedric:insideproject}

	The project come with a doxygen documentation:

	\$ cd projects/PolyhedralModel/docs \\
	\$ doxygen doxy.conf

	\section{What I need to compile this project with Rose ?}
	\label{polyhedric:getstarted}

	To enable the Polyhedral Model project, you need PPL (Parma Polyhedra Library) (May 2011: version 0.11.2). The configure options are: \\
	\emph{--enable-ppl --with-ppl=[path to PPL]}\\

	Code generation is one of the main future of this project. It needs the library Cloog (Chunky Loop Generator) (May 2011: version 0.16.2). The configure options are: \\
	\emph{--enable-cloog --with-cloog=[path to Cloog]}\\

	Some other works depend on ScopLib and Candl, they can be enable by the same kind of options.

	\section{And now a few maths !}
	\label{polyhedric:maths}

More about "polyhedral representation of dependences".

		\subsection{Definitions and Notations}
		\label{polyhedric:maths:defs}

			\subsubsection{Integer Polyhedron}
			\label{polyhedric:maths:defs:polyhedron}

A integer polyhedron (which would be the only polyhedron that we will consider) is a set of integers points.

$\forall n \in \mathbb{N}, \mathbb{P} = \{\bar{z} \in \mathbb{Z}^{n} | \mathcal{P}(\bar{z}) \geq 0 \}$ where:
\begin{itemize}
	\item $n$ is the dimension of the polyhedron
	\item $\mathcal{P}: \mathbb{Z}^{n} \rightarrow \mathbb{Z}^{m}$ is an affine application
	\item $m$ is the number of intersected half-space
\end{itemize}
($\bar{z} \geq 0$ means $\forall i, z_{i} \geq 0$)

			\subsubsection{Notation}
			\label{polyhedric:maths:defs:notation}

\begin{itemize}
	\item $ \Box $ is the vector concatenation operator;
	\item $ 1_{i} $ is a vector with all component equal to zero but the $i^{th}$;
	\item $ Row_{i}^{\mathbb{P}}(\bar{z}) \geq 0 $: the $i^{th}$ inequalitie that define the polyhedron $\mathbb{P}$.\\
	$Row_{i}^{\mathbb{P}}(\bar{z}) = \mathcal{P}(\bar{z}).1_{i}$ where $\mathcal{P}$ is the affine application associate
	to $\mathbb{P}$.
\end{itemize}
		\subsection{Program}
		\label{polyhedric:maths:}

All considered programs are on the scope of Affine Control Loop (ACL), follow some notation:
\begin{itemize}
	\item $\mathbb{G}$ the set of Global variables: scalars from global scope (ex: array size).\\
	We will note $\bar{g}$ an element of $\mathbb{G}$.
	\item A set of statement $\mathbb{S}$, define by:
	\begin{itemize}
		\item An iteration domain: $\mathbb{D} = \{\bar{z} | f(\bar{z}, \bar{g}) \geq 0\}$ where f is an affine
				form $f: \mathbb{Z}^{n} \times \mathbb{G} \rightarrow \mathbb{Z}$ where n is the number
				of iterators.
		\item Two set of variables: read and write.\\
		A variable can be a scalar or an array element. In second case, the access vector to this element need to
		be an affine application $f: \mathbb{D} \times \mathbb{G} \rightarrow \mathbb{Z}^{dim(array)}$.\\
		NB: for a pseudo-multi-dimensionnal array, index needs to be
		$\sum_{i=1}^{dim(array)} f_{i} \times \prod_{j=1}^{i} size(dim_{j})$
	\end{itemize}
\end{itemize}

		\subsection{Dependence Graph}

We consider the graph of dependence $\Gamma$, in this directed graph, nodes are statement and vertex are dependence.
Vertex are qualified with the conditions of the dependence.\\
Given two statements $S_{1}$ and $S_{2}$ with $\mathbb{D}_{1}$ and $\mathbb{D}_{2}$ theirs iterations domains.
A vertex from $S_{1}$ to $S_{2}$ will be qualified by:
\begin{itemize}
	\item a condition $q_{1 \rightarrow 2}(\bar{z}, \bar{g}) \geq 0$ where $q_{1 \rightarrow 2}$ is an affine application
		$q_{1 \rightarrow 2}: \mathbb{D}_{1} \times \mathbb{G} \rightarrow \mathbb{Z}^{n}$ where is the number of
		sub-conditions.
	\item an affine application $f_{1 \rightarrow 2} : \mathbb{D}_{1} \rightarrow \mathbb{D}_{2}$
\end{itemize}
This mean that the statement $S_{1}$ at iteration point $\bar{z} \in \mathbb{D}_{1}$ depends on $S_{2}$ at iteration point
$f_{1 \rightarrow 2}(\bar{z}) \in \mathbb{D}_{1}$ if $q_{1 \rightarrow 2}(\bar{z}, \bar{g}) \geq 0$.\\

		\subsection{Polyhedron associate to a dependence}

An affine application $f: \mathbb{E}_{1} \rightarrow \mathbb{E}_{2}$ can be represent as a polyhedron
$\mathbb{P} \subset \mathbb{E}_{1} \times \mathbb{E}_{2}$.\\
Given $\mathbb{P}_{1 \rightarrow 2} \subset \mathbb{Z}^{n_{1} + n_{2} + n_{g}}$ where $n_{1}$ and $n_{2}$ and $n_{g}$ are
respectively the number of iterators of $S_{1}$ and $S_{2}$, and the number of globals.\\

$\mathbb{P}_{1 \rightarrow 2}$ will be the polyhedron associate to the dependence between $S_{1}$ and $S_{2}$ iff:
%\begin{center}
%$\forall (\bar{z_{1}}, \bar{z_{2}}, \bar{g}) \in \mathbb{D}_{1} \times \mathbb{D}_{2} \times \mathbb{G}$
%
%( $q_{1 \rightarrow 2}(\bar{z_{1}}, \bar{g}) \geq 0 \wedge \bar{z_{2}} = f_{1 \rightarrow 2}(\bar{z_{1}})$ )
%
%$\Leftrightarrow$
%
%$(\bar{z_{1}}, \bar{z_{2}}, \bar{g}) \in \mathbb{P}_{1 \rightarrow 2}$
%\end{center}
%An other version of $\mathbb{P}_{1 \rightarrow 2}$, more restrictive, that consider also the domains (this one is use ):
\begin{center}
$\forall (\bar{z_{1}}, \bar{z_{2}}, \bar{g}) \in \mathbb{Z}^{n_{1} + n_{2} + n_{g}}$

($\bar{z_{1}} \in \mathbb{D}_{1} \wedge \bar{z_{2}} \in \mathbb{D}_{2} \wedge \bar{g} \in \mathbb{G} \wedge q_{1 \rightarrow 2}(\bar{z_{1}}, \bar{g}) \geq 0 \wedge \bar{z_{2}} = f_{1 \rightarrow 2}(\bar{z_{1}})$ )

$\Leftrightarrow$

$ \bar{z_{1}} \,\Box\, \bar{z_{2}} \,\Box\, \bar{g} \in \mathbb{P}_{1 \rightarrow 2}$
\end{center}

		\subsection{Example}
		\label{polyhedric:maths:example}

\begin{figure}[!h]
\begin{lstlisting}[language=C, numbers=left]
for (i = 0; i < n; i++) {
    s[i] = 0
    for (j = 0; j < n; j++) {
        s[i] = s[i] + a[i][j] * x[j]
    }
}
\end{lstlisting}
\caption{A sample program}
\label{polyhedric:sampleprogram}
\end{figure}

			\subsubsection{Program}

Consider the small program of figure \ref{polyhedric:sampleprogram}. It contains a reference to one global variable
$\mathbb{G} = \{ n \}$ and we have two statement $S_{1}$ and $S_{2}$, respectively at line 2 and 4, with associated domains:

$\mathbb{D}_{1} = \{ i | 0 \leq i \leq n - 1 \}$

$\mathbb{D}_{2} = \{ i, j | 0 \leq i, j \leq n - 1 \}$

			\subsubsection{Dependences}

A dependence analysis (from FADAlib for example) would give us:
\begin{itemize}
	\item $S_{1}$ has no dependence (it never read a variable)
	\item $S_{2}$ has dependence at iteration point $( i, j )$:
	\begin{itemize}
		\item on $S_{1}$ at iteration point $( i )$: if $j = 0$
		\item on $S_{2}$ at iteration point $( i, j - 1 )$: if $j \geq 1$
	\end{itemize}
\end{itemize}

So, the dependence graph is composed from two nodes $\{S_{1}, S_{2}\}$ and two edges:
\begin{itemize}
	\item $S_{2} \rightarrow S_{1}$
	\begin{itemize}
		\item $q_{S_{2} \rightarrow S_{1}}(i, j, n) = (j, -j)$
		\item $f_{S_{2} \rightarrow S_{1}}(i, j) = (i)$
	\end{itemize}
	\item $S_{2} \rightarrow S_{2}$
	\begin{itemize}
		\item $q_{S_{2} \rightarrow S_{2}}(i, j, n) = j - 1$
		\item $f_{S_{2} \rightarrow S_{2}}(i, j) = (i, j - 1)$
	\end{itemize}
\end{itemize}

			\subsubsection{Polyhedron}

We have two dependances, then two polyhedrons
\begin{itemize}
	\item $\mathbb{P}_{S_{2} \rightarrow S_{1}}$ associate with the following affine application:\\
	$\mathcal{P}_{S_{2} \rightarrow S_{1}}: \mathbb{Z}^{2 + 1 + 1} \rightarrow  \mathbb{Z}^{4 + 2 + 2 + 2} $\\
	$$\mathcal{P}_{S_{2} \rightarrow S_{1}}(i, j, i', n) =
	\begin{pmatrix}
		1  & 0  & 0  & 0 \\
		-1 & 0  & 0  & 1 \\
		0  & 1  & 0  & 0 \\
		0  & -1 & 0  & 1 \\
		0  & 0  & 1  & 0 \\
		0  & 0  & -1 & 1 \\
		0  & 1  & 0  & 0 \\
		0  & -1 & 0  & 0 \\
		1  & 0  & -1 & 0 \\
		-1 & 0  & 1  & 0 \\
	\end{pmatrix}
	\times
	\begin{pmatrix}
		i  \\
		j  \\
		i' \\
		n  \\
	\end{pmatrix}
	+
	\begin{pmatrix}
		0 \\
		1 \\
		0 \\
		1 \\
		0 \\
		1 \\
		0 \\
		0 \\
		0 \\
		0 \\
	\end{pmatrix}$$
	\item $\mathbb{P}_{S_{2} \rightarrow S_{2}}$ associate with the following affine application:\\
	$\mathcal{P}_{S_{2} \rightarrow S_{2}}: \mathbb{Z}^{2 + 2 + 1} \rightarrow  \mathbb{Z}^{4 + 4 + 1 + 4} $\\
	$$\mathcal{P}_{S_{2} \rightarrow S_{2}}(i, j, i', j', n) =
	\begin{pmatrix}
		1  & 0  & 0  & 0  & 0  \\
		-1 & 0  & 0  & 0  & 1  \\
		0  & 1  & 0  & 0  & 0  \\
		0  & -1 & 0  & 0  & 1  \\
		0  & 0  & 1  & 0  & 0  \\
		0  & 0  & -1 & 0  & 1  \\
		0  & 0  & 0  & 1  & 0  \\
		0  & 0  & 0  & -1 & 1  \\
		0  & 1  & 0  & 0  & 0  \\
		1  & 0  & -1 & 0  & 0  \\
		-1 & 0  & 1  & 0  & 0  \\
		0  & 1  & 0  & -1 & 0  \\
		0  & -1 & 0  & 1  & 0  \\
	\end{pmatrix}
	\times
	\begin{pmatrix}
		i  \\
		j  \\
		i' \\
		j' \\
		n  \\
	\end{pmatrix}
	+
	\begin{pmatrix}
		0  \\
		-1 \\
		0  \\
		-1 \\
		0  \\
		-1 \\
		0  \\
		-1 \\
		-1 \\
		0  \\
		0  \\
		-1 \\
		1  \\
	\end{pmatrix}$$
\end{itemize}

	\section{Generating affine schedule with polyhedric model}
	\label{polyhedric:schedule}

		\subsection{What is this ?}

	Says  that an operation is the execution of statement $S \in \mathbb{S}$ at an iteration point
$\bar{z} \in \mathbb{D}_{S}$, we will note it $<\!S, \bar{z}\!>$.\\
	A schedule $\Theta$ will associate a date to an operation $<\!S, \bar{z}\!>$. An affine schedule is a set (one
by statement) of affine applications $\Theta_{S}: \mathbb{D}_{S} \times \mathbb{G} \rightarrow \mathbb{Z}^{n}$ where $n$
is the dimension of the schedule (date).

	We look for 1-dimensional schedules only. In this case:
$$\Theta_{S}(\bar{z}, \bar{g}) = \bar{\alpha_{S}} . \bar{z} + \bar{\beta_{S}} . \bar{g} + \kappa_{S}$$

		\subsection{Generating the set of Valid Schedule}
		\label{polyhedric:schedule:vss}

			\subsubsection{Valid Schedule ?}
			\label{polyhedric:schedule:vss:vss}

	A valid schedule is a schedule that respect the dependences of a program. So, one of such schedule need to respect:
	$$ \forall (S_{1}, S_{2}) \in \mathbb{S}, \; \mathbb{P}_{S_{2} \rightarrow S_{1}} \neq \emptyset $$
	$$ \implies $$
	$$ \forall \, \bar{z_{2}} \,\Box\, \bar{z_{1}} \,\Box\, \bar{g} \in \mathbb{P}_{S_{2} \rightarrow S_{1}}, \;
	\Theta_{S_{1}}(\bar{z_{1}}, \bar{g}) + \Delta_{S_{1}} \leq \Theta_{S_{2}}(\bar{z_{2}, \bar{g}}) $$
	$$ \implies $$
	$$ \forall \, \bar{z_{2}} \,\Box\, \bar{z_{1}} \,\Box\, \bar{g} \in \mathbb{P}_{S_{2} \rightarrow S_{1}}, \;
	\Theta_{S_{2}}(\bar{z_{2}, \bar{g}}) - \Theta_{S_{1}}(\bar{z_{1}}, \bar{g}) - \Delta_{S_{1}} \geq 0 $$
	where:
\begin{itemize}
	\item $ \mathbb{P}_{S_{2} \rightarrow S_{1}} $ is the polyhedron associated to the dependence from $S_{2}$ to $S_{1}$,
	empty if this dependence do not exist;
	\item $ \Delta_{S_{1}} $ is the latence of the $S_{1}$ statement execution, considered equal to 1 in following
	devellopement.
\end{itemize}

			\subsubsection{Farkas algorithm}
			\label{polyhedric:schedule:vss:farkas}

	\paragraph{Theorem:} Affine Form of Farkas Lemma

\begin{quote}
	Let $\mathbb{P}$ be a nonempty polyhedron defined by $p$ affine inequalities:
	$$ \bar{z} \in \mathbb{P} \iff \forall k \in [\![ 1, p ]\!], a_{k}.\bar{z} + b_{k} \geq 0 $$

	Then an affine form $\psi$ is non negative everywhere in $\mathbb{P}$ iff it is a positive affine combination:

	$$ \forall \bar{z} \in \mathbb{P}, \psi(\bar{z}) \geq 0 $$
	$$ \iff $$
	$$ \forall k \in [\![ 0, p ]\!], \; \exists \lambda_{k} \geq 0 \;\text{such as}\;
	\forall \bar{z} \in \mathbb{P}, \; \psi(\bar{z}) = \lambda_{0} + \sum_{k = 1}^{p} \lambda_{k}.(a_{k}.\bar{z} + b_{k}) $$
\end{quote}

	With the valid schedule definition and the above theorem, we have:
	$$ \forall (S_{1}, S_{2}) \in \mathbb{S}, \; \mathbb{P}_{S_{2} \rightarrow S_{1}} \neq \emptyset $$
	$$ \implies $$
	$$ \forall k \in [\![ 0, p ]\!], \; \exists \lambda_{k} \geq 0 \;\text{such as}\;
	\forall \; \bar{z_{2}} \,\Box\, \bar{z_{1}} \,\Box\, \bar{g} \in \mathbb{P}_{S_{2} \rightarrow S_{1}}, \; $$
	$$ \Theta_{S_{2}}(\bar{z_{2}, \bar{g}}) - \Theta_{S_{1}}(\bar{z_{1}}, \bar{g}) - 1 =
	\lambda_{0} + \sum_{k = 1}^{p} \lambda_{k} . Row_{k}^{\mathbb{P}}(\bar{z_{2}} \,\Box\, \bar{z_{1}} \,\Box\, \bar{g}) $$

			\subsubsection{Example}
			\label{polyhedric:schedule:vss:example}

	We still consider the small program of figure \ref{polyhedric:sampleprogram}.\\
	Firstly, we need to remove from polyhedron some redondante constraints:
$$\mathcal{P}_{S_{2} \rightarrow S_{1}}(i, j, i', n) =
	\begin{pmatrix}
		1  & 0  & 0  & 0 \\
		-1 & 0  & 0  & 1 \\
		0  & 1  & 0  & 0 \\
		0  & -1 & 0  & 0 \\
		1  & 0  & -1 & 0 \\
		-1 & 0  & 1  & 0 \\
	\end{pmatrix}
	\times
	\begin{pmatrix}
		i  \\
		j  \\
		i' \\
		n  \\
	\end{pmatrix}
	+
	\begin{pmatrix}
		0  \\
		-1 \\
		0  \\
		0  \\
		0  \\
		0  \\
	\end{pmatrix}$$
$$\mathcal{P}_{S_{2} \rightarrow S_{2}}(i, j, i', j', n) =
	\begin{pmatrix}
		1  & 0  & 0  & 0  & 0  \\
		-1 & 0  & 0  & 0  & 1  \\
		0  & -1 & 0  & 0  & 1  \\
		0  & 1  & 0  & 0  & 0  \\
		1  & 0  & -1 & 0  & 0  \\
		-1 & 0  & 1  & 0  & 0  \\
		0  & 1  & 0  & -1 & 0  \\
		0  & -1 & 0  & 1  & 0  \\
	\end{pmatrix}
	\times
	\begin{pmatrix}
		i  \\
		j  \\
		i' \\
		j' \\
		n  \\
	\end{pmatrix}
	+
	\begin{pmatrix}
		0  \\
		-1 \\
		-1 \\
		-1 \\
		0  \\
		0  \\
		-1 \\
		1  \\
	\end{pmatrix}$$

	We also consider the following expressions for $\Theta_{S_{1}}$ and $\Theta_{S_{2}}$:
$$\Theta_{S_{1}}(i, n) = \alpha_{S_{1}}^{1} \times i + \beta_{S_{1}}^{1} \times n + \kappa_{S_{1}}$$
$$\Theta_{S_{2}}(i, j, n) = \alpha_{S_{2}}^{1} \times i + \alpha_{S_{2}}^{2} \times j + \beta_{S_{2}}^{1} \times n + \kappa_{S_{2}}$$

	The application of Farkas Lemma give us:
\begin{itemize}
	\item for $S_{2} \rightarrow S_{1}$:
	$$ \Theta_{S_{2}}(i, j, n) - \Theta_{S_{1}}(i', n) - 1 =
	\lambda_{0}^{S_{2} \rightarrow S_{1}} + \sum_{k = 1}^{6} \lambda_{k}^{S_{2} \rightarrow S_{1}} . Row_{k}^{\mathbb{P}}(i, j, i', n) $$
	$$ \left\{
	\begin{array}{l}
	\begin{array}{rcl}
			\alpha_{S_{2}}^{1} & = &
				\lambda_{1}^{S_{2} \rightarrow S_{1}}
				- \lambda_{2}^{S_{2} \rightarrow S_{1}}
				+ \lambda_{5}^{S_{2} \rightarrow S_{1}}
				- \lambda_{6}^{S_{2} \rightarrow S_{1}}\\
			\alpha_{S_{2}}^{2} & = &
				\lambda_{3}^{S_{2} \rightarrow S_{1}}
				- \lambda_{4}^{S_{2} \rightarrow S_{1}} \\
			- \alpha_{S_{1}}^{1} & = &
				\lambda_{6}^{S_{2} \rightarrow S_{1}}
				- \lambda_{5}^{S_{2} \rightarrow S_{1}} \\
			\beta_{S_{2}}^{1} - \beta_{S_{1}}^{1} & = &
				\lambda_{2}^{S_{2} \rightarrow S_{1}} \\
			\kappa_{S_{1}} - \kappa_{S_{2}} - 1 & = &
				\lambda_{0}^{S_{2} \rightarrow S_{1}}
				- \lambda_{2}^{S_{2} \rightarrow S_{1}}
		\end{array}\\
		\forall k \in [\![ 0, 6 ]\!], \quad \lambda_{k}^{S_{2} \rightarrow S_{1}} \geq 0
	\end{array}
	\right. $$
	\item for $S_{2} \rightarrow S_{2}$:
	$$ \Theta_{S_{2}}(i, j, n) - \Theta_{S_{2}}(i', j', n) - 1 =
	\lambda_{0}^{S_{2} \rightarrow S_{2}} + \sum_{k = 1}^{6} \lambda_{k}^{S_{2} \rightarrow S_{2}} . Row_{k}^{\mathbb{P}}(i, j, i', j', n) $$
	$$ \left\{
	\begin{array}{l}
		\begin{array}{rcl}
			 \alpha_{S_{2}}^{1} & = &
				\lambda_{1}^{S_{2} \rightarrow S_{2}}
				- \lambda_{2}^{S_{2} \rightarrow S_{2}}
				+ \lambda_{5}^{S_{2} \rightarrow S_{2}}
				- \lambda_{6}^{S_{2} \rightarrow S_{2}} \\
			\alpha_{S_{2}}^{2} & = &
				- \lambda_{3}^{S_{2} \rightarrow S_{2}}
				+ \lambda_{4}^{S_{2} \rightarrow S_{2}}
				+ \lambda_{7}^{S_{2} \rightarrow S_{2}}
				- \lambda_{8}^{S_{2} \rightarrow S_{2}} \\
			- \alpha_{S_{2}}^{1} & = &
				- \lambda_{5}^{S_{2} \rightarrow S_{2}}
				+ \lambda_{6}^{S_{2} \rightarrow S_{2}} \\
			- \alpha_{S_{2}}^{2} & = &
				- \lambda_{7}^{S_{2} \rightarrow S_{2}}
				+ \lambda_{8}^{S_{2} \rightarrow S_{2}} \\
			0 & = & \lambda_{2}^{S_{2} \rightarrow S_{2}}
				+ \lambda_{3}^{S_{2} \rightarrow S_{2}}  \\
			-1 & = & \lambda_{0}^{S_{2} \rightarrow S_{2}}
				- \lambda_{2}^{S_{2} \rightarrow S_{2}}
				- \lambda_{3}^{S_{2} \rightarrow S_{2}}
				- \lambda_{4}^{S_{2} \rightarrow S_{2}}
				- \lambda_{7}^{S_{2} \rightarrow S_{2}}
				+ \lambda_{8}^{S_{2} \rightarrow S_{2}}
		\end{array}\\
		\forall k \in [\![ 0, 8 ]\!], \quad \lambda_{k}^{S_{2} \rightarrow S_{2}} \geq 0
	\end{array}
	\right. $$
\end{itemize}

			\subsubsection{Dimension of the Valid Schedule space}


         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
         % Information sheet for the Matlab lab - Maths 6111 %
         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentclass[10pt]{article}
\input ma_no_html_header

\usepackage{color}
\usepackage{hyperref}
\hypersetup{breaklinks=true,colorlinks=true}
%\input ma_header
\setlength{\parindent}{0pt}
\pagestyle{myheadings}
% \markright{
% \protect {\protect \epsfxsize=0.2 true cm \protect \epsffile {dolph.line.eps}}
% \it Maths 3018/6111 - Numerical methods \hfill}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\thispagestyle{empty}
\begin{center}
\textbf{\Large Maths 3018/6111 - Numerical Methods \\*[8mm]
Worksheet 7}\\*[.8cm]
\end{center}

\section*{Theory}

\begin{enumerate}
\item Find the linear system that results from using central
  differencing to solve the elliptic equation
  \begin{equation*}
    u_{xx} + u_{yy} = \sin ( \pi y ) \left( 2 - 6 x - (\pi x)^2 (1 -
      x) \right).
  \end{equation*}
  Central differencing, trivial boundary conditions, and a $3 \times
  2$ grid should be used.
\item Explain the strategy for solving evolutionary PDEs using finite
  differencing methods.
\item Write out a FTCS method for the equations
  \begin{align*}
    \partial_t y - \partial_{xx} y & = \sin(x), & x & \in [0,1], &
    y(t, 0) & = 0 = y(t, 1), \\
    \partial_t y + \partial_{x} y & = e^{-(x-1/2)^2}, & x & \in [0,1], &
    y(t, 0) & = 0 = y(t, 1).
  \end{align*}
  In both cases the initial data should be taken to be $y(0, x) =
  f(x)$. Full details of the grid, the discrete initial and boundary
  conditions, and the discrete update algorithm should be given.
\item Using von Neumann analysis, find when the BTCS method is stable
  for the advection equation.
\end{enumerate}

\section*{Coding}

\begin{enumerate}
\item Implement the finite difference method above to solve the
  elliptic equation
  \begin{equation*}
    u_{xx} + u_{yy} = \sin ( \pi y ) \left( 2 - 6 x - (\pi x)^2 (1 -
      x) \right).
  \end{equation*}
  Check how it converges with resolution (the exact solution is $u =
  x^2 (1 - x) \sin(\pi y)$.
\item Implement a FTBS algorithm for the advection equation with
  periodic boundaries. Use a sine wave on $x \in [0,1]$ as initial
  data. Check how it converges with resolution.
\item{} [Additional] Apply the FTBS algorithm to the traffic flow equation
  \begin{equation*}
    \partial_t y + \partial_x \left( y ( 1 - y) \right) = 0.
  \end{equation*}
  Use initial data $y(0, x) = \exp(-x^2)$, $y(0, x) = 0.2 + 1.2 H(x)$
  and $y(0, x) = 1 - H(x)$ on $x \in [-10, 10]$ where $H(x)$ is the
  Heaviside step function. By comparing against the results for
  Burger's equation, explain your results. You may want to restrict to
  $t \leq 0.5$.
\end{enumerate}

\end{document}


         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
         % Information sheet for the Matlab lab - Maths 6111 %
         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentclass[10pt]{article}
\input ma_no_html_header

\usepackage{color}
\usepackage{hyperref}
\hypersetup{breaklinks=true,colorlinks=true}
%\input ma_header
\setlength{\parindent}{0pt}
\pagestyle{myheadings}
% \markright{
% \protect {\protect \epsfxsize=0.2 true cm \protect \epsffile {dolph.line.eps}}
% \it Maths 3018/6111 - Numerical methods \hfill}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\thispagestyle{empty}
\begin{center}
\textbf{\Large Maths 3018/6111 - Numerical Methods \\*[8mm]
Worksheet 2}\\*[.8cm]
\end{center}

\section*{Theory}

\begin{enumerate}
\item Perform the $LU$ decomposition of
  \begin{equation*}
    A =
    \begin{pmatrix}
      1 & 3 \\ 4 & 16
    \end{pmatrix}.
  \end{equation*}
  Use both standard factorisation methods.
\item{} [Additional] Write out the Thomas algorithm for a tridiagonal
  system.
\item Write down the general framework for iterative methods for
  linear systems. Give the convergence matrix. If the linear system
  uses the matrix $A$ above, will an iterative method converge? [Hint:
  remember what to do with the diagonal entries]
\item Check which of the matrices on this sheet are diagonally dominant.
\item Briefly explain what is meant by quadrature methods based on
  polynomial interpolation.
\item{} [3018 only] Write down the contraction mapping theorem. Check
  that $g(x) = \cos(x)$ is contracting on the unit interval.
\end{enumerate}

\section*{Coding}

\begin{enumerate}
\item Write a code to do $LU$ decomposition. Check the decomposition of
  \begin{equation*}
    B =
    \begin{pmatrix}
      64 & 8 & 48 \\
      24 & 28 & 53 \\
      32 & 49 & 91
    \end{pmatrix}.
  \end{equation*}
\item{} [Additional] Implement the Thomas algorithm for a tridiagonal
  system.
\item Implement the Jacobi method for linear systems. Investigate the
  behaviour of the method for the problem
  \begin{equation*}
    B {\bf x} =
    \begin{pmatrix}
      3 \\ 2 \\ 1
    \end{pmatrix}
  \end{equation*}
  starting from the trivial initial guess. Should this be expected?
  [Hint: Check the convergence theorem] Try instead the matrix
  \begin{equation*}
    C =
    \begin{pmatrix}
      \frac{119}{108} & -\frac{14}{27} & -\frac{8}{9} \\
      \frac{7}{54} & \frac{46}{27} & \frac{7}{9} \\
      \frac{5}{108} & \frac{1}{27} & \frac{23}{18}
    \end{pmatrix}.
  \end{equation*}
\item Implement the Gauss-Seidel method, applying it to the
  convergent problem above. Compare the convergence rate to Jacobi.
\item {} [3018 only] Implement the chord method to find the root of
  \begin{equation*}
    f(x) = \tan x - e^{-x}, \quad x \in [0,1].
  \end{equation*}
\end{enumerate}

\end{document}


\section{Stratified sampling by union of unstratified probability bounds}\label{section:old_statistics}

The minimisation of existing EBBs (such as Theorems \ref{MandPsEBB} and \ref{AudibertsEBB}) cannot be directly be used as a method of choosing samples in stratified sampling, since the population mean estimate does not have a sample variance but only the strata have sample variances. Such EBBs can be applied as a bound for the mean of an individual stratum but cannot bind the aggregation of the mean estimates of all the strata.
In this section, we show that these EBBs can be combined by probability unions to create a bound for the error of the stratified mean estimate, which can then be minimised to create new stratified sampling methodologies.
The combined probability bound will not depend on the variance of the strata, but only depend on the sample variances of the strata.
And thus the resulting sampling methodologies will implicitly be sensitive to the variances of the strata but not requiring foreknowledge of them - unlike Neyman sampling.
%And thus it will be possible to conduct such sampling in a single step, rather than in two steps - as is necessary in Neyman sampling.

\subsection{Stratified sampling via EBBs and sequential unions}\label{section:unionising_ebbs}

To create a bound on the error in stratified sampling from EBBs, it is necessary to use probability unions to bind the EBBs together.
We derive the following two theorems \ref{triangle_theorem1} and \ref{triangle_theorem2} that give derivations on the error, and the absolute error, of the stratified mean estimator.

\begin{theorem}\label{triangle_theorem1}
If we have $m$ strata of sizes $N_i$. If we have taken $n_i$ samples $X_{i,1},X_{i,2},\dots,X_{i,n_i}$ from each stratum, resulting in a stratum sample mean $\hat{\mu}_i = \frac{1}{n_i}\sum_{j=1}^{n_i}X_{i,j}$ and stratum sample variance $\hat{\sigma}_i^2=\frac{1}{n_i}\sum_{j=1}^{n_i}(X_{i,j}-\hat{\mu}_i)^2 $.
If the error in the sample mean of a stratum is bounded by an Empirical Bernstein Bound:
$\quad \p(\hat{\mu}_i-\mu_i \ge Z(n_i,D_i,\hat{\sigma}^2_i,t)) \le t $\\
Then the error in our stratified estimation of the population mean itself is probability bounded:
$$ \p\left(\hat{\mu}-\mu \ge \sum_{i=1}^m\frac{N_i}{\sum_kN_k} Z(n_i,D_i,\hat{\sigma}^2_i,t/m)\right)\le t $$
\end{theorem}

\input{exerpts/old_statistics_equation.tex}

%\begin{proof}
%We begin by considering that the stratified mean estimate is given by:
%$$ \hat{\mu} = \sum_{i=1}^m\frac{N_i}{\sum_kN_k} \hat{\mu}_i ~~~~~~~\text{and thus:}~~~~~~~ \hat{\mu}-\mu = \sum_{i=1}^m\frac{N_i}{\sum_kN_k} (\hat{\mu}_i-\mu_i)$$
%Thence because we can scale (by positive factor) the inside of the EBB inequality (for any $j\in\{1,\dots, m\}$):
%\begin{equation}\label{equation_partial_union1} \p\left(\frac{N_j}{\sum_kN_k}(\hat{\mu}_j-\mu_j) \ge \frac{N_j}{\sum_kN_k}Z(n_j,D_j,\hat{\sigma}^2_j,t)\right) \le t \end{equation}
%adding identical terms to both sides of the inner inequality gives:
%\begin{equation}\label{equation_partial_union2} \p\left(\sum_{i=1}^m\frac{N_i}{\sum_kN_k}(\hat{\mu}_i-\mu_i) \ge \frac{N_j}{\sum_kN_k}Z(n_j,D_j,\hat{\sigma}^2_j,t) + \sum_{\substack{i=1 \\ i\ne j}}^m\frac{N_i}{\sum_kN_k}(\hat{\mu}_i-\mu_i)\right) \le t \end{equation}
%now since equation \ref{equation_partial_union1} also holds for any $l\in\{1,\dots, m\}$ other $j$ then:
%$$ \p\left(\frac{N_l}{\sum_kN_k}(\hat{\mu}_l-\mu_l) \ge \frac{N_l}{\sum_kN_k}Z(n_l,D_l,\hat{\sigma}^2_l,t)\right) \le t $$
%hence by adding terms to both sides of the inner inequality:
%\begin{equation}\label{equation_partial_union3} \p\left(\begin{matrix*}\frac{N_j}{\sum_kN_k}Z(n_j,D_j,\hat{\sigma}^2_j,t)\\ + \sum_{\substack{i=1 \\ i\ne j}}^m\frac{N_i}{\sum_kN_k}(\hat{\mu}_i-\mu_i)\end{matrix*} \ge \begin{matrix*} \frac{N_j}{\sum_kN_k}Z(n_j,D_j,\hat{\sigma}^2_j,t) +\\ \frac{N_l}{\sum_kN_k}Z(n_l,D_l,\hat{\sigma}^2_l,t) +\\ \sum_{\substack{i=1 \\ i\ne j\\i\ne l}}^m\frac{N_i}{\sum_kN_k}(\hat{\mu}_i-\mu_i)\end{matrix*}\right) \le t \end{equation}
%Applying probability union (lemma \ref{prob_union}) to equations \ref{equation_partial_union2} and \ref{equation_partial_union3} gives:
%$$ \p\left(\hat{\mu}-\mu \ge \begin{matrix*} \frac{N_j}{\sum_kN_k}Z(n_j,D_j,\hat{\sigma}^2_j,t) +\\ \frac{N_l}{\sum_kN_k}Z(n_l,D_l,\hat{\sigma}^2_l,t) +\\ \sum_{\substack{i=1 \\ i\ne j\\i\ne l}}^m\frac{N_i}{\sum_kN_k}(\hat{\mu}_i-\mu_i)\end{matrix*}\right) \le 2t $$
%We can repeat this process of taking equation \ref{equation_partial_union1} for a new index $q\ne j\ne l$, adding appropriate terms to the inner inequality and using probability union lemma \ref{prob_union}, gives:
%$$ \p\left(\hat{\mu}-\mu \ge \begin{matrix*} \frac{N_q}{\sum_kN_k}Z(n_q,D_q,\hat{\sigma}^2_q,t) +\\\frac{N_j}{\sum_kN_k}Z(n_j,D_j,\hat{\sigma}^2_j,t) +\\ \frac{N_l}{\sum_kN_k}Z(n_l,D_l,\hat{\sigma}^2_l,t) +\\ \sum_{\substack{i=1 \\ i\ne j\\i\ne l\\i\ne q}}^m\frac{N_i}{\sum_kN_k}(\hat{\mu}_i-\mu_i)\end{matrix*}\right) \le 3t $$
%and so on, and ultimately gives:
%$$ \p\left(\hat{\mu}-\mu \ge \sum_{i=1}^m\frac{N_i}{\sum_kN_k} Z(n_i,D_i,\hat{\sigma}^2_i,t)\right)\le mt $$
%And scaling $t$ gives result.
%\end{proof}

The result of this novel proof is an inequality bounding the error of the stratified mean estimate by the number of samples and the width and sample variance of each of the strata.
If instead of the error, we are concerned about the absolute error of the stratified estimate $|\hat{\mu}-\mu|$ then by a similar procedure (inspired by the utilisation of the triangle inequality in the proofs of \cite{2013arXiv1306.4265M}) we can derive a similar probability bound:

\begin{theorem}\label{triangle_theorem2}
In the same context of the statement of theorem \ref{triangle_theorem1} the absolute error of the stratified estimation is probability bounded:
$$ \p\left(|\hat{\mu}-\mu| \ge \sum_{i=1}^m\frac{N_i}{\sum_kN_k} Z(n_i,D_i,\hat{\sigma}^2_i,t/2m)\right)\le t $$
\end{theorem}
\begin{proof}
If $ \p(\hat{\mu}_i-\mu_i \ge Z(n_i,D_i,\hat{\sigma}^2_i,t)) \le t$ then
$ \p(|\hat{\mu}_i-\mu_i| \ge Z(n_i,D_i,\hat{\sigma}^2_i,t)) \le 2t$.\\
Then by repeated application of probability unions (similar to that process used in the proof of theorem \ref{triangle_theorem1}) we get:
\begin{equation}\label{partial_equation_part11} \p\left(\sum_{i=1}^m\frac{N_i}{\sum_kN_k} |\hat{\mu}_i-\mu_i| \ge \sum_{i=1}^m\frac{N_i}{\sum_kN_k} Z(n_i,D_i,\hat{\sigma}^2_i,t) \right) \le 2mt \end{equation}
Now, via the triangle inequality:
$$\hat{\mu}-\mu = \sum_{i=1}^m\frac{N_i}{\sum_kN_k} (\hat{\mu}_i-\mu_i) ~~~~~~~~~~\text{implies}~~~~~~~~~~ |\hat{\mu}-\mu| \le \sum_{i=1}^m\frac{N_i}{\sum_kN_k} |\hat{\mu}_i-\mu_i| $$
then $ \p(|\hat{\mu}-\mu| > \sum_{i=1}^m\frac{N_i}{\sum_kN_k}|\hat{\mu}_i-\mu_i|) \le 0 $ and by probability union with \eqref{partial_equation_part11}:
$$ \p\left(|\hat{\mu}-\mu| \ge \sum_{i=1}^m\frac{N_i}{\sum_kN_k} Z(n_i,D_i,\hat{\sigma}^2_i,t)\right)\le 2mt $$
And the result follows by scaling $t$.
\end{proof}

These two theorems make clear that sampling to minimise either the error or the absolute error, essentially amounts to minimising the same target.
It is possible to apply these theorems with a choice of EBB to create a bound for the stratified mean error, which can then be minimised by choosing appropriate sample numbers.

This method is made clear in pseudocode in technically-novel algorithm \ref{alg3}.
Where a scan is conducted over the possible strata in lines 4-10, and the improvement that would be offered by taking an additional sample from the stratum's respective EBB is calculated.
The strata that gives the most reduction of its EBB for a prospective additional sample is chosen to have an additional sample taken, and its sample variance is recalculated in lines 11-13; this process repeats until the sample budget is exhausted.
%This algorithm \ref{alg3} is much more simple than algorithm \ref{alg2} primarily because the scan through the strata only needs to consider the change in the EBB associated with that stratum.
%This is because Theorem \ref{triangle_theorem2} identifies that the bound on the stratified mean estimate is the sum of the EBBs of the strata, hence they can be considered as minimising factors in isolation.

\begin{algorithm}
\caption[Stratified Error bound reduction algorithm by unionised EBBs]{Stratified Error bound reduction algorithm by unionised EBBs - by Theorem \ref{triangle_theorem2}}
\label{alg3}
\begin{algorithmic}[1]
    \REQUIRE probability $t$, number of strata $N$, initial sample numbers $m_i$, initial stratum sample variances $\hat{\sigma}_i^2$, widths $D_i$, maximum sample budget $B$,
    for an EBB per the following form: $\p(\hat{\mu}_i-\mu_i \ge Z(m_i,D_i,\hat{\sigma}^2_i,t)) \le t $
    \WHILE{$\sum_k{m_k}<B$}
        \STATE $beststrata \leftarrow -1$
        \STATE $bestimprovement \leftarrow 0$
    	\FOR{$i=1$ to $N$}
    		\STATE $improvement \leftarrow \frac{n_i}{\sum_kn_k}\left(Z(m_i,D_i,\hat{\sigma}^2_i,t) - Z(m_i+1,D_i,\hat{\sigma}^2_i,t)\right)$
    	    \IF{$improvement > bestimprovement$}
    	        \STATE $beststrata \leftarrow i$
    	        \STATE $bestimprovement \leftarrow improvement$
    	    \ENDIF
    	\ENDFOR
    	\STATE take an extra sample from strata: $beststrata$
	    \STATE $m_{beststrata} \leftarrow m_{beststrata} + 1$
    	\STATE recalculate $\hat{\sigma}_{beststrata}^2$
    \ENDWHILE
\end{algorithmic}
\end{algorithm}

The numerical performance of this process in the context of stratified sampling is shown in section \ref{section:statistics_results}.
In the next section we derive and numerically generate a stronger EBB for application in the context of these theorems for stratified sampling.

\newpage
\section{The derivation of a stronger EBB}\label{section:new_EBB}\label{derivation}

There are various EBBs which place variance-sensitive bounds on the mean, and it remains an outstanding task is to see how much these EBBs can be improved; we take inspiration from the work of \cite{Maurer50empiricalbernstein} to develop a new and stronger EBB.
However, due to its analytic intractability, we complete the derivation by discussing how to numerically determine the bound.

%The evaluations in section \ref{} show that our EBB significantly tightens existing bounds.
%Specifically, our EBB can shrink the best existing EBBs by about a third. This represents half of the distance between the best existing EBBs and an unattainable Bernstein bound constructed with perfect variance information.

In this section, we derive two Chernoff bounds, for the sample mean and the mean of sample squares, (Theorem~\ref{sample_squares} and Lemma~\ref{variance2}, respectively).
These are fused using a probability union (Theorem \ref{prob_union}) and variance decomposition (Theorem \ref{variance1}) to derive a bound for the sample variance. This bound is then used to derive our new EBB, as presented in Theorem~\ref{ebb1}.

Within this section there are multiple parts:
\begin{enumerate}
\item	subsection \ref{subsection:bennetts_inequality} presents and provides a derivation of Bennett's inequality
\item	subsection \ref{subsection:sample_squares_bound} presents and derives a Chernoff bound on the error of the sample squares
\item	subsection \ref{subsection:sample_variance_bound} shows how these two can be used to create a bound on the sample variance
\item	subsection \ref{subsection:general_ebb_creation} shows how a bound on the sample variance and a bound on the mean can be used to derive an EBB
\item	subsection \ref{numerical-implementation} we give details on the numerical determination of a new EBB using all appropriate elements, and present a numerically fitted symbolic envelope over the numerical determinations.
\end{enumerate}

\subsection{A presentation of Bennett's inequality}\label{subsection:bennetts_inequality}

Our first part of the derivation of our new EBB is a Chernoff bound on the sample mean called \textit{Bennett's inequality}.
This bound is not new and was derived by \cite{hoeffding1} and \cite{10.2307/2282438} and has subsequently been a subject of discussion and many further developments; it is known to be quite strong \citep{Bentkus08boundsfor,Pinelis2014,zbMATH00812598};
We derive and present a refactored summary proof of Bennett's inequality as the theorem \ref{hoeffdings1}, whose proof involves the use of an intermediate theorem \ref{thm:parabola}.
We present it here in a way conducive to the manipulation we will subsequently perform with it (modifying it into proof of Theorem \ref{expectation1}).

\begin{theorem}[Bennett's inequality]\label{hoeffdings1}
Let $X$ be a real-valued random variable with a mean of zero and variance $\sigma^2$, that is bounded $a\le X\le b$.
Then for $t>0$, the mean $\hat{\mu}$ of $n$ samples of $X$ is probability bounded by:
\begin{equation}\label{eq_no2}\p(\hat{\mu}\ge t)\le H_1^n\left(\frac{\sigma^2}{b^2},\frac{t}{b}\right),
\end{equation}
where:
\begin{equation*}
H_1^n\left(\frac{\sigma^2}{b^2},\frac{t}{b}\right) =
\left(\left(\frac{\frac{\sigma^2}{b^2}}{\frac{\sigma^2}{b^2}+\frac{t}{b}}\right)^{\frac{\sigma^2}{b^2}+\frac{t}{b}}
\left(1-\frac{t}{b}\right)^{\frac{t}{b}-1}\right)^{\frac{n}{\frac{\sigma^2}{b^2}+1}}
\end{equation*}
\end{theorem}
\begin{proof}As random variable X is bounded $a\le X\le b$, for any $s>0$, by Lemma \ref{thm:parabola}, there exist parameters $\alpha,\beta,\gamma$ such that, $\alpha s^2X^2+\beta sX+\gamma\ge \exp(sX)$ is always satisfied, hence for these we have:
$$\E\left[\exp(sX)\right] \le \E[\alpha s^2X^2+\beta sX+\gamma] \le \alpha s^2\E[X^2]+\gamma \le \alpha  s^2\sigma^2+\gamma$$
$$ \le (\sigma^2\exp(sb) + b^2\exp(-s\sigma^2/b))(\sigma^2 + b^2)^{-1}$$
Hence by application of lemma \ref{chernoff1}:\\
$$\p(\hat{\mu}\ge t) \;\le\; (\sigma^2\exp(sb) + b^2\exp(-s\sigma^2/b))^n((\sigma^2 + b^2)\exp(st))^{-n}$$
minimising with respect to $s$ completes the proof, minimum $s$ occurs at:
$$ s = \frac{b}{\sigma^2 + b^2}\log\left(\frac{b(\sigma^2 + tb)}{\sigma^2(b-t)}\right) $$
\end{proof}

\begin{lemma}[Parabola Fitting]\label{thm:parabola}
For $b>0$, $a<b$ and $z>0$, there exists an $\alpha,\beta,\gamma$ such that: $\alpha x^2+\beta x+\gamma\ge \exp(x)$ for all $a\le x\le b$, and:
$$z\alpha+\gamma = (z\exp(b) + b^2\exp(-z/b))(z + b^2)^{-1}$$
\end{lemma}
\begin{proof}
A example parabola $\alpha x^2+\beta x+\gamma$ which that satisfies these requirements tangentially touches the exponential curve at one point (at $x=f<b$) and intersects it at another (at $x=b$), as illustrated in Figure \ref{fig:graph1}.
Thus the parabola's intersection at $x=b$ and its tangential intersection at $x=f$ can be written in matrix algebra:
$$
\begin{bmatrix}
    \alpha \\
    \beta \\
	\gamma
\end{bmatrix}
=
\begin{bmatrix}
    b^2 & b & 1 \\
    f^2 & f & 1 \\
	2f  & 1 & 0
\end{bmatrix}^{-1}
\begin{bmatrix}
    \exp(b) \\
    \exp(f) \\
	\exp(f)
\end{bmatrix}$$
This gives our parabola parameters $\alpha,\beta,\gamma$, in terms of $f$ and $b$, hence:
$$z\alpha+\gamma = (((z+fb-b)(f-b-1)-b)e^f+(f^2+z)e^b)(b-f)^{-2}$$
Minimizing with respect to $f$ occurs at $f=\frac{-z}{b}$ and gives the result.
\end{proof}

\input{figs/parabola_illustrations.tex}

The derivation of Bennett's inequality is separated into two parts as the above Lemma \ref{thm:parabola} will also be reused to derive a weaker but more mathematically manipulatable inequality later (as Lemma \ref{expectation1}).

The essential difference between Bennett's inequality and Hoeffding's inequality is the fitting of a parabola instead of a linear term over the exponential function.
And this second order parabolic term makes Bennett's inequality sensitive to the variance of the distribution whereas Hoeffding's inequality is not.
Bennett's inequality provides a probability bound for the difference of the sample mean from the true mean given the variance, however the true variance is not often known in practice.

\subsection{A Chernoff bound on the sample squares}\label{subsection:sample_squares_bound}

As already stated, the variance is often unknown in practice, but can only be estimated via the sample variance statistic.
In order to derive a bound for the error between the sample variance and the variance we derived a concentration inequality for the sample squares to use in conjunction with the variance decomposition Lemma \ref{variance1}.
The following novel concentration inequality was derived:

\begin{lemma}[Sample square bound]\label{sample_squares}
Let $X$ be a real-valued random variable with a mean of zero and variance $\sigma^2$, that is bounded $a\le X\le b$, if $d=\max(b,-a)$ then for $y>0$, the mean of sample squares $\hat{\sigma}_0^2=\frac{1}{n}\sum_ix_i^2$ is probability bounded:
\begin{equation}\label{equation_squares}\p(\sigma^2 - \hat{\sigma}_0^2> y) \le H_2^n\left(\frac{\sigma^2}{d^2},\frac{y}{d^2}\right),
\end{equation}
where:
\begin{align*} H_2^n\left(\frac{\sigma^2}{d^2},\frac{y}{d^2}\right) = \left(
\left(\frac{1-\frac{\sigma^2}{d^2}}{1+\frac{y}{d^2}-\frac{\sigma^2}{d^2}}\right)^{1+\frac{y}{d^2}-\frac{\sigma^2}{d^2}}
\left(\frac{\frac{\sigma^2}{d^2}}{\frac{\sigma^2}{d^2}-\frac{y}{d^2}}\right)^{\frac{\sigma^2}{d^2}-\frac{y}{d^2}}
\right)^n
\end{align*}
\end{lemma}

\begin{proof}
There exist parameters $\alpha,\gamma$ such for all $a\le X\le b$ that $\alpha X^2 + \gamma \ge \exp(-qX^2)$ whence:\\
$$\E[\exp(-qX^2)] \le \E[\alpha x^2 +\gamma] \le \alpha\sigma^2 + \gamma $$
With $d=\max(b,-a)$, we choose (see Fig \ref{fig:graph111}) $\alpha=(\exp(-qd^2)-1)d^{-2}$ and $\gamma=1$\\
Then applying lemma \ref{chernoff1} to the mean of the negated sample squares gives:\\
$$
\p(-\hat{\sigma}_0^2\ge t) \le \left(\frac{\sigma^2}{d^2}\exp(-qd^2)+1-\frac{\sigma^2}{d^2}\right)^n\exp(-qnt)
$$
Substituting $t$ for $y-\sigma^2$ and minimising with $q$ completes the proof, minimum $q$ occurs at:
$$ q = \frac{1}{d^2}\log\left(\frac{\sigma^2(-\sigma^2 + d^2 + y)}{(\sigma^2-d^2)(y-\sigma^2)}\right) $$
\end{proof}

Note that the application of this inequality (and thence the domain of function $H_2^n$) are only sensibly considered in certain settings, such as when:
(i) it is defined for $a<0<b$ (because otherwise the mean could not be zero), and (ii) $\sigma^2\le-ab\le (b-a)^2/4$ by Popoviciu's inequality\footnote{see \cite{zbMATH05780164}} (as it is not possible for the variance to be larger given the width of the data bounds).

\subsection{A new bound on the sample variance}\label{subsection:sample_variance_bound}

By Theorem \ref{hoeffdings1} and Lemma \ref{sample_squares} we have a probability bound on the mean squared and a probability bound on the sample squares, and from these it is possible to create novel a bound on the sample variance using lemma \ref{variance1}, as follows:

\begin{theorem}[Sample Variance Bound]\label{variance2}
For a random variable that is bounded $a\le X\le b$ with variance $\sigma^2$ and a mean of zero, if $d=\max(b,-a)$ then for $w>0$, the sample variance $\hat{\sigma}^2$ of $n$ samples is probability bounded by:
\begin{equation}\label{eq_no8}
\p(\sigma^2 - \hat{\sigma}^2 > w) \le H_3^n(a,b,w,\sigma^2),
\end{equation}
where:
\begin{align*} H_3^n(a,b,w,\sigma^2) =\min_{\phi\in[0,1]}
\begin{Bmatrix}
	H_1^n\left(\frac{\sigma^2}{b^2},\frac{\sqrt{\phi(\frac{n-1}{n}w+\frac{1}{n}\sigma^2)}}{b}\right)\\
	+H_1^n\left(\frac{\sigma^2}{a^2},\frac{-\sqrt{\phi(\frac{n-1}{n}w+\frac{1}{n}\sigma^2)}}{a}\right)\\
	+H_2^n\left(\frac{\sigma^2}{d^2},\frac{(1-\phi)(\frac{n-1}{n}w+\frac{1}{n}\sigma^2)}{d^2}\right)
\end{Bmatrix}\end{align*}
\end{theorem}
\begin{proof}By Lemmas \ref{sample_squares} and \ref{variance1}:
\begin{equation}\label{eq_no44}\p\left(\sigma^2 - \hat{\sigma}^2 > \frac{n}{n-1}\left(\hat{\mu}^2+y- \frac{1}{n}\sigma^2\right)\right)
\le H_2^n\left(\frac{\sigma^2}{d^2},\frac{y}{d^2}\right)\end{equation}
By inspection of equation \ref{eq_no2} we can convert to a double-sided version:
\begin{equation}\label{eq_no1}\p(\hat{\mu}^2\ge r^2)= \p(\hat{\mu}\ge r)+\p(\hat{\mu}\le -r) \le H_1^n\left(\frac{\sigma^2}{b^2},\frac{r}{b}\right) + H_1^n\left(\frac{\sigma^2}{a^2},\frac{-r}{a}\right) \end{equation}
Also, by manipulating the inner inequality of this equation: \begin{equation}\label{eq_no33}\p\left(\frac{n}{n-1}\left(\hat{\mu}^2+y-\frac{1}{n}\sigma^2\right)\ge \frac{n}{n-1}\left(r^2+y-\frac{1}{n}\sigma^2\right)\right) \le H_1^n\left(\frac{\sigma^2}{b^2},\frac{r}{b}\right)+H_1^n\left(\frac{\sigma^2}{a^2},\frac{-r}{a}\right)\end{equation}
Applying lemma \ref{prob_union} to the equations \ref{eq_no33} and \ref{eq_no44} gives:
$$\p\left(\sigma^2 - \hat{\sigma}^2 > \frac{n}{n-1}\left(r^2+y-\frac{1}{n}\sigma^2\right)\right) \le H_2^n\left(\frac{\sigma^2}{d^2},\frac{y}{d^2}\right)+H_1^n\left(\frac{\sigma^2}{b^2},\frac{r}{b}\right)+H_1^n\left(\frac{\sigma^2}{a^2},\frac{-r}{a}\right)$$
For a choice of parameter $w=\frac{n}{n-1}\left(r^2+y-\frac{1}{n}\sigma^2\right)$ there is a range of possible $r,y>0$ which we can parameterise by value $\phi$, such that $0\le\phi\le 1$:
$$y(\phi) = (1-\phi)\left(\frac{n-1}{n}w+\frac{1}{n}\sigma^2\right) \quad\text{and}\quad
r(\phi)^2 = \phi\left(\frac{n-1}{n}w+\frac{1}{n}\sigma^2\right)$$
Thus:
$$\p\left(\sigma^2 - \hat{\sigma}^2 > w\right)
\le H_2^n\left(\frac{\sigma^2}{d^2},\frac{y(\phi)}{d^2}\right)
  + H_1^n\left(\frac{\sigma^2}{b^2},\frac{r(\phi)}{b}\right) + H_1^n\left(\frac{\sigma^2}{a^2},\frac{-r(\phi)}{a}\right)$$
The result of this proof follows by taking the minimum over $\phi$.
\end{proof}

The use of this Theorem \ref{variance2} (and thus implicitly the domain of function $H_3^n$) is subject to the same restrictions as Lemma \ref{sample_squares} (and its domain as $H_2^n$); specifically that it is defined for $a<0$ and $b>0$ and for $\sigma^2\le-ab$; as otherwise the configuration is senseless.

\subsection{Generalised EBB creation process}\label{subsection:general_ebb_creation}

Theorem \ref{hoeffdings1} presents a bound for the sample mean given the variance, and Theorem \ref{variance2} presents a probability bound for the error of the sample variance from the variance. The task remaining is to create a new EBB by binding these two together.
To combine these two theorems to create a bound for the sample mean given the sample variance, we give a theorem that is representative of a slightly improved derivation process than that followed by \cite{Maurer50empiricalbernstein}, where they use this process to create their EBB.

Before beginning this theorem, we need to introduce some notation to ease presentation.
For a function $f$ with ordered inputs, we denote the inverse of $f$ with respect to its $i${th} input (counting from one) as $f^{-(i)}$, assuming it exists.
We summarily denote probability bounds on the differences of the sample mean from the mean, and the sample variance from the variance, by
$\p(\hat{\mu}-\mu>t)\le h(\sigma^2,t)$ and $\p(\sigma^2-\hat{\sigma}^2>w)\le f(\sigma^2,w)$, respectively.
And note that functions $h$ and $f$ may have additional arguments not limited to $\sigma^2$ and $t$, and $\sigma^2$ and $w$, respectively; but that these are not considered in the theorem and proof for brevity.

\begin{theorem}[Essential EBB]\label{ebb1}
For probability bounds $\p(\hat{\mu}-\mu>t)\le h(\sigma^2,t)$ and $\p(\sigma^2-\hat{\sigma}^2>w)\le f(\sigma^2,w)$,
if $f^{-(2)}$ and $h^{-(2)}$ both exist, and also if $h^{-(2)}$ is monotonically increasing in its first argument, so that we can define:
\[
z(\sigma^2,w) = \sigma^2-f^{-(2)}\left(\sigma^2,w\right)
\]
If $z^{-(1)}$ exists and is monotonic increasing in its first argument, then for any $x\in[0,y]$, the following relationship holds:
\[
\p\left(\hat{\mu}-\mu>h^{-(2)}\left(z^{-(1)}\left(\hat{\sigma}^2,y-x\right),x\right)\right)
\le y
\]
\end{theorem}
%
\begin{proof}
Substituting $w$ for $f^{-(2)}(\sigma^2,w)$ gives:
%$\p(\sigma^2-\hat{\sigma}^2>f^{-2}(\sigma^2,w))\le w$\\
%$\p(z(\sigma^2,w)>\hat{\sigma}^2)\le w$\\
%$\p(\sigma^2>z^{-1}(\hat{\sigma}^2,w))\le w$\\
%$\p(h^{-2}(\sigma^2,t)>h^{-2}(z^{-1}(\hat{\sigma}^2,w),t))\le w$\\
\begin{align*}
w & \ge \p\left(\sigma^2-\hat{\sigma}^2>f^{-(2)}\left(\sigma^2,w\right)\right)\\
 & \ge \p\left(z\left(\sigma^2,w\right)>\hat{\sigma}^2\right)\\
 & \ge \p\left(\sigma^2>z^{-(1)}\left(\hat{\sigma}^2,w\right)\right)\\
 & \ge \p\left(h^{-2}\left(\sigma^2,t\right)>h^{-(2)}\left(z^{-(1)}\left(\hat{\sigma}^2,w\right),t\right)\right)
\end{align*}
% $w  \ge \p(\sigma^2-\hat{\sigma}^2>f^{-2}(\sigma^2,w))$\\
% \-\hspace{4mm}$\ge \p(z(\sigma^2,w)>\hat{\sigma}^2)$\\
% \-\hspace{4mm}$\ge \p(\sigma^2>z^{-1}(\hat{\sigma}^2,w))$\\
% \-\hspace{4mm}$\ge \p(h^{-2}(\sigma^2,t)>h^{-2}(z^{-1}(\hat{\sigma}^2,w),t))$\\
Substituting $t$ for $h^{-(2)}(\sigma^2,t)$ gives:
\[
\p\left(\hat{\mu}-\mu>h^{-(2)}\left(\sigma^2,t\right)\right)
\le t.
\]
Applying probability union (lemma \ref{prob_union}) gives:\\
\[
\p\left(\hat{\mu}-\mu>h^{-(2)}\left(z^{-(1)}\left(\hat{\sigma}^2,w\right),t\right)\right)
\le t+w.
\]
Letting $y=t+w$ and $x=y-w$ completes the proof.
\end{proof}

The result of this Theorem is an Empricial Bernstein Bound. And our novel EBB is completed by substituting  $h(\sigma^2,t)=H_1^n\left(\sigma^2/b^2,t/b\right)$ (from Theorem \ref{hoeffdings1}) and $f(\sigma^2,w)=H_3^n\left(a,b,w,\sigma^2\right)$ (from Theorem \ref{variance2}) into Theorem \ref{ebb1}.
In this process care was taken in applying this theorem that all the assumptions hold, the necessary inverses exist, and that the domains of the functions were propagated through the analysis.

\subsection{Numerical determination}\label{numerical-implementation}

The Empirical Bernstein Bound described in the previous subsection \ref{subsection:general_ebb_creation} consists of inversions and compositions of functions $H_1$ and $H_3$ from theorems \ref{hoeffdings1} and \ref{variance2}, is identified to be challenging to analytically determine but much more possible to numerically determine.
Calculating this EBB numericaly consisted of three primary parts:
\begin{enumerate}
\item  Computating function $f(\sigma^2,w)=H_3^n(a,b,y,\sigma^2)$
\item  Verifying the assumptions of Theorem \ref{ebb1} hold for $h(\sigma^2,t)=H_1$ and $f(\sigma^2,w)=H_3$
\item  Calculating the subsequent result of Theorem \ref{ebb1}
\end{enumerate}

First, the function $f(\sigma^2,w)=H_3^n(a,b,w,\sigma^2)$ (per Theorem \ref{variance2}) was identified as the solution to an optimisation problem that solves for the minima of an objective function subject to constraint $\phi\in[0,1]$.
Despite its complexity, solutions of this sort can be found quickly using a single variable parameter sweep.

Second, it was necessary to verify the assumptions that $h^{-(2)}$, $f^{-(2)}$ and $z^{-(1)}$ exist and that $z^{-(1)}$ and $f^{-(2)}$ are monotonically increasing in their first argument.
It was easy to note that $h(\sigma^2,t)=H_1^n\left(\sigma^2/b^2,t/b\right)$ is a closed-form function that is monotonically decreasing from $1$ to $0$ on the second argument, so $h^{-(2)}$ exists and is monotonically increasing in its first argument.  However the remaining of these assumptions are more difficult to verify.
For any function, the values that the function takes can be plotted as an array of points and the values that the inverse of that function takes can be determined by conducting coordinate swapping on those points.
The values of $f(\sigma^2,w)=H_3^n(a,b,w,\sigma^2)$ were computed and were seen to be monotonically decreasing in its second argument confirming that $f^{-(2)}$ exists.
The function $z(\sigma^2,w)=\sigma^2-f^{-(2)}\left(\sigma^2,w\right)$ is then a manipulation on the coordinate swapped points of $f(\sigma^2,w)=H_3^n(a,b,w,\sigma^2)$.
By coordinate swapping again, $z^{-(1)}$ was seen to be a regular function monotonically increasing on its first argument, hence satisfying assumptions.

Third, to numerically calculate the result of Theorem \ref{ebb1} the functions $h^{-(2)}$ and $z^{-(1)}$ were numerically evaluated by direct parameter searches and then composed as:
$h^{-(2)}(z^{-(1)}(\hat{\sigma}^2,y-x),x)$ - which was the inner part of the expression of the new EBB parameterised by $x$ explicitly and also $a,b$ implicitly.
%
However we typically don't know the values of $a$ and $b$, but instead know the mean is somewhere within a finite interval of width $D=b-a$, in this context was taken the worst case values of $a$ and $b$ consistent with a given $D$,
and then the best $x\in[0,y]$ was taken subject to all other bounds.

Throughout this three stage process the new EBB was numerically determined by a series of coordinate manipulations and mundane parameter searches.\footnote{sourcecode available at:\\\url{https://github.com/Markopolo141/Engineered-Empirical-Bernstein-Bound}}
For the ease of this application of our EBB, we hand-tuned an envelope of our EBB's probability 0.5 bound,
where the process of creating such an expression involved plotting the numerical data, and manually fitting a symbolic expression above the data:
\begin{equation}\label{eq:prob_bound} \p\left(\mu-\hat{\mu}\ge \frac{D}{\sqrt{n}}\; \min\left[ \sqrt{2\log 2},
\footnotesize
\left(\begin{matrix*}[c]
\frac{3}{5}\sqrt{\min\left[1,\frac{\hat{\sigma}^2}{D^2}+\frac{25}{n}\right]} \\+ \ln\left(\max\left[1,n\left(1-\frac{\hat{\sigma}^2}{D^2}\right)\right]\right)^{-4}\end{matrix*}
\normalsize
\right)\right]\right)
\lessapprox 0.5 \end{equation}

The strength of this new EBB against existing EBBs are discussed in section \ref{sec:discussion}, and the performance of this new EBB in the context of stratified sampling is reported in section \ref{section:statistics_results}.
We now turn away from the subject matter of the unionisation of EBBs for stratified sampling (in the previous section \ref{section:old_statistics}) and the creation of a new EBB for this purpose in this section, to the creation of a completely novel bound directly tailored for stratified sampling.


\documentclass[12]{scrartcl}
\usepackage{amssymb,amsmath,gensymb,dsfont,calc,multicol,fullpage}
\makeatletter
\newcommand\Aboxed[1]{
   \@Aboxed#1\ENDDNE}
\def\@Aboxed#1&#2\ENDDNE{%
   &
   \settowidth\@tempdima{$\displaystyle#1{}$}
   \setlength\@tempdima{\@tempdima+\fboxsep+\fboxrule}
   \kern-\@tempdima
   \boxed{#1#2}
}
\makeatother

\begin{document}

\title{Homework 12, Section 2.7: 6(do only 1,...,6, do 7,...,12 for extra credit), 10(a,b) (trial and error may work) }
\author{Alex Gordon}
\date{\today}
\maketitle
\subsection*{6.}
1:1\\
2:7\\
3:9\\
4:10\\
5:8\\
6:11\\
7:2\\
8:5\\
9:3\\
10:4\\
11:6\\
12:12

\subsection*{10. A)}
All $x \equiv_{15} 2 $will work in this situation.

\subsection*{10. B)}
All $x \equiv_{8} 0 $will work in this situation.

\end{document}


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\hyphenation{ISTATS}
\begmath 15.1 Basic Statistics and Histogram

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

Computes the count, minimum, maximum, mean, standard deviation, and histogram
for a one-dimensional array of numbers. Can be called multiple times to update
these statistics to include additional data. Subroutines are provided to
display the results on a screen or printer. Versions are provided for REAL,
DOUBLE PRECISION, and INTEGER data.

\subsection{Usage}

\subsubsection{Computation of statistics, REAL data}

\paragraph{Program Prototype}

\begin{description}
\item[INTEGER]  \ {\bf NX, IHIST}($\geq $NCELLS){\bf , NCELLS}

\item[REAL]  \ {\bf XTAB}($\geq $NX){\bf , STATS}(5){\bf , X1, X2}
\end{description}

Assign values to XTAB(), NX, NCELLS, X1, X2, and on the first call of
a set, set STATS(1) = 0.0.

\begin{center}
\fbox{%
\begin{tabular}{@{\bf }c@{}}
CALL SSTAT1(XTAB, NX, STATS,\\
IHIST, NCELLS, X1, X2)\\
\end{tabular}}
\end{center}

New or updated quantities are returned in STATS() and IHIST().

\paragraph{Argument Definitions}

\begin{description}
\item[XTAB()]  \ [in] Array of NX values of $x$ whose statistics are to be
computed.

\item[NX]  \ [in] Number of values given in XTAB(). If NX $\leq $ 0, the
subroutine returns taking no action.

\item[STATS()]  \ [inout] Array of length~5 into which statistics are or
will be stored. On entry STATS(1) must be zero or positive.

If STATS$(1)=0$, the subroutine assumes there are no prior results in
STATS() and IHIST(). Results will be computed just for the data given in
XTAB().

If STATS$(1)>0$, the subroutine assumes there are prior results in STATS()
and IHIST(). Results will be updated to include the data given in XTAB().

The statistics stored in STATS() are:

\begin{tabbing}
\hspace{.5in}\= STATS(1) \== $count$\\
\>STATS(2) \>= $minimum$\\
\>STATS(3) \>= $maximum$\\
\>STATS(4) \>= $mean$\\
\>STATS(5) \>= $standard\ deviation$
\end{tabbing}

\item[IHIST()]  \ [inout] Integer array of length at least NCELLS, to hold
histogram results. Interpretation of contents on entry depends on the value
of STATS(1).

Used to store counts of occurrences of $x$-values in the different
classification intervals. Values less than X1 are counted in IHIST(1).
Values greater than X2 are counted in IHIST(NCELLS). The interval from X1 to
X2 is divided into NCELLS $-$ 2 equal-length subintervals, for which the
counts are stored in IHIST(2 : NCELLS $-$ 1).  Each of these subintervals
includes its left endpoint but not its right endpoint, with the exception that
the subinterval counted in IHIST(NCELLS $-$ 1) includes both of its endpoints.

\item[NCELLS]  \ [in] Number of elements to be used in the array IHIST().
Require NCELLS $\geq 3.$

\item[X1, X2]  \ [in] Lower and upper boundaries, respectively, defining the
range of $x$ values to be classified into NCELLS $-$ 2 equal intervals. Require
X1 $<$ X2.
\end{description}

\subsubsection{Display of statistics, REAL data}

\paragraph{Program Prototype}

All arguments should be declared as for SSTAT1 and should contain values
resulting from a previous call to SSTAT1.

\begin{center}
\fbox{%
\begin{tabular}{@{\bf }c@{}}
CALL SSTAT2( STATS, IHIST,\\
NCELLS, X1, X2)\\
\end{tabular}}
\end{center}

This subroutine writes to the standard system output the
contents of STATS() with labels, and a representation of the
histogram specified by the contents of IHIST(), NCELLS, X1, and X2.
The output is produced using statements PRINT and WRITE(*,~...).

\subsubsection{Computation of statistics, INTEGER data, REAL results}

\paragraph{Program Prototype}

\begin{description}
\item[INTEGER]  \ {\bf NI, ILOW, NCELLS}{\bf , ITAB}($\geq $NI)

\item[INTEGER]  \ {\bf ISTATS}(3){\bf , IHIST}($\geq $NCELLS)

\item[REAL]  \ {\bf XSTATS}(2)
\end{description}

Assign values to ITAB(), NI, ILOW, NCELLS, and on the first call of
a set, set ISTATS(1) = 0.

\begin{center}
\fbox{%
\begin{tabular}{@{\bf }c@{}}
CALL ISSTA1(ITAB, NI, ISTATS, XSTATS,\\
IHIST, ILOW, NCELLS)\\
\end{tabular}}
\end{center}

New or updated quantities are returned in ISTATS(), XSTATS(), and IHIST().

\paragraph{Argument Definitions}

\begin{description}
\item[ITAB()]  \ [in] Array of NI integer values whose statistics are to be
computed.

\item[NI]  \ [in] Number of values given in ITAB(). If NI $\leq $ 0, the
subroutine returns taking no action.

\item[ISTATS()]  \ [inout] Array of length~3 into which statistics are or
will be stored. On entry ISTATS(1) must be zero or positive.

If ISTATS$(1)=0$, the subroutine assumes there are no prior results in
ISTATS(), XSTATS(), and IHIST(). Results will be computed just for the data
given in ITAB().

If ISTATS$(1)>0$, the subroutine assumes there are prior results in
ISTATS(), XSTATS(), and IHIST(). Results will be updated to include the data
given in ITAB().

The statistics stored in ISTATS() are:

\begin{tabbing}
\hspace{.5in}\=ISTATS(1) \== $count$\\
\>ISTATS(2)\>= $minimum$\\
\>ISTATS(3)\>= $maximum$
\end{tabbing}

\item[XSTATS()]  \ [inout] Array of length~2 into which statistics are or
will be stored. Interpretation of contents on entry depends on the value of
ISTATS(1). The statistics stored in XSTATS() are:

\begin{tabbing}
\hspace{.5in}\=XSTATS(1) \== $mean$\\
\>XSTATS(2)\>= $standard\ deviation$
\end{tabbing}

\item[IHIST()]  \ [inout] Integer array of length at least NCELLS, to hold
histogram results. Interpretation of contents on entry depends on the value
of ISTATS(1).

Used to store counts of occurrences of integer data according to their
values. Values less than ILOW are counted in IHIST(1). Values greater than
ILOW + NCELLS $-$ 3 are counted in IHIST(NCELLS). The occurrences of the
NCELLS $-$ 2 consecutive integer values ILOW, ILOW $+1$, ..., ILOW + NCELLS $%
-$ 3, are counted in the cells IHIST(2 : NCELLS $-$ 1),
respectively.

\item[ILOW]  \ [in] Specifies the integer value whose occurrences are to be
counted in IHIST(2).

\item[NCELLS]  \ [in] Number of elements to be used in the array IHIST().
Require NCELLS $\geq 3.$
\end{description}

\subsubsection{Display of statistics, INTEGER data, REAL results}

\paragraph{Program Prototype}

All arguments should be declared as for ISSTA1 and should contain values
resulting from a previous call to ISSTA1.

\begin{center}
\fbox{%
\begin{tabular}{@{\bf }c@{}}
CALL ISSTA2(ISTATS, XSTATS,\\
IHIST, ILOW, NCELLS)\\
\end{tabular}}
\end{center}

This subroutine writes to the standard system output the
contents of ISTATS() and XSTATS() with labels, and a representation
of the histogram specified by the contents of IHIST(), ILOW, and
NCELLS.  The output is produced using statements PRINT and WRITE(*,~...).

\subsubsection{Modifications for DOUBLE PRECISION data and results}

For double-precision usage change the names SSTAT1, SSTAT2, ISSTA1
and ISSTA2, to DSTAT1, DSTAT2, IDSTA1, and IDSTA2, and change the
REAL declarations to DOUBLE PRECISION.

\subsection{Examples and Remarks}

See demonstration drivers and sample output in Chapter~3.3 for examples of
the use of these subroutines.

Note that to compute statistics for a set of $n$ REAL numbers one could call
SSTAT1 once giving the whole set of $n$ numbers, or call SSTAT1 $n$ times
giving just one new datum at each call, or use any strategy of grouping
the data intermediate between these two extremes. Giving all data on one
call will be most efficient and will introduce slightly less round off
error. The SQRT function is referenced once on each call.

In ISSTA1 the histogram records a separate count for each distinct integer
value from ILOW through ILOW + NCELLS $-$ 3. If this is a finer resolution
than one wants, the data should be converted from INTEGER to REAL or DOUBLE
PRECISION, and one should use SSTAT1 or DSTAT1 to let each counting cell
cover a range of values of the data.

\subsection{Functional Description}

\subsubsection{Method}

The sample mean, $\mu _n$, and standard deviation, $\sigma _n$, of a set of
numbers, $x_i$, $i=1$, ..., $n$, are defined by%
\begin{equation*}
\mu _n=\frac 1n\sum_{i=1}^nx_i,\quad \text{and\quad }\sigma _n^2=\frac
1{n-1}\sum_{i=1}^n\left( x_i-\mu _n\right) ^2.
\end{equation*}
To discuss recursive computation of these quantities it is useful to define
the auxiliary quantity%
\begin{equation*}
S_n=\sum_{i=1}^n\left( x_i-\mu _n\right) ^2.
\end{equation*}
The quantities $\mu _n$ and $S_n$ satisfy the recursive relations
\begin{align}
\label{O1}\mu _n&=\mu _{n-1}+\frac{\delta _n}n,\quad \text{and}\\
\label{O2}S_n&=S_{n-1}+\frac{n-1}n\delta _n^2,\quad \text{where}\\
\delta _n&=x_n-\mu _{n-1}.\notag
\end{align}
Formulas (1) and~(2) apply for $n\geq 2$ after initiating the sequences by setting
$\mu _1=x_1$ and $S_1=0.$

Formulas (1) and~(2) are satisfactory from a numerical accuracy point of view,
however, since $S_n$ is about the magnitude of the square of $\sigma _n$
there is a remote possibility that $S_n$ could be outside the exponent range
of a computer's floating-point arithmetic even when $\sigma _n$ is within
range.

The range of data that can be processed without encountering this problem
can be substantially increased by introducing scaling. Let $c_n$, $n=2$, 3,
..., be an as yet unspecified sequence of positive numbers and define $%
\delta _n^{^{\prime }}=\delta _n/c_n$ and $S_n^{^{\prime }}=S_n/c_n^2$. Then
from Eq.\,(2) we may write
\begin{equation}
\label{O3}S_n^{\prime }=\left[ \frac{c_{n-1}}{c_n}\right] ^2S_{n-1}^{\prime
}+\frac{n-1}n\delta _n^{\prime 2}
\end{equation}
Let $k$ denote the first index for which $\delta _i\neq 0$. No scaling is
needed for $i<k$, but formally we may set $c_i=1$ for $i<k$. Define $%
c_k=|\delta _k|$. A satisfactory scaling would be provided by
setting $c_i=\max (c_{i-1}$, $|\delta _i|)$ for $i>k$. This would
assure that $|\delta _i^{^{\prime }}|\leq 1$ and $1/2\leq
S_i^{^{\prime }}\leq i$ for all $i\geq k$. Note, however, that when $%
c_i=c_{i-1}$ it is not necessary to compute and apply the factor $(c_{i-1}/c_i)
$. To reduce substantially the number of times the scale factor is changed
we choose to leave it unchanged, $i.e.$, set $c_i=c_{i-1}$, until some $%
\delta _i$ satisfies $|\delta _i|>64c_{i-1}$. At that point the
scale factor will be changed to $c_i=|\delta _i|.$

The standard deviation is recovered from $S_n^{^{\prime }}$ by%
\begin{equation*}
\sigma _n=c_n\left[ \frac{S_n^{\prime }}{n-1}\right] ^{1/2}.
\end{equation*}
Subroutines SSTAT1 and DSTAT1 use Eqs.\,(1) and (3) and the scaling technique
just described. Subroutine ISSTA1 uses the simpler Eqs.\,(1) and (2) since
the integer data cannot reach such extremes of magnitude.

\subsubsection{Accuracy tests}

These subroutines have been tested for correctness against simpler
algorithms. SSTAT1 and DSTAT1 have been tested successfully using very large
and very small data values that would have led to failure on overflow or
underflow without the scaling.

\subsection{Error Procedures and Restrictions}

For useful processing one must have NX $\geq 1$, STATS$(1) \geq 0$, and
NCELLS\ $\geq 3$ on entry to SSTAT1 or DSTAT1, and NI $\geq 1$, ISTATS$(1)
\geq 0$, and NCELLS $\geq 3$ on entry to ISSTA1.

If NX $\leq $ 0 or NI $\leq $ 0, the entered subroutine will return
immediately, doing no processing. If STATS$(1) < 0$, ISTATS$(1) < 0$, or
NCELLS $< 3$, the results will be unpredictable.

When calling any of these subroutines to update previous statistics with new
data, the contents of all arguments must remain unchanged from a previous
call except for XTAB() and NX in SSTAT1 or DSTAT1, and ITAB() and NI in
ISSTA1.

\subsection{Supporting Information}

The source language is ANSI Fortran~77.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.2in} {\bf Required Files}\vspace{2pt}\\
DSTAT1 & \hspace{.35in} DSTAT1\rule[-5pt]{0pt}{8pt}\\
DSTAT2 & \hspace{.35in} DPRPL, DSTAT2\rule[-5pt]{0pt}{8pt}\\
IDSTA1 & \hspace{.35in} IDSTA1\rule[-5pt]{0pt}{8pt}\\
IDSTA2 & \hspace{.35in} DPRPL, IDSTA2\rule[-5pt]{0pt}{8pt}\\
ISSTA1 & \hspace{.35in} ISSTA1\rule[-5pt]{0pt}{8pt}\\
ISSTA2 & \hspace{.35in} ISSTA2, SPRPL\rule[-5pt]{0pt}{8pt}\\
SSTAT1 & \hspace{.35in} SSTAT1\rule[-5pt]{0pt}{8pt}\\
SSTAT2 & \hspace{.35in} SPRPL, SSTAT2\\
\end{tabular}

Designed by C. L. Lawson and F. T. Krogh, JPL, Apr.~1987. Programmed by C.
L. Lawson and S. Y. Chiu, JPL, Apr.~1987. Changed Nov.~1988 (MATH77
Release~2.2) to use SPRPL and DPRPL instead of PRPL.

\end{multicols}
\end{document}