\section{LongReal: 64-bit IEEE Reals}
\subsection{Types}
Why rename LONGREAL?  Because numerical analysis is tuned to specific
resolutions for specific classes of problems.  If you need a 64-bit real
to do a job, you'd better be sure that is exactly what you are getting.
Of course the language definition says that is the meaning of LONGREAL,
but why trust it over time?  Maybe Modula-4 will use LONGREAL for
the 80-bit reals.  By using REAL64, we get to fix it one place.
Of course, with structural equivalence, you can still intermingle
LONGREAL as desired.

A minor point:  Since these routines do not lend themselves to
generics, using REAL64 and REAL32 makes it fairly easy to clone
a 64-bit routine and generate a 32-bit version.  One still has to
convert the remaining literals from ``D'' to ``E''.

\subsection{Constants}
These are largely taken from Math.i3 and CRC91.
We added others as needed when trying
to eliminate literals from routines.

\subsection{Math.i3 Functions}
Why redirect them to here?  Because Math.i3 is inadequate, and
Real64 can serve as a replacement.

\subsection{Other Functions}
\subsection*{sgn}
The signum function is occasionally handy.  However, for
performance purpose, you usually have to do it inline.

\subsection{Special Functions}
These are implemented in SpecialFunction.m3 but exported via LongReal.i3.

\subsection*{factorial}
Following NR92, this is implemented with a cached table for
$0! \dots 34!$.
Originally 34 was chosen because it could be handled by
REAL32.  Having moved to REAL64, we could extend the
table, but see no need.  Beyond $34!$ we use the $\Gamma$ function.

\subsection*{ln\_factorial}
This one needs a cache also, but it can't be a
CONST [HGG: At least, I'm not willing to build it].  So we need a VAR
cache, which means it could be in global space, in NT-thread
space, or in an object.  Since all threads should agree on
the table values, I'll go global space.  However, this
assumes that assigning a real to a table cell is an atomic
action.  Seems like a safe assumption for me.

How big should the cache be?  We typically use ln\_factorial when
factorial overflows.  That goes to $34!$, so the cache better be bigger
than 34.  On the
other hand, too big a cache takes extra room (hey, this is
Modula-3, so who cares about that? :-), and it takes extra
initialization on the startup.  We notice that $70!$ is 1
googol, so that seems a handy cutoff.  [HGG: Personally, I've
never run into a problem using googols.]

\subsection*{gamma, ln\_gamma}
The basic idea is to implement Stirling's formula for gamma(n+1).
That is described in Grah81, pg 112, and in
CRC91, pg 351.  CRC91 has the most complete treatment:
\begin{equation}
  \Gamma(x+1)=\sqrt{2\pi x}x^{x}e^{-x}(1+\frac{1}{12x}
    +\frac{1}{288x^2}+\dots)
\end{equation}

However, for numerical analysis, Lanczos (cited in NR92, pg 213, 6.1.5,)
provides a better approximatrixion:
\begin{equation}
  \Gamma(z+1)=(z+ \gamma +1/2)^{z+1/2}e^{-(z+ \gamma +1/2)}
  \sqrt{2\pi}*(c_0+\frac{c_1}{z+1}+\frac{c_2}{z+2}
       +\cdots+\frac{c_N}{z+N}+\epsilon)\ (z>0)
\end{equation}

With all these exponentials and multiplies, the obvious implementation
is to work in logs.  As NR92 notes, $\Gamma$ is typically used
in multiply and divide situations, leading to normal (small) numbers.
So it makes sense to get ln\_gamma, and then get other functions from that.

The exponential needs work.  First, there
is a repeated factor $x=z+\gamma+0.5$.  Second, since we are
returning ln, we really want:
\begin{equation}
  \ln(x)(z+1/2)(-x)+\ln(\sqrt{2\pi})+\ln(\mbox{series})
\end{equation}
This simplifies to:
\begin{equation}
  \ln(x)*(z+1/2)+(-x)+0.918938533204673+\ln(\mbox{series})
\end{equation}
[HGG: I got the $0.918\ldots $ number from MathCad.]

NR92 handles the coeffs in a loop.  We have unrolled the loop here.

\subsection*{gamma\_p, gamma\_q}
Incomplete Gamma Functions.  Per Krey88, pg A78:
\begin{eqnarray}
 \Gamma(a) & =& P(a,x) + Q(a,x)\\
 P(a,x) & = & \int_0^x e^{-t}t^{a-1}dt\\
 Q(a,x) & = & \int_x^\infty e^{-t}t^{a-1}\,dt
\end{eqnarray}
NR92, pg 216, also gives:
\begin{eqnarray}
  P(a,x) & = & \frac{\gamma (a,x)}{\Gamma (a)}\\
  Q(a,x) & = & \frac{\Gamma (a,x)}{\Gamma (a)}\\
         & = & \frac{1}{\Gamma(a)} \int_x^\infty e^{-t}t^{a-1}\,dt
\end{eqnarray}
In other words, NR92's $Q$ is Krey88's $Q/\Gamma$.  We will use
NR92's treatment, because it leads to the useful:
\begin{equation}
  Q(a,x) = 1 - P(a,x)
\end{equation}
This is helpful because depending on the ratio of a and x, Q or P could
be the better mechanism for computation.  That is, Q is flat when
x is small compared to a, and P is flat when x is large compared to a.
Either case makes numerical analysis difficult.  So we will choose
via the criteria given by NR92:
\begin{verbatim}
     if x<(a+1) then
          use P approach
     else
          use Q approach
     end
\end{verbatim}

The next problem is of course to determine the $\gamma(a,x)$ for P and
the $\Gamma(a,x)$ for Q.  NR92, pg 217, provides a series approximatrixion
for $\gamma(a,x)$ and a continued fraction approximatrixion
for $\Gamma(a,x)$. These deserve helper functions of their own.
While we have used the same
names for these functions (gamser for the series and gamcf for the
continued fraction), these are original implementations.

\subsubsection*{gamser}
\begin{equation}
  \gamma(a,x) = e^{-x}x^a\sum_{n=0}^\infty
                    \frac{\Gamma(a)}{\Gamma(a+1+n)}x^n
\end{equation}
The initial factor can be captured as follows, using ln\_gamma(a)
which we pass as lga:
\begin{verbatim}
    factor= exp(-lga-x+a*log(x))
\end{verbatim}
Next, we need to iterate the series to convergence.  How
converged?  We will go until the absolute value of the last
term is less than eps=5*EPS.

Notice that each additional term can be produced by multiplying the
previous term:
\begin{verbatim}
  term[i+1]=term[i]*x/(a+1+i)
\end{verbatim}

To initialize this, we can do the zeroth term by hand, then
start iterating.  Note that in the each term,
$\Gamma(a)/\Gamma(a+1)$ cancels to $1/a$.  In the zeroth term that
is it.  In later terms, we need $(a+1)(a+1+1)(a+1+2)\dots$ in
the denominator.  The $(a+1+n)$ item, called a1n, is
initialized to $a+1.0$.  For the x's, we start with $x^0=1$, and
then multiply x's onto the term
\begin{verbatim}
     term:=1.0/a;
     sum:=term;
     a1n:=a;
     FOR i:=1 TO MaxIter DO
          a1n:=a1n+1.0;
          term:=term*x/a1n;
          sum:=sum+term;
          IF ABS(term)<eps THEN
               RETURN factor*sum
          END;
     END;
\end{verbatim}

\subsubsection*{gamcf}
\begin{equation}
  \Gamma(a,x) = e^{-x}x^a \left(
                    \frac{1}{\displaystyle x+1-a
                    - \frac{1\cdot(1-a)}{\displaystyle x+3-a
                      - \frac{2\cdot(2-a)}{\displaystyle x+5-a
                        - \cdots}}}
                    \right)
\end{equation}
Continued fractions are tedious to set up.
We start by copying the betacf code and then
working out the new meanings of the variables here:

We need to use n in the calculations, so we need to track the
iterator with a real, call it m.
\begin{verbatim}
     m[j]=m[j-1]+1.0
     b[j]=m*(m-a);  b[0]=0 so actually start with b[0]=tiny;
     b[1]=1
     a[j]=a[j-1]+2.0;  a[1]=x+1.0-a
     the global TINY will work here
     eps=5.0e-7
     MaxIter=90
\end{verbatim}

The initialization conditions are so odd, I decided to set
up a[1], b[1] by hand, and then update them at the end of
the loop.
\begin{verbatim}
j=0:
     b0=0
     f=b0=0 ==> f=TINY
     C=f=TINY
     D=0
j=1:
     bj=x+1-a
     aj=1
     D=bj+aj*D=bj=x+1+a
     C=bj+aj/C=x+1+a+1/TINY
     D=1/D=1/(x+1+a)
     delta=C*D=(x+1+a+1/TINY)/(x+1+a)
     f=f*delta=TINY*(x+1+a+1/TINY)/(x+1+a)
          =(x*TINY+1*TINY+a*TINY+1)/(x+1+a)
          = approx 1/(x+1+a) = D
j>=2:
     m=m+1; m2p1=2*m+1
     bj=x+m2p1-a
     aj=m*(m-a)
\end{verbatim}
We can turn m2p1 into an add: m2p1:=m2p1+2.0.  We can turn x-
a into a precalc: xa:=x-a.  Then, with m=0 and m2p1=1 at the
start:
\begin{verbatim}
     m:=m+1.0; m2p1:=m2p1+2.0;
     by:=xa+m2p1;
     aj:=-m*(m-a);
\end{verbatim}

Next, I see I actually need that x-a business in the
initialization, so I can skip the m2p1 stuff and just do
\begin{verbatim}
     init: m:=0.0; xa:=x+1-a
     m:=m+1.0; xa:=xa+2.0;
\end{verbatim}

Then I see that aj=-m*(m-a)=m*(a-m).  I'm not sure if the
compiler would have caught that, so I'll do it here.  But it
is a bit obscure, so folks better read this note before
fooling with it.

\subsection*{binomial}
The formula is:
\begin{equation}
  {n \choose k} = \frac{n!}{k!(n-k)!}
\end{equation}

Even if we weren't doing factorials, we would
probably want to do a divide and a multiply as logs.  So the
formula becomes:
\begin{equation}
  \exp(\ln(n!)-\ln(k!)-\ln((n-k)!))
\end{equation}
Since we already have ln\_factorial, we can use that and just take
the exponential at the end.

\subsection*{beta}
The standard formula is:
\begin{equation}
  B(z,w) = \frac{\Gamma(z) \Gamma(w)}{\Gamma(z+w)}
\end{equation}
However, we implement using logs, which means we can use ln\_gamma.

\subsection*{betai}
According to NR92, pp 226-227,
the incomplete beta function is defined as:
\begin{eqnarray}
    I_x(a,b) & = & \frac{B_x(a,b)}{B(a,b)}
               = \frac{1}{B(a,b)}\int_0^xt^{a-1}(1-t)^{b-1}dt\\
             & = & 1 - I_{1-x}(b,a)
\end{eqnarray}

NR92 provides both a series and a continued fraction approximatrixion,
and statistices the continued fraction approach is good for
$x < (a+1)/(a+b+2)$.  For cases where this does not hold, NR92
recommends using the $I_{1-x}$ formula.  The continued fraction
form is:
\begin{eqnarray}
  I_x(a,b) & = & \frac{x^a(1-x)^b}{aB(a,b)}
                 \left[ \frac{1}{\displaystyle 1
                        + \frac{d_1} {\displaystyle 1
                          + \frac{d_2} {displaystyle 1
                            + \cdots}}} \right]\\
  d_{2m+1} & = & -\frac{(a+m)(a+b+m)x} {(a+2m)(a+2m+1)}\\
  d_{2m}   & = & \frac{m(b-m)x}{(a+2m-1)(a+2m)}
\end{eqnarray}

The initial factor needs to be
calculated using $B(a,b),x$ or $B(b,a),1-x$, depending on the convergence
condition:
\begin{verbatim}
     if x < (a+1)/(a+b+2) then
          factor = x^a*(1-x)^b)/(a*beta(a,b))
     else
          factor = (1-x)^b*(1-(1-x))^a/(b*beta(b,a))
     end
\end{verbatim}

Notice that $\mbox{beta}(b,a)=\mbox{beta}(a,b)$,
and that $(1-(1-x))^a = x^a$.
So the two forms are the same except for dividing by a vs b.
So let's make the factor just the common part:
\begin{equation}
  \mbox{factor} = (1-x)^b*x^a/\mbox{beta}(a,b)
\end{equation}
This has lots of exps and muls, so we use log forms:
\begin{equation}
  \mbox{factor} = \exp(a \cdot \ln(x)+b \cdot \ln(1.0-x)
      -\ln(\mbox{beta}(a,b)))
\end{equation}
But notice that beta is already an exp function.  So we can
use the raw form and save some calls and calcs:
\begin{equation}
  \mbox{factor} = \exp(a \cdot \ln(x) + b \cdot \ln(1.0-x)
          -(\mbox{ln\_gamma}(a)+\mbox{ln\_gamma}(b)-\mbox{ln\_gamma}(a+b))
\end{equation}

For the second part of the equation, we follow NR92's lead
in using the continued fraction approach.   And we use the recommended
modified Lentz (NR92, pg 172).

Lentz is a bit tricky.
To avoid using b for both the ftn
parameter, and the Lentz b[j] item, we change the Lenz
a's to aj and the Lentz b's to bj.  That makes it much
cleaner.  Because the iteration variable is lost after a
loop completes, we return f directly when found,
and raise the error if we actually complete the loop.
\begin{verbatim}
     f=b0=0, so make f=TINY
     C=f
     D=0
     b[0]=0; b[j>0]= 1
     a[0]=1; a[1]=1; a[2m+1>1]=d1; a[2m>1]=d2
\end{verbatim}

We have a tricky initialization for j=0,1,2 and
then get rolling with j=3.  That suggests doing the j0,j1
cases by hand and then starting the loop with j=2.  That
would be done with a loop on i=1 to Max, with m=2*i and then
m=2*i+1 for the pair.
\begin{verbatim}
j=0:
     b0=0
     f=b0=0, so f=TINY
     C=f=TINY
     D=0
j=1:
     aj=1
     bj=1
     D=bj+aj*D=1+TINY
     C=bj+aj/C=1+1/TINY
     D=1/D=1/(1+TINY)= approx 1
     delta=C*D=C*1=C=1+1/TINY=very large
     f=f*delta=TINY(1+1/TINY)=TINY+1= approx 1
j=2:
     j=2 is equiv of d1, so the d[2m+1) formula applies,
     using m=0
     aj=-(a+0)*(a+b+m)*x/((a+2*0)*(a+2*0+1))=-
     a*(a+b)*x/(a*(a+1))
          =-(a+b)*x/(a+1)
     bj=1
     D=bj+aj*D=1+aj*1=1+aj
     C=bj+aj/C=1+aj/(very large)=1+aj*TINY=approx 1
     D=1/D=1/(1+aj)
     delta=C*D=1*(1/(1+aj))=1/(1+aj)
     f=f*delta=1/(1+aj) = D
\end{verbatim}
These are the actual initialization values.  After that we
can begin the normal 2m, 2m+1 cycle.

\subsection*{erf,erfc}
NR92, pg 220, provides:
\begin{eqnarray}
  \mbox{erf}(x)  & = & P\left( \frac{1}{2},x^2 \right) \, (x \ge 0)\\
  \mbox{erf}(-x) & = & -\mbox{erf}(x)\\
  \mbox{erfc}(x) & = & Q\left( \frac{1}{2},x^2 \right) \, (x \ge 0)\\
  \mbox{erfc}(-x) & = & 2-\mbox{erfc}(x)
\end{eqnarray}
We implement these as is.


\chapter{CT Reconstruction}

The essential goal any reconstruction is to use the information gathered to find out what the object looks like.  In the case of CT, we use the information gathered on the detector over angle to reconstruct the x-ray attenuation coefficient of the object.  CT system as evolved tremendously over the years and reconstruction algorithm has also evolved dramatically to support the types of systems. Over the years, reconstruction techniques has evolved with two general approaches to reconstruction, analytical and iterative.  In this chapter we attempt to give a brief overview of these general techniques used for CT reconstruction, then we will explain the method that was used for our system.

\subsection{Analytical Reconstruction Techniques}
In general, analytical reconstruction approaches try to formulate the solution in a closed-form equation.  The foundation of analytical reconstruction methods is the Radon transform, which relates a function $f(x,y)$ to the collection of line integrals of that function $L(\rho)$.  Shown in equation X.

\begin{equation}
\label{eq:radon}
\end{equation}

\subsection{Approximate analytic reconstruction techniques}
Since the first clinical CT scanner that was installed 1971 up until 1990, most clinical CT scanners employ the fan-beam geometry with axial rotation.  The dominant mode of image reconstruction for these CT scanners were the fan-beam axial filtered back-projection algorithm in order to solve for the object function (slice).  Essentially the projection data on the detector at each angle goes through a Fourier transform, filtered in the spatial frequency domain then back-projected onto each slice to reconstruct the original object slice.  In 1989 the first spiral CT scanner was installed with the clinical desire to cover an entire human organ, and ever since then, the spiral geometry became the main configuration for all clinical CT scanners.  The introduction of spiral CT forced the development of helical FBP algorithm and later with the introduction of multi-slice and cone-beam CT, the helical FBP is replaced by the cone-beam FBP, commonly known as the FDK algorithm \citep{Feldkamp1984}.  The helical and cone-beam FBP are adaptations of the 2D FBP to a cone-beam geometry.  Since they do not conform to the Radon transform assumptions, therefore the projection data requires rebinning and interpolation prior to FBP reconstruction
.  Although these algorithms are approximate in nature, they do offer distinct advantages such as volume reconstruction in single half-scan acquisition.  Because of their flexibility, their ability to intrinsically handle data from long objects and their computational efficiency, the approximate reconstruction algorithms are still the dominant force behind most commercial CT reconstruction engines.\citep{Wang1993}, \citep{Hsieh}, \citep{Hsieh2007}, \citep{Tang2006}, \citep{Silver1998}

\subsection{Exact analytic reconstruction techniques}
With the introduction of spiral CT and cone-beam CT, another approach was to solve the three-dimension Radon transform analytically, which is known as the exact reconstruction methods.  These methods try to derive analytical solutions that match the scanned object if the input projections are the true line integrals. These algorithms were first developed by Katsevich on the exact FBP algorithm \citep{Katsevich2002} \citep{Katsevich2003} \citep{Katsevich2002SIAM}.  Since its invention, various sophisticated formulas have been proposed and developed to allow for more efficient use of the projection samples, region-of-interest reconstruction and for more general scanning trajectories\citep{Chen2003}, \citep{Pan2004}, \citep{Zou2004}, \citep{Ye2005}, \citep{Zhuang2006}, \citep{Wang2008}.  Although these algorithms offer exact solution to the cone-beam spiral CT problem, due to their noise properties, limited robustness to patient motion, and are far less computationally efficient as compared to the popular filtered back-projection, they have yet to be implemented widely in commercial products.

\subsection{Iterative Reconstruction Techniques}
Iterative reconstruction techniques try to solve for the object using projection images in an iterative fashion, where the solution to the object at the current iteration will be used for the next iteration.  All iterative reconstruction methods consist of three major steps which are repeated iteratively until a final solution is reached.  First, a forward projection of the object create an estimate of the raw data.  Second the estimated raw data and the real measured data are compared and a correction term is calculated.  Third, the correction term is back projected onto the object to create a new estimate of the object.  This process is repeated until either a fixed umber of iteration is reached or the object estimation reaches a predefined criterion.

\subsection{Algebraic Iterative reconstruction techniques}
From a historical perspective, the very first reconstructed CT image was created using the algebraic iterative reconstruction (ART) method where it tries find the object by solving a series of linear equations Ax = b, where in terms of image reconstruction x are the voxels of the volume to be reconstructed, A is the system matrix used for producing the raw data and b are the pixels of the measured raw data.  The entries of the matrix A correspond to rays from the x-ray source through the object volume to the detector pixels.  Starting from ART, there evolved a series of algorithms that would try to converge faster, reduce noise, reduce problems with streak artifacts, but in general all ART-based methods are non-statistical and model the geometry of the acquisition process better than common analytical methods based on FBP.  Therefore ART-based methods can better deal with sparse data and an irregular sampling of acquisition positions\citep{Beister2012}.

\subsection{Statistical Iterative reconstruction techniques}
Statistical iterative reconstruction techniques has been used extensively in single photon emission computed tomography (SPECT) and positron emission tomography (PET) to where low photon rates and noise are major issues.  With the increase of public awareness of CT-induced radiation, and the need to reduce the associated risks. Transition of statistical IR to CT carries high promises of lowering radiation dose while suppressing noise for low-dose clinical protocols and techniques.  However, accurate modeling is necessary to ensure that modeling errors do not grow during the iterative convergence process that would form artifacts in the reconstructed images.  There are two parts when it comes to modeling the CT system, the physical model that involves the geometry of the system, and the statistical model that involves the detector statistics or noise of the system.

Unlike analytical reconstruction where numerous physical assumptions were used to make the mathematics manageable.  For example, the size of the x-ray focal spot is assumed to be infinitely small, the shape and dimension of each detector cell are ignored, and all x-ray photon interactions are assumed to take place at a point located at the geometric center of the detector cell.  Iterative reconstruction requires no prior assumptions about the geometry of the system.  Iterative reconstruction allows for better modeling of the image forming process and can account for any physical process limited by the computational power of the current computer.  For example, one commonly used model is to cast multiple pencil rays through the x-ray focal spot, image voxel, and the detector pixel to mimic different x-ray photon paths going through the object and the summation of the rays on each detector are used to approximate the CT image forming system. Needless to say, this method is extremely time consuming.  Another commonly used approach is to model the "shadow" cast by each image voxel onto the detector and rely on the point-response function to model the forward projection process of the CT system, where the response function is non-stationary and changes with the voxel location to account for different magnifications and orientations.  This method is computationally more efficient.  Shown in figure \ref{fig:physical_modeling}.

\begin{figure}
\centering
	\begin{subfigure}[b]{0.4\linewidth}
	\centering
	\placeholderimage[width=4cm,height=4cm]{pencilbeammodel.png}
	\end{subfigure}
\hspace{0.2cm}
	\begin{subfigure}[b]{0.4\linewidth}
	\centering
	\placeholderimage[width=4cm,height=4cm]{psfmodel.png}
	\end{subfigure}
\caption{examples of physical modeling in Hsieh's paper}
\label{fig:physical_modeling}
\end{figure}

The statistical model or the noise model tries to incorporate the statistics of the detected photon into the reconstruction process.  This may include incident photon flux variations (known as the Poisson distribution, though it is approximate due to the poly-energetic x-ray source used in CT), energy-dependent light production in the scintillator, shot noise in photo-diodes, electronic noise in readout electronics. The most common model used is the maximum likelihood expectation maximization (MLEM) algorithm where it assumes the noise is a zero-mean Gaussian.

Once both the geometry and the noise of CT system are properly modeled, we need to choose a cost function to minimize.  The iterative process is used to search for the minimum of the cost function and the solution that minimizes the cost function is the reconstructed object.  This cost function, sometimes called an objective functional, typically consist of two parts, a data agreement part, such as least squares, tries to see the difference between the experimental data and the object that generated the data based on the model, and a regularizing term sometimes called a penalty function that depends only on the object and serves as a prior such as positivity.  If the functional is strictly convex, then the minimum is unique and all algorithms should obtain the same image if run to convergence.  The only issues are then which algorithm will find the minimum most efficiently and with the least computing resources.  In practice, however, iterative algorithms may not be run to convergence, and the resulting image depends on the algorithm, the initial estimate and the stopping rule.

\subsubsection{Circular orbit}
In the analytical reconstruction algorithm side, there are two types of reconstruction geometry, circular trajectory most commonly called cone-beam step-and-shoot reconstruction and multi-slice helical reconstruction.  The advantage of SSCB is its simplicity and robustness, especially when data collection needs to be synchronized with a physiological signal, such as an EKG.  If the anticipated data acquisition is not aligned with the desired physiological state, as in the case of arrhythmia in coronary artery imaging, the CT system can simply wait for the next physiological cycle to collect the desired data.    When the z-coverage is sufficiently large, the entire organ, such as a heart or a brain, can be covered during a single gantry rotation.

The inherent drawback of the SSCB is its incomplete sampling since that data acquisition does not satisfy the necessary condition for exact reconstruction.  This data incompleteness can be viewed in terms of the local Fourier transform of a given image voxel.  In the case of the SSCB acquisition, image artifacts arise from three major causes: missing frequencies, frequencies mishandle during reconstruction, and axial truncation.  \comment{need citations or learn about it??}
One of the most commonly used reconstruction algorithm is the Feldkamp-Davis-Kress (FDK) reconstruction.  This algorithm differs from the conventional fan-beam reconstruction in replacing the two-dimensional back-projection with a three-dimensional back-projection to mimic the x-ray path as it traverses the object, and in including a cosine weighting in the cone direction to account for path length differences of oblique rays \citep{Feldkamp1984}.
\subsubsection{Helical orbit}
\comment{notes taken from Hiseh's recent advances in CT image reconstruction paper}
The algorithmic development can generally be classified into three major areas: analytical reconstruction, model-based iterative reconstruction, and application-specific reconstruction.
\comment{citation must include Jiang Hsieh's recent advances in CT image reconstruction, his Computed Tomography book, Marcel Beister's iterative reconstruction methods in x-ray CT}

The use of an iterative reconstruction to solve the inverse problem of x-ray computed tomography has a long history.  Reconstruction of the very first clinical image utilized an iterative technique called algebraic reconstruction technique(ART) to invert a large matrix.  statistical IR has been used extensively in single photon emission computed tomography (SPECT) and positron emission tomography (PET) to combat photon starvation issues and to show great benefits in noise reduction and improved accuracy in the reconstructed image.  Unlike the analytical reconstruction algorithms where each projection sample is weighted, filtered, and backprojected to formulate an image, iterative reconstruction arrives at the final solution in an iterative manner.  The initial reconstructed images are refined and modified iteratively until certain criteria are met.  The criteria are written in the form of a cost function to be minimized, which measures the fit of the image to the data according to a model of the imaging system.  Each iteration typically involves the forward and backward projection of an intermediate image volume.  The forward projection step simulates the x-ray interaction with the "object" (intermediate image volume) and produces a set of synthesized projections.  The synthesized projections are compared against the real projection measurements collected by the CT scanner.  The differences between the two are attributed to the "error" or "bias".  This error is then used to update the intermediate image volume to reduce the discrepancy between the image and the acquired data.  The new intermediate image volume is then forward-projected again in the next iteration.  Provided with the choice of an adequate cost function and optimization algorithm, the image volume will converge to an estimate near the minimizer of the cost function after several iterations.
There are both non-statistical and statisical based iterative algorithms.  non-statistical algorithms include ART, SIRT, SART, ....  Statistical based iterative algorithm tries to incorporate the counting statistics of the detected photons into the reconstruction process.  The most common statistical based IR include the maximum likelihood expectation maximization algorithm (MLEM) where it has its roots in the reconstruction algorithms for emission tomography, like PET and SPECT.
\newline
Advantages of statistical IR:\\
provide accurate physics models that can incorporate x-ray spectrum, beam-hardening effects, scatter.  It can model the x-ray detector spatial response, focal spot size and detector spectral response that results in improved spatial resolution contrast and reduce artifacts.  It can reduce image noise or x-ray dose.
\newline
Disadvantage: \\
long computation time; must reconstruct entire FOV over and over; model complexity.

There are "statistical" methods in the image (object) domain where it's simply a denoising algorithm that typically assume white noise to get rid of streaks in FBP. examples are ASIR IRIS and the technical detaisl are often a mystery...  The magical iterative denoiser comes after the noisy reconstruction to produce a "better" final image.
The second "statistical" method is applied at the sinogram level in an attempt to restore the sinogram to reduce noise. (Adaptive, Iterative,  J. Hsieh, med. Phys. 1998; Kachelrie$\beta$, Med. Phys. 2001; P. La Rivere, IEEE T-MI, 2000, 20005, 2006, 2008; Wang et al. T-MI, 2006, using PWLS-GS on sinogram.

There are five choices when it comes to statistical image reconstruction, object model, system physical model, measurement statistical model, cost function: data-mismatch and regularization, algorithm/initialization.
In Object model, we model the object as a linear series expansion approach where we parameterize the object as a summation of basis functions each weighted by coefficients.  Numerous basis functions exists, the primary contenders are voxels, and blobs (kaiser-bessel functions).

In system model, we can model the scan geometry, source (intensity fluctuations, spectrum), detector with finite size, finite x-ray spot size, detector after glow, scatter, etc.  challenges are computation times, accuracy.

In statistical model: physical model describes measurement mean. This is very complicated, particularly at low doses.  Include incident photon flux variations (Poisson), energy-dependent light production in scintillator, shot noise in photodiodes, electronic noise in readout electronics.

The interest in applying statistical methods for CT reconstruction may have been motivated by their success in emission tomography (PET and SPECT).  Contributing to this success is the fact that the maximum-likelihood expectation-maximization (ML-EM) algorithm has a closed-form expression for the emission case.
Another motivation for exploring statistical reconstruction for CT is the industry's drive towards non-Radon scanning geometries where data from several slices are acquired and reconstructed simultaneously.  For example, measurements acquired with cone-beam geometry with helical trajectory do not conform to the Radon transform assumptions, and require rebinning and interpolation prior to FBP reconstruction.  Rebinning and interpolation degrade image resolution and introduces image artifacts.  Iterative methods require no prior assumptions about the geometry of the system.  The system matrix used in iterative image reconstruction can incorporate an arbitrary geometry such as a cone-beam system, or even a cone-beam system following a helical path.  Because statistical methods outperform FBP in low count situations in PET and SPECT, where SNR is low, they carry the promise of lower dose for CT patients.  With FBP CT reconstrucion, images produced from low dose scans suffer from noise-induced streaking \citep{Hsieh1998}.  Since statistical methods are based on a probability distribution for measurement noise, they tend to suppress more the contributions of low signal rays in the data, leading to better image quality.  Major CT manufacturers are currently exploring iterative reconstruction, and are partly motivated by the desire to deliver lower dose scans. PhD thesis :\citep{Elbakri2003}

Accurate statistical modeling forms the foundation of statistical iterative reconstruction.  This statistical model leads to a cost function that is optimized by an iterative algorithm under certain constrains.

There are countless number of iterative reconstruction algorithms out there.  They all try to estimate the object function by minimize some objective functional.  Then the iterative algorithm tries to find the object that minimizes the functional. The functional is usually presented as $Q(\theta, g)$ has two terms, a data agreement portion that tries to see the difference between the data and the object that generated data based on a model, the second term can be treated as a regularizing term that provides whatever prior information we know about the object, such as positivity.!
From Harry's book: once we have chosen an objective functional to minimize, we next need an algorithm to find the minimum.  If the functional is strictly convex, then the minimum is unique and all algorithms should obtain the same image if run to convergence.  The only issues are which algorithm will find the minimum most efficiently and with the least computing resources.  In practice, however, iterative algorithms may not be run to convergence, and the resulting image depends on the algorithm, the initial estimate and the stopping rule.

 Ideally we want the functional to be convex so we can first find a solution to the problem.  Second, we want to make sure that the local minima is the global minima.  One of the common algorithm is based on the least square method that tries to find a solution that agrees with the data.  Though anytime we try to fit an object to noisy data will often result in noisy object.  Another common algorithm is MLEM where we try to find the solution that will maximizes the likelihood.  IT is a nonlinear algorithm that has magical properties but it is simple to implement.
Maximum-likelihood expectation maximization, it can be derived by alternating expectation and maximization steps, and because it maximizes the likelihood for a Poisson data model.  MLEM is but one example of a broad class of algorithms that alternate expectation and maximization steps.
\comment{Include forward models examples}

\comment{i'm completely lost on cost functions and regularizations, later?}
In cost functions:
Regularization

after talking with Eric: \\
The probability is the statistics, it is the noise part of getting the data.  The physics is the model, aka, the mean of the data, from x-ray source to the values on the detector (mean).  The functional is the thing you are trying to minimize, there's the data agreement term, i.e. least squares, then there is the prior term that's based on the object, though a lot of algorithms don't necessarily make the prior statistics based, it's just what they do not want the object to do, like have sharp edges.  The algorithm is the method of finding the minimum of the functional, whether it is conjugate gradient or coordinate descent or even surrogate figure of merit.

\comment{probably get get some most common methods, as examples}

\comment{notes taken from Beister paper}
The concept of iterative reconstruction was established in single-photon emission CT in the 1960s and was used in the first transmission CT efforts in the early 1970s (\citep{Housfield1973}).  They were successfully used in the first clinical CT products when relatively small amounts of measured data were generated per scan and reconstructed into crude 128 x 128 image matrices.
The simplest form of iterative reconstruction is the algebraic reconstruction technique (ART) \citep{Gordon1970}, it is a method for solving linear systems of equations Ax = b, where in terms of image reconstruction x are the voxels of the volume to be reconstructed, A is the system matrix used for producing the raw data dn b are the pixels of measured raw data.  The entries of the matrix A correspond to rays from the x-ray source through the volume to the detector pixels, i.e. the line integral of the linear attenuation coefficient.  Often a positivity constraint is applied to the voxels based on the assumption that negative attenuation values are not possible.  The original ART algorithm works on single rays and thus single pixels, the simultaneous algebraic reconstruction technique (SART) performs updates for complete raw data projections.  This leads to a much faster convergence of volumetric images towards a stable solution.

%Not sure if it's necessary to include subsections of recon algorithms, seems redundent!
\subsection{Filtered back projection algorithms}
\subsection{Iterative reconstruction algorithms}
\comment{These are the subsections that needs to be included, use the information above and put them into individual sections.}

Iterative reconstruction methods in X-ray CT~\cite{Beister}
\section{Maximum Likelihood Expectation Maximization (MLEM) }
In this project, we used the Maximum likelihood expectation maximization algorithm for the reconstruction process.  We did not use any regularization function.
We used a very simple model, with infinitely small x-ray point source, the x-rays travel in a straight line.  The object voxels were points centered at the voxel location.  The projection through the object volume is calculated by doing a 3D interpolation of the object as the ray passes through each plane in the object.  The summation of the contributions of the object in each of the x-ray direction is taken as the value on the detector.
The MLEM algorithm requires H and $H^{t}$ matrix, which in our case, is the forward and backward projectors.
\comment{provide assumptions} in this section
\comment{provide MLEM equation?}

\subsection{Forward Projection calculation}

\begin{enumerate}
\item forward projector calculated on the GPU
\item each thread launched is for each x-ray originate from the source to the center of each detector pixel.  Each detector is treated like a delta point
\item when the x-ray is passed through the object, the object value at each plane is calculated by 3D interpolation (machine interpolation on the GPU), so we know the (x,y,z) coordinate of the object and can retrieve its value using tex3D function provided by the CUDA SDK (Software Development Kit).  The summations of all the object value along the ray is calculated when we iterate through all object z? planes.  The number of z planes is take to be four time of the object z dimension.  The value at the detector is the summation of the object value at each plane times the distance between each planes.  The procedure is repeated for each CT rotation angle where the (x,y,z) coordinate of the objects were calculated at each angle and the object values were retrieved through 3D machine interpolation.  Figure ~\ref{fig:forwardprojection} shows the forward projection procedure.
\end{enumerate}

\begin{figure}
\centering
	\placeholderimage[width=3cm,height=3cm]{forwardprojection.png}
	\label{fig:forwardprojection}
	\caption{Forward Projection model}
\end{figure}

The equation used in forward and backward projector are identical to the ones used for calibration.

\subsection{Backward Projection calculation}
The treatment for the backward projector is similar to the forward projector, except in this case, each object voxel location is used to calculate the x-ray that would pass through the center of the object voxels and a 2D interpolation is used to back fill the value on the detector onto the object voxel.  Threads were launched for each object voxel so one iteration would be able to back fill the detector values at one CT angle onto all object voxels (assuming that the number of object voxels are fairly small, i.e. we were able to use $128^3$ object volume).  Figure ~\ref{fig:backwardprojectior} shows the procedure for back projection.
\begin{figure}
\centering
\placeholderimage[width=3cm, height=3cm]{backwardprojector.png}
\label{fig:backwardprojectior}
\caption{Backward Projection model}
\end{figure}

\subsection{Sensitivity}
The sensitivity used in the MLEM algorithm is essentially the contribution of all detector values onto the object voxel.  This was calculated by setting all detector value to a constant number, in our case all pixel values were set to 1.  The the sensitivity volume was calculated by back filling the object value using the back projector described in the above section.  Note that the sensitivity changes depending on the geometry of the CT system, number of object volume and number of detector pixels.  However if one were to use a different data set while keeping geometry, object volume, and detector pixels constant, then sensitivity volume does not need to be recalculated.

Figure \ref{fig:reconstructedimage} shows the center slice of a reconstructed object after X iterations.  The MLEM algorithm used can be speed up by using ordered-subset technique, we have also included this function in our code.

\begin{figure}
\centering
\placeholderimage[width=3cm, height=3cm]{reconstructedimage.png}
\label{fig:reconstructedimage}
\caption{Center slice of reconstructed object after x iterations}
\end{figure}

The equation for a point object located at ($x_0$, $y_0$, $z_0$) in a circular scan trajectory is given by:

\begin{equation}\label{eq:circularorbit}
\begin{split}
x_{\alpha}& = x_0 \cos \alpha \pm y_0 \sin \alpha, \\
y_{\alpha}& = \mp x_0 \sin \alpha + y_0 \cos \alpha, \\
z& \equiv z_0.
\end{split}
\end{equation}

On a perfectly well aligned detector the projection orbit of the point, $(u_{\alpha}^{(id)}, v_{\alpha}^{(id)})$, in the (x,y,z) coordinate is then given by its projections onto the detector plane,
\begin{equation}\label{eq:idealUVpoints}
u_{\alpha}^{(id)} = \frac{R x_{\alpha}}{y_{\alpha} + R_F}, \hspace{0.4cm} v_{\alpha}^{(id)} = \frac{Rz_0}{y_{\alpha}+R_F},
\end{equation}
where $R$ and $R_F$ are the distances from the focus to the detector and the axis of rotation, respectively.  For a misaligned detector the point (u,v) with (x', y', z') coordinates, the point locations on the detector is given by:

\begin{equation}\label{eq:circularorbit}
\begin{split}
x_{\alpha}& = x_0 \cos \alpha \pm y_0 \sin \alpha, \\
y_{\alpha}& = \mp x_0 \sin \alpha + y_0 \cos \alpha, \\
z& \equiv z_0.
\end{split}
\end{equation}

On a perfectly well aligned detector the projection orbit of the point, $(u_{\alpha}^{(id)}, v_{\alpha}^{(id)})$, in the (x,y,z) coordinate is then given by its projections onto the detector plane,
\begin{equation}\label{eq:idealUVpoints}
u_{\alpha}^{(id)} = \frac{R x_{\alpha}}{y_{\alpha} + R_F}, \hspace{0.4cm} v_{\alpha}^{(id)} = \frac{Rz_0}{y_{\alpha}+R_F},
\end{equation}
where $R$ and $R_F$ are the distances from the focus to the detector and the axis of rotation, respectively.  For a misaligned detector the point (u,v) with (x', y', z') coordinates, the point locations on the detector is given by:

\begin{equation}\label{eq:nonidealUVpoints}
\begin{split}
\begin{pmatrix}
u \\ v
\end{pmatrix} = & \frac{1}{\det \mathbf{Q}}
\begin{pmatrix}
o_{33}-o_{23}v^{(id)}/R & -(o_{13} - o_{23} u^{(id)}/R) \\
-(o_{31} - o_{21} v^{(id)}/R) & o_{11}-o_{21} u^{(id)}/R \\
\end{pmatrix} \\
\times &
\begin{pmatrix}
u^{(id)'} - d_x \\
v^{(id)'} - d_z
\end{pmatrix},
\end{split}
\end{equation}
where the primes denote a simple rescaling of the ideal orbit by

\begin{equation}
\begin{pmatrix}
u^{(id)'} \\ v^{(id)'}
\end{pmatrix}
= \frac{R_y'}{R}
\begin{pmatrix}
u^{(id)} \\ v^{(id)}
\end{pmatrix}.
\end{equation}
where
\begin{equation}
\begin{split}
\det \mathbf{Q} =& (o_{11} - o_{21} u^{(id)}/R)(o_{33}-o_{23}v^{(id)}/R) \\
& -(o_{13}-o_{23}u^{(id)}/R)(o_{31}-o_{21} v^{(id)}/R).
\end{split}
\end{equation}

and $\mathbf{O}$ is the rotation matrix of the misaligned detector shown by:
\begin{equation}
\mathbf{O} =
\begin{pmatrix}
\cos \eta \cos \varphi - \sin \eta \sin \theta \sin \varphi & -\cos \theta \sin \varphi & -\cos \varphi \sin \eta - \cos \eta \sin \theta \sin \varphi \\
\cos \varphi \sin \eta \sin \theta + \cos \eta \sin \varphi & \cos \theta \cos \varphi & \cos \eta \cos \varphi \sin \theta - \sin \eta \sin \varphi \\
\cos \theta \sin \eta & -\sin \theta & \cos \eta \cos \theta
\end{pmatrix}
\end{equation}

\section{Filtered Back-Projection (FBP) }


\documentclass[11pt]{report}
\usepackage{amsmath, bm, xcolor, graphicx, listings}
\usepackage{courier} %for listings
\usepackage{caption}
\usepackage{subcaption}
\lstset{basicstyle=\footnotesize\ttfamily,breaklines=true} %listing font settings
%\lstset{framextopmargin=50pt} %listing font settings
\usepackage[english]{babel}
\setlength\parindent{0pt} %remove indentation in new paragraphs

\begin{document}
\title{\textbf{Poisson Solver (2D)}}
\author{Nikos Tryfonidis}
\date{2015}

\maketitle

\begin{abstract}
The current report summarizes the structure, use and results of a two-dimensional Poisson solver,
created for the purpose of being integrated into a 2D Particle-In-Cell (PIC) code. After a brief introduction
regarding the numerical solution of the 2D Poisson PDE in chapter 1, the structure of the code will be
described in chapter 2. Verification of the code will be given in chapter 3, through a number of test cases.
The different iterative methods that have been implemented are discussed in chapter 4, along with a brief review of their performance. Finally, the parallel implementation of the iterative solvers, using OpenMP threads, will be described in the fifth chapter, with a scaling performance review for each method.
\end{abstract}

\tableofcontents

\chapter{2D Poisson Equation: Numerical Solution}
The Poisson Equation and the associated boundary conditions form what is generally known as a
\emph{Poisson Problem}:

\begin{equation}
\nabla ^2 u = u_{xx} + u_{yy} = f(x,y)
\end{equation}

The Poisson Problem is one of the most well-known Elliptic Partial Differential Equations (PDE). A brief description of the numerical method followed in the code will be given here. Detailed analysis of the Poisson PDE and other methods can be found in most numerical analysis textbooks (for example \cite{leveque}).

\section{The Finite Difference Scheme}
The 5-point stencil discretization scheme is followed. A uniform Cartesian grid is used, where each point $x_i, y_j$ describes a point in 2D space, with $x_i = i\Delta x$ and $y_j = j\Delta y$. We assume that $0 \leq x \leq 1$ and $0 \leq y \leq 1$ for simplicity. Replacing the derivatives $u_{xx}$ and $u_{yy}$ in (1.1) with centered finite difference schemes, we end up with the following discretization scheme:

\begin{equation}
\frac{1}{(\Delta x)^2}(u_{i-1,j} -2u_{i,j} + u_{i+1,j}) +
\frac{1}{(\Delta y)^2}(u_{i,j-1} -2u_{i,j} + u_{i,j+1}) = f_{i,j}
\end{equation}

Assuming that $\Delta x = \Delta y = h$, we get the simple finite difference scheme

\begin{equation}
u_{i-1,j} + u_{i+1,j} + u_{i,j-1} + u_{i,j+1} - 4u_{i,j} = h^2 f_{i,j}
\end{equation}

For a grid of $N \times N$ points in total, this scheme gives a linear system of $(N-2) \times (N-2)$ equations for the $(N-2) \times (N-2)$ unknowns (the excluded values are the boundaries, which are considered known). As an example to illustrate the system of equations, suppose we have a $5 \times 5$ grid of points, including boundaries, that represent our 2D space.

\begin{figure}[h]
\centering
\includegraphics[scale=0.35]{images/grid}
\caption{A $5 \times 5$ two-dimensional grid.
Boundary points, for which the solution is known, are shown in blue. }
\label{fig:Space grid 5x5}
\end{figure}

Applying the finite difference scheme (1.3) to each inner grid point $u_{i,j}$ leads us to a system of $(5-2) \times (5-2) = 9$ equations:

\begin{subequations}
\begin{align}
\bm{u_{1,1}}:
\quad  & \textcolor{blue}{u_{0,1}} + u_{2,1} + \textcolor{blue}{u_{1,0}} + u_{1,2} - 4u_{1,1} = h^2 f_{1,1} \\
\bm{u_{1,2}}:
\quad  & \textcolor{blue}{u_{0,2}} + u_{2,2} + u_{1,1} + u_{1,3} - 4u_{1,2} = h^2 f_{1,2} \\
\bm{u_{1,3}}:
\quad  & \textcolor{blue}{u_{0,3}} + u_{2,3} + u_{1,2} + \textcolor{blue}{u_{1,4}} - 4u_{1,3} = h^2 f_{1,3} \\
\bm{u_{2,1}}:
\quad  & u_{1,1} + u_{3,1} + \textcolor{blue}{u_{2,0}} + u_{2,2} - 4u_{2,1} = h^2 f_{2,1} \\
\bm{u_{2,2}}:
\quad  & u_{1,2} + u_{3,2} + u_{2,1} + u_{2,3} - 4u_{2,2} = h^2 f_{2,2} \\
\bm{u_{2,3}}:
\quad  & u_{1,3} + u_{3,3} + u_{2,2} + \textcolor{blue}{u_{2,4}} - 4u_{2,3} = h^2 f_{2,3} \\
\bm{u_{3,1}}:
\quad  & u_{2,1} + \textcolor{blue}{u_{4,1}} + \textcolor{blue}{u_{3,0}} + u_{3,2} - 4u_{3,1} = h^2 f_{3,1} \\
\bm{u_{3,2}}:
\quad  & u_{2,2} + \textcolor{blue}{u_{4,2}} + u_{3,1} + u_{3,3} - 4u_{3,2} = h^2 f_{3,2} \\
\bm{u_{3,3}}:
\quad  & u_{2,3} + \textcolor{blue}{u_{4,3}} + u_{3,2} + \textcolor{blue}{u_{3,4}} - 4u_{3,3} = h^2 f_{3,3}
\end{align}
\end{subequations}

\vspace{0.3cm}
We have arranged the equations row-wise, grouping elements with the same row number ($i$ index) together. We
also see the boundary grid points in blue. These are considered to be known from the given boundary conditions.
\newline

We can now present the linear system of equations in matrix form,

\begin{equation}
\overleftrightarrow{\textbf{A}} \cdot \overrightarrow{\textbf{u}} = \overrightarrow{\textbf{b}},
\end{equation}

\vspace{0.3cm}

where $\overleftrightarrow{\textbf{A}}$ is the matrix equation, $\overrightarrow{\textbf{u}}$ is the vector of unknowns and $\overrightarrow{\textbf{b}}$ is the right-hand-side of the system of equations. Note that we will move the known boundary grid points to the right-hand-side of the matrix equation (again, shown in blue). The resulting matrix equation is

%%%Matrix with labels (uses parentheses):
\begin{equation*}
\let\quad\thinspace
\bordermatrix{\text{}& \bm{u_{1,1}} & \bm{u_{1,2}} & \bm{u_{1,3}} & \bm{u_{2,1}} & \bm{u_{2,2}} & \bm{u_{2,3}} & \bm{u_{3,1}} & \bm{u_{3,2}} & \bm{u_{3,3}} \cr
   & -4 & 1 & 0 & 1 & 0 & 0 & 0 & 0 & 0  \cr
   & 1 & -4 & 1 & 0 & 1 & 0 & 0 & 0 & 0  \cr
   & 0 & 1 & -4 & 0 & 0 & 1 & 0 & 0 & 0  \cr
   & 1 & 0 & 0 & -4 & 1 & 0 & 1 & 0 & 0  \cr
   & 0 & 1 & 0 & 1 & -4 & 1 & 0 & 1 & 0  \cr
   & 0 & 0 & 1 & 0 & 1 & -4 & 0 & 0 & 1  \cr
   & 0 & 0 & 0 & 1 & 0 & 0 & -4 & 1 & 0  \cr
   & 0 & 0 & 0 & 0 & 1 & 0 & 1 & -4 & 1  \cr
   & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & -4
    }
\cdot
\begin{pmatrix}
u_{1,1} \\ u_{1,2} \\ u_{1,3} \\ u_{2,1} \\ u_{2,2} \\ u_{2,3} \\ u_{3,1} \\ u_{3,2} \\ u_{3,3}
\end{pmatrix}
=
\begin{pmatrix}
h^2f_{1,1} + \textcolor{blue}{u_{0,1}} + \textcolor{blue}{u_{1,0}} \\
h^2f_{1,2} + \textcolor{blue}{u_{0,2}} \\
h^2f_{1,3} + \textcolor{blue}{u_{0,3}} + \textcolor{blue}{u_{1,4}} \\
h^2f_{2,1} + \textcolor{blue}{u_{2,0}} \\
h^2f_{2,2} \\
h^2f_{2,3} + \textcolor{blue}{u_{2,4}} \\
h^2f_{3,1} + \textcolor{blue}{u_{4,1}} + \textcolor{blue}{u_{3,0}} \\
h^2f_{3,2} + \textcolor{blue}{u_{3,2}} + \textcolor{blue}{u_{4,2}} \\
h^2f_{3,3} + \textcolor{blue}{u_{4,3}} + \textcolor{blue}{u_{3,4}}
\end{pmatrix}
\end{equation*}
\vspace{0.75cm}

We see that the resulting $\overleftrightarrow{\textbf{A}}$ matrix is block-tridiagonal:

\begin{equation}
\overleftrightarrow{\textbf{A}} =
\begin{pmatrix}
\bm{T} & \bm{I} & \bm{0} \\
\bm{I} & \bm{T} & \bm{I} \\
\bm{0} & \bm{I} & \bm{T}
\end{pmatrix}
\end{equation}
\vspace{0.5cm}
\begin{equation}
\bm{T} =
\begin{pmatrix}
-4 & 1 & 0 \\
1 & -4 & 0 \\
0 & 1 & -4
\end{pmatrix}
, \quad
\bm{I} =
\begin{pmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{pmatrix}
, \quad
\bm{0} =
\begin{pmatrix}
0 & 0 & 0 \\
0 & 0 & 0 \\
0 & 0 & 0
\end{pmatrix}
\end{equation}

\vspace{0.5cm}

This linear system of equations can be solved with a variety of methods (direct, iterative etc). Our focus will be on iterative methods.

\section{Error and Stability}
We will not go into the details of error and stability analysis in this report, since these are well-known
and can be found in most relevant texts (e.g. \cite{leveque}). We should simply keep in mind that the local truncation error of the 5-point stencil finite difference scheme we are using is $O(h^2)$ and that the method is stable, so the global error of the method is also $O(h^2)$. In case we do not assume $\Delta x = \Delta y = h$, this
simply means that the error is $O\left( (\Delta x)^2, (\Delta y)^2 \right)$.

\chapter{The Code}
In this chapter, the structure and use of the code will be described. The project has been arranged into
separate directories. The source code is in "src", with corresponding header files in "headers", one for each
.c source file (other than "main.c"). Object files (".o") are automatically placed into the "obj" directory by the makefile. Output files are also placed into the "output" directory. The python visualization script used
for plotting the output is also in "output". Finally, "documentation" contains the current report.

In the following sections, we will briefly go through instructions on how to use the code, and we will review its structure.

\section{Compilation and Execution}
The code does not contain any external libraries and should compile without problems on any linux system with a C compiler. To compile, go to the central directory of the project, where the makefile is located, and type "make". This will create the executable file in the same directory, with object files in "obj". To delete the executable and the object files, type "make clean".

To run, simply run the executable from the command line, giving two arguments (number of grid points in dimensions X and Y, respectively). For example, the following will run the program with 101 grid points in X and Y:

\begin{lstlisting}
./poisson2D 101 101
\end{lstlisting}

\section{Code Structure}
Here we will briefly go through the code structure, by explaining the work done by each of the source files.

\begin{description}
  \item[main.c] The main file simply wraps all pieces of the program together and can be seen as a program outline. First it reads the command line arguments and it allocates the required memory. Then, it sets up the problem, by setting the appropriate Boundary Conditions and right-hand-side function f(x,y) (see "setup.c").
  \item[io.c] Contains functions that read command line arguments and write file output.
  \item[setup.c] Contains functions that set boundary conditions and the right-hand-side function. This is where the user can specify the Poisson problem he/she wants to solve. \emph{Boundary conditions} can be set for each one of the four boundaries separately, by the $"setBoundaries2D"$ function. The RHS function is set by $"setRHS2D"$.
  \item[memory.c] Contains the $"array2D_contiguous"$ function, that dynamically allocates a two-dimensional array of doubles, contiguous in memory. Also contains the $"free_array2D_contiguous"$ function, that frees memory allocated by the previous function.
  \item[poisson2D.c] The solver function. It iterates the solution for the appropriate number of iterations, until the desired accuracy is reached (or a maximum number of iterations is done). The iteration method of choice is passed into the function as an argument (function pointer). Error tolerance, maximum iterations, iterations per residual check are all set in this function. Also contains the residual calculation function.
  \item[iterative.c] Contains the different iterative methods that may be used, and that are called by $"solvePoisson2D"$. Currently, the Jacobi, Gauss-Seidel and Successive Overrelaxation methods have been implemented.
\end{description}

The space dimensions $(x_0, x_L), (y_0, y_L)$ are set in \emph{definitions.h}, inside the \emph{headers} directory.

\section{Data Structures}
The main data structure used is a two-dimensional array of doubles, that represents the discretized space grid. The dimensions of the grid are given from the command line arguments the user provides. Memory for the 2D array is allocated by the $"array2D_contiguous"$ function in \emph{memory.c}.
\newline

Also, another two-dimensional array of the same dimensions is the right-hand-side function (RHS) of the Poisson Equation, $f(x,y)$. Memory for the RHS function is allocated in the same way as for the grid. The values of RHS for each grid point is set by the $"setRHS2D"$ function in \emph{setup.c}.

\section{Solving a Poisson Problem}
To solve a given Poisson problem, the user must perform the following actions:

\begin{enumerate}
  \item Set Boundary Conditions in \emph{setBoundaries2D} (setup.c).
  \item Set the RHS function of the Poisson problem in \emph{setRHS2D} (setup.c).
  \item Choose the desired iterative method in main function (main.c).
  \item Compile the program by typing "make" in the main directory (where the makefile is located).
  \item Run the executable with the desired number of grid points in each of the two dimensions.
\end{enumerate}

\section{Output}
Output is written in the "output" directory. Two output files are written:

\begin{description}
  \item[output.txt] Contains the 2D solution grid.
  \item[gridData.txt] Contains grid data $\left( (x_0, x_L), (y_0, y_L) \right) $, step sizes ($dx, dy$) and number of grid points in each dimension ($N_x, N_y$). Data from this file is read by the visualization python script in the same folder, in order to create a plot of the results.
\end{description}

The solution is visualized by the python script \emph{visualize.py}, in the output directory.
The script requires \emph{numpy} and \emph{matplotlib}, two well-known packages for scientific computing in Python.

These can be easily installed in any Linux system, for example in Ubuntu:

\begin{lstlisting}
sudo apt-get install python-numpy python-matplotlib
\end{lstlisting}

This python script reads grid meta-data from \emph{gridData.txt} and the solution from \emph{output.txt} and creates a 3D plot of the solution.

To create the plot, the user must simply run the script by typing:

\begin{lstlisting}
python visualize.py
\end{lstlisting}

\chapter{Verification}
In this chapter we will verify the correctness of the 2D Poisson solver, through three basic examples that have known analytic solutions. By comparing the numerical solution with the analytic one, we will show that the solver produces correct results.
\newline

Each example has detailed instructions on how to set up the Poisson problem studied, so the following sections can also be used as examples on how to use the code. The \emph{Gauss-Seidel} iterative method was chosen for these examples, with a tolerance of $\Delta x^2$ for the residual check.
\newline

For the following examples, the domain was chosen to be $0 \leq x,y \leq 1$, which is set in \emph{headers/definitions.h} as follows:

\begin{lstlisting}
#define X0 0.0
#define XL 1.0
#define Y0 0.0
#define YL 1.0
\end{lstlisting}

We also used 101 grid points for each dimension, dividing each dimension into $(101 - 1)$ pieces, so that
\begin{equation}
\Delta x = \Delta y = (Length) / (101 - 1) = 1.0 / 100 = 0.01
\end{equation}
\newpage

\section{Zero RHS (Laplace Equation)}
In this problem, we will solve a Poisson problem with no right-hand-side function. Let us assume that the (known) solution $u(x,y)$ is

\begin{equation}
u(x,y) = 3x + 4y
\end{equation}

The corresponding Poisson problem is then

\begin{subequations}
\begin{align}
\nabla ^2 u = &u_{xx} + u_{yy} = 0 \\
u(x,0) = 3x \quad \quad \quad & \quad \quad \quad u(x,y_L) = 3x + 4y_L \\
u(0,y) = 4y \quad \quad \quad & \quad \quad \quad u(x_L,y) = 3x_L + 4y
\end{align}
\end{subequations}

Equation $(3.2a)$, along with boundary conditions $(3.2b)$ and $(3.2c)$ are also known as a Laplace problem, a special case of the Poisson problem with a zero right-hand-side function. We will now solve this Poisson problem and compare the numerical solution with the analytic $(3.1)$.

\subsection*{Setting Up The Problem}
As explained in paragraph \emph{2.4}, the functions that are used to set up the desired Poisson problem are in \emph{setup.c}. Boundary conditions are given by the known solution \emph{(3.1)}, while the RHS function is zero for this problem.
\newline

We set the Boundary Conditions:

\begin{lstlisting}
double fBC(double x, double y)
{
    return 3.0*x + 4.0*y;
}
\end{lstlisting}

and the RHS function:

\begin{lstlisting}
double f(double x, double y)
{
    return 0.0;
}
\end{lstlisting}

\subsection*{Running the Solver}
After we set up the problem as shown, we compile the program and run. To specify 101 grid points for $x$ and $y$, we run the executable as follows:

\begin{lstlisting}
./poisson2D 101 101
\end{lstlisting}

Once the program has finished, output is ready in \emph{output/output.txt}. We can plot the solution by running the python script \emph{visualize.py}, which will create a 3D surface plot of our solution. In the next figure, the analytic and numerical solutions are shown side by side for comparison:

\begin{figure}[h!]
\centering
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example1_analytic}
  \caption{Analytic solution}
  \label{fig:sub1}
\end{subfigure}%
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example1_numerical}
  \caption{Numerical solution}
  \label{fig:sub2}
\end{subfigure}
\begin{subfigure}{0.6\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example1_error}
  \caption{Error}
  \label{fig:sub2}
\end{subfigure}
\caption{Analytic (a) and numerical (b) solution for the current Poisson problem, with a surface plot of the  error in (c).}
\label{fig:example1}
\end{figure}

We see that the numerical solution is identical to the analytic one, and the error is shown to be of the order of $10^{-15}$ (machine precision, since with a constant right-hand-side we are effectively calculating averages).

\section{Simple Linear RHS}
Now, we will solve a Poisson problem with a simple linear right-hand-side function. Let us assume that the (known) solution $u(x,y)$ is

\begin{equation}
u(x,y) = x^2 + y^2
\end{equation}

The corresponding Poisson problem is now

\begin{subequations}
\begin{align}
\nabla ^2 u = &u_{xx} + u_{yy} = 2 + 2 = 4 \\
u(x,0) = x^2 \quad \quad \quad & \quad \quad \quad u(x,y_L) = x^2 + (y_L)^2 \\
u(0,y) = y^2 \quad \quad \quad & \quad \quad \quad u(x_L,y) = (x_L)^2 + y^2
\end{align}
\end{subequations}

We will now solve this Poisson problem and compare the numerical solution with the analytic $(3.4)$.

\subsection*{Setting Up The Problem}
We set up the desired Poisson problem, as shown in the previous example, in \emph{setup.c}. Boundary conditions are given by the known solution \emph{(3.4)}, while the RHS function is equal to $4$ for this problem.
\newline

We set the Boundary Conditions:

\begin{lstlisting}
double fBC(double x, double y)
{
    return x*x + y*y;
}
\end{lstlisting}

and the RHS function:

\begin{lstlisting}
double f(double x, double y)
{
    return 4.0;
}
\end{lstlisting}

\subsection*{Running the Solver}
We compile and run, once again, with 101 grid points in each dimension, as before:
\newline

\begin{lstlisting}
./poisson2D 101 101
\end{lstlisting}

We produce the surface plot of the results with \emph{visualize.py}. In the next figure, the analytic and numerical solutions are shown side by side for comparison:

\begin{figure}[h!]
\centering
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example2_analytic}
  \caption{Analytic solution}
  \label{fig:sub2.1}
\end{subfigure}%
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example2_numerical}
  \caption{Numerical solution}
  \label{fig:sub2.2}
\end{subfigure}
\begin{subfigure}{0.6\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example2_error}
  \caption{Error}
  \label{fig:sub2.3}
\end{subfigure}
\caption{Analytic (a) and numerical (b) solution for the current Poisson problem, with a surface plot of the  error in (c).}
\label{fig:example2}
\end{figure}

We see that the numerical solution is identical to the analytic one, and the error is shown to be of the order of $10^{-16}$ (machine precision again, since with a constant right-hand-side we are effectively calculating averages).

\newpage
\section{Sinusoidal RHS}
Now, we will solve a Poisson problem with a slightly more complex right-hand-side function. Let us assume that the (known) solution $u(x,y)$ is

\begin{equation}
u(x,y) = sin\left[ 2\pi \left( x + y\right) \right]
\end{equation}

The corresponding Poisson problem is now

\begin{subequations}
\begin{align}
\nabla ^2 u = u_{xx} + u_{yy} &= -8\pi^2 sin\left[ 2\pi \left( x + y\right) \right] \\
u(x,0) = sin\left[ 2\pi \left( x \right) \right] \quad \quad & \quad \quad
u(x,y_L) = sin\left[ 2\pi \left( x + y_L\right) \right]\\
u(0,y) =  sin\left[ 2\pi \left( y \right) \right] \quad \quad & \quad \quad
u(x_L,y) =  sin\left[ 2\pi \left( x_L + y\right) \right]
\end{align}
\end{subequations}

We will now solve this Poisson problem and compare the numerical solution with the analytic $(3.6)$.

\subsection*{Setting Up The Problem}
We set up the desired Poisson problem, as shown in the previous examples, in \emph{setup.c}. Boundary conditions are given by the known solution \emph{(3.6)}, while the RHS function is $f(x,y) =  -8\pi^2 sin\left[ 2\pi \left( x + y\right) \right]$ for this problem.
\newline

We set the Boundary Conditions:

\begin{lstlisting}
double fBC(double x, double y)
{
    return sin(2.0*M_PI*(x+y));
}
\end{lstlisting}

and the RHS function:

\begin{lstlisting}
double f(double x, double y)
{
    return -8.0*M_PI*M_PI*sin(2.0*M_PI*(x+y));
}
\end{lstlisting}

\subsection*{Running the Solver}
We compile the program and run with 101 grid points in each dimension, as before:
\newline

\begin{lstlisting}
./poisson2D 101 101
\end{lstlisting}

We produce the surface plot of the results with \emph{visualize.py}. In the next figure, the analytic and numerical solutions are shown side by side for comparison. Also, the error surface plot is shown; this was created by plotting the difference of the analytic minus the numerical solution.

\begin{figure}[h!]
\centering
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example3_analytic}
  \caption{Analytic solution}
  \label{fig:sub3.1}
\end{subfigure}%
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example3_numerical}
  \caption{Numerical solution}
  \label{fig:sub3.2}
\end{subfigure}
\begin{subfigure}{0.6\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example3_error}
  \caption{Error}
  \label{fig:sub3.3}
\end{subfigure}
\caption{Analytic (a) and numerical (b) solution for the current Poisson problem, with a surface plot of the  error in (c).}
\label{fig:example3}
\end{figure}

We see that the numerical solution is identical to the analytic one, and the error is shown to be, as expected, of the order of $O(h^2) = O(0.0001)$, where $h = \Delta x = \Delta y = 0.01$.

\section{Minimizing the Error}
As a last verification test, we will show that our solution becomes better (i.e. the error becomes smaller), as we refine our grid. We will use the previous section's Poisson problem to show this.
\newline

We will plot the error, using an increasing number of grid points. We expect the error to get smaller as $dx, dy$ become smaller. The results are shown in the following two figures:

\begin{figure}[h!]
\centering
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example4_10}
  \caption{$N = 10 \quad (h = 0.1)$}
\end{subfigure}
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example4_100}
  \caption{$N = 100  \quad (h = 0.01)$}
\end{subfigure}
\caption{Error ($u_{analytic} - u_{numerical}$) for decreasing $h = dx = dy$. The error gets smaller as we go from 10 to 100 grid points (see Error axis range).}
\label{fig:example4.1}
\end{figure}

We notice that the error does get smaller as $h$ decreases, and also it is of order $O(h^2)$, as expected. Even smaller $h$ choices are shown in the next figure:

\begin{figure}[h!]
\centering
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example4_200}
  \caption{$N = 200 \quad (h = 0.005)$}
\end{subfigure}\\
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example4_500}
  \caption{$N = 500 \quad (h = 0.002)$}
\end{subfigure}
\begin{subfigure}{0.5\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/example4_1000}
  \caption{$N = 1000 \quad (h = 0.001)$}
\end{subfigure}
\caption{Error ($u_{analytic} - u_{numerical}$) for decreasing $h = dx = dy$. The error gets smaller as we go from 200 to 1000 grid points (see Error axis range). }
\label{fig:example4.2}
\end{figure}

\chapter{Iterative Methods}
In this chapter, we will briefly describe the different iterative methods that have been implemented for the solution of the linear system of equations of the Poisson problem. After that, a convergence review will be given for the methods. A detailed analysis of the iterative methods is outside the scope of this report, but can be found in most numerical PDE books (e.g. \cite{leveque}).
\newline

The code for the iterative methods described in this chapter can be found in the \emph{iterative.c} source file. The desired iterative method can be chosen in \emph{main.c}, where it is passed as function pointer to the Poisson solver function.

\section{Jacobi Method}
The Jacobi method is the simplest iterative method that we will examine, and also the least efficient, as it has the slowest convergence. However, it is useful, not only as an example, but as a smoother in multigrid methods \cite{leveque}.
\newline

The Poisson problem we are solving can be written as follows, assuming that $dx = dy = h$, from equation (1.3):

\begin{equation}
u_{i,j} = \frac{1}{4} \left(  u_{i-1, j} + u_{i+1, j} + u_{i, j-1} + u_{i, j+1}  \right) - \frac{h^2}{4}f_{i,j}
\end{equation}

The Jacobi method can be described by the following algorithm:

\begin{equation}
u_{i,j}^{k+1} = \frac{1}{4} \left(  u_{i-1, j}^k + u_{i+1, j}^k + u_{i, j-1}^k + u_{i, j+1}^k  \right) - \frac{h^2}{4}f_{i,j}
\end{equation}

where $u^{k}$ is the \emph{current} state of the solution, while $u^{k+1}$ is the new estimation. It is obvious that this method requires more memory, as the previous estimation of the solution has to be stored.
\newline

Alternatively, if we do not assume that $dx = dy$, as in equation (1.2), we write the Poisson problem as follows:

\begin{equation}
\begin{split}
u_{i,j} \quad = & \quad \frac{1}{2(dx^2 + dy^2)} \left(  (u_{i-1, j} + u_{i+1, j})dy^2 + (u_{i, j-1} + u_{i, j+1})dx^2  \right)\\
 & - \frac{dx^2 dy^2}{2(dx^2+dy^2)}f_{i,j}
\end{split}
\end{equation}

and the Jacobi method is now

\begin{equation}
\begin{split}
u_{i,j}^{k+1} \quad = & \quad \frac{1}{2(dx^2 + dy^2)} \left(  (u_{i-1, j}^k + u_{i+1, j}^k)dy^2 + (u_{i, j-1}^k + u_{i, j+1}^k)dx^2  \right) \\
& - \frac{dx^2 dy^2}{2(dx^2+dy^2)}f_{i,j}
\end{split}
\end{equation}

\section{Gauss - Seidel Method}
The Gauss-Seidel method is a variation of the Jacobi method; it improves the solution by using the already calculated values for the new estimation. This simply translates to the following algorithm:

\begin{equation}
u_{i,j}^{k+1} = \frac{1}{4} \left(  u_{i-1, j}^{k+1} + u_{i+1, j}^k + u_{i, j-1}^{k+1} + u_{i, j+1}^k  \right) - \frac{h^2}{4}f_{i,j}
\end{equation}

or, keeping $dx \neq dy$:

\begin{equation}
\begin{split}
u_{i,j}^{k+1} \quad = & \quad \frac{1}{2(dx^2 + dy^2)} \left(  (u_{i-1, j}^{k+1} + u_{i+1, j}^k)dy^2 + (u_{i, j-1}^{k+1} + u_{i, j+1}^k)dx^2  \right) \\
 & - \frac{dx^2 dy^2}{2(dx^2+dy^2)}f_{i,j}
\end{split}
\end{equation}

The Gauss-Seidel method converges roughly twice as fast as the Jacobi method, and it does not require additional memory. However, it is less straightforward to parallelize (but can be parallelized using a red-black scheme \cite{karniadakis}).

\section{Successive Overrelaxation}
Successive overrelaxation (SOR) is an improvement on the Gauss-Seidel method, using an additional scalar parameter $\omega$, which is essentially moves the solution farther than Gauss-Seidel if $\omega > 1$ (and nearer if $\omega < 1$). SOR can be described by the following algorithm:

\begin{equation}
u_{i,j}^{k+1} = \frac{\omega}{2} \left(  u_{i-1, j}^{k+1} + u_{i+1, j}^k + u_{i, j-1}^{k+1} + u_{i, j+1}^k  \right) - \frac{h^2}{4}f_{i,j}
\end{equation}

and, in case that $dx \neq dy$:

\begin{equation}
\begin{split}
u_{i,j}^{k+1} \quad = & \quad \omega \left[ \frac{1}{2(dx^2 + dy^2)}
\left(  (u_{i-1, j}^{k+1} + u_{i+1, j}^k)dy^2 + (u_{i, j-1}^{k+1} + u_{i, j+1}^k)dx^2  \right) \right.\\
& \left. - \frac{dx^2 dy^2}{2(dx^2+dy^2)}f_{i,j} \right] + (1 - \omega) u_{i,j}^k
\end{split}
\end{equation}

It should be mentioned that SOR is unstable for $\omega > 2$, while for $\omega = 1$ it is equivalent to the Gauss-Seidel method. Regarding the optimal choice for $\omega$, in general it depends on the problem (and may be hard to find). However, for the Poisson problem it can be shown that $\omega_{opt} \simeq 2 - 2 \pi h$ (\cite{leveque}, par. 4.2.2).

\section{Residual Checking}
Before commenting on the convergence of the previous methods, it will be useful to show the method followed in order to validate our iterative solution. The method followed is the well-known \emph{residual checking} method. This involves calculating the estimated solution between a fixed number of steps checking if the residual is smaller than a predetermined tolerance.
\newline

If $\bm{A}$ is the equation matrix and $\bm{u}$ is the estimated solution after a number of iterations, then

\begin{equation}
\bm{A} \bm{u} = \bm{f}_{est}
\end{equation}

But since we already know the right-hand values of our equations, we can quantify the correctness of our estimation by measuring how much our estimation differs from the actual right-hand side, by calculating the \emph{residual}:

\begin{equation}
r = \bm{A} \bm{u} - \bm{f}_{est}
\end{equation}

and stopping when it becomes smaller than some value that we set:

\begin{equation}
\sqrt{\sum_{i,j}r_{i,j}^2} < tolerance
\end{equation}

This procedure is carried out by the \emph{residual} function in \emph{src/poisson.c}.

\section{Convergence Comparison}
In this section we will show a brief comparison of the three iterative methods described in the revious sections. For this comparison, the Poisson problem shown in section (3.3) was solved, using $201 \times 201$ grid points.
\newline

To produce the following results, the program was altered slightly in order to output the residual in every iteration. Checking for the residual in every iteration is probably not worth the effort in general, but it is useful in order to evaluate the convergence of the three iterative methods.
\newline

We have set the maximum number of iterations to 1000 and the tolerance to zero; in this way, we can see the convergence of all three methods for 1000 iterations. Results are shown below using semi-log axes:
\newline

\begin{figure}[h!]
\centering
\includegraphics[width=0.75\linewidth]{images/residual}
\caption{Residual versus number of iterations, for 3 different iterative methods (Jacobi, Gauss-Seidel, SOR). Problem size: $201 \times 201$}
\label{fig:residual}
\end{figure}

We see that SOR is by far the most effective method. However, we should keep in mind that we are using the optimal $\omega$ value for SOR, which is known for this problem. Gauss-Seidel converges roughly twice as fast compared to Jacobi, something that is well known in general. Further analysis regarding the convergence of the iterative methods is outside the scope of this report, but can be found in \cite{leveque}, \cite{karniadakis}.

\chapter{Parallel Poisson Solver}
In this chapter, we will briefly show the parallel implementation of the Poisson solver in 2D, using OpenMP threads. A good review of OpenMP can be found in \cite{usingopenmp}. In this report, we will simply show the parallel implementation of the iterative solvers, along with their performance.

It should be mentioned that the code is maintained with the OpenMP directives, which can just be omitted if serial execution is desired. Also, for the Jacobi method, a number of OpenMP optimizations have been made, which are described in the next section; the uploaded code only features the optimized version. All other methods have their corresponding OpenMP parallel functions, starting with "redBlack" (from the red-black partitioning scheme, explained in section 5.2). To run in parallel, the user should choose the desired parallel function and compile.

OpenMP code can run as usual. First, we set the desired number of threads, for example for 4 threads we type:

\begin{lstlisting}
export OMP_NUM_THREADS=4
\end{lstlisting}

It is recommended, for systems using hyperthreading, to tie the threads to specific cores, using the command below (for this example using 4 threads):

\begin{lstlisting}
export GOMP_CPU_AFFINITY="0 1 2 3"
\end{lstlisting}

This ties the threads to the corresponding cores, ensuring that two different threads will not be assigned to a single hyperthreading core.

\section{Parallel Jacobi}
An initial parallelization of the Jacobi algorithm is very straightforward, and requires no further work. The OpenMP parallel version of the Jacobi method is shown below:

\begin{lstlisting}
#pragma omp parallel for schedule(static)\
                         shared(u, u_old, f)\
                         private(i,j)\
                         firstprivate(nX, nY, dx, dy)\
                         default(none)
for(i=1;i<nX-1;i++) {
  for(j=1;j<nY-1;j++) {
     u[i][j] = ((dy*dy)*(u_old[i-1][j] + u_old[i+1][j]) +
               (dx*dx)*(u_old[i][j-1] + u_old[i][j+1]) -
               (dx*dx)*(dy*dy)*f[i][j])/(2.0*((dx*dx) + (dy*dy)));
  }
}
\end{lstlisting}

Arrays are declared as \emph{shared} and no additional synchronization is required, since each thread only reads from and writes to its own part of the arrays. Array boundaries \emph{(nX, nY)} and values for \emph{(dx, dy)} are declared as \emph{firstprivate}, so that each thread has its own copy of these, with the correct initialized values. Finally, a static schedule is chosen with no chunksize specified. This results in the workload being equally distributed to the threads, in chunks equal to threads in number (for example, if we use 4 threads, the arrays will be divided into 4 parts).
\newline

We also parallelize the residual check in \emph{poisson.c}, in the following way:

\begin{lstlisting}
#pragma omp parallel for schedule(static)\
                         shared(u, rhs)\
                         private(i, j, Ax_r)\
                         firstprivate(nX, nY, dx, dy)\
                         reduction(+:res)\
                         default(none)
for (i=1; i<nX-1; i++) {
  for (j=1; j<nY-1; j++) {
    Ax_r = ( 1.0/(dx*dx)*(u[i-1][j]-2.0*u[i][j]+u[i+1][j]) +
            1.0/(dy*dy)*(u[i][j-1]-2.0*u[i][j]+u[i][j+1]) -
            rhs[i][j] );
    res += Ax_r*Ax_r;
  }
}
\end{lstlisting}

The residual parallelization also takes place for the other iterative methods examined in the next sections. Apart from that, for the Jacobi method, we also parallelize the \emph{copyArray2D} function in \emph{iterative.c}. Without parallelizing the copy function, we end up in worse performance, due to poor cache memory distribution among threads.
\newline

Below we can see performance results for the OpenMP Jacobi method, for two different problem sizes.

\begin{figure}[h!]
\centering
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/jacobi_time_101}
  \caption{$Time$}
\end{subfigure}
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/jacobi_speedup_101}
  \caption{$Speedup$}
\end{subfigure}
\caption{Execution time and speedup for the OpenMP Jacobi implementation (problem size: $101 \times 101$) }
\label{fig:example5.1}
\end{figure}

\begin{figure}[th!]
\centering
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/jacobi_time_201}
  \caption{$Time$}
\end{subfigure}
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/jacobi_speedup_201}
  \caption{$Speedup$}
\end{subfigure}
\caption{Execution time and speedup for the OpenMP Jacobi implementation (problem size: $201 \times 201$) }
\label{fig:example5.2}
\end{figure}

\newpage

\subsection{An OpenMP Performance Optimization}
We notice that performance is worse than ideal for the initial Jacobi OpenMP implementation, something that may be attributed to the additional parallel regions by the \emph{copyArray2D} function, and also by another important factor: the initialization of the "u\_ old" arrays in memory ("\emph{array2D\_ contiguous}" function), in each Jacobi iteration. In the above experiments, arrays were initialized serially. This, however, leads to poor cache memory utilization; if all threads took part in the initialization, then they would have the memory in their cache for the subsequent iterations. Also, as NUMA architectures are prevalent, it is possible that memory is placed in the NUMA region where it is initialized, leading to cluttered communication for threads, reading and writing to NUMA regions further than optimal.
\newline

To improve performance and bypass the obstacles above, we can simply do two things: first, join the copy and Jacobi iterations into one parallel region, using two separate OpenMP worksharing directives, one for each loop. In this way, we end up with half as many parallel regions opened and closed in each iteration, hopefully removing much of the overhead.
\newline

Second, we can do the array initialization to zero in parallel, in the \emph{array2D\_ contiguous}, in \emph{memory.c}. In this way, each thread will initialize the part of the array it will work on in the Jacobi iterations, both keeping that memory in cache and also in its corresponding NUMA region. It is, of course, important to keep the same worksharing schedule in all parallel regions, for obvious reasons; otherwise, each thread would possibly initialize another thread's array elements, negating the cache memory and NUMA advantages mentioned.
\newline

Doing the above, we repeat the performance tests and summarize the results in the following figures, showing both initial and optimized OpenMP versions (and also ideal/linear scaling).

\begin{figure}[h!]
\centering
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/jacobi_time_101_opt}
  \caption{$Time$}
\end{subfigure}
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/jacobi_speedup_101_opt}
  \caption{$Speedup$}
\end{subfigure}
\caption{Execution time and speedup for the optimized OpenMP Jacobi implementation (problem size: $101 \times 101$). The initial performance is also shown, for comparison. }
\label{fig:example5.3}
\end{figure}

\newpage

\begin{figure}[th!]
\centering
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/jacobi_time_201_opt}
  \caption{$Time$}
\end{subfigure}
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/jacobi_speedup_201_opt}
  \caption{$Speedup$}
\end{subfigure}
\caption{Execution time and speedup for the optimized OpenMP Jacobi implementation (problem size: $201 \times 201$). The initial speedup is also shown, for comparison. }
\label{fig:example5.4}
\end{figure}

It is obvious that performance has clearly improved and is actually near the ideal performance, for the larger problem size. This was a good example of OpenMP optimization. Although that the Jacobi method was trivial to parallelize, it required a number of alterations to achieve good performance. It should be mentioned that the code is maintained \emph{with} the optimizations described here.

\section{Parallel Gauss-Seidel}
The Gauss-Seidel is less trivial to parallelize than Jacobi. In general, a \emph{red-black} (or \emph{checkerboard}) approach is followed, in which we update the grid elements in two separate groups ("red" and "black"), as in a checkerboard. We will not describe the method further in this report, but a good description can be found at \cite{karniadakis}.
\newline

The parallelized Gauss-Seidel algorithm can be found in the function \emph{redBlackGaussSeidelIteration2D}, in \emph{iterative.c}. Below we can see performance results for the Gauss-Seidel method in parallel, following the red-black scheme.

\begin{figure}[th!]
\centering
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/gauss_time_101}
  \caption{$Time$}
\end{subfigure}
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/gauss_speedup_101}
  \caption{$Speedup$}
\end{subfigure}
\caption{Execution time and speedup for the OpenMP Gauss-Seidel implementation (problem size: $101 \times 101$). }
\label{fig:example5.5}
\end{figure}

\begin{figure}[th!]
\centering
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/gauss_time_201}
  \caption{$Time$}
\end{subfigure}
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=1.0\linewidth]{images/gauss_speedup_201}
  \caption{$Speedup$}
\end{subfigure}
\caption{Execution time and speedup for the OpenMP Gauss-Seidel implementation (problem size: $201 \times 201$). }
\label{fig:example5.6}
\end{figure}

\newpage

We see that OpenMP performance for the Gauss-Seidel method is decent, especially for the larger problem size. Regarding the smaller problem size, the execution times show that the method requires few iterations to converge and, along with the small grid, this means that the workload is not so heavy to fully utilize all threads, for higher thread counts. Still, the OpenMP code manages to achieve speedup for all cases. We expect even larger problem sizes to scale even better, however such tests were not carried out for this report, in order to save core hours.

\section{Parallel Successive Overrelaxation}
Parallelizing the SOR method is very similar to parallelizing the Gauss-Seidel method. Again, we follow a \emph{red-black} scheme and the SOR OpenMP function is named \emph{"redBlackSORIteration2D"}, in \emph{iteration.c}, where the OpenMP code can be found.
\newline

Due to the fact that SOR converges very fast for this Poisson problem, it is sensible (and possible) to show scaling tests for larger grids with this method. Performance results are shown in the following figures.

\begin{figure}[th!]
\centering
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=0.8\linewidth]{images/sor_time_101}
  \caption{$Time$}
\end{subfigure}
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=0.8\linewidth]{images/sor_speedup_101}
  \caption{$Speedup$}
\end{subfigure}
\caption{Execution time and speedup for the OpenMP SOR implementation (problem size: $101 \times 101$). }
\label{fig:example5.7}
\end{figure}

\begin{figure}[th!]
\centering
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=0.8\linewidth]{images/sor_time_201}
  \caption{$Time$}
\end{subfigure}
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=0.8\linewidth]{images/sor_speedup_201}
  \caption{$Speedup$}
\end{subfigure}
\caption{Execution time and speedup for the OpenMP SOR implementation (problem size: $201 \times 201$). }
\label{fig:example5.7}
\end{figure}

\begin{figure}[th!]
\centering
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=0.8\linewidth]{images/sor_time_1001}
  \caption{$Time$}
\end{subfigure}
\begin{subfigure}{0.45\textwidth}
  \centering
  \includegraphics[width=0.8\linewidth]{images/sor_speedup_1001}
  \caption{$Speedup$}
\end{subfigure}
\caption{Execution time and speedup for the OpenMP SOR implementation (problem size: $1001 \times 1001$). }
\label{fig:example5.8}
\end{figure}

\newpage

Again, sizes $101 \times 101$ and $201 \times 201$ follow a similar trend, achieving better scaling as the grid size increases. It is worth noticing how much faster the SOR method converges, taking 5\% of a second with a single thread for the smaller size.
\newline

However, the larger size of $1001 \times 1001$ is where the OpenMP parallel implementation truly shines, achieving practically ideal speedup, as shown in figure 5.9.b. This leads us to expect similar scaling behavior in the other iterative methods (although with much larger execution times).
\newline

\begin{thebibliography}{1}

  \bibitem{leveque} Randall J. LeVeque {\em Finite Difference Methods for Ordinary and Partial Differential Equations.}  2007.

  \bibitem{karniadakis} George Em. Karniadakis, Robert M. Kirby {\em Parallel Scientific Computing in C++ and MPI.} 2003

  \bibitem{usingopenmp} Barbara Chapman, Gabrielle Jost, Ruud Van Der Pas {\em Using OpenMP.} 2008.

  \end{thebibliography}

\end{document}


\documentclass[Physics.tex]{subfiles}
\begin{document}
\chapter{Oscillations}
Oscillation is the repetitive variation, typically in time, of some measure about a central value (often a point of equilibrium) or between two or more different states.
\section{Simple harmonic motion}
\sldef{Simple harmonic motion} is the oscillatory motion of a particle whose acceleration is always directed towards a fixed point and is directly proportional to its displacement from that fixed point i.e. it is the motion that satisfies the equation \begin{equation}\mathbf{a} = -\omega^2\mathbf{x}\end{equation}

The \sldef{period} \(T\) of an object in simple harmonic motion is the time it takes to make one complete oscillation. Likewise, the frequency \(f\), with units \si{\hertz}, is the number of complete oscillations per unit time.

\sldef{Phase} \(\varphi\) is the wave angle, or the fraction of, which has elapsed relative to some defined point, typically the last complete oscillation. It is usually given in radians, between \SIrange{0}{2\pi}{\radian}. \begin{equation}\varphi = 2\pi\frac{t}{T}\end{equation} Phase difference is simply the difference in phase between two particles, or one particle at different times.

The \sldef{amplitude} \(x_0\) is the maximum magnitude of displacement of the oscillating particle from equilibrium.

The displacement of a particle at a given time is \begin{equation}\begin{split}\mathbf{x}(t) &= x_0\sin(\omega t)\\&= x_0\cos(\omega t)\end{split}\end{equation}

The velocity of a particle is given by \begin{equation}\begin{split}\mathbf{v}(t) &= \frac{\mathrm{d}\mathbf{x}}{\mathrm{d}t} = \omega x_0\cos(\omega t)\\&= -\omega x_0\sin(\omega t)\\\mathbf{v}(\mathbf{x}) &= \pm \omega\sqrt{{x_0}^2 - \mathbf{x}^2}\end{split}\end{equation}

The acceleration of a particle is given by \begin{equation}\begin{split}\mathbf{a}(t) &= \frac{\mathrm{d}\mathbf{v}}{\mathrm{d}t} = -\omega^2x_0\sin(\omega t)\\&= -\omega^2x_0\cos(\omega t)\\\mathbf{a}(\mathbf{x}) &= -\omega^2x\end{split}\end{equation}

Based on the above equations, it can be said that \begin{align*}\mathbf{x} &= 0 \implies v = \max_\mathbf{x} v = \omega x_0 \vee \mathbf{a} = 0\\
\mathbf{x} &= \pm x_0 \implies \mathbf{v} = 0 \vee a = \max_\mathbf{x} a = \omega^2x_0\end{align*}

If the displacement an object in uniform circular motion is broken into two perpendicular axes coplanar with the plane of circular motion, the motion in both axes separately are simple harmonic.

Examples of simple harmonic motion include the simple pendulum for small angles, where \begin{equation}\omega^2 = \frac{g}{l}\end{equation} and a mass on a spring for a spring obeying Hooke's law, where \begin{equation}\omega^2 = \frac{k}{m}\end{equation}

The energies of an object in simple harmonic motion are given by \begin{equation}\begin{split}T &= \frac{1}{2}m\omega^2{x_0}^2\cos^2(\omega t)\\
&= \frac{1}{2}m\omega^2{x_0}^2\sin^2(\omega t)\\
&= \frac{1}{2}m\omega^2({x_0}^2-\mathbf{x}^2)\end{split}\end{equation} If we assume the potential energy at equilibrium to be zero, then \begin{equation}\begin{split}U &= \frac{1}{2}m\omega^2{x_0}^2\sin^2(\omega t)\\
&= \frac{1}{2}m\omega^2{x_0}^2\cos^2(\omega t)\\
&= \frac{1}{2}m\omega^2\mathbf{x}^2\end{split}\end{equation} This gives us total energy \begin{equation}E = \frac{1}{2}m\omega^2{x_0}^2\end{equation}
\section{Damping}
Dissipative forces like air resistance and friction typically oppose oscillating systems in real life. The amplitude and total mechanical energy of real systems generally decrease with time i.e. they are damped.

Examples of damped oscillations include a note played on a piano gradually fading away after being played, or a bell being loud when it is struck but gradually fades away after.

There are three degrees of damping. \begin{itemize}
\item \sldef{Light damping} occurs when there are still oscillations, but the amplitude of oscillation decreases gradually over time.
\item \sldef{Critical damping} occurs when the system returns to its equilibrium position in the shortest possible time without any oscillation i.e. it does not cross the equilibrium position at all.
\item \sldef{Heavy damping} occurs when the system returns to its equilibrium position very slowly, without any oscillation. The system does not cross the equilibrium position either.\end{itemize}

Critical damping is important and seen in many places in our daily lives. In instruments like balances and electric meters, the pointer is critically damped so that it moves quickly to the reading without oscillating. In car suspension systems, the shock absorbers critically damp the suspension and resist the setting up of vibrations that could impair control or cause damage.
\section{Forced oscillations and resonance}
A system undergoing free oscillation will oscillate with a frequency known as its \sldef{natural frequency} \(f_0\).

A \sldef{forced oscillation} is the motion produced when a system (the driven oscillator) is acted upon by an external periodic force (the driving force). The system will oscillate at the driving frequency.

The amplitude of oscillation of a system depends on the driving frequency, and reaches a maximum when the driving frequency matches the natural frequency. When this occurs, it is known as \sldef{resonance}, and there is a maximum transfer of energy from the driving system into the oscillating system.

If the system is not damped, the amplitude of the driven system in resonance will tend towards infinity. Otherwise, the peak amplitude and natural frequency decreases as the degree of damping increases, as greater damping means a greater period and smaller frequency.

Resonance can be useful. It is exploited to produce sound in wind instruments and is also the basis of modern radio telecommunications, where electromagnetic resonance occurs between the radio circuit and the signal. It also allows for cooking of food using microwaves, where microwaves resonate with water molecules.

However, resonance can be harmful. It can cause damage to human organs -- internal organs can resonate with sounds of frequency below \SI{100}{\hertz}. It can also cause bridges to collapse due to resonance -- like with the wind e.g. the Tacoma Narrows Bridge, or when people march in step over bridges (which is why armies break step when crossing bridges).
\end{document}