\documentclass[UTF8]{article}
\usepackage{amsmath}
\usepackage{setspace}
\usepackage{caption}
\usepackage{graphicx, subfig}
%\usepackage{noindent}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{enumerate}
\usepackage[colorlinks,linkcolor=blue,citecolor=blue]{hyperref}
% biblio
\usepackage[authoryear]{natbib}

\lstset{
    basicstyle   =   \sffamily,          % code style
    keywordstyle =   \bfseries,          % keyword
    commentstyle =   \rmfamily\itshape,  % style of comments
    stringstyle  =   \ttfamily,  % style of strings
    flexiblecolumns,                %
    numbers      =   left,   % location of line number
    showspaces   =   false,  %
    numberstyle  =   \tiny \ttfamily,
    showstringspaces  =   false,
    captionpos          =   t,
    frame               =   lrtb,   % frame
}

\title{\textbf{Manual of JDSurfG}}
\author{Nanqiao Du \\ Version 3.0}
\date{\today}

\setlength{\listparindent}{0pt}
\begin{document}
\maketitle
\tableofcontents
\newpage

\section{Introduction}
This document describes how to use \texttt{JDSurfG} package
to perform 3-D joint inversion of shear wave velocity
in crust and upper mantle by surface wave dispersion and
gravity anomaly data. This package is consisted of three
independent modules: generating gravity matrix in spherical
coordinates, conducting direct surface wave tomographic
inversion, and performing tomographic joint inversion of
dispersion and gravity anomaly data. In joint inversion, we
utilize direct surface wave tomography method \citep{Fang2015}
to compute 3-D (pseudo) sensitivity kernel of surface waves
traveltime, and the adaptive Gauss-Legendre integration
method \citep{RN35} is adopted to calculate the gravity
forward matrix. Based on empirical relations between seismic
velocity and density of rocks \citep{Brocher05}, this
package would be a useful tool to investigate 3-D shear
wave structure in the crust and upper mantle. For more details,
please refer to the paper \citep{Du2021} \\

This package is written by \texttt{C++} and \texttt{Fortran}.
\texttt{C++} is responsible for the main logic such as
handling I/O, providing useful data structures and
interfaces. And the \texttt{Fortran} part, derived from
package \href{https://github.com/HongjianFang/DSurfTomo/tree/stable/src}{DSurfTomo}
with several important modifications (by adding
parallel mode and analytical derivatives), is used to
compute surface wave traveltimes and corresponding 1D/3D
sensitivity kernels.\\

I should note that this package was only tested by
several people on similar computational conditions,
so it might contain some bugs in it. I'd appreciate
to receive bug reportings and modify some part of
it when obtaining good suggestions. In the long term,
I would add some new functions to this package,
you could see the \texttt{TODO LIST} in \texttt{README}.

\section{Preliminaries}
This package utilizes \texttt{C++} library \href{http://eigen.tuxfamily.org/index.php?title=Main_Page}{Eigen}
to handle multi-dimensional arrays, and \href{http://www.gnu.org/software/gsl/}{GSL}
to compute Gauss-Legendre nodes' locations and weights. So you need
to install these two libraries in your own computer before
compiling this package. Also, because some of this package
depend on template expressions, you should make sure
that your \texttt{C++} compiler support \texttt{C++11}
standards (\texttt{GCC} >= 4.8).

\section{Installation}
After you have downloaded the .zip file from Github,
you can unzip it by using:
\begin{lstlisting}[language=bash]
    unzip JDSurfG.zip
\end{lstlisting}
Before preceding to compile this package, you should
add the Include path of \texttt{Eigen} and \texttt{GSL}
and the Library path of \texttt{GSL} in
\texttt{JDSurfG/include/Makefile} if you don't modify your
\textasciitilde \texttt{/.bashrc} when you install these
two libraies. For \texttt{Eigen}, you could add include path
as
\begin{lstlisting}[language=bash]
    EIGEN_INC = -I/path/to/your/eigen
\end{lstlisting}
For \texttt{GSL} library, a convinient way is to
use \texttt{gsl-config} in the \texttt{bin}
directory in the \texttt{GSL} Installation path:
\begin{lstlisting}[language=c]
    GSL_LIB = $(shell gsl-config --libs)
    GSL_INC = $(shell gsl-config --cflags)
\end{lstlisting}

Then compile this package by:
\begin{lstlisting}[language=bash]
    cd JDSurfG
    make
\end{lstlisting}
Then you will find four excecutable files in the
directory \texttt{bin}:
\begin{lstlisting}[language=bash]
    mkmat DSurfTomo JointSG syngrav
\end{lstlisting}
That means you have finished installation. \\

\section{Direct Surface Wave Tomography Module}
This module need 3 (4 if checkerboard or other test
is required) input files:
\begin{itemize}
\item \verb!DSurfTomo.in!: contains information of
        input model, dispersion data and inversion
        parameters.
\item \verb!surfdataSC.dat!: dispersion data for
            each station pair
\item \verb!MOD!: Initial model
\item \verb!MOD.true!: True model (if
                        additional tests are required )
\end{itemize}
There are some points to note before we precede to the
format of each file:
\begin{enumerate}[(1)]
    \item Input model is homogeneous in latitudinal and
            longitudinal direction, but the grid size
            could vary in depth direction.
    \item The input model should be slightly larger than
          the target one in order to perform \texttt{B-Spline}
          smoothing for forward computation. To be specific,
         if our target model is in region
         $(lat_0 \sim lat_1,lon_0\sim lon_1,0\sim dep_1)$,
        you should edit your input model at least in
        region $(lat_0-dlat \sim lat_1+dlat, lon_0-dlon \sim lon_1+dlon,0 \sim dep_1+dz)$.
    \item Stations should be far away from target model
            boundaries, or some warnings will be given
            when you running this package. This is to
            avoid that surface wave train travels along
            the models' boundaries.
\end{enumerate}
Now there are the formats of every file below:
\subsection{Parameter File} \label{DSurfTomo}
The parameter file is called \texttt{DSurfTomo.in}
in this instruction. It is a self-explanatory file.
Here is a template of this file below:
\\
\begin{lstlisting}
################################
# DSurfTomo Program Parameters
################################

# input model description:
33 36 25       # nlat nlon nz (no. of points in lat lon and depth direction)
35.3 99.7      # lat0 lon0 (upper left point,[lat,lon], in degree)
0.3 0.3        # dlat dlon (grid interval(degree) in lat and lon direction)

# dispersion forward/adjoint computation parameters
# Insert 'ninsert' points in the grid-based model
# to compute dispersion curves
3              # ninsert (insert several layers, 2~5)
8              # no.of threads used

# inversion parameters
20.0 1.0       # smooth damp
2.0 4.7        # minimum velocity, maximum velocity (a priori information)
10             # maximum iteration
2              # no. of threads used in Linear System Solver

# dispersion data description
16             # kmaxRc (followed by periods)
4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0 22.0 24.0 26.0 28.0 30.0 32.0 34.0
16             # kmaxRg (followed by periods)
4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0 22.0 24.0 26.0 28.0 30.0 32.0 34.0
0              # kmaxLc (followed by periods)
0              # kmaxLg (followed by periods)

# synthetic test
1              # synthetic flag(0:real data,1:synthetic)
3.5            # noise level, std value of gaussian noise

\end{lstlisting}
When you run this program, it will automatically skip
all the empty lines and comments (denoted by '\#'). So you
could modify this file (by adding your own comments) as long
as you don't change the order of these parameters. Now let's
see the meaning of each parameter in the template file.

\begin{enumerate}
    \item \texttt{Input Model Description Block}. This block
    contains information of input model. The first line contains
    three integers which denote the number of grid points
    in latitude,longitude,and in
    depth respectively. The second line denotes coordinates
    of the origin point (NorthWest point), in degree. The last
    line denotes the grid interval in latitudinal and
    longitudinal direction, both in degree.

    \item \texttt{Forward/Adjoint Computation Block}. This
    block contains parameters for forward/adjoint computation.
    The first line is an integer we call it "ninsert".
    As metioned above, we use grid nodes to parameterize
    our model and thus all physical parameters are defined directly
    at grid points. However, in order to compute surface wave
    dispersion by Thomas-Haskell Matrix Method \citep{Haskell53,schwab1972},
    we need to convert grid-based model to layer-based one. In that
    case, we insert 'ninsert' points between adjacent points to do
    that conversion. The next integer is how many threads you
    want to use in calculation of dispersion curves and do
    ray tracing with Fast Marching Method \citep{rawlinson2004}.

    \item \texttt{Inverse Parameters Block}. This block contains
    all parameters used in the inverse problem. The first line indicates
    the smoothing and damping factor in solving the linear systems.
    Then the followed line is the restriction of your
    inverted model (the minimum and maximum velocity). The next line
    is an integer which is the maximum iterations you want to iterate. The
    last line is the number of threads used in solving linear
    systems, usually 2 is enough.

    \item \texttt{Dispersion Data Block}. This block describes
    the information of your dispersion data. The first line is
    the number of periods your Rayleigh wave phase velocity
    disperion data spans. If this number is not zero, then you
    should write all the periods in the next line. Other lines
    are for Rayleigh group dispersion, Love phase, and Love group
    respectively.

    \item \texttt{Synthetic Test Blcok}. This block contains all
    parameters used in synthetic test. The first one is a flag variable
    to denote whether you will conduct synthetic test when using this
    program. The next line is the noiselevel of gaussian noise,
    defined as below:
    \[
        d_{obs}^i = d_{syn}^i +  noiselevel * N(0,1)
    \]

\end{enumerate}

\subsection{Initial Model and True Model File}
\texttt{Please Note that the format of MOD.true in current version
is different from previous versions!} \\

The initial and true model are called \verb!MOD!
and \verb!MOD.true! respectively in this instruction.
They are used for initial input model,
and for synthetic test dataset (like in checkerboard test).
The format of these two files are listed below: \\

The first line of each files are depths (in km) of each grid points,
i.e. there would be \texttt{nz} numbers in this line.
The subsequent lines of \verb!MOD! are shear wave velocities.
If we store the velocity with the format V(nz,nlon,nlat)
(Row-major,where $V(0,0,0)$ is the shear-wave velocity of the
\texttt{top NorthWest} point at \texttt{dep(0)} km, and V(nz-1,nlon-1,nlat-1) is
the velocity of \texttt{bottom SouthEast} point at dep(nz-1) km), then it
should be print to MOD as following:
\begin{lstlisting}[language=c++]
//c++
for(int i=0;i<nz;i++){
for(int j=0;j<nlon;j++){
    for(int k=0;k<nlat;k++){
        fprintf(fileptr,"%f ",V(i,j,k));
    }
    fprintf(fileptr,"\n");
}}
\end{lstlisting}

\subsection{Dispersion Data File}
This dispersion data in \verb!example/! is surfdataSC.dat,
which contains dispersions (distance/traveltime) obtained
surface wave or cross-correlation functions\\
Here we list its first 9 lines:\\
\verb!#!  31.962600 108.646000 1 2 0\\
33.732800 105.764100 3.0840\\
34.342500 106.020600 3.1154\\
33.357400 104.991700 3.0484\\
33.356800 106.139500 3.0820\\
34.128300 107.817000 3.1250\\
33.229300 106.800200 3.0575\\
\verb!#! 29.905000 107.232500 1 2 0\\
34.020000 102.060100 2.7532\\

Obviously, this file contains two kinds of infomation:
surface wave source or master station (lines begin with
\verb!#!) and receiver stations (without \verb!#! in each
line). For a source or master station, the format of
this line is:\\
\begin{center}
    \verb!#! \texttt{latitude longitude period-index wavetype
     velotype}
\end{center}

\begin{itemize}
    \item \texttt{wavetype}: the type of surface wave, for
                            Rayleigh it is 2, and it is 1 for Love wave.
    \item \texttt{velotype}: velocity type, 0 for phase velocity
                            and 1 for group velocity.
    \item \texttt{period-index}: The period index number.
\end{itemize}
For example, if in our dataset, the dispersion periods
for Rayleigh phase velocity is at periods 4s,5s,6s,7s,
and this line contains information only at 5s, then
the last 3 numbers
are \texttt{2,2,0}.\\

Then the following lines contain all the receiver stations
that you have successfully extracted dispersion curves
at this period, this wave type and this velocity type.
And the format of each line is:
\begin{center}
    \texttt{latitude longitude velocity}
\end{center}

\subsection{Run This Module}
After you have prepared all the required files in a folder,
then you could enter into this folder,and run this command
in your shell:
\begin{lstlisting}[language=bash]
./DSurfTomo -h
\end{lstlisting}
Then you could type in all required files according to the
order on the screen.

\subsection{Output Files}\label{surface wave output files}
This module will output two (temporary) folders: \verb!kernel/!
and \verb!results/!. \texttt{kernel} is a tempoaray folder ,
and it contains 3-D pseudo sensitivity kenrels of surface wave
travel times. \verb!results/! contains all the intermediate models
after each iteration. Here are information about the format
of these files:

\begin{itemize}
\item \verb!kernel/!: 3-D pseudo sensitivity kernel. The number of
txt files is equal to no. of threads used in your program
(see \ref{DSurfTomo}), and start from 0.
The kernel matrix is Compressed Sparse Row matrix. It looks like:
\begin{lstlisting}
# 0 1283
24292 2.18522
24293 2.10063
24322 2.29456
24323 2.90529
24324 2.77207
24353 2.45755
24354 3.14743
24355 2.99618
24384 2.14853
\end{lstlisting}

Note there are two numbers followed '\#'. The first one is the
row-index, and another one is the no. of nonzeros in this row
(1283 for current case). Then the following (1283) rows
are column-index and matrix elements. It should be noted that
this folder is only a temporary folder in the inversion process,
and if will be deleted after each iteration.

\item \verb!results/!:contains models (\verb!mod_iter*!)
and surface wave travel time (\verb!res*.dat!) of each iteration.
The format of model files is:
\begin{center}
    \texttt{longitude latitude depth Vs},
\end{center}
And the format of traveltime file is (9 columns):
\begin{center}
    \small{\texttt{distance observation synthetic lat1 lon1 lat2 lon2
    period wavetype}}.
\end{center}

\end{itemize}

\section{Joint Inversion Module}
This module needs 4-6 files in total:
\begin{itemize}
    \item \verb!JointSG.in!: contains information of input model,
            dispersion data and inversion parameters.
    \item \verb!surfdataSC.dat!: surface wave dispersion data.
    \item \verb!obsgrav.dat!: Gravity anomaly dataset.
    \item \verb!gravmat.dat!: sensitivity kernel of
                gravity to density.
    \item \verb!MOD!: Initial model.
    \item \verb!MOD.true!: checkerboard model.
    \item \texttt{MOD.ref}: gravity reference model, used
            for compute gravity anomaly.
\end{itemize}
Here are the formats of each file.

\subsection{Parameter File}
The difference between \verb!JointSG.in! and \verb!DSurfTomo.in!
is little except last three blocks:
\begin{lstlisting}
# synthetic test
1             # synthetic flag(0:real data,1:synthetic)
3.5 0.05      # noiselevel for traveltime and Bourguer

# weight parameter
0.5           # parameter p (Julia(2000)) for 2 datasets, 0-1
3.5 0.05      # std parameters for two datasets

# gravity processing
# whether to remove average value of synthetic gravity data
0             # 1 remove, 0 not remove
\end{lstlisting}

In order to know the meaning of each parameter in \texttt{
Weight Parameters Block},
here I list the weighted cost function of joint inverse
problem \citep{Julia2000}:
\[
    L = \frac{p}{N_1 \sigma_1^2}\sum_{i=1}^{N_1} (d_{1,i}-
        d_{1,i}^o)^2 +
        \frac{1-p}{N_2 \sigma_2^2} \sum_{i=1}^{N_2}
        (d_{2,i}- d_{2,i}^o)^2 \tag{1}
\]
where $\mathbf{d}_{1,2}$ is the data vector for each dataset. $N_{1,2}$
is the size of each dataset and $\sigma_{1,2}$ is the corresponding
errors. $p$ is a parameter in the range [0,1] to control the contribution
of each dataset manually. \\

The additional \texttt{gravity processing block} contains only one integer.
When this integer is equal to 1, the program will automatically
remove the average value of observed and synthetic gravity data.
This option is applied to remove large-scale density anomalies
in the deep mantle. For more details please refer to \ref{Gravref}.

\subsection{Gravity Data File}
The format of this file is quite simple, just print all
the gravity data to this file. Each line of it contains
the longitude,latitude and gravity anomaly at this point. \\

Here I list the first 9 lines of this file to give you
an example:\\
100.000000 35.000000 -224.206921\\
100.000000 34.950000 -224.110699\\
100.000000 34.900000 -241.774731\\
100.000000 34.850000 -240.245968\\
100.000000 34.800000 -234.187542\\
100.000000 34.750000 -246.670605\\
100.000000 34.700000 -238.575939\\
100.000000 34.650000 -241.357250\\
100.000000 34.600000 -260.059429\\

\subsection{Gravity Matrix File}
This file is generated by gravity module, please refer to
chapter \ref{Gravity Matrix}.

\subsection{Gravity Reference Model File}\label{Gravref}
Based on the definition of gravity anomaly, it reflects the
density perturbance according to a normal elliptoid. Thus we need
to convert the absolute anomalies to local anomalies in our research
area. Before inversion, we will remove the average value of
oberserved anomalies. So in the real data
invesion, there are 3 steps to compute relative anomalies:
\begin{itemize}
    \item Compute density distribution of
        current S-wave model and reference model (through
        empirical relations).
    \item Compute density anomaly, then utilize gravity matrix
          to get gravity anomalies.
    \item Remove average value of the synthetic anomaly data.
\end{itemize}
This model could be a user self-defined model, and you could also use the
default one (the average of initial model in every depth). Please
remember, when you are tackling real data, to set the integer to 1 in the gravity processing block
in \texttt{JointSG.in} file.

\subsection{Run This Module}
After you have prepared all these required files in a folder,
then you could enter into this folder, and run this command
in your shell:
\begin{lstlisting}[language=bash]
./JointSG -h
\end{lstlisting}
Then you could type in all required files according to the
order on the screen.

\subsection{Output Files}
This module will output two temporary folders:
\verb!kernel/! and \verb!results/!. \\

The format of \verb!kernel/! is the same as the files
in \ref{surface wave output files}. In the folder \verb!results/!,
it contains models (\verb!joint_mod_iter*!),
surface wave travel time (\verb!res_surf*.dat!) and
graivty anomaly (\verb!res_grav*.dat!)
of each iteration. The format of gravity anomaly is:
\begin{center}
    \texttt{lon lat observed-anomalies synthetic-anomalies}
\end{center}

\section{Gravity Module}
This module requires three input files:
\begin{itemize}
    \item \verb!JointSG.in! or \verb!DSurfTomo.in!: contains model information
    \item \texttt{MOD} : only first line of it are needed.
    \item \texttt{obsgrav.dat}: contains coordinates of each
                     obeservation points.
\end{itemize}
These formats have been illustrated in previous chapters.

\subsection{Run This Module}
After you prepare all these required files in a folder,
then you could enter into this folder,and print in your shell:
\begin{lstlisting}[language=bash]
./mkmat -h
\end{lstlisting}
Then you could type in all required files according to
the order on the screen.

\subsection{Output Files}\label{Gravity Matrix}
This module output only one file: \verb!gravmat.dat!.
This is the density kernel for gravity anomaly. It is also
a Compressed Sparse Row Matrix, the format of it is the same
as surface wave kernel in \ref{surface wave output files}.

\section{Advanced Options}

\subsection{Gauss-Legendre Quadrature and
the Sparcity of Gravity Matrix}
In \verb!src/gravity/gravmat.cpp!, you could modify the
max and min order of Gauss-Legendre Quadrature,
and the maximum distance beyond which gravity matrix is zero (The attraction between two
points is negligible if beyond this distance)
\begin{lstlisting}
const int NMIN=3,NMAX=256;
const float maxdis = 100.0;
\end{lstlisting}
Also, because we use a preallocated memory to store gravity matrix
in the program, you should make sure this memory is large enough. If
your memory is small, this program will complain and stop with
infomation like \texttt{"Please increase the sparse ratio!"}. In fact,
the size of the memory (bytes) used is from:
\[
    Mem = sparse * m * n * 4
\]
Where \texttt{m} is the number of observed gravity data,
\texttt{n} is the total size of your
model, and \texttt{sparse} is the sparse ratio of the gravity matrix.
To change that memory used, you should modify this line in
\texttt{src/gravity/gravmat.cpp}:
\begin{lstlisting}
const float sparse = 0.2;
\end{lstlisting}
where sparse should be a floating number between 0 and 1. Note that
this number is dependent on the \texttt{maxdis} in the same file.
After you change (one of ) these parameters, please remember
to recompile this program.

\subsection{Empirical Relations}
Empirical relations are utilized to connect density and
velocity of rocks in our program. If you have better relations
in your research area, you could change this relation
in \verb!src/utils/bkg_model.cpp!
\begin{lstlisting}[language=c++]
void MOD3d::
empirical_relation(float vsz,float &vpz,float &rhoz)
{
    vpz = 0.9409 + 2.0947*vsz - 0.8206*pow(vsz,2)+
            0.2683*pow(vsz,3) - 0.0251*pow(vsz,4);
    rhoz = 1.6612 * vpz - 0.4721 * pow(vpz,2) +
            0.0671 * pow(vpz,3) - 0.0043 * pow(vpz,4) +
            0.000106 * pow(vpz,5);
}
\end{lstlisting}
And if you want to conduct joint inversion, please remember to
change the derivative function in the same file.

\subsection{Random Seed}
In our program, to make sure the generated random number the same
in each independent numerical experiments, we fix the random seed
in \texttt{src/utils/gaussian.cpp}:
\begin{lstlisting}[language=c++]
static default_random_engine e(11);
\end{lstlisting}
You could change 11 to other integer if required.

\section{Tips and Tools}
\subsection{Warnings}
In fact, \texttt{DSurfTomo} and \texttt{JointSG} are all dependent
on two modules. First one is the forward computation of dispersion
curves from 1-D model, and another one is the ray-tracing on
2-D spherical surface. There would be some possible errors in these
two modules due to incompatible input model format, terrible initial
model, and even the inappropriate location of your stations. To tackle possible
problems, the two modules will print some information on the screen
or in a output file to help you debug.

The first possible warning is \texttt{"WARNING:improper initial value in
disper-no zero found"} on the screen. Then a \texttt{fort.66} file will
be generated in the running directory. This error is due to
a strange 1-D model that the dispersion module cannot find
dispersion curves for this model. There are some possible reasons for
this warning. If you find this problem for the first iteration,
this problem may from that the format of your initial model
is different from the required one. You could check the model in
file \texttt{fort.66} and compare that with your model.
Another possible reason is that you take inappropriate
Tickonov regulariztion parameters (i.e. damping and smooth factors)
which lead to abrupt change of previous model.

The second warning is \texttt{"Source lies outside bounds
of model (lat,long)="}.
This reason is caused by that the receivers are too close to
the boundary of the current model that ray paths are along
this boundary. If this warning happens, please enlarge your
model to avoid this problem.

\subsection{L-curve Analysis}
Usually L-curve analysis is applied to determine the best
suitable smoothing and damping factor for a inverse problem. In our
numerical experiment, we find that the damping factor affects the
final result smaller than smoothing factor. So we \texttt{recommend}
you to keep the damping factor to a small value (such as 0.001) during
your inversion, and only tune the smoothing factor.

In the folder \texttt{utils/}, we have prepared two scripts to help
you tune the smoothing factor. The first one is a bash script
\texttt{\text{tune\_L\_curve.sh}}
that will generate a series of inversion results in the folder
\texttt{storage/}. Please read
this script carefully and then try your own examples.

The second python script \texttt{utils/lcurve.py} will help you
generate a L-curve based on your results. Before running
this script, please change all the
required parameters in the function \texttt{main()}.

\subsection{Checkpoint Restart}
Sometimes you want to start from one of the results (like mod\_iter5.dat)
instead of the initial model. We prepared a python script
\texttt{utils/out2init.py}, you could run this script like this
\begin{lstlisting}
python out2init.py mod_iter5.dat MOD05
\end{lstlisting}
Then this script will convert the output model to the format
of the initial model. This script will be useful in joint inversion.
And also, it may help you restart from some checkpoint if some issues
happend.

% biblio GJI
\bibliographystyle{abbrvnat}
\bibliography{bibliography}
\end{document}


%!TEX root = vorlage.tex

\section{Models}

In this practical, we were examining deep neural network models for semantic
segmentation. The most basic neural network model is a so called
\textit{multilayer Perceptron} (\textit{MLP}). MLPs consist of an input layer,
multiple hidden layers and an output layer. Each layer consists of nodes. The
number of nodes of the input layer is determined by the features and the number
of output nodes is determined by the classes. The number of nodes in the hidden
layers can be arbitrary large, but is typically between 0.25 and 3 times the
number of the layer before. The parameters which get adjusted during the
training of a model are called \textit{weights}. One weight is between two
neurons of neighboring layers.

Each node has an activation function. Two functions which are used very often
are the \textit{sigmoid function} and the \textit{rectified linear function}.

The sigmoid activation function is

\[\varphi: \mathbb{R} \rightarrow (0, 1) \qquad \varphi(x) = \frac{1}{1 + e^{-x}}\]

while rectified linear units (commonly abbreviated with ReLU) have the
activation function

\[\varphi: \mathbb{R} \rightarrow [0, \infty) \qquad \varphi(x) = \max(0, x)\]

A more detailed introduction to MLPs can be found in~\cite{Mitchell1997}.

\subsection{Convolutional Neural Networks}
One of the most important extensions of MLPs are so called
\textit{Convolutional Neural Networks} (CNNs). Those introduce two new layer types
besides the fully connected layers: convolutional layers and pooling layers.
Each convolutional layer has $k \in \mathbb{N}_{\geq 1}$ filters of size $p
\times p \times c$, where $p \in \mathbb{N}_{\geq 1}$ is a design choice and $c
\in \mathbb{N}_{\geq 1}$ is the number of channels of the layer before. The
input layer has typically three channels: Red, green, blue (RGB). When $k$
filters get applied in layer $A$, then the output of layer $A$ has $k$
channels.\\
The parameters which get adjusted during training in convolutional layers are
the weights of the filters. A layer with $k$ filters of size
$p \times p \times c$ has $k \cdot p^2 \cdot c$ weights which get adjusted.

Convolutions produce an output which is smaller than their input, as one can see
in~\cref{fig:convolution-size}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{images/convolution-schematic.png}
    \caption{A $5 \times 5$ image convolved with a $3 \times 3$ filter becomes
             a $3 \times 3$ image as the filter size minus one gets removed
             from the input image size.\\
             Image source:~\cite{Cyfoo}}
    \label{fig:convolution-size}
\end{figure}

Sometimes it is desired to get an output of the same size as the input. In this
case the image is padded with $\frac{\text{filter size} - 1}{2}$ in each
direction. Commonly, the padding is done with zeroes.

The pooling layers do not learn any parameter. They reduce the data being
processed by the network by grouping it. Pooling layers have four hyperparameters:
the pooling function, the pooling size, the strides and the border mode.
The pooling size gives the size of the region to which the pooling function
gets applied. Typically, $3 \times 3$ pooling is chosen. For those four~pixels
the maximum function is chosen. Another option is average pooling. The stride
is the parameter which reduces the amount of processed data. A stride of $(2, 2)$
is used which has the effect of reducing the data to $\nicefrac{1}{4}$th of
the original amount.

There are a couple of well-known image classification networks for which both,
the architecture and the weights are available. One of those networks is called
VGG-16 (for \textit{Visual Geometry Group})~\cite{KarenSimonyan,simonyan2014very}.

\subsection{Fully Convolutional Neural Networks}

Fully Convolutional Networks (FCNs) were introduced by~\cite{long2015fully}.
They were shown to improve performance in semantic segmentation tasks over
standard convolutional neural networks (CNNs).

FCNs combine four ideas:

\begin{enumerate}
    \item Train an image classification network for $k$ classes,
    \item interpret this network as $k$ filters of a single, non-linear
          convolution,
    \item apply those filters to images of arbitrary size to receive a
          coarse heat map for the $k$ classes.
    \item Finally, train an upsampling network to get a fine-grained semantic
          segmentation.
\end{enumerate}

Upsampling is realized by padding and convolution as visualized
in~\cref{fig:upsampling}. It is worth emphasizing that this upsampling is
learned.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{images/upsampling2-16.png}
    \caption{Upsampling: The blue input gets padded by the white cells. Then
             a convolution is applied --- visualized in gray --- to produce
             the green output.\\
             Image source:~\cite{vdumoulin2016}}
    \label{fig:upsampling}
\end{figure}


%!TEX root = da2020-01.tex

\Chapter{1}{Warm-Up}

\noindent
We will start this course with an informal introduction to distributed algorithms. We will formalize the model of computing later but for now the intuitive idea of computers that can exchange messages with each others is sufficient.

\section{Running Example: Coloring Paths}

Imagine that we have $n$ computers (or \emph{nodes} as they are usually called) that are connected to each other with communication channels so that the network topology is a \emph{path}:
\begin{center}
    \includegraphics[page=\PIntroTopo]{figs.pdf}
\end{center}
The computers can exchange messages with their neighbors. All computers run the \emph{same} algorithm\mydash this is the \emph{distributed algorithm} that we will design. The algorithm will decide what messages a computer sends in each step, how it processes the messages that it receives, when it stops, and what it outputs when it stops.

In this example, the task is to find a proper \emph{coloring} of the path with $3$ colors. That is, each node has to output one of the colors, $1$, $2$, or $3$, so that neighbors have different colors\mydash here is an example of a proper solution:
\begin{center}
    \includegraphics[page=\PIntroCol]{figs.pdf}
\end{center}

\section{Challenges of Distributed Algorithm}\label{sec:intro-challenges}

With a bird's-eye view of the entire network, coloring a path looks like a very simple task: just start from one endpoint and assign colors $1$ and $2$ alternately. However, in a real-world computer network we usually do not have all-powerful entities that know everything about the network and can directly tell each computer what to do.

Indeed, when we start a networked computer, it is typically only aware of itself and the communication channels that it can use. In our simple example, the endpoints of the path know that they have one neighbor:
\begin{center}
    \includegraphics[page=\PIntroDegOne]{figs.pdf}
\end{center}
All other nodes along the path just know that they have two neighbors:
\begin{center}
    \includegraphics[page=\PIntroDegTwo]{figs.pdf}
\end{center}
For example, the second node along the path looks no different from the third node, yet somehow they have to produce \emph{different} outputs.

Obviously, the nodes have to exchange \emph{messages} with each other in order to figure out a proper solution. Yet this turns out to be surprisingly difficult even in the case of just $n = 2$ nodes:
\begin{center}
    \includegraphics[page=\PIntroTwo]{figs.pdf}
\end{center}
If we have two \emph{identical} computers connected to each other with a single communication link, both computers are started simultaneously, and both of them run the same deterministic algorithm, how could they ever end up in \emph{different} states?

The answer is that it is not possible, without some additional assumptions. In practice, we could try to rely on some real-world imperfections (e.g., the computers are seldom perfectly synchronized), but in the theory of distributed algorithms we often assume that there is some \emph{explicit} way to \emph{break symmetry} between otherwise identical computers. In this chapter, we will have a brief look at two common assumption:
\begin{itemize}[noitemsep]
    \item each computer has a unique name,
    \item each computer has a source of random bits.
\end{itemize}
In subsequent chapters we will then formalize these models, and develop a theory that will help us understand precisely what kind of tasks can be solved in each case, and how fast.

\section{Coloring with Unique Identifiers}\label{sec:algo-p3c}

There are plenty of examples of real-world networks with globally unique identifiers: public IPv4 and IPv6 addresses are globally unique identifiers of Internet hosts, devices connected to an Ethernet network have globally unique MAC addresses, mobile phones have their IMEI numbers, etc. The common theme is that the identifiers are globally unique, and the numbers can be interpreted as natural numbers:
\begin{center}
    \includegraphics[page=\PIntroId]{figs.pdf}
\end{center}
With the help of unique identifiers, it is now easy to design an algorithm that colors a path. Indeed, the unique identifiers already form a coloring with a large number of colors! All that we need to do is to reduce the number of colors to $3$.

We can use the following simple strategy. In each step, a node is active if it is a ``local maximum'', i.e., its current color is larger than the current colors of its neighbors:
\begin{center}
    \includegraphics[page=\PIntroIdA]{figs.pdf}
\end{center}
The active nodes will then pick a new color from the color palette $\{1,2,3\}$, so that it does not conflict with the current colors of their neighbors. This is always possible, as each node in a path has at most $2$ neighbors, and we have $3$ colors in our color palette:
\begin{center}
    \includegraphics[page=\PIntroIdAA]{figs.pdf}
\end{center}
Then we simply repeat the same procedure until all nodes have small colors. First find the local maxima:
\begin{center}
    \includegraphics[page=\PIntroIdB]{figs.pdf}
\end{center}
And then recolor the local maxima with colors from $\{1,2,3\}$:
\begin{center}
    \includegraphics[page=\PIntroIdBB]{figs.pdf}
\end{center}
Continuing this way we will eventually have a path that is properly colored with colors $\{1,2,3\}$:
\begin{center}
    \includegraphics[page=\PIntroIdC]{figs.pdf}\\
    \includegraphics[page=\PIntroIdCC]{figs.pdf}\\
    \includegraphics[page=\PIntroIdD]{figs.pdf}\\
    \includegraphics[page=\PIntroIdDD]{figs.pdf}\\
    \includegraphics[page=\PIntroIdE]{figs.pdf}\\
    \includegraphics[page=\PIntroIdEE]{figs.pdf}
\end{center}
Note that we may indeed be forced to use all three colors.

So far we have sketched an algorithm idea, but we still have to show that we can actually implement this idea as a distributed algorithm. Remember that there is no central control; nobody has a bird's-eye view of the entire network. Each node is an independent computer, and all computers are running the \emph{same} algorithm. What would the algorithm look like?

Let us fix some notation. Each node maintains a variable $c$ that contains its current color. Initially, $c$ is equal to the unique identifier of the node. Then computation proceeds as shown in Table~\ref{tab:algo-p3c}.

\begin{table}
    \raggedright
    \algtoprule
    \begin{descriptionb}
        \item[Repeat forever:] \mbox{}
        \begin{itemize}
            \item Send message $c$ to all neighbors.
            \item Receive messages from all neighbors. \\
                  Let $M$ be the set of messages received.
            \item If $c \notin \{1,2,3\}$ and $c > \max M$: \\
                  Let $c \gets \min\,(\{1,2,3\} \setminus M)$.
        \end{itemize}
    \end{descriptionb}
    \algbottomrule
    \caption{A simple $3$-coloring algorithm for paths.}\label{tab:algo-p3c}
\end{table}

This shows a typical structure of a distributed algorithm: an infinite send--receive--compute loop. A computer is seen as a state machine; here $c$ is the variable that holds the current state of the computer. In this algorithm, we have three \emph{stopping states}: $c = 1$, $c = 2$, and $c = 3$. It is easy to verify that the algorithm is indeed correct in the following sense:
\begin{enumerate}
    \item In any path graph, for any assignment of unique identifiers, all computers will eventually reach a stopping state.
    \item Once a computer reaches a stopping state, it never changes its state.
\end{enumerate}
The second property is very important: each computer has to know when it is safe to announce its output and stop.

Our algorithm may look a bit strange in the sense that computers that have ``stopped'' are still sending messages. However, it is fairly straightforward to rewrite the algorithm so that you could actually turn off computers that have stopped. The basic idea is that nodes that are going to switch to a stopping state first inform their neighbors about this. Each node will memorize which of its neighbors have already stopped and what were their final colors. Implementing this idea is left as Exercise~\ref{ex:intro-stopped}, and you will later see that this can be done for \emph{any} distributed algorithm. Hence, without loss of generality, we can play by the following simple rules:
\begin{itemize}
    \item The nodes are state machines that repeatedly send messages to their neighbors, receive messages from their neighbors, and update their state\mydash all nodes perform these steps synchronously in parallel.
    \item Some of the states are stopping states, and once a node reaches a stopping state, it no longer changes its state.
    \item Eventually all nodes have to reach stopping states, and these states must form a correct solution to the problem that we want to solve.
\end{itemize}
Note that here a ``state machine'' does not necessarily refer to a \emph{finite}-state machine. We can perfectly well have a state machine with infinitely many states. Indeed, in the example of Table~\ref{tab:algo-p3c} the set of possible states was the set of all positive integers.

\section{Faster Coloring with Unique Identifiers}\label{sec:algo-p3cbit}

So far we have seen that with the help of unique identifiers, it is \emph{possible} to find a $3$-coloring of a path. However, the algorithm that we designed is not particularly efficient in the worst case. To see this, consider a path in which the unique identifiers happen to be assigned in an increasing order:
\begin{center}
    \includegraphics[page=\PIntroIdBad]{figs.pdf}
\end{center}
In such a graph, in each round there is only one node that is active. In total, it will take $\Theta(n)$ rounds until all nodes have stopped.

However, it is possible to color paths \emph{much} faster. The algorithm is easier to explain if we have a \emph{directed} path:
\begin{center}
    \includegraphics[page=\PIntroIdDir]{figs.pdf}
\end{center}
That is, we have a consistent orientation in the path so that each node has at most one ``predecessor'' and at most one ``successor''. The orientations are just additional information that we will use in algorithm design\mydash nodes can always exchange information along each edge in either direction. Once we have presented the algorithm for directed paths, we will then generalize it to undirected paths in Exercise~\ref{ex:intro-undir-path}.

\subsection{Algorithm Overview}

For the sake of concreteness, let us assume that the nodes are labeled with $128$-bit unique identifiers\mydash for example, IPv6 addresses. In most real-world networks $2^{128}$ identifiers is certainly more than enough, but the same idea can be easily generalized to arbitrarily large identifiers if needed.

Again, we will interpret the unique identifiers as colors; hence our starting point is a path that is properly colored with $2^{128}$ colors. In the next section, we will present a fast color reduction algorithm for directed paths that reduces the number of colors from $2^x$ to $2x$ in one round, for any positive integer $x$. Hence in one step we can reduce the number of colors from $2^{128}$ to $2 \cdot 128 = 256$. In just four iterations we can reduce the number of colors from $2^{128}$ to $6$, as follows:
\begin{align*}
    2^{128} &\to 2 \cdot 128 = 2^8, \\
    2^8 &\to 2 \cdot 8 = 2^4, \\
    2^4 &\to 2 \cdot 4 = 2^3, \\
    2^3 &\to 2 \cdot 3 = 6.
\end{align*}
Once we have found a $6$-coloring, we can then apply the algorithm of Table~\ref{tab:algo-p3c} to reduce the number of colors from $6$ to $3$. It is easy to see that this will take at most $3$ rounds. Overall, we have an algorithm that reduces the number of colors from $2^{128}$ to $3$ in only $7$ rounds\mydash no matter how many nodes we have in the path. Compare this with the simple 3-coloring algorithm, which may take millions of rounds for paths with millions of nodes.

\subsection{Algorithm for One Step}\label{ssec:fast-color-red-one-step}

Let us now show how to reduce the number of colors from $2^x$ to $2x$ in one round; this will be achieved by doing some bit manipulations. First, each node sends its current color to its predecessor. After this step, each node $u$ knows two values:
\begin{itemize}[noitemsep]
    \item $c_0(u)$, the current color of the node,
    \item $c_1(u)$, the current color of its successor.
\end{itemize}
If a node does not have any successor, it just proceeds \emph{as if} it had a successor of some color different from $c_0(u)$.

We can interpret both $c_0(u)$ and $c_1(u)$ as $x$-bit binary strings that represent integers from range $0$ to $2^x-1$. We know that the current color of node $u$ is different from the current color of its successor, i.e., $c_0(u) \ne c_1(u)$. Hence in the two binary strings $c_0(u)$ and $c_1(u)$ there is at least one bit that differs. Define:
\begin{itemize}
    \item $i(u) \in \{0,1,\dotsc,x-1\}$ is the \emph{\textbf{index}} of the first bit that differs between $c_0(u)$ and $c_1(u)$,
    \item $b(u) \in \{0,1\}$ is the \emph{\textbf{value}} of bit number $i(u)$ in $c_0(u)$.
\end{itemize}
Finally, node $u$ chooses
\[
    c(u) = 2i(u) + b(u)
\]
as its new color.

\begin{table}
    \newcommand{\hl}[1]{\mathbf{\color{hlcolor}#1}}
    \newcommand{\node}{\raisebox{-1pt}{$\bigcirc$}}
    \newcommand{\mylf}{\\[-2pt]}
    \center
    \begin{tabular}{@{}c@{\qquad}ccccc@{}}
    \toprule
    node & input &&&& output \\
    $u$ & $c_0(u)$ & $c_1(u)$ & $i(u)$ & $b(u)$ & $c(u)$ \\
    \midrule
    $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ \mylf
    $\downarrow$ \mylf
    \node & $01111\hl{0}11_2$ & $00101\hl{1}11_2$ & $2$ & $0$ & $4$ \mylf
    $\downarrow$ \mylf
    \node & $00101\hl{1}11_2$ & $01101\hl{0}11_2$ & $2$ & $1$ & $5$ \mylf
    $\downarrow$ \mylf
    $\node$ & $01101011_2$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ \mylf
    $\downarrow$ \mylf
    $\cdots$ & $\cdots$ \mylf
    \midrule
    $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ \mylf
    $\downarrow$ \mylf
    \node & $01111\hl{0}11_2$ & $00101\hl{1}11_2$ & $2$ & $0$ & $4$ \mylf
    $\downarrow$ \mylf
    \node & $0\hl{0}101111_2$ & $0\hl{1}101111_2$ & $6$ & $0$ & $12$ \mylf
    $\downarrow$ \mylf
    $\node$ & $01101111_2$ & $\cdots$ & $\cdots$ & $\cdots$ & $\cdots$ \mylf
    $\downarrow$ \mylf
    $\cdots$ & $\cdots$ \mylf
    \bottomrule
    \end{tabular}
    \caption{Fast color reduction algorithm for directed paths: reducing the number of colors from $2^x$ to $2x$, for $x = 8$. There are two interesting cases: either $i(u)$ is the same for two neighbors (first example), or they are different (second example). In the first case, the values $b(u)$ will differ, and in the second case, the values $i(u)$ will differ. In both cases, the final colors $c(u)$ will be different.}\label{tab:algo-p3cbit}
\end{table}

\subsection{An Example}

Let $x = 8$, i.e., nodes are colored with $8$-bit numbers. Assume that we have a node $u$ of color $123$, and $u$ has a successor $v$ of color $47$; see Table~\ref{tab:algo-p3cbit} for an illustration. In binary, we have
\begin{align*}
    c_0(u) &= 01111011_2, \\
    c_1(u) &= 00101111_2.
\end{align*}
\begin{samepage}
Counting from the least significant bit, node $u$ can see that:
\begin{itemize}[noitemsep]
    \item bit number $0$ is the same in both $c_0(u)$ and $c_1(u)$,
    \item bit number $1$ is the same in both $c_0(u)$ and $c_1(u)$,
    \item bit number $2$ is different in $c_0(u)$ and $c_1(u)$.
\end{itemize}
\end{samepage}
Hence we will set
\[
    i(u) = 2, \quad
    b(u) = 0, \quad
    c(u) = 2\cdot2 + 0 = 4.
\]
That is, node $u$ picks $4$ as its new color. If all other nodes run the same algorithm, this will be a valid choice\mydash as we will argue next, both the predecessor and the successor of $u$ will pick a color that is different from~$4$.

\subsection{Correctness}

Clearly, the value $c(u)$ is in the range $\{0,1,\dotsc,2x-1\}$. However, it is not entirely obvious that these values actually produce a proper $2x$-coloring of the path. To see this, consider a pair of nodes $u$ and $v$ so that $v$ is the successor of $u$. By definition, $c_1(u) = c_0(v)$. We need to show that $c(u) \ne c(v)$. There are two cases\mydash see Table~\ref{tab:algo-p3cbit} for an example:
\begin{enumerate}
    \item $i(u) = i(v) = i$: We know that $b(u)$ is bit number $i$ of $c_0(u)$, and $b(v)$ is bit number $i$ of $c_1(u)$. By the definition of $i(u)$, we also know that these bits differ. Hence $b(u) \ne b(v)$ and $c(u) \ne c(v)$.
    \item $i(u) \ne i(v)$: No matter how we choose $b(u) \in \{0,1\}$ and $b(v) \in \{0,1\}$, we have $c(u) \ne c(v)$.
\end{enumerate}
We have argued that $c(u) \ne c(v)$ for any pair of two adjacent nodes $u$ and $v$, and the value of $c(u)$ is an integer between $0$ and $2x-1$ for each node $u$. Hence the algorithm finds a proper $2x$-coloring in one round.

\subsection{Iteration}

The algorithm that we presented in this section can reduce the number of colors from $2^x$ to $2x$ in one round; put otherwise, we can reduce the number of colors from $x$ to $O(\log x)$ in one round.

If we iterate the algorithm, we can reduce the number of colors from $x$ to $6$ in $O(\log^* x)$ rounds (please refer to Section~\ref{sec:intro-app} for the definition of the $\log^*$ function if you are not familiar with it).

Once we have reduced the number of colors to $6$, we can use the simple color reduction algorithm from Section~\ref{sec:algo-p3c} to reduce the number of colors from $6$ to $3$ in $3$ rounds. The details of the analysis are left as Exercises \ref{ex:logstar} and~\ref{ex:logstar-tight}.

\section{Coloring with Randomized Algorithms}\label{sec:algo-p3crand}

So far we have used unique identifiers to break symmetry. Another possibility is to use randomness. Here is a simple randomized distributed algorithm that finds a proper $3$-coloring of a path: nodes try to pick colors from the palette $\{1,2,3\}$ uniformly at random, and they stop once they succeed in picking a color that is different from the colors of their neighbors.

\subsection{Algorithm}

Let us formalize the \emph{simple randomized 3-coloring algorithm} that we sketched above. Each node $u$ has a flag $s(u) \in \{0,1\}$ indicating whether it has stopped, and a variable $c(u) \in \{1,2,3\}$ that stores its current color. If $s(u) = 1$, a node has stopped and its output is $c(u)$.

In each step, each node $u$ with $s(u) = 0$ picks a new color $c(u) \in \{1,2,3\}$ uniformly at random. Then each node sends its current color $c(u)$ to its neighbors. If $c(u)$ is different from the colors of its neighbors, $u$ will set $s(u) = 1$ and stop; otherwise it tries again in the next round.

\subsection{Analysis}

It is easy to see that in each step, a node $u$ will stop with probability at least $1/3$: after all, no matter what its neighbors do, there is at least one choice for $c(u) \in \{1,2,3\}$ that does not conflict with its neighbors.

Fix a positive constant $C$. Consider what happens if we run the algorithm for
\[
    k = (C+1) \log_{3/2} n
\]
steps, where $n$ is the number of nodes in the network. Now the probability that a given node $u$ has not stopped after $k$ steps is at most
\[
    (1 - 1/3)^k = \frac{1}{n^{C+1}}.
\]
By the union bound, the probability that there is a node that has not stopped is at most $1/n^C$. Hence with probability at least $1-1/n^C$, all nodes have stopped after $k$ steps.

\subsection{With High Probability}

Let us summarize what we have achieved: for any given constant $C$, there is an algorithm that runs for $k = O(\log n)$ rounds and produces a proper $3$-coloring of a path with probability $1-1/n^C$. We say that the algorithm runs in time $O(\log n)$ \emph{with high probability}\mydash here the phrase ``high probability'' means that we can choose any constant $C$ and the algorithm will succeed at least with a probability of $1-1/n^C$. Note that even for a moderate value of $C$, say, $C = 10$, the success probability approaches $1$ very rapidly as $n$ increases.

\section{Summary}

In this chapter we have seen three different distributed algorithms for $3$-coloring paths:
\begin{itemize}
    \item A simple 3-coloring algorithm, Section~\ref{sec:algo-p3c}: A deterministic algorithm for paths with unique identifiers. Runs in $O(n)$ rounds, where $n$ is the number of nodes.
    \item A fast 3-coloring algorithm, Section~\ref{sec:algo-p3cbit}: A deterministic algorithm for \emph{directed} paths with unique identifiers. Runs in $O(\log^* x)$ rounds, where $x$ is the largest identifier.
    \item A simple randomized 3-coloring algorithm, Section~\ref{sec:algo-p3crand}: A randomized algorithm for paths without unique identifiers. Runs in $O(\log n)$ rounds with high probability.
\end{itemize}
We will explore and analyze these algorithms and their variants in more depth in the exercises.

\section{Quiz}

Construct a directed path of 3 nodes that is labeled with unique identifiers (of any size) such that the following holds: After two iterations of the fast color reduction algorithm from Section~\ref{ssec:fast-color-red-one-step}, the color of the first node is 7.

It is enough to just list the three unique identifiers (in decimal); there is no need to explain anything else.

\section{Exercises}

\begin{ex}[maximal independent sets]\label{ex:intro-mis}
    A \emph{maximal independent set} is a set of nodes $I$ that satisfies the following properties:
    \begin{itemize}[noitemsep]
        \item for each node $v \in I$, none of its neighbors are in $I$,
        \item for each node $v \notin I$, at least one of its neighbors is in $I$.
    \end{itemize}
    Here is an example\mydash the nodes labeled with a ``1'' form a maximal independent set:
    \begin{center}
        \includegraphics[page=\PIntroMis]{figs.pdf}
    \end{center}
    Your task is to design a distributed algorithm that finds a maximal independent set in any path graph, for each of the following settings:
    \begin{subex}
        \item\label{subex:intro-mis-a} a deterministic algorithm for paths with arbitrarily large unique identifiers,
        \item\label{subex:intro-mis-b} a fast deterministic algorithm for \emph{directed} paths with $128$-bit unique identifiers,
        \item\label{subex:intro-mis-c} a randomized algorithm that does not need unique identifiers.
    \end{subex}
    In part \ref{subex:intro-mis-a}, use the techniques presented in Section~\ref{sec:algo-p3c},
    in part \ref{subex:intro-mis-b}, use the techniques presented in Section~\ref{sec:algo-p3cbit}, and
    in part \ref{subex:intro-mis-c}, use the techniques presented in Section~\ref{sec:algo-p3crand}.
\end{ex}

\begin{ex}[stopped nodes]\label{ex:intro-stopped}
    Rewrite the greedy algorithm of Table~\ref{tab:algo-p3c} so that stopped nodes do not need to send messages. Be precise: explain your algorithm in detail so that you could easily implement it.
\end{ex}

\begin{ex}[undirected paths]\label{ex:intro-undir-path}
    The fast 3-coloring algorithm from Section~\ref{sec:algo-p3cbit} finds a $3$-coloring very fast in any directed path. Design an algorithm that is almost as fast and works in any path, even if the edges are not directed. You can assume that the range of identifiers is known.

    \hint{Use the local maxima and minima to partition the path in subpaths so that within each subpath we have unique identifiers given in an increasing order. Use this ordering to orient each subpath. Then we can apply the fast color reduction algorithm in each subpath. Finally, combine the solutions.}
\end{ex}

\begin{ex}[randomized and fast]
    The simple randomized 3-coloring algorithm finds a $3$-coloring in time $O(\log n)$ with high probability, and it does not need any unique identifiers. Can you design a randomized algorithm that finds a $3$-coloring in time $o(\log n)$ with high probability? You can assume that $n$ is known.

    \hint{Design a randomized algorithm that finds a coloring with a large number of colors quickly. Then apply the technique of the fast 3-coloring algorithm from Section~\ref{sec:algo-p3cbit} to reduce the number of colors to $3$ quickly.}
\end{ex}

\begin{ex}[asymptotic analysis]\label{ex:logstar}
    Analyze the fast 3-coloring algorithm from Section~\ref{sec:algo-p3cbit}:
    \begin{subex}
        \item Assume that we are given a coloring with $x$ colors; the colors are numbers from $\{1,2,\dotsc,x\}$. Show that we can find a $3$-coloring in time $O(\log^* x)$.
        \item Assume that we are given unique identifiers that are polynomial in $n$, that is, there is a constant $c = O(1)$ such that the unique identifiers are a subset of $\{1,2,\dotsc,n^c\}$. Show that we can find a $3$-coloring in time $O(\log^* n)$.
    \end{subex}
\end{ex}

\begin{exs}[tight analysis]\label{ex:logstar-tight}
    Analyze the fast 3-coloring algorithm from Section~\ref{sec:algo-p3cbit}:
    Assume that we are given a coloring with $x$ colors, for any integer $x \ge 6$; the colors are numbers from $\{1,2,\dotsc,x\}$. Show that we can find a $6$-coloring in time $\log^*(x)$, and therefore a $3$-coloring in time $\log^*(x) + 3$.
    \hint{Consider the following cases separately:
    \begin{center}
    (i)~$\log^* x \le 2$,\quad
    (ii)~$\log^* x = 3$,\quad
    (iii)~$\log^* x \ge 4$.
    \end{center}
    In case~(iii), prove that after $\log^*(x)-3$ iterations, the number of colors is at most $64$.}
\end{exs}

\begin{exs}[oblivious algorithms]\label{ex:oblivious}
    The simple 3-coloring algorithm works correctly even if we do not know how many nodes there are in the network, or what is the range of unique identifiers\mydash we say that the algorithm is \emph{oblivious}. Adapt the fast 3-coloring algorithm from Section~\ref{sec:algo-p3cbit} so that it is also oblivious.
    \hint{One possible strategy is this: Choose some threshold, e.g., $d = 10$. Focus on the nodes that have identifiers smaller than $d$, and find a proper $3$-coloring in those parts, in time $O(\log^* d)$. Remove the nodes that are properly colored. Then increase threshold $d$, and repeat. Be careful with the way in which you increase $d$. Show that you can achieve a running time of $O(\log^* x)$, where $x$ is the largest identifier, without knowing $x$ in advance.}
\end{exs}

\section{Bibliographic Notes}

The fast 3-coloring algorithm (Section~\ref{sec:algo-p3cbit}) was originally presented by Cole and Vishkin~\cite{cole86deterministic} and further refined by Goldberg et al.~\cite{goldberg88parallel}; in the literature, it is commonly known as the ``Cole--Vishkin algorithm''. Exercise~\ref{ex:oblivious} was inspired by Korman et al.~\cite{korman11removing}.


\documentclass{subfile}

\begin{document}
	\section{APMO}\label{sec:apmo}

		\begin{problem}[$2018$, problem $2$]
			Let $f(x)$ and $g(x)$ be given by
				\begin{align*}
					f(x)
						& = \dfrac{1}{x}+\dfrac{1}{x-2}+\ldots+\dfrac{1}{x-2018}\\
					g(x)
						& = \dfrac{1}{x-1}+\dfrac{1}{x-3}+\ldots+\dfrac{1}{x-2017}
				\end{align*}
			Prove that
				\begin{align*}
					|f(x)-g(x)|
						& > 2
				\end{align*}
		\end{problem}

		\begin{problem}[$2012$, problem $5$]
			Let $n\geq2$ be an integer and $a_{1},\ldots,a_{n}$ be real numbers such that $a_{1}^{2}+\ldots+a_{n}^{2}=n$. Prove that
				\begin{align*}
					\sum\limits_{1\leq i < j\leq n}\dfrac{1}{n-a_{i}a_{j}}
						& \leq \dfrac{n}{2}
				\end{align*}
		\end{problem}

		\begin{problem}[$2007$, problem $4$]
			Let $x,y,z$ be positive real numbers such that
				\begin{align*}
					\sqrt{x}+\sqrt{y}+\sqrt{z}
						& = 1
				\end{align*}
			Prove that
				\begin{align*}
					\dfrac{x^{2}+yz}{\sqrt{2x^{2}(y+z)}}+\dfrac{y^{2}+zx}{\sqrt{2y^{2}(z+x)}}+\dfrac{z^{2}+xy}{\sqrt{2z^{2}(x+y)}}
						& \geq 1
				\end{align*}
		\end{problem}

		\begin{problem}[$2005$, problem $2$]
			Let $a,b,c$ be positive real numbers such that $abc=8$. Prove that
				\begin{align*}
					\dfrac{a^{2}}{\sqrt{(1+a^{3})(1+b^{3})}}+\dfrac{b^{2}}{\sqrt{(1+b^{3})(1+c^{3})}}+\dfrac{c^{2}}{\sqrt{(1+c^{3})(1+a^{3})}}
						& \geq \dfrac{4}{3}
				\end{align*}
		\end{problem}

		\begin{problem}[APMO $2004$, problem $5$]
			Prove that the inequality
				\begin{align*}
					(a^{2}+2)(b^{2}+2)(c^{2}+2)
						& \geq9(ab+bc+ca)
				\end{align*}
			holds for all positive real numbers $a,b,c$.
		\end{problem}

		\begin{problem}[APMO $2003$, problem $4$]
			Let $a,b,c$ be the sides of a triangle with $a+b+c=1$ and $n\geq2$ be an integer. Show that
				\begin{align*}
					\sqrt[n]{a^{n}+b^{n}}+\sqrt[n]{b^{n}+c^{n}}+\sqrt[n]{c^{n}+a^{n}}
						& < 1+\dfrac{\sqrt[n]{2}}{2}
				\end{align*}
		\end{problem}

		\begin{problem}[APMO $2002$, problem $1$]
			Let $n$ be a positive integer and $a_{1},\ldots,a_{n}$ be a sequence of non-negative integers. Let
				\begin{align*}
					A_{n}
						& = \dfrac{a_{1}+\ldots+a_{n}}{n}
				\end{align*}
			Prove that
				\begin{align*}
					a_{1}!\cdots a_{n}!
						& \geq (\lfloor A_{n}\rfloor!)^{n}
				\end{align*}
		\end{problem}

		\begin{problem}[APMO $2002$, problem $4$]
			Let $x,y,z$ be positive numbers such that
				\begin{align*}
					\dfrac{1}{x}+\dfrac{1}{y}+\dfrac{1}{z}
						& = 1
				\end{align*}
			Show that
				\begin{align*}
					\sqrt{x+yz}+\sqrt{y+zx}+\sqrt{z+xy}
						& \geq\sqrt{xyz}+\sqrt{x}+\sqrt{y}+\sqrt{z}
				\end{align*}
		\end{problem}

		\begin{problem}[APMO $1998$, problem $3$]
			Let $a,b,c$ be positive real numbers. Prove that
				\begin{align*}
					\left(1+\dfrac{a}{b}\right)\left(1+\dfrac{b}{c}\right)\left(1+\dfrac{c}{a}\right)
						& \geq 2\left(1+\dfrac{a+b+c}{\sqrt[3]{abc}}\right)
				\end{align*}
		\end{problem}

		\begin{problem}[APMO $1996$, problem $5$]
			Let $a,b,c$ be the lengths of the sides of a triangle. Prove that
				\begin{align*}
					\sqrt{a+b-c}+\sqrt{b+c-a}+\sqrt{c+a-b}
						& \leq \sqrt{a}+\sqrt{b}+\sqrt{c}
				\end{align*}
			and determine when equality occurs.
		\end{problem}

		\begin{problem}[APMO $1991$, problem $3$]
			Let $a_{1},\ldots,a_{n},b_{1},\ldots,b_{n}$ be positive real numbers such that
				\begin{align*}
					a_{1}+\ldots+a_{n}
						& = b_{1}+\ldots+b_{n}
				\end{align*}
			Show that
				\begin{align*}
					\dfrac{a_{1}^{2}}{a_{1}+b_{1}}+\ldots+\dfrac{a_{n}^{2}}{a_{n}+b_{n}}
						& \geq\dfrac{a_{1}+\ldots+a_{n}}{2}
				\end{align*}

				\begin{solution}
					\nameref{thm:engel} can be used to solve this easily.
						\begin{align*}
							\dfrac{a_{1}^{2}}{a_{1}+b_{1}}+\ldots+\dfrac{a_{n}^{2}}{a_{n}+b_{n}}
								& \geq \dfrac{(a_{1}+\ldots+a_{n})^{2}}{a_{1}+b_{1}+\ldots+a_{n}+b_{n}}\\
								& = \dfrac{(a_{1}+\ldots+a_{n})^{2}}{2(a_{1}+\ldots+a_{n})}\\
								& = \dfrac{a_{1}+\ldots+a_{n}}{2}
						\end{align*}
				\end{solution}
		\end{problem}

		\begin{problem}[APMO $1990$, problem $2$]
			Let $a_{1},\ldots,a_{n}$ be positive real numbers and $S_{k}$ be the sum of the products of $a_{1},\ldots,a_{n}$ taken $k$ at a time. Show that
				\begin{align*}
					S_{k}S_{n-k}
						& \geq\binom{n}{k}^{2}a_{1}\cdots a_{n}
				\end{align*}
		\end{problem}

		\begin{problem}[APMO $1989$, problem $1$]
			Let $x_{1},\ldots,x_{n}$ be positive real numbers and
				\begin{align*}
					S
						& = x_{1}+\ldots+x_{n}
				\end{align*}
			Prove that
				\begin{align*}
					(1+x_{1})\cdots(1+x_{n})
						& \leq 1+S+\dfrac{S^2}{2!}+\ldots+\dfrac{S^n}{n!}
				\end{align*}
		\end{problem}
\end{document}


\documentclass[]{article}

\usepackage{graphicx}

\usepackage[margin=1in]{geometry}

\setlength\parindent{0pt}

\usepackage{physics}
% \usepackage{amsmath}

\usepackage{listings}

\usepackage{enumitem}
\renewcommand{\theenumi}{\alph{enumi}}

%opening

\title{MATH 5301 Elementary Analysis - Homework 1}

\author{Jonas Wagner}

\date{2021, September 01}

\begin{document}

\maketitle

\section{Problem 1}
\textbf{Problem:}
Prove the following tautologies by writing the true-false table.
Also translate each of these statements into human language.

\begin{enumerate}
	\item $A \lor \sim A$
	\item $(A \lor B) \Rightarrow A$
	\item $(A \land B) \Rightarrow A$
	\item $(A \Rightarrow B) \iff (\sim B \Rightarrow \sim A)$
	\item $\sim (A \lor B) \iff (\sim A \land \sim B)$
	\item $((A \Rightarrow B) \Rightarrow A) \Rightarrow A$
	\item $(A \Rightarrow (B \Rightarrow C)) \Rightarrow ((A \Rightarrow B) \Rightarrow (A \Rightarrow C))$
\end{enumerate}

\textbf{Solution:}\\
\begin{figure}[h]
	\centering
	\includegraphics*[width=0.7\textwidth]{fig/pblm1.png}
\end{figure}

\begin{enumerate}
	\item A or not A
	\item A or B implies A
	\item A and B implies A
	\item A implying B implies not B implying not A
	\item Neither A nor B occurs if and only if not A and not B
	\item A implying B implies A which implies A
	\item A implying B implies C implies that A implying B implies A implying C
\end{enumerate}

\newpage
\section{Problem 2}
\textbf{Problem:}
Prove the following identities for the set operations.
\begin{enumerate}
	\item $A \cap (B \cup C) = (A \cap B) \cup (A \cap C)$
	\item $(A \backslash B) \cup C = ((A \cup C) \backslash B) \cup (B \cap C)$
\end{enumerate}

\textbf{Solution:}
\begin{figure}[h]
	\centering
	\includegraphics*[width = \textwidth]{fig/pblm2a.png}
\end{figure}

\begin{figure}[h]
	\centering
	\includegraphics*[width = \textwidth]{fig/pblm2b.png}
\end{figure}

\newpage
\section{Problem 3}
\textbf{Problem:}
Write the following statements using quantifiers.
\begin{enumerate}
	\item Even elements of the sequence $\{a_n\}$ may be arbitrarly large.
	\item The sequence $\{a_n\}$ contains arbitary large even elements.
	\item The sequence $\{a_n\}$ contains infinitely many even elements.
\end{enumerate}

\textbf{Solution:}

\begin{enumerate}
	\item $\forall n \in \mathbf{N} : a_n \ \vdots \ 2 \Rightarrow \exists a \in \mathbf{N} : a_n \geq a$
	\item $\exists n \in \mathbf{N} : a_n \ \vdots \ 2 \Rightarrow \forall a \in \mathbf{N} : a_n > a$
	\item $\forall n \in \mathbf{N} : a_n \ \vdots \ 2 : \exists m \in \mathbf{N} : m > n : a_m \ \vdots \ 2$
\end{enumerate}

\newpage
\section{Problem 4}
\textbf{Problem:}
Show that:
\begin{enumerate}
	\item $\exists_x : (p(x) \lor q(x)) \iff (\exists_x : p(x)) \lor (\exists_x : q(x))$
	\item $(\forall_x p(x) \lor \forall_x q(x)) \Rightarrow \forall_x (p(x) \lor q(x))$
	\item Why is there no left arror implication on the previous line?
\end{enumerate}

\textbf{Solution:}\\
Part a)
\begin{figure}[h]
	\centering
	\includegraphics[width = \textwidth]{fig/pblm4a.png}
\end{figure}

Part b)
\begin{figure}[h]
	\centering
	\includegraphics[width = \textwidth]{fig/pblm4b.png}
\end{figure}

Part c)\\
This is becouse there are cases when $p(x)$ and $q(x)$ themselves are not
satisfied $\forall x$, but together at least one of them are true $\forall x$.

\newpage
\section{Problem 5}
\textbf{Problem:}
Show that one needs only one logic operation to construct all the 16 binary operations on statements A and B.\\
Define $A \star B$ via the following table:
\includegraphics[width=0.3\textwidth]{fig/pblm5_star_TF_table.png}

Show that one can construct $\sim A, A \lor B, $and$ A \land B$ using only $\star$.
Then show that any other binary operation can be obtained from $\{\sim, \lor, \land\}$.\\

\textbf{Solution:}
\begin{figure}[h]
	\centering
	\includegraphics[width=0.7\textwidth]{fig/pblm5_star2op_soln.png}
	\includegraphics[width=0.9\textwidth]{fig/pblm5_star2op_table.png}
\end{figure}

\newpage
(using the notation learned in digital circuits)\\
\includegraphics[width=\textwidth]{fig/pblm5_op2all_soln.png}

\newpage
\section{Problem 6}
\textbf{Problem:}
How many subsets does the set $A = \{a,p,p,l,e\}$ have?

\textbf{Solution:}
On the surface this problem can be solved simply by considering the selection of
each letter to be a part of a subset as a binary diget of T/F for inclusion,
resulting in $2^n = 2^5 = 32$.
However, due to the repeaterd element, $p$, this is not as simple but a similar
combinatorics proccess can be used to account for this.\\

\textbf{General Solution:}\\
Let the following be defined:\\
$n :=$ number of elements within $A$\\
$m :=$ number of unique elements within $A$\\
${a_i} :=$ sequence of the unique elements within $A$ ordered from
the most occuring to the least occuring\\
${n_i} :=$ sequence of the number of unique elements, $a_i$, within $A$.\\
${(a_i,n_i)} := $ set of ordered pairs of unique elements of a unique
element, $a_i$, within $A$ paired with the number of $a_i$ elements contained
within $A$, $n_i$.\\

The calculation can split into the sum of all the possible subsets of lengths, $l$,
from 0 to n.\\
For $l=0$ the only possible subset is $\emptyset$,
so $$N_0 = \mqty(n \\ 0) = 1.$$

Similarily, for $l=n$ the only possible subset of $A$ is $A$,
so $$N_n = \mqty(n \\ n) = 1.$$

For $l=1$ there exists $m$ unique sets consisting of the elements in $\{a_i\}$.
Alternativly, this can be calculated as
$$N_1 = \mqty(n\\1) - \sum_{i=1}^m (n_i - 1)$$.

For $l=2$ (and all $l>1$) the computation becomes more complicated.\\
The collection of subsets with repeated elements for $l=2$ would be all possible
combinations of the elements of $A$ with $l=2$ elements, $\qty(N_{l=2}^{(all)} = \mqty(n\\2))$.\\

The number of duplicated elements can be done by constructing the ordered pairs
$\{(b_i,n_i-1)\}: b_i = a_i \forall i : n_i > 1$. $B$ can then be defined as the collection
of $n_i - 1$ copies of $b_i$. \footnote{How do you do this with quantifiers?}\\

The number of elements in $B$, $n^{(l=2)}$ can then be used to determine the number of duplicated
subsets included in $N_{l=2}^{(all)}$, $N_{l=2}^{(duplicates)} = n^{(l=2)} * N_{l=1}$. Therefore,
$$N_{l=2} = N_{l=2}^{(all)} - N_{l=2}^{(duplicates)}
= \mqty(n\\2) - n^{(l=2)} * N_{l=1}
= \mqty(n\\2) - n^{(l=2)} * \qty(\mqty(n\\1) - \sum_{i=1}^m (n_i - 1))$$

This proccess can be repeated for $l > 2$ until the newly constructed set (labeled $B$ for $l=2$) is empty,
in which case $N_{l} = \mqty(n\\l)$.\\

For the given finite case of $A = \{a,p,p,l,e\}$ the result is calculated as:
\begin{align*}
	N_0 &= 1\\
	N_1 &= 4\\
	N_2 &= 13\\
	N_3 &= 6\\
	N_4 &= 4\\
	N_5 &= 1\\
	\intertext{Therefore,}
	N &= 29
\end{align*}
% One solution is to define the ordered pairs
% $\{(b_i,n_i-1)\}: b_i = a_i \forall i : n_i > 1$.
% The new sequence, $\{b_i\}$ can then be used to add the nessicary
% extra repeated element subsets.
% Additionally, let the number of elements within sequence $\{b_i\}$
% be defined as $m_2$.
% The collection of subsets with $l=2$ would be all possible combinations of
% $a_i$, $\qty(N_{l=2}^{(all)} = \mqty(n\\2))$, minus the number of repeated
% subsets constructed with ${b_i,a_j} \forall i \in \{1,2,\cdots,m_2\},
% j \in \{1,2,\cdots,m_2\}$, $\qty(N_{l=2}^{(duplicates)} = m_2 * m)$. Therefore,
% $$N_2 = \mqty(m\\2) - m_2.$$

% Next, this proccess can be repeated for $2 < l < n$, such that a new set of
% ordered pairs is defined, $\{c_i, n + (l-1)\} : c_i = a_i \forall n > (l-1)$,
% and then the number of subsets with $l$ elements is calculated as
% the sum of all possible combinations of the elements of $A$ of length $l$,
% $\qty(N_{l}^{(all)} = \mqty(n\\2))$, minus the number of repeated subsets
% constructed with ${c_i,\cdots,b_j,a_k} \forall i \in \{1,2,\cdots,m_l\},
% \cdots, j \in \{1,2,\cdots,m_2\}, k \in \{1,2,\cdots,m\}$,
% $\qty(N_l)^{(duplicates)} = m_l * \cdots * m_2 * m$. Therefore,
% $$N_l = \mqty(n\\l) - m_l * \cdots * m_2 * m.$$

% The total number of possible subsets can then be calculated as the sum
% $$N = \sum_{l=1}^{n} N_l = \sum_{l=1}^n \mqty(n\\l) - \prod_{1}^{l} m_l$$

% Note: I am not overly confident of how well this scales when it grows
% larger then I actually tested it at...

\end{document}


\input{../header.tex}
\title{\vspace{-2cm}INF3490/INF4490 Exercise Solutions - Evolutionary Computing}
\author{Ole Herman S. Elgesem\input{../author_footnote.tex}}
\date{}

% Removing paragraph indents is sometimes useful:
\setlength\parindent{0pt}
% ==============================================================================

% ================================= DOCUMENT ===================================
\begin{document}
    \renewcommand\marginsymbol[1][0pt]{%
  \tabto*{0cm}\makebox[-1cm][c]{$\mathbb{P}$}\tabto*{\TabPrevPos}}

\maketitle
\input{../intro.tex}

\section{Evolution strategy(ES)} % 1
\subsection{} % a
A common variant of evolution strategies used for (local) search is the \((1 + 4)\) ES.
How would this differ from the \((1 + 1)\) ES in how the search space is explored?
How does this, and \((1 + \lambda)\) in general, compared to hill climbing and greedy search?\\

\textit{Answer:}\\

The (1 + 4) ES would generate four candidate solutions from the same origin before potentially changing parent instead of one.
This would make for a more informed choice in which direction to move in the search space.
The higher \(\lambda\) is, the more information is gathered about the neighborhood of the current solution.
Recall that greedy search checks out all neighbors before making a move, while hill climbing on makes one.
So we would expect the (1 + \(\lambda\)) ES to behave increasingly like a greedy search as we increase \(\lambda\).

\subsection{} % b
What effect does an adaptive search strategy have on optimization performance?\\

\textit{Answer:}\\

An adaptive search strategy will, in most cases, increase the convergence rate of the search especially in the late stages.
However, it does not by itself help avoid getting stuck in local optima.

\subsection{} % c
How would it affect the search if the strategy parameters were mutated after the solution parameters instead of before?\\

\textit{Answer:}\\

When strategy parameters are mutated first, the change in strategy has immediate effect on the new solution that is created.
Thus the fitness of this solution also indirectly rates the strategy to some degree.
If the strategy parameters are mutated after the solution parameters this link is weaker, and we would expect the strategies to adapt slower, if at all.

\section{ES Implementation} % 2
\subsection{\marginsymbol} % a
\label{subsec:w4e3a}
Ignoring mutation, and starting with the population \(\{1, 2, 3, 4\}\), implement and run 3 generations of a \((4 + 8)\) ES maximizing \(g(x) = x\), and observe what the end population looks like (use intermediary recombination).\\

\textit{Code: (\texttt{inf3490-sol4-p2a.py})}\\
\inputminted{Python}{inf3490-sol4-p2a.py}

\subsection{} % b
If a \((4, 8)\) ES had been used in Problem \ref{subsec:w4e3a},
what would the probability of the optimal solution \((x = 4)\) surviving the first generation have been?\\

\textit{Answer:}\\

The best solution will always survive as an offspring, so it just needs to be created
at least once in the first place. If parents are drawn without replacement the
probability is zero - 4 can only be one of the parents. Otherwise the probability of
any one offspring having 4 as both parents is
\( \frac{1}{4} \times \frac{1}{4} = \frac{1}{16} \).
The chance of none of the eight offspring having 4 as both parents is
\( (\frac{15}{16})^8 \approx 0.597 \)
and so the probability of the solution surviving is
\(1 - 0.597 = 0.403\).

\subsection{\marginsymbol} % c
Repeat Problem \ref{subsec:w4e3a} with an EP with \(q = 2\).
How do the two algorithms compare?\\

\textit{Code: (\texttt{inf3490-sol4-p2c.py})}\\
\inputminted{Python}{inf3490-sol4-p2c.py}

\section{Knapsack problem}
In a 0-1 \href{https://en.wikipedia.org/wiki/Knapsack_problem}{\textbf{knapsack problem}},
how could you implement a repair mutation to transform infeasible solutions into feasible ones
(i.e. make the sum of costs of the selected items go below the budget)?\\

\textit{Answer:}\\

One way would be to randomly deselect items until the solution is under budget.
One might also consider a greedy approach that iteratively deselects the items of
least worth until the budget is met. One could even consider this as a smaller
0-1 knapsack problem from only the selected items and optimize that using some
potentially completely different algorithm.

\input{../contact.tex}
\end{document}
% ==============================================================================


\input{../common/common.tex}

\title{Math notes - How many trailing zeros}
\author{Uwe Hoffmann}
\hypersetup{colorlinks, pdftitle={Math notes - How many trailing zeros}}

\begin{document}

\setcounter{chapter}{1}
\section*{How many trailing zeros in $n!$}

\newthought{Greatest dividing exponent} and its properties is the topic of the problem in this note.\index{greatest dividing exponent}

\vspace{10 mm}
\begin{problem}
Write a program that calculates for an arbitrary positive integer $n$ how many trailing zeros there are in $n!$.
\end{problem}

Let's first try to figure out for any natural number $n$ what the number of trailing zeros is. A useful concept here is the greatest dividing exponent \cite{mathworld01}:

\begin{defn}
The greatest dividing exponent  $gde(n, b)$ of a base  $b$ with respect to a number  $n$ is the largest integer value of  $k$ such that $b^k \mid n$, where $b^k \leq n$.
\end{defn}

\begin{lem}\label{mingde}
\begin{equation*}
gde(n, a b) = min(gde(n, a), gde(n, b))  \text{, with } (a, b) = 1
\end{equation*}
\end{lem}

\begin{proof}
Assume $gde(n,a) \leq gde(n, b)$. Then $a^{gde(n, a)} \mid n$ and $b^{gde(n, a)} \mid n$, with $(a^{gde(n, a)}, b^{gde(n, a)}) = 1$, so $(ab)^{gde(n, a)} \mid n$. By definition of $gde$ we then have $gde(n, a) \leq gde(n, ab)$.

We also have $(ab)^{gde(n, ab)} \mid n$, so $a^{gde(n, ab)} \mid n$. By definition of $gde$ we then have $gde(n, a) \geq gde(n, ab)$.

It follows that $gde(n, a)= gde(n, ab)$.
\end{proof}

It's clear that the number of trailing zeros of $n$ equals $gde(n, 10)$. From lemma \ref{mingde} we are looking for $min(gde(n!, 2), gde(n!, 5))$.\\

\begin{lem}\label{facgde}
\begin{equation*}
gde(n!, p) = \sum_{k = 1}^{\lfloor log_p n \rfloor} \left\lfloor\frac{n}{p^k}\right\rfloor \text{, for a prime } p \leq n
\end{equation*}
\end{lem}

\begin{proof}
We define the following subsets of $\{1,\ldots,n\}$:

\begin{equation*}
M_p^k = \{i : 1 \leq i \leq n: p^k \mid i\}
\end{equation*}

For $k > \lfloor log_p n \rfloor$ the sets $M_p^k$ are empty, so we only consider $k \leq \lfloor log_p n \rfloor$. Each member of one set $M_p^k$ contributes $k$ to $gde(n!, p)$, so the whole set contributes $k | M_p^k |$.  From $p^k \mid i$ it follows that also $p^{k - 1} \mid i$, so $M_p^k \subseteq M_p^{k - 1}$ for $k = 2,\ldots,\lfloor log_p n \rfloor$. Being careful not to count the contributions more than once we get:
\begin{equation*}
gde(n!, p) = \sum_{k = 1}^{\lfloor log_p n \rfloor}  | M_p^k |
\end{equation*}
With $| M_p^k | = \left\lfloor\frac{n}{p^k}\right\rfloor$ we conclude the proof.
\end{proof}

\begin{lem}\label{gdetwo}
\begin{equation*}
gde(n!, 2) \geq gde(n!, 5) \text{ for any } n \geq 1
\end{equation*}
\end{lem}

\begin{proof}
Plugging in the expression of $gde$ from lemma \ref{facgde} into the claim of this lemma we get:
\begin{equation*}
   gde(n!, 2) \geq gde(n!, 5) \Leftrightarrow  \sum_{k = 1}^{\left\lfloor log_2 n\right\rfloor} \left\lfloor\frac{n}{2^k}\right\rfloor  \geq \sum_{k = 1}^{\left\lfloor log_5 n\right\rfloor} \left\lfloor\frac{n}{5^k}\right\rfloor
\end{equation*}

\noindent We establish:
\begin{equation*}
\begin{split}
   log_2 n  \geq log_5 n & \Leftrightarrow  log_2 n  \geq log_2 n\ log_5 2 \\
   & \Leftrightarrow  1 \geq  log_5 2 \text{, which is true}
\end{split}
\end{equation*}

\noindent For each $1 \leq k \leq \left\lfloor log_5 n\right\rfloor$ we have:
\begin{equation*}
 \left\lfloor\frac{n}{2^k}\right\rfloor  \geq \left\lfloor\frac{n}{5^k}\right\rfloor
\end{equation*}
and for $\left\lfloor log_5 n\right\rfloor + 1 \leq k \leq \left\lfloor log_2 n\right\rfloor$ we have:
\begin{equation*}
 \left\lfloor\frac{n}{2^k}\right\rfloor > 0
\end{equation*}

\noindent Adding up the inequalities establishes the claim.
\end{proof}

\noindent From the three lemmas we found that:
\begin{equation*}
\begin{split}
\text{(number of trailing zeros in } n! )  & = gde(n!, 10)  \\
&= min(gde(n!, 2), gde(n!, 5))  \\
& = gde(n!, 5)  \\
& = \sum_{k = 1}^{\lfloor log_5 n \rfloor}   \left\lfloor\frac{n}{5^k}\right\rfloor
\end{split}
\end{equation*}
\noindent so our program needs to calculate the expression:
\begin{equation*}
\sum_{k = 1}^{\lfloor log_5 n \rfloor}   \left\lfloor\frac{n}{5^k}\right\rfloor
\end{equation*}

\noindent The following small Haskell function does it:

\lstinputlisting[language=Haskell, basicstyle=\small, frame=trBL, caption={Haskell code}]{gdefac.hs}

\noindent It works on tuples of numbers. It keeps dividing the second number in the tuple by 5 until zero and adding the division results together into the first number of the tuple. In the end it returns the first number in the tuple.

\bibliographystyle{plainnat}
\bibliography{../common/math}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                                 %
%  GEANT manual in LaTeX form                                     %
%                                                                 %
%  Version 1.00                                                   %
%                                                                 %
%  Last Mod. 12 June 1993 1800   MG                               %
%                                                                 %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Origin{L.Urb\'{a}n}
\Revision{G.Azuelos}
\Version{Geant 3.16}\Routid{PHYS351}
\Submitted{26.10.84}    \Revised{16.12.93}
\Makehead{Simulation of e+e- annihilation}
\section{Subroutines}
\Shubr{GANNI}{}

\Rind{GANNI} generates the annihilation of a positron into either one
or two photons. It uses the following input and output:
\begin{DLtt}{MMMMMMM}
\item[input:]  via common \FCind{/GCTRAK/}
\item[output:] via common \FCind{/GCKING/}
\end{DLtt}
The routine is called automatically from the tracking routine
\Rind{GTELEC}, when the positron reaches its interaction
point during the tracking.
\Shubr{GANNIR}{}
\Rind{GANNIR} generates the positron annihilation at rest. It is
called from the tracking routine \Rind{GTELEC}, if the positron
energy is below the cut-off energy {\tt CUTELE} in common block \FCind{/GCCUTS/}.
\section{Method }
The type of annihilation is sampled from the total cross-sections
for the annihilation into two photons and into one photon (see
section {\tt [PHYS350]}).
\subsubsection{Annihilation into two photons}
The differential cross-section of the two-photon
positron-electron annihilation can be written as
\cite{bib-HEIT,bib-EGS3}:
\begin{equation}
\label{eq:phys351-1}
    \frac{d \sigma (Z, \epsilon)} {d \epsilon}=
       m \: a [S(a \epsilon) + S (a(1- \epsilon))]
\end{equation}
where $m$ is the  electron mass
$Z$ is the atomic number of the material. If we indicate with $E$ the
initial energy of the positron, with $r_{0}$ the classical electron
radius and with $k$ the energy of the less
energetic photon generated, we have:
\[
\begin{array}{LCL@{\hspace{4cm}}LCL}
\gamma   & = & \frac{E}{m} & a    & = & \gamma+1             \\
\epsilon & = & \frac{k}{E+m} &
S(x)     & = & C_1 \left[ -1 + \frac{C_2}{x} -\frac{1} {x^2}\right]        \\
C_1 & = &  \frac{Z \pi r^2_0}{a(E-m)} &
               C_2 & = & a + \frac{e \gamma}{a}
\end{array}
\]
The kinematical limits for the variable $\epsilon$ are:
\[
\epsilon_0 = \frac{1} {a+ \sqrt{\gamma^2 -1}}\leq \epsilon  \leq \frac{1} {2}
\]
Due to the symmetry of the formula (\ref{eq:phys351-1})
in $\epsilon$, the range of
$\epsilon$ can be expanded from ($\epsilon_0,1/2$) to
($\epsilon_0,1-\epsilon_0$)
and the second function $S$ can be eliminated from the formula.
Having done this, the differential
cross-section can be decomposed (apart from the normalisation) as:
\begin{equation}
\label{eq:phys351-2}
\frac{d \sigma} {d \epsilon}=\underbrace{\frac{1}{\ln
            \frac{1 - \epsilon_0}
            {\epsilon_0}}\frac{1}{\epsilon}}_{f(\epsilon)}
\underbrace{\frac{(a^2 +2a-2)-a^2 \epsilon -\frac{1}{\epsilon }}
                  {a^2 -2\vphantom{{\ln \frac{1 - \epsilon_0}
                          {\epsilon_0}}}}}_{g(\epsilon)}
\end{equation}

\begin{figure}[hbt]
   \centering
   \epsfig{file=eps/phys351-1.eps,width=14cm}
   \caption{Comparison between the K-shell binding energies
            given by the expression in the text
            and the tabulated values. }
   \label{fg:phys351-1}
\end{figure}

Using the expression (\ref{eq:phys351-2}) with random numbers
$r_i \in ]0,1[, i=1,2$, the secondary photon energy is sampled by
the following steps:
\begin{enumerate}
\item sample $\epsilon$ from $f(\epsilon)$:
\begin{equation}
\epsilon =\epsilon_0 \exp  \left[ \ln \left(\frac{1- \epsilon_0}
{\epsilon_0}
          \right)  r_1  \right]
\end{equation}
\item
compute the rejection function $g(\epsilon)$ and
\begin{enumerate}
\item if $r_2 \leq g(\epsilon)$   accept $\epsilon$
\item if $r_1 > g(\epsilon)$  go back to 1.
\end{enumerate}
\end{enumerate}
After the successful sampling of $\epsilon$, the photon energy is computed as
 \begin{equation}
k = (E+m)\epsilon
\end{equation}
and then \Rind{GANNI} generates the polar angles of the photon with respect
to an axis defined by the momentum of the positron. The azimuthal angle
$\Phi$ is generated isotropically and $\Theta$ is computed from the
energy-momentum conservation. With this information, the momentum
vector of both photons can be calculated and transformed into the {\tt GEANT}
coordinate system.

The routine \Rind{GANNIR} treates the special case when a positron falls
below the cut-off energy ({\tt CUTELE} in
common block \FCind{/GCCUTS/}) before annihilating.
In this case, it is assumed that the positron comes to rest before annihilating.
\Rind{GANNIR} generates two photons with energy $k=m$. The
angular distribution is isotropic.
\subsubsection{Annihilation into one photon}
The generated photon is assumed to be collinear with the positron.
Its energy will be
\begin{equation}
k = E + m_e - E_{bind}
\end{equation}
where $E_{bind}$ is the binding energy of the K-shell electron.
It can be estimated as follows
\begin{equation}
\label{eq:phys315-1}
E_{bind} = 0.5 (Z \, \alpha )^2  \, m_e
\end{equation}
where $\alpha$ is the fine stucture constant. The comparison
of this expression with the experimental data from
\cite{bib-XRAY} is shown in figure \ref{fg:phys351-1}.

\section{Restrictions}
\begin{enumerate}
\item
The annihilation processes producing three or more photons are
ignored, because these processes are negligible compared to the
annihilation into two photons~\cite{bib-EGS3,bib-MES1};
\item
In calculating the process it is assumed that the atomic electron initially
is free and at rest. This is the usual assumption used in shower programs
\cite{bib-EGS3};
\end {enumerate}


\section{Language Variation Model with discrete time}


\chapter{List of Symbols and Abbreviations}%
\label{chp:list_of_symbols_and_abbreviations}
% chapter list_of_symbols_and_abbreviations

\begin{longtable}{l l}
  crv         & continuous random variable \\
  DFR         & Decreasing Failure Rate \\
  DMRL        & Decreasing Mean Residual Lifetime \\
  drv         & discrete random variable \\
  $e_X(d)$    & Mean Excess Loss / Mean Residual Lifetime \\
  $G_N(t)$    & probability generating function of random variable $N$ \\
  $h_X$       & hazard rate of random variable $X$ \\
  IFR         & Increasing Failure Rate \\
  IMRL        & Increasing Mean Residual Lifetime \\
  LER         & Loss Elimination Ratio \\
  mgf         & moment generating function \\
  pf          & probability function \\
  pdf         & probability density function \\
  pmf         & probability mass function \\
  pgf         & probability generating function \\
  rv          & random variable \\
  $\bar{F}_X$ & survival function of random variable $X$ \\
  $T_L$       & Amount Paid per Loss \\
  $T_P$       & Amount Paid Per Payment \\
  $\TVaR$     & Tail-Value-at-Risk \\
  $\VaR$      & Value-at-Risk
\end{longtable}

% chapter list_of_symbols_and_abbreviations (end)


\section{NASA C-MAPSS Dataset}
\label{sec:rul_dataset}

The NASA \gls{cmaps} dataset is used to evaluate performance of the proposed method. The \gls{cmaps} dataset contains simulated data produced using a model based simulation program developed by NASA. The dataset is further divided into 4 subsets composed of multi-variate temporal data obtained from 21 sensors.

For each of the 4 subsets a training and a test set are provided. The training sets include run-to-failure sensor records of multiple aero-engines collected under different operational conditions and fault modes as described in Table \ref{table:cmapss}.

\begin{table}[!htb]
\centering
\begin{tabular}{l | l l l l}
	\hline
	 & \multicolumn{4}{c}{C-MAPSS}\\
	 Dataset & FD001 & FD002 & FD003 & FD004\\
  	\hline
  	Train Trajectories & 100 & 260 & 100 & 248\\
  	Test Trajectories & 100 & 259 & 100 & 248\\
  	Operating Conditions & 1 & 6 & 1 & 6\\
  	Fault Modes & 1 & 1 & 2 & 2\\
  	\hline
\end{tabular}
\caption{C-MAPSS Dataset details}
\label{table:cmapss}
\end{table}

The data is arranged in an $n\times26$ matrix where $n$ corresponds to the number of data points in each subset. The first two variables represent the engine and cycle numbers respectively. The following three variables are operational settings which correspond to the operating conditions in Table \ref{table:cmapss} and have a substantial effect on engine performance. The remaining variables represent the 21 sensors readings that model the engine degradation throughout time.

Each trajectory within the train and test trajectories is assumed to represent the life-cycle of an engine. Each engine in the test set is simulated with different initial health conditions (no faults), for each trajectory of an engine the last data entry corresponds to the moment the engine is declared faulty. On the other hand the trajectories within the test sets terminate at some point prior to failure. The aim of the regressor, e.g. \gls{mlp}, is then to predict the \gls{rul} of each engine in the test set. The actual \gls{rul} value of each test trajectories are also included in the dataset for verification purposes. Further discussion of the dataset and details on how the data is generated are given in \citep{Saxena2008}.

\subsection{Performance evaluation}
\label{sec:rul_metrics}

To evaluate the performance of the proposed approach on the \gls{cmaps} dataset we make use of two scoring indicators, namely the Root Mean Squared Error (\gls{rmse}) which we will refer as $e_{rms}(d)$ and a score function proposed in \citep{Saxena2008} which we address in this work as \gls{rul} Health Score (\gls{rhs}) and will be referred as $s_{rh}(d)$.

\pagebreak

The scores are defined as follows,

\begin{equation}
e_{rms}(d) = \sqrt{ \frac{1}{N} \sum_{i=1}^{N}{d_i^2}}
\label{eq:rmse}
\end{equation}

\begin{align}
s_{rh}(d) &= \frac{1}{N} \sum_{i=1}^{N}{s_i} \nonumber \\
s_i &= \begin{cases}
      e^{-\frac{d_i}{13}} - 1, & d_i < 0 \\
      e^{\frac{d_i}{10}} - 1, & d_i \geq 0,
\end{cases}
\label{eq:rhs}
\end{align}
where $N$ is the total number of testing data samples and $d = \hat{y} - y$ is the error between the estimated \gls{rul} values $\hat{y}$, and the actual \gls{rul} values $y$ for the engines within the test set. It is important to notice that $s_{rh}(d)$ penalizes late predictions more than early predictions since usually late predictions lead to more severe consequences in fields such as aerospace.