\section{Variable Byte}

Variable Byte (VByte) is an easy to implement algorithm, which makes it a
commonly used compression algorithm. VByte is a byte-aligned, i.e compression
at the byte level, algorithm that compress lists of values one integer at a
time. A compressed byte is divided in two parts
\begin{enumerate}
  \item the most significant bit is a \emph{flag} bit.
  \item the 7 bits left are used to store the integer.
\end{enumerate}
For an integer v to be compressed, the lower 7 bits are stored in one byte. The
flag is put at
\begin{inparaenum}[(a)]
\item 1 if there is still bits remaining in v or
\item 0 otherwise.
\end{inparaenum}
This process is repeated until no more bits are left. The Table~\ref{tab:vbyte}
reports the algorithms used for VByte. The Algorithm~\ref{lst:vbyte-cmp}
compress an array $L$ of integers into an array $LC$ of bytes. The loop from
line 3 to 7 continues as long as there are still 7 bits left int the current
integer $int$, where it outputs the 7 lower bits of int into LC and put the flag
at 1. The Algorithm~\ref{lst:vbyte-dcmp} decompress all the array LC. The
line 5 to 9 loops as long as the flag is at 1, meaning that the next byte data
belongs to a same integer value.

This algorithm presents the positive aspects of an easy implementation and of
good compression and decompression overall. However this algorithm possesses two
drawbacks.
The branching condition leads to branch mispredictions which makes it slower
than CPU optimised techniques such as \emph{Frame Of Reference}-based
algorithms presented next. Moreover, VByte has a poor compression ratio since
it requires one full byte to encode small integers (i.e., $\forall n < 2^7$).

\begin{table}
\resizebox{\linewidth}{!}{%
\begin{tabular}[T]{p{0.5\linewidth}p{0.5\linewidth}}
\begin{minipage}[T]{\linewidth}
\vspace{-1.9cm}
\begin{algorithm}[H]
\SetAlFnt{\tiny}
\DontPrintSemicolon
\KwIn{An array L of integers}
\KwOut{An array LC of bytes}

$i \leftarrow 0$\;
\For{$int \in L$}{
  \While{$(int \:\&\: 127) \neq 0$}{
      $LC[i] \leftarrow (int \:\&\: 127) \mid 128$\;
      $x \leftarrow int \gg 7$\;
      $i \leftarrow i + 1$\;
  }
  $LC[i] \leftarrow int$\;
}
\caption{Compression algorithm.}
\label{lst:vbyte-cmp}
\end{algorithm}
\end{minipage}
&
\begin{minipage}[T]{\linewidth}
\begin{algorithm}[H]
\SetAlFnt{\tiny}
\DontPrintSemicolon
\KwIn{An array LC of bytes}
\KwOut{An array L of integers}
\SetKwFunction{len}{len}
\SetKwFunction{poll}{poll}

\For{$i$ \KwTo $\len(L)$}{
  $b \leftarrow \poll(LC)$\;
  $dec \leftarrow b \:\&\: 127$\;
  $shift \leftarrow 7$\;
  \While{$(b \:\&\: 128 = 1)$}{
  	$b \leftarrow \poll(LC)$\;
  	$dec \leftarrow dec \mid ((b \:\&\:127) \ll shift)$\;
  	$shift \leftarrow shift+7$\;
  }
}
\caption{Decompression algorithm. The functions \texttt{len} returns the size of
an array, and \texttt{poll} retrieves and removes the first element of an array.}
\label{lst:vbyte-dcmp}
\end{algorithm}
\end{minipage}\\
\end{tabular}
}%
\caption{Variable Byte algorithms. L denotes the array to be compressed, LC its
compressed array. The Algorithm~\ref{lst:vbyte-dcmp} decompress the values in
LC, returned by the Algorithm~\ref{lst:vbyte-cmp}.}
\label{tab:vbyte}
\end{table}

\section{Rice}

In Rice, an integer $n$ is encoded in two parts: a quotient $q = \lfloor
\frac{n}{2^b} \rfloor$ and a remainder $r = n \bmod 2^b$. The quotient is
stored in unary format using $q + 1$ bits while the remainder is store in
binary format using $b$ bits. In unary format, a integer $n$ is represented
with n consecutive bits at 1, and a final bit at 0, serving as the termination
criterion. In our implementation, the parameter $b$ is chosen per block such
that $2^b$ is close to the average value of the block.

The main advantages of Rice is to achieve a very good compression ratio.
However, it is in general the slowest method in term of compression and
decompression. The main reason is that Rice needs to manipulate the unary word
one bit at a time during both compression and decompression, which is very
demanding in CPU cycles.

\section{Simple Family}

The idea behind the Simple coding is to pack as many integers as possible into
one machine word (being 32 or 64 bits). We describe one Simple coding method
(S-64) based on 64-bit machine words~\cite{anh:2010:simple64}. In the
experiments, we report only S-64 results since its overall performance was
always superior to Simple9~\cite{anh:2005:simple9}. In S-64, each word
consists of 4 status bits and 60 data bits. The 4 status bits are used to
encode one of the 16 possible configurations for the data bits. The
Table~\ref{tab:s64-status} reports the 14 configurations used in S-64. For
instance, 12 integers of 5 bits each at maximum can be encoded into a
machine word using the Status 4. The \emph{Wasted Bits} row highlights the
problem encountered with some configurations, as some bits are left unused with
a straight forward approach. A solution is to allow the last integer of the
configuration to use these extra bits.

On top of providing a good compression ratio, decompression is done efficiently
by reading one machine word and by using a precomputed lookup table over the
status bits in order to execute the appropriate optimised routine (one routine
per configuration) to decode the data bits using shift and mask operations
only. However, Simple coding performs one table lookup per machine word which
costs more CPU cycles than the other highly CPU optimised techniques presented
next.

One disadvantage is that compression cannot be done efficiently. The typical
implementation is to use a sliding window over the stream of integers and to
find the best configuration, i.e., the one providing the best compression
ratio, for the current window. This generally requires repetitive try and
error iterations over the possible configurations at each new window.

\begin{table}
\centering
\begin{tabular}{lc@{\hs}rrrrrrrrrrrrrr}
\toprule
Status & \phantom{a} &0&1&2&3&4&5&6&7&8&9&10&11&12&13\\
\cmidrule{3-16}
Bits & \phantom{a} &1&2&3&4&5&6&7&8&10&12&15&20&30&60\\
Group& \phantom{a} &60&30&20&15&12&10&8&7&6&5&4&3&2&1\\
WastedBits& \phantom{a} &0&0&0&0&0&0&4&4&0&0&0&0&0&0\\
\bottomrule
\end{tabular}
\caption{Status options used in Simple-64 coding scheme. Bits refers to the
number of bits an integer is coded with. Group refers to the number of integers
that can be put into one machine word. WastedBits reports the number of bits
wasted for each status.}
\label{tab:s64-status}
\end{table}

\section{Frame Of Reference}

Frame Of Reference (FOR) determines the range of possible values in a sub-block,
called a \emph{frame}, and maps each value into this range by storing just
enough bits to distinguish the values~\cite{goldstein:1998:icde}. In
the case of the delta-encoded list of values, since the probability
distribution generated by taking the delta tends to be naturally monotonically
decreasing, one common practice~\cite{Nzukowski:2006:pfor,anh:2010:simple64}
is to choose as frame the range $[0, max]$ where $max$ is the largest number
in the group of delta values.\footnote{This assumes that a group of values
will always contain $0$, which is not always the case. However, we found that
taking the real range $[min, max]$ was only reducing the index size by 0.007\%
while increasing the complexity of the algorithm.}

Given a frame $[0, max]$, FOR needs $\lceil \log_2(max + 1) \rceil$ bits,
called a \emph{bit frame}, to encode each integer in a
block. The main disadvantage of FOR is that it is sensitive to outliers in the
group of values. For example, if a block of 1024 integers contains 1023
integers inferior to 16, and one value superior to 128, then the bit frame will
be $\lceil \log_2(128 + 1) \rceil = 8$, wasting 4 bits for each other values.

However, compression and decompression is done very efficiently using
highly-optimised routines~\cite{Nzukowski:2006:pfor} which avoid branching
conditions. Each routine is loop-unrolled to encode or decode $m$ values using
shift and mask operations only. Listing~\ref{lst:compression-routine} and
\ref{lst:decompression-routine} show the routines to encode or decode 8
integers with a bit frame of 3. There is a compression and decompression
routines for each bit frame.

Given a block of $n$ integers, FOR determines a frame of reference for the
block and encodes the block by small iterations of $m$ integers using the same
compression routine at each iteration. Usually, and for questions of
performance, $m$ is chosen to be a multiple of 8 so that the routines match
byte boundaries. In our implementation, FOR relies on routines to encode and
decode 32 values at a time.

The selection of the appropriate routine for a given bit frame is done using a
precomputed lookup table. The compression step performs one pass only over the
block to determine the bit frame. Then, it selects the routine associated to
the bit frame using the lookup table. Finally, the bit frame is stored using
one byte in the block header and the compression routine is executed to encode
the block. During decompression, FOR reads the bit frame, performs one table
lookup to select the decompression routine and executes iteratively the
routine over the compressed sub-blocks.

\begin{fileformat}
  \centering
  \begin{minipage}[t]{0.47\linewidth}
\begin{lstlisting}[frame=lines,language=Java,caption=Loop unrolled compression routine that encodes 8 integers using 3 bits each,label=lst:compression-routine]
encode3(int[] i, byte[] b)
	b[0] = (i[0] & 7)
	     | ((i[1] & 7) << 3)
	     | ((i[2] & 3) << 6);
	b[1] = ((i[2] >> 2) & 1)
	     | ((i[3] & 7) << 1)
	     | ((i[4] & 7) << 4)
	     | ((i[5] & 1) << 7);
	b[2] = ((i[5] >> 1) & 3)
	     | ((i[6] & 7) << 2)
	     | ((i[7] & 7) << 5);
\end{lstlisting}
  \end{minipage}
  \quad%
  \begin{minipage}[t]{0.47\linewidth}
\begin{lstlisting}[frame=lines,language=Java,caption=Loop unrolled decompression routine that decodes 8 integers represented by 3 bits each,label=lst:decompression-routine]
decode3(byte[] b, int[] i)
	i[0] = (b[0] & 7);
	i[1] = (b[0] >> 3) & 7;
	i[2] = ((b[1] & 1) << 2)
	      | (b[0] >> 6);
	i[3] = (b[1] >> 1) & 7;
	i[4] = (b[1] >> 4) & 7;
	i[5] = ((b[2] & 3) << 1)
	      | (b[1] >> 7);
	i[6] = (b[2] >> 2) & 7;
	i[7] = (b[2] >> 5) & 7;
\end{lstlisting}
  \end{minipage}
\end{fileformat}

\section{Patched Frame Of Reference}

Patched Frame Of Reference (PFOR)~\cite{Nzukowski:2006:pfor} is an extension of
FOR less vulnerable to outliers in the value distribution. PFOR stores
outliers as exceptions such that the frame of reference $[0, max]$ is greatly
reduce. PFOR first determines the smallest $max$ value such that the best
compression ratio is achieved based on an estimated size of the frame and of
the exceptions. Compressed blocks are divided in two: one section where the
values are stored using FOR, a second section where the exceptions, i.e., all
values superior to $max$, are encoded using 8, 16 or 32 bits. The unused slots
of the exceptions in the first block section are used to store the offset of
the next exceptions in order to keep a linked list of exception offsets. In
the case where the unused slot is not large enough to store the offset of the
next exceptions, a \emph{compulsive exception}~\cite{Nzukowski:2006:pfor} is
created.

For large blocks, the linked list approach for keeping track of the position
of the exceptions is costly when exceptions are sparse since a large number of
compulsory exceptions has to be created. \cite{Nzukowski:2006:pfor} proposes
to use blocks of 128 integers to minimise the effect. \cite{yan:2009:www}
proposes a non-compulsive approach where the exceptions are stored along with
their offset in the second block section. We choose the latest approach since
it has been shown to provide better performance~\cite{yan:2009:www}.

The decompression is performed efficiently in two phases. First, the list of
values are decoded using the FOR routines. Then, the list of values is
\emph{patched} by:
\begin{inparaenum}
\item decompressing the exceptions and their offsets and
\item replacing in the list the exception values.
\end{inparaenum}
However, the compression phase cannot be efficiently implemented. The main
reason is that PFOR requires complex heuristics to find the best bit frame and
set of exceptions for a block.


\input{../header_function}

%---------- start document ---------- %
 \section{ecpp -- elliptic curve primality proving}\linkedzero{ecpp}
 The module consists of various functions for ECPP (Elliptic Curve Primality Proving).

 It is probable that the module will be refactored in the future so that each function be placed in other modules.\\
\spacing
The ecpp module requires mpmath.

%
  \subsection{ecpp -- elliptic curve primality proving}\linkedone{ecpp}{ecpp}
   \func{ecpp}
   {%
     \hiki{n}{integer},\ %
     \hikiopt{era}{list}{None}%
   }{%
     \out{bool}%
   }\\
   \spacing
   % document of basic document
   \quad Do elliptic curve primality proving.\\
   If \param{n} is prime, return True. Otherwise, return False. \\
   \spacing
   % added document
   The optional argument \param{era} is a list of primes (which stands for ERAtosthenes).\\
   \spacing
   % input, output document
   \quad \param{n} must be a big integer.\\
%
  \subsection{hilbert -- Hilbert class polynomial}\linkedone{ecpp}{hilbert}
   \func{hilbert}
        {\hiki{D}{integer}}
        {\out{(integer, list)}}\\
   \spacing
   % document of basic document
   \quad Return the class number and Hilbert class polynomial for the imaginary quadratic field with fundamental discriminant \param{D}.\\
   \spacing
   % added document
   \quad Note that this function returns Hilbert class polynomial as a list of coefficients.\\
   \negok If the option \linkingone{config}{HAVE\_NET} is set, at first try to retrieve the data in \url{http://hilbert-class-polynomial.appspot.com/}.
   If the data corresponding to \param{D} is not found, compute the Hilbert polynomial directly (for a long time).\\
   \spacing
   % input, output document
   \quad \param{D} must be negative int or long. See \cite{Pomerance}.\\
%
  \subsection{dedekind -- Dedekind's eta function}\linkedone{ecpp}{dedekind}
   \func{dedekind}
        {\hiki{tau}{mpmath.mpc}, \
         \hiki{floatpre}{integer}}
        {\out{mpmath.mpc}}\\
   \spacing
   % document of basic document
   \quad Return Dedekind's eta of a complex number \param{tau} in the upper half-plane. \\
   \spacing
   % added document
   \quad Additional argument \param{floatpre} specifies the precision of calculation in decimal digits.\\
   \spacing
   % input, output document
   \quad \param{floatpre} must be positive int. \\
%
  \subsection{cmm -- CM method}\linkedone{ecpp}{cmm}
   \func{cmm}
        {\hiki{p}{integer}}
        {\out{list}}\\
   \spacing
   % document of basic document
   \quad Return curve parameters for CM curves.\\
   \spacing
   % added document
   \quad If you also need its orders, use \linkingone{ecpp}{cmm\_order}.\\
   \spacing
   % input, output document
   \quad A prime \param{p} has to be odd.\\
   This function returns a list of (\param{a},\ \param{b}), where (\param{a},\ \param{b}) expresses Weierstrass' short form.
%
  \subsection{cmm\_order -- CM method with order}\linkedone{ecpp}{cmm\_order}
   \func{cmm\_order}
        {\hiki{p}{integer}}
        {\out{list}}\\
   \spacing
   % document of basic document
   \quad Return curve parameters for CM curves and its orders.\\
   \spacing
   % added document
   \quad If you need only curves, use \linkingone{ecpp}{cmm}.\\
   \spacing
   % input, output document
   \quad A prime \param{p} has to be odd.\\
   This function returns a list of (\param{a},\ \param{b},\ \param{order}), where (\param{a},\ \param{b}) expresses Weierstrass' short form and \param{order} is the order of the curve.
%
  \subsection{cornacchiamodify -- Modified cornacchia algorithm}\linkedone{ecpp}{cornacchiamodify}
   \func{cornacchiamodify}
        {\hiki{d}{integer},\
        \hiki{p}{integer}}
        {\out{list}}\\
   \spacing
   % document of basic document
   \quad Return the solution $(u,\ v)$ of $u^2 - \param{d}v^2 = 4\param{p}$.\\
   \spacing
   % added document
   \quad If there is no solution, raise ValueError.\\
   \spacing
   % input, output document
   \quad  \param{p} must be a prime integer and \param{d} be an integer such that $\param{d} < 0$ and $\param{d} > -4\param{p}$ with $\param{d} \equiv 0, 1 \pmod{4}$.\\
%
\begin{ex}
>>> ecpp.ecpp(300000000000000000053)
True
>>> ecpp.hilbert(-7)
(1, [3375, 1])
>>> ecpp.cmm(7)
[(6L, 3L), (5L, 4L)]
>>> ecpp.cornacchiamodify(-7, 29)
(2, 4)
\end{ex}%Don't indent!(indent causes an error.)
\C

%---------- end document ---------- %

\input{../footer}


\problemname{Neighborhood Watch}

Jennifer was nominated to be neighborhood watch captain and is now in charge of managing the watch for her street.

Jennifer's street consists of houses on only one side of the road. She has a plan of which houses will be a \textit{neighborhood watch house} and wants to know how safe the plan is. A \textit{walk} from one house to another house (not necessarily distinct) is considered \textit{safe} if there is at least one house along the walk that is a neighborhood watch house. The \textit{safety rating} of a plan is the number of walks that are safe on the street. Since a walk is either safe or not safe, when traveling in either direction, it is not counted twice in the safety rating.

\begin{figure}[h]
\begin{center}
 \includegraphics[width=0.7\textwidth]{houses}
 \caption{Sample input. One example of a safe walk is traveling from house $1$ to house $5$.
}
\end{center}
\end{figure}

Tell Jennifer the safety rating of her plan.

\section*{Input}

The first line of input contains two integers $N$~($1 \leq N \leq 200\,000$), which is the number of houses on the street, and $K$~($0 \leq K \leq N$), which is the number of neighborhood watch houses in Jennifer's plan. The houses are numbered $1, \dots , N$.

The next $K$ lines describe the neighborhood watch houses. Each of these lines contains a single integer $H$~($1 \leq H \leq N$), which is the house number of a neighborhood watch house. The house numbers are given in strictly increasing order.

\section*{Output}

Display the safety rating of Jennifer's plan.


Diffusion and Dynamics on Statistical Manifolds for Monte Carlo

* Spaghetti diagram. MC projected paths for uncertainty *

Stat. uncertainty in mathematical models: account for, quantify and propagate uncertainty

Goal: Stochastic sampling from induced prob measures (Bayesian posterior measures),
inferring the parameters, unobserved states, etc.

Models:
\begin{itemize}
    \item Circadean control in a cell
    \item Heat conductivity (pde with fem)
\end{itemize}

Properties:
\begin{itemize}
    \item known (more or less) dynamic equation
    \item unknown parameters
    \item unknown initial conditions
    \item uncertain measurements
\end{itemize}

Problems:
\begin{itemize}
    \item posterior is highly concentrated and in a non-linear fashion
    \item (Metropolis-Hastings) MH MCMC completely fails (poor mixing for the induced posterior)
\end{itemize}

Goal: Design MC samples which exploit intrinsic geometry of the posterior
H-MC on Riemanniann manifolds \url{https://rss.onlinelibrary.wiley.com/doi/full/10.1111/j.1467-9868.2010.00765.x}

Hamiltonian dynamics on geodesic flows (with differential geometric foundations)

$$
\int \phi(\theta) \pi(\theta) d\theta
    \approx \tfrac1n \sum_i \phi(\theta^i) + \mathca{O}(n^{-\tfrac12})
\,. $$

Draw each $\theta_i$ from an ergodic Markov Chain the stationary distribution of which is $\pi$.

Key components of the MH:
\begin{itemize}
    \item Proposal $p_p$ generates a $\theta$ given $\theta'$
    \item Accept~/~reject probability
    $$
    p_a(\theta'\vert \theta)
        = \min\bigl\{
            1, \frac{
                \pi(\theta') p_p(\theta \vert \theta')
            }{
                \pi(\theta) p_p(\theta'\vert \theta)
            }
        \bigr\}
        \,. $$
\end{itemize}

The rate of convergence is related to the proposal: appropriate proposal design.

Choosing a good proposal -- borrow from differential geometry.

Fisher information is $
G(\theta)
    = \mathbb{V} \nabla_\theta \log \ell(\theta)
$

The natural distance between distribution is \textbf{not} the
distance between the parameters:
\begin{itemize}
    \item total-variation $
    \|\mu - \nu\|_\mathrm{tv}
        = \sup_{f\in C^b} \lvert \int f d\mu - \int f d\nu \rvert
    $

    \item Kullback-Leibler divergence (iff $\mu\ll \nu$) $
    KL(\mu\|\nu)
        = \int \tfrac{d\mu}{d\nu} \log \tfrac{d\mu}{d\nu} d\nu
    $
\end{itemize}

Rao, 1945: $\chi^2$ distance (approximation of the KL)
$$
\chi^2(\delta
    = \int \frac{
            \lvert p(\omega; \theta + \delta) - p(\omega; \theta) \rvert^2
        }{
            p(\omega; \theta)
        } d\omega
    \approx \delta^\top G(\theta) \delta
    \,. $$

Jeffreys, 1948:
$$
KL(p(\omega; \theta + \delta)\|p(\omega; \theta))
    = \int p(\omega; \theta + \delta)
        \log\frac{
            p(\omega; \theta + \delta)
        }{
            p(\omega; \theta)
        } d\omega
    \approx \delta^\top G(\theta) \delta
    \,. $$

Basic of Differential (Remanniann) manifolds. A tangent space,
an exponential map, local metric
(inner product)
$
\langle u, v \rangle_x
    = u^\top G_x v
$, a Riemannian connection, a geodesic path (the shortest path on the manifold).

The geodesic is a solution to the following ODE:
$$
\tfrac{d^2 \theta^i}{d t^2}
    + \sum_{kl} \Gamma^i_{kl}
        \tfrac{d \theta^k}{d t}
        \tfrac{d \theta^l}{d t}
    = 0
\,. $$

The natural Riemannian metric for the manifold of distributions is
given by Fisher's information matrix, and known as the Fisher-Rao metric:
$
G(\theta)
    = - \mathbb{E}_{\omega \sim p(\omega\vert \theta)}
        \nabla^2_{\theta\theta} \log p(\omega\vert \theta)
$

For a univariate Gaussian $\mathcal{N}(x\vert \mu, \sigma^2)$:
$
\delta^\top G(\theta) \delta
    = \tfrac1{\sigma^2} \bigl(
        \delta \mu^2 + 2 \sigma^2 \delta
    \bigr)
$

So let's define a proposal distribution that \textbf{adapts}
to the space that is being explored (and keep the normal traversal
of Hamiltonian MC).

How do? HMC!

\subsection{Langevin} % (fold)
\label{sub:langevin}

Diffusion processes, driven by BM: SDE
$
d\theta_t
    = \tfrac12 \nabla_\theta \mathcal{L}(\theta(t)) dt
    + dB_t
$
with the $\log$-likelihood $\mathcal{L}$.

Euler-Maruyama discrete SDE
$
\theta_{t + \delta}
    = \theta_t
    + \tfrac{\delta^2}2 \nabla_\theta \mathcal{L}(\theta(t)) dt
    + \delta d \xi_t
$
for $\xi_t \sim \mathcal{N}_n(0, I)$

Isotropic proposal:
$$
p_p(\theta'\vert \theta)
    = \mathcal{N}_n(
        \theta'
        \vert
        \mu(\theta, \delta), \delta^2 I
        )
    \,, \mu(\theta, \delta)
    = \theta_t + \tfrac{\delta^2}2 \nabla_\theta \mathcal{L}(\theta(t))
    \,. $$

Is it any good?

Diffusion in the natural manifold:
$$
d\theta_t
    = \tfrac12 G^{-1}(\theta) \nabla_\theta \mathcal{L}(\theta(t)) dt
    - \sum_{kl} G^{-1}(\theta)_{kl}
    + G^{-\tfrac12}(\theta) dB_t
    \,, $$

The proposal on the manifold of the constant curvature (so that
$\Gamma$'s go away):
$$
p_p(\theta'\vert \theta)
    = \mathcal{N}_n(
        \theta'
        \vert
        \mu(\theta, \delta),
        \delta^2 G^{-1}(\theta)
        )
    \,, \mu(\theta, \delta)
    = \theta_t + \tfrac{\delta^2}2 G^{-1}(\theta)
        \nabla_\theta \mathcal{L}(\theta(t))
    \,. $$

% subsection langevin (end)

Sampling from mixtures:
$$
\theta
    \sim \sum_k \pi_k \mathcal{N}_n(\theta\vert \mu_k, \Sigma_k)
    \,, $$
\begin{itemize}
    \item choose a component
    \item sample from it
\end{itemize}


\chapter{Appendix Numerical Software}
Computers are getting more powerful over time but size of the problems we're solving scales with the increased performance. Tools for acquiring and storing data are improving at an even faster pace than processors.  It turns out the communication is the real bottleneck to scaling many algorithms. The capacity of fast memory close to the computing resource (cache) grows very slowly in time.  Commodity hardware is now being used to deploy large distributed computing systems running Apache Hadoop and Spark. The possibility for computing on ever larger data sets is tantalizing for the machine learning community.
These notes discuss the a number of numerical computing libraries the author has used in various machine learning and image processing projects.  There is a lot of effort underway to integrate and distribute some of the algorithms in these libraries to the Hadooop ecosystem.

Many of the libraries discussed here are available in Matlab. Intel MLK \& IPP, Arpack, UMFPACK, and SDPA have been integrated in the autor's klMatrix library see the section on the KL Libraries below for specifics.

\section*{BLAS MLK \& Atlas}

\section*{Graphviz - Graph Visualization Software}
Graphviz ( http://www.graphviz.org ) is open source graph visualization software. Graph visualization is a way of representing structural information as diagrams of abstract graphs and networks. It has important applications in networking, bioinformatics,  software engineering, database and web design, machine learning, and in visual interfaces for other technical domains.

Graphviz is used to generate collaboration, inheritance, and call diagrams the KL documentation. There is an API that is used in the KL framework to facilitate graph visualization.

\section*{ARPACK}
ARPACK++ is an object-oriented version of the Fortran ARPACK package. ARPACK is designed to compute a few eigenvalues and eigenvectors of large scale sparse matrices and pencils via the Arnoldi process for finding eigenvalues called. These methods utilize Krylov Subspace Projections for iterative solution that avoids matrix multiplication.  ARPACK implements the implicit restarted Arnoldi method which reduces the storage requirements of the traditional Lanczos iteration for Hermitian matrices and Arnoldi iteration for general matrices.  The key to the Krylov method is to calculate the linear subspace of $\Real^{(n,n)}$ induced by span of the first m powers of the image of $b$ under a linear operator $A$, $\kappa_m(A,b) | A \in \mathbb R^{(n,n)}
b\ in \mathbb R^n = \{b, Ab (A)^2b, \ldots (A)^mb \}$.  This avoids direct matrix matrix operations when finding the first few eigenvector, eigenvalue pairs in a large system of linear
equations.

\section*{ATLAS}
Automatically Tuned Linear Algebra software.

\section*{METIS}
METIS is a software library for finite element analysis and graph partitions.  It also can be used to reduce the fill order of
sparse matrices.

\section*{SDPA}
SDPA is a software library for solving SDPs using on the Mehrotra-type predictor-corrector infeasible primal-dual interior-point method. It is implemented C++ language and utilizes the machine dependent BLAS such as Intel MKL, ATLAS. LAPACK routines are used for matrix computations.  Efficient methods to compute the search directions exploiting the sparsity of the data matrices are implemented. Sparse or dense Cholesky factorization for the Schur complemetn matrix is automatically selected. The calculation of the Schur complement
matrix is implemented in reentrant code. A sparse version of SDPA is available that uses METIS and SPOOLES libraries for finding a proper sparse structure of the problem.

\section*{SPOOLS}
SPOOLES is a library for solving sparse real and complex linear systems of equations. SPOOLES can factor and solve square linear systems of equations with symmetric structure, and it can compute multiple minimum degree, generalized nested dissection and multisection orderings of matrices with symmetric structure.  SPOOLES utilizes a variety of Krylov iterative methods. The preconditioner is a drop tolerance factorization.

\section*{SuperLU}
SuperLU ( http://crd-legacy.lbl.gov/~xiaoye/SuperLU/) is a general purpose library for the direct solution of large, sparse, nonsymmetric systems of linear equations on high performance machines. The library is written in C and is callable from either C or Fortran. The library routines will perform an LU decomposition with partial pivoting and triangular system solves through forward and back substitution. The LU factorization routines can handle non-square matrices but the triangular solves are performed only for square matrices. The matrix columns may be preordered (before factorization) either through library or user supplied routines. This pre-ordering for sparsity is completely separate from the factorization. Working precision iterative refinement subroutines are provided for improved backward stability. Routines are also provided to equilibrate the system, estimate the condition number, calculate the relative backward error, and estimate error bounds for the refined solutions.

\section*{SuiteSparse}
Tim Davis' ( http://www.cise.ufl.edu/~davis/welcome.html) collection of sparse matrix software.  Tim is also the curator of The University of Florida Sparse Matrix Collection (http://www.cise.ufl.edu/research/sparse/matrices/) a must see for anyone interested in sparse
matrices and visualization.

AMD: symmetric approximate minimum degree
BTF: permutation to block triangular form
CAMD: symmetric approximate minimum degree
CCOLAMD: constrained column approximate minimum degree
COLAMD: column approximate minimum degree
CHOLMOD: sparse supernodal Cholesky factorization and update/downdate
CSparse: a concise sparse matrix package
CXSparse: an extended version of CSparse
KLU: sparse$ LU$ factorization, for circuit simulation
LDL: a simple $LDL^T$ factorization
UMFPACK: sparse multifrontal $LU$ factorization
RBio: MATLAB toolbox for reading/writing sparse matrices
UFconfig: common configuration for all but CSparse
SuiteSparseQR: multifrontal sparse $QR$

\section*{SuiteSparse AMD}
AMD is a set of routines for pre-ordering a sparse matrix prior to numerical factorization. It uses an approximate minimum degree ordering algorithm to find a permutation matrix P so that the Cholesky factorization $PAP^\dag =LL^\dag$ has fewer (often much fewer) nonzero entries than the Cholesky factorization of A. The algorithm is typically much faster than other ordering methods and minimum degree ordering algorithms that compute an exact degree . Some methods, such as approximate deficiency [Rothberg and Eisenstat 1998] and graph-partitioning based methods [Hendrickson and Rothberg 1999; Karypis and Kumar 1998; Pellegrini et al. 2000; Schulze 2001] can produce better orderings, depending on the matrix. The algorithm starts with an undirected graph representation of a symmetric sparse matrix . Node $i$ in the graph corresponds to row and column i of the matrix, and there is an edge $(i,j)$ in the graph if $a_{ij}$ is nonzero. The degree of a node is initialized to the number of off diagonal non-zeros in row $i$, which is the size of the set of nodes adjacent to $i$ in the graph.

\section*{SuiteSparse UMFPACK}
UMFPACK is a set of routines for solving systems of linear equations, $Ax = b$, when $A$ is sparse and unsymmetric. It is based on the Unsymmetric-pattern MultiFrontal method. UMFPACK factorizes $PAQ$, $PRAQ$ and $PR^{-1}AQ$, into the product $LU$, where $L$ and $U$ are lower and upper triangular, respectively, $P$ and $Q$ are permutation matrices, and $R$ is a diagonal matrix of row scaling factors (or $R = I$ if row-scaling is not used). Both $P$ and $Q$ are chosen to reduce fill-in (new nonzeros in $L$ and $U$ that are not present in $A$). The permutation $P$ has the dual role of reducing fill-in and maintaining numerical accuracy (via relaxed partial pivoting and row interchanges). The sparse matrix $A$ can be square or rectangular, singular or non-singular, and real or complex (or any combination). Only square matrices $A$ can be used to solve $Ax = b$ or related systems. Rectangular matrices can only be factorized.

\section*{fftw}
fftw is a highly optimized library for calculating the discrete Fourier transform. \cite{G_ieeetrans_FFTW}.  Generally fftw performs better than Intel IPP.  fftw can be sped up by compiling with the Intel Compiler.

\section*{Simulation and modeling with the kl Software
Framework}Class, interaction and collaboration diagrams are presented below for a modeling framework implemented by the author. The framework is implemented in C++.  The simulation of various univariate and multivariate random number generators along with the distribution tests from the CDHC library are included as well.
Features of this framework include:
\begin{itemize}
\item utilizing optimized BLAS libraries
\item the up to date methods for univariate random number generation
\item wrappers for Intel performance primitives and GSL
\item multiple memory management facilities
\end{itemize}
We will use the term $EPDF_X$ to mean the empirical probability density function. There are a variety of univariate tests to help determine which parametric distribution your data belongs to. These fall under the category of Goodness of Fit testing. For a parametric family the null hypothesis $H_o : X=_d p(x| \theta)$ is tested against the alternative that $X$ does not belong to the family $p(x|\theta)$ There are also family of test to determine whether two $EPDF$'s come from the same distribution.


\chapter{Introduction to Vehicle Routing Problem}\label{introduction_vrp}

    The problem objective of \gls{vrp} is simply finding the shortest route for multiple vehicles to serve all the given set of customers. The shortest route can be differently interpreted based on your minimalization criteria, e.g, traveled distance, time, or a combination of both. It was first proposed by Dantzig and Ramser \cite{truck-dispatching-problem} in 1959, and since then researchers are coming up with different approaches how to solve the problem.

    \section{Vehicle Routing Problem Definition}

    The general \gls{vrp} can be defined as a problem in a complete graph $G=(V,E)$ of finding the optimal permutation $\pi_l = (\pi_0, \cdots, \pi_m)$ of nodes $V$ all starting from a node $v_0$ for given number of paths $k$ which results in minimal traversal cost where $\forall v \in (V \setminus v_0)$ are visited only once. \gls{vrp} is generalization of \gls{tsp} which only has one path.

    \begin{figure}[ht]
        \centering
        \includegraphics[width=0.75\textwidth]{resources/intro/vrp-graph.png}
        \caption{Intuitive view of \gls{vrp} instance on left and proposed solution for 5 vehicles on the right \cite{vrp-malaga}}
        \label{fig:vrp-graph}
    \end{figure}

    \gls{vrp}s are classified as NP-Hard problems which was proved by Lenstra and Kan \cite{time-complexity-vrp}. It means that in the worst case, adding new nodes, i.e., customers results in an exponential increase of computational complexity.

    \subsection{VRP Notation}
    Let's introduce our used notation and its real-world interpretation.
    \begin{itemize}
        \item $G=(V,E)$ is a complete undirected graph
        \begin{itemize}
            \item Network of routes
        \end{itemize}
        \item $v_0$ is the initial node
        \begin{itemize}
            \item A depot
        \end{itemize}
        \item $V^{\prime} = (v_1, \cdots, v_n)$ nodes expect the initial node
        \begin{itemize}
            \item Geographically scattered location of customers
        \end{itemize}
        \item $E = \{(v_i, v_j)| v_i, v_j \in V, i \neq j\}$ with associated weight as a cost $c: E \to \mathbb{N}^+$
        \begin{itemize}
            \item A single route between two locations with associated cost, e.g., distance.
        \end{itemize}
        \item $C$ is a matrix of edge weights indexed by nodes. $c_i_j$ where $i,j \in V$
        \begin{itemize}
            \item Matrix of costs between customers
        \end{itemize}
        \item $R_i \subset V$ is a path that starts and ends at $v_0$. $(r_0 = v_0 \land r_{|R_i|} = v_0)$
        \begin{itemize}
            \item Route visit a subset of customers starting and ending at the depot, it can be referred to it as a delivery plan.
        \end{itemize}
        \item $k$ number of paths
        \begin{itemize}
            \item Number of vehicles
        \end{itemize}
        \item $R = R_1, \cdots, R_k$ is a set of paths
        \begin{itemize}
            \item All routes (delivery plans) for a given instance of \gls{vrp}.
        \end{itemize}
        \item $\pi = (\pi_1, \cdots, \pi_k)$ solution for a given instance of \gls{vrp}.
        \begin{itemize}
            \item Customer locations in visiting order for multiple vehicles.
        \end{itemize}
    \end{itemize}

    \textbf{Feasibility of \gls{vrp} solution} for \gls{vrp} of routes $R$ is feasible only if each node $V_1$ is visited exactly once.

    \textbf{The cost of route $R_i$} which we aim to minimize is the sum of its weights (costs). If we operate in Euclidean space, then it is L2 norm of route locations.
    \begin{equation}
        C(R_i) := \sum_{k = 0}^{|R_i|} c_{r_k}_{r_k+1}
    \end{equation}

    \textbf{The cost of \gls{vrp} solution} is the sum of route costs.
    \begin{equation}
        C(R) := \sum_{i = 1}^{|R|} C(R_i)
    \end{equation}

\section{Vehicle Routing Flavors}
Our modern world heavily relies on complex logistics networks. It requires to synchronize multimodal planning to ship your goods from one side of the world to your doorstep. In order to achieve this, multiple variants and flavours of \gls{vrp} had to be studied and implemented in the real world use cases. It goes from ordinary variants like measuring the capacity of cars to a more niche problem like eVRP where vehicles are required to make stops to recharge.

All the flavours of \gls{vrp} can be mutually combined, which is usually the main area of research.

    \begin{figure}[ht]
        \centering
        \includegraphics[width=1.0\textwidth]{resources/intro/vrp-flavours.png}
        \caption{Taxonomy of VRPs \cite{bono-stochastic-vrp}}
        \label{fig:vrp-flavours}
    \end{figure}

The sections below are describing each flavour shown in \ref{fig:vrp-flavours}.

    \subsection{Capacitated Vehicle Routing Problem}
    The \gls{cvrp} extends the regular \gls{cvrp} in introducing a capacity element for each customer. In the literature, it is sometimes referred to as a demand. The customer's demand is $d \in \mathbb{N}^+$ which may represent capacity in the form of weight, size but also in some abstract concepts such as a basket of apples. Additionally, each vehicle has a predefined capacity $Q > 0$.

    The \gls{cvrp} extends the solution feasibility formula by the following capacity constrain.
    \begin{equation}
        q(R) := \sum_{i \in R} d_i \leq Q
    \end{equation}

    If the vehicle capacity of the fleet stays the same, we are dealing with \gls{cvrp} with homogeneous fleet. A fleet with varying capacity for each vehicle is a heterogeneous fleet.

    \subsection{Vehicle Routing Problem with Time Windows}
    The \gls{vrptw} \cite{vrptw-solomon} extends the regular \gls{vrp} by time constraint for each customer. Customers have assigned time window interval $[e_i, l_i]$ where $e_li < l_i$. The time interval is the request within a vehicle is supposed to visit the node.

    The time window can be either implemented as a hard constraint or a soft constraint. Hard constraint forces the vehicle to visit the node, i.e., the customer either in the given time interval or the solution is not feasible. Soft constrains are not strictly enforcing the vehicle to visit the customer, but they introduce a penalty for a violated interval barring a penalty cost. The penalty becomes a part of the cost function which \gls{vrp} aims to minimize.

    In this thesis, we will be focusing on soft constraints for time windows since it is a better reflection of real world use cases. Most businesses allow couriers to arrive late or early, but these types of arrivals are supposed to be minimized.

    \subsection{Pick and Deliver}\label{pick-and-delivery}
    The \gls{pdp} extends the regular \gls{vrp} by pairing pick and drop with precedence relationships, in which a pickup point must precede the paired delivery point. This flavour of \gls{vrp} is one of the most complex and even challenging for conventional methods like optimization heuristics algorithms.

    The feasibility of a \gls{pdp} solution is checking whether all delivery points have preceded pickup point.

    \subsection{Static vs. Dynamic}\label{dynamic}
    When solving the vehicle routing model, usually we assume that all the input data are static and known with certainty. However, this is not the case in real-life applications where data such as customer demand or travel time are often incomplete or not precise during the planning phase, they are only gradually revealed and specified.

    \textbf{Static \gls{vrp}} does not assume that the input data could be subject to change. The \textbf{dynamic \gls{vrp}} is aware about the information evolution\cite{psaraftis} and its goal is to obtain a robust routing planner that will be able to solve already seen instances with subject to small changes without the need of recalculating the whole instance again. This is called a priori optimization, after solving a given instance of a combinatorial optimization problem, it becomes necessary to repeatedly solve many other instances with a small variation from the original instance but without reconsideration of the entire problem \cite{apriori-optimization}.

    In this thesis, the \gls{vrp} based on \gls{ai} could be a great candidate for dynamic \gls{vrp} even though, the entire instance is being recalculated. The reason is that the problem solution is calculated in a seconds instead of minutes and the \gls{ai} technically already seen the instance in some variation during the training phase.

    Dynamic \gls{vrp} can be achieved with enough robust architecture around the core planner and periodically recalculating the instance with newly revealed information. The planner needs to take its previous solution as an input so the part of the problem does not need to be recalculated. This approach tends to be more exploitative since it is finding a solution in a predefined search space. It would benefit from introducing an explorative element which would diversify the search and could find better cost in a different local optimum.

    \subsection{Deterministic vs. Stochastic}\label{dynamic}
    Psaraftis \cite{psaraftis} stated that there are two important dimensions of input data, the information evolution which is used in dynamic \gls{vrp} and quality of information for stochastic \gls{vrp}. The majority of studied \gls{vrp} models are under the assumption that all the information necessary to formulate the problems is known and readily available. This is true but only for the deterministic settings \cite{vrp-bible}.

    A \textbf{\gls{vrp} is stochastic} \cite{stochastic-vrp} when some of its data behave as random variables, and the routes must be defined before the values of these random variables become known. Based on the probability distribution of the random variables, we may extract some hidden information and use it to our advantage in the planning process. The newly created plans will have incorporated stochastic information and the routing decisions may lead to different decisions because of the stochastic information being part of the cost function.

    A specific real-life example of stochastic \gls{vrp} would be if we consider an electric fleet of shared mobility vehicles and treat the locations of Blinkee electric scooters as random variables. Based on the probability distribution of Blinkee scooter, we may predict the time and location where a courier will transfer to a new fully charged Blinkee scooter. This action will be incorporated into the planned routes.

    In contrast, \textbf{deterministic \gls{vrp}} has no random information which could be leveraged before the execution of routes and all the given information are known with certainty. In this thesis, we are focusing on deterministic \gls{vrp}.

    \subsection{Other Flavours}
    Dial-a-Ride (DARP) proposed by Wilson et al. \cite{darp-proposed} in 1971 is a special case of dynamic \gls{vrp} with pick and deliver. Passengers request a ride at a specified origin and drop location with an optional time window.

    \label{split-delivery}Split Delivery \gls{vrp} \cite{split-deliver} is a variant where customers are allowed to be visited more than once. This can be convenient for deliveries of large capacity or stocking fulfillment centers.

    Multi Depot \gls{vrp} is a simplification of the vehicle routing problem with pick and deliver, where pick can happen only on predefined depot locations. This simplification of pickup location is making the problems less complex then \gls{vrppd}.

\section{VRP in a Real-world}
Consumer habits have been shifting towards online and the pandemic situation only accelerated this process. Delivery option is nowadays taken for granted and consumers are demanding a perfect delivery experience. In 2020, there has been shipped over 5.5 billion packages around the world \cite{num-shipped-packages}.

Solving various flavors of \gls{vrp} efficiently in a reasonable time plays a crucial role for multiple businesses. For example, urban logistics is an essential part of the delivery process, not only it is the last part of the delivery chain, but frequently the courier interacts with the customer and delivery on time with proper ETA prediction is a must. It is also the most expansive part which makes up about 53\% of shipment’s total cost\cite{num-shipped-packages}. Urban logistics by large benefits from a better and more optimized \gls{vrp} which increase the delivery efficiency and reduces the delivery cost.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.75\textwidth]{resources/intro/hofkorb.png}
    \caption{Grocery delivery planning with multiple depots from the GoDeliver system}
    \label{fig:hofkorb}
\end{figure}

At GoDeliver, we are building an autonomous last-mile delivery system and at the core is a planning system which is solving various \gls{vrp}. The flavour which we are focusing on is Dynamic Capacitated Vehicle Routing Problem with Time Windows and Pick and Delivery (CVRPDPTW). GoDeliver typical use case is on-demand food delivery with multiple depots (pick and deliver), this means that the system has to be dynamic and flexible because a new customers are ordering stochastically for a chosen time window. Another our common use case shown in \ref{fig:hofkorb} is grocery delivery with multiple depots, time windows, and capacity for customers.


%\chapter{Space Decomposition and Coordinate Descent Methods}
\subsection{Space decomposition versus function decomposition}
For simplicity of exposition, we assume that
$$
\mathcal X=\mathbb R^n.
$$

\subsubsection{Space decomposition}
$$
\mathcal X=\sum_{j=1}^m\mathcal X_j
$$

Observation: The space-alternating iterative optimization algorithm
usually converges if the subspace optimization iterative method
converges.  Randomization does improve convergence, see
Xu~\cite{xu1992iterative}, \cite{xu2002method},  Tai and Xu
\cite{tai2002global}, and Hu and Xu (2017).

Many optimization algorithms are those type, especially some multigrid
method for optimization. And coordinate descent type methods. By the
way, the popular ADMM method is also closely related to this type.

Chen, He, Ye and Yuan \cite{chen2016direct} gave a counter example for ADMM for
$m=3$, that is divergent.  But the randomized version converges.  see
Xu, Xu and Ye \cite{xu2017derandomized} and the reference cited
therein.

\subsubsection{Function decomposition}
If we write
\begin{equation}
\label{function-decomposition}
f=\frac{1}{N}\sum_{i=1}^Nf_i
\end{equation}
Here is a basic question:
\begin{quote}
	Can we find $\argmin f$ by alternatively finding $\argmin f_i$?
\end{quote}

We first state the consistency and inconsistency definition for those kinds questions:
\begin{definition}
	Denote
	$$
	X^* = \mathop{\arg\min}_{x} f(x),
	$$
	and
	$$
	X_i^* =  \mathop{\arg\min}_{x} f_i(x) \quad i = 1:m.
	$$
	We say that the function decomposition
	\eqref{function-decomposition} is consistent if
	\begin{equation}
	\label{xxi0}
	X^* \subset \bigcap_{i=1}^m X_i^*.
	\end{equation}
	Or equivalently
	\begin{equation}
	\label{xxi1}
	X^* = \bigcap_{i=1}^m X_i^*,
	\end{equation}
	since
	$$
	\min_x f(x) = \min_x \sum_{i=1}^m f_i(x) \ge \sum_{i=1}^m \min_x f_i(x).
	$$
\end{definition}

For each $i$, we assume that we are given an iterative algorithm for
optimizing $f_i$ such as
\begin{equation}
\label{alg-i}
x\leftarrow x+g_i(x),
\end{equation}
We note that a necessary condition for convergence of \eqref{alg-i} is
that
\begin{equation}\label{single-conv}
g_i(x) = 0, \quad\forall x \in X_i^*.
\end{equation}

It is then natural to consider the following algorithm:
\begin{algorithm}\caption{Alternative subfunction iteration (constant step-size)}\label{CIGD-constant}
	For $t = 1,2,\cdots, T$
	$$
	x\leftarrow x+  g_{i_t}(x)
	$$
	with
	$$
	i_t \equiv t \mod(m), \quad i_t \in 1:m,
	$$
\end{algorithm}

Observation 1:  If gradient type methods are applied(i.e $g_i = -\eta_t \nabla f_i$), the alternating optimization algorithm with constant stepsize does not
converge for inconsistent case, even if the underlying iterative method converges for each
$i$.

\begin{comment}
There is a counter example(see Bertsekas's book ``Convex optimization
algorithms" in chapter 2.1.5) that: $m=3$ and
$$
f_i = (c_i \cdot x + b_i)^2,
$$
with $c_i, x \in \mathbb{R}^2$ and $b_i \in \mathbb{R}$ with $i \in 1:m$.
\begin{theorem}
There exist some choices of  $c_i, b_i$ such that this system is inconsistent and this Algorithm~\ref{CIGD-constant} does not converge with any constant step-size.
\end{theorem}
\end{comment}
A more abstract and simpler result is that:
\begin{theorem}
	If $f_i$ is convex and $X^* = \{x^*\}$ is a single point set and the
	function decomposition is inconsistent, the above
	Algorithm~\ref{CIGD-constant} with gradient descent type cannot
	converge.
\end{theorem}
\begin{proof}
	If it converges, we have $x_t \to x^* $, which means that
	$$
	x^* = x^* + \alpha g_i(x^*), \quad \forall i = 1:m.
	$$
	Because of the inconsistency, exist a $1 \le i \le m$ such
	that $x^* \notin X_i^*$, this is contrary to the convexity of
	$f_i$ and \ref{single-conv}.
\end{proof}

Observation 2:  by introducing diminishing damping parameter, the
aforementioned divergent method can be made convergent:
\begin{algorithm}\caption{Alternative subfunction iteration(diminishing step-size)}\label{CIGD-diminishing}
	For $t = 1,2,\cdots, T$
	$$
	x\leftarrow x+ \eta_t g_{i_t}(x), \quad \eta_t \to 0,
	$$
	with
	$$
	i_t \in 1:m,
	$$
	randomly or alternatively.
\end{algorithm}

\begin{theorem} For inconsistent function decomposition, a necessary
	condition for converge with both choosing $i_t$ randomly or alternatively is that the learning rate
	$\eta_t\to 0$ as $t\to\infty$.
\end{theorem}

\subsection{Connection between space decomposition and function decomposition}
When linear case, i.e each $f_i$ is like $(a_i \cdot x + b_i)^2$, those two methods will consistent. An simple idea is that,
$$
\nabla f_i \in span\{a_i\}.
$$
Examples:  Kaczmarz method. The original idea of Kacamarz algorithm is to solve the next consistent linear system:
$$
Ax = b,
$$
with full rank $A$ and $A \in \mathbb{R}^{m\times n}$ for $m \ge n$.
The iterative scheme of Kaczmarz algorithm is:
\begin{algorithm}\caption{Kaczmarz Algorithm for $Ax = b$}\label{algo:Kacamarz}
	For $t = 1,2,\cdots, T$
	\begin{equation}\label{KacamarzAlgo}
	x_{t+1} = x_t -  \frac{1}{\|A_{i_t}\|^2}(A_{i_t} \cdot x_t - b_{i_t})A_{i_t}.
	\end{equation}
	with
	$$
	i_t \in 1:m,
	$$
	randomly or alternatively.
\end{algorithm}

From the optimization point of view, we have the next two strategies to understand it as:
\begin{itemize}
	\item Using CIGD to optimize $f = \sum_{i=1}^m f_i(x)$ with
	$$
	f_i(x) = \frac{1}{2}(A_i \cdot x + b_i)^2
	$$
	with learning rate $\eta_{i_t} = \frac{1}{\|A_{i_t}\|^2}$.
	\item Using CIGD to optimize $f = \sum_{i+1}^m f_i(x)$ with
	$$
	f_i(x) = \frac{1}{2\|A_{i_t}\|^2}(A_i \cdot x + b_i)^2
	$$
	with learning rate $\eta_{i_t} = 1$.
\end{itemize}

However, we can also understand a this algorithm using the method of subspace correction. Denote the new system as:
\begin{equation}\label{eq:KacamarzLinear}
AA^T y = b.
\end{equation}
If we apply the stander Gauss-Seidel iterative scheme for the above problem, we have
\begin{equation}\label{eq:G-SKaca}
y_{t+1} = y_t -  \frac{1}{\|A_{i_t}\|^2}e_{i_t}^T(AA^Ty_t - b) e_{i_t}.
\end{equation}
and
$$
A^Ty_{t+1} =A^T y_t -  \frac{1}{\|A_{i_t}\|^2}e_{i_t}^T(AA^Ty_t - b) A^Te_{i_t}.
$$
which means we also go the Kacamarz iteration:
$$
x_{t+1} =x_t -  \frac{1}{\|A_{i_t}\|^2}e_{i_t}^T(Ax_t - b) A^Te_{i_t}.
$$

Because of the well-studied of MSC, many result in traditional
analysis of Kacamarz method can be obtained from MSC framework.

Details of the above results and generalization can be found in He and
Xu \cite{he2018iterative}.


\documentclass[main.tex]{subfiles}
\begin{document}

\section*{Fri Dec 06 2019}

From last time we have the metric
%
\begin{align}
  \dd{s^2} = \eta_{\mu \nu } \dd{x^{\mu }} \dd{x^{\nu }}  - \frac{4GJ}{r} \sin^2 \theta \dd{t} \dd{\varphi }
\,,
\end{align}
%
and since in polar coordinates the \(z\) axis is singular we use cartesian ones:
%
\begin{subequations}
\begin{align}
  x &= r \sin \theta \cos \varphi  \\
  y &= r \sin \theta \sin \varphi
\,,
\end{align}
\end{subequations}
%
which means \(\varphi = \arctan (y/x)\). Then
%
\begin{align}
  \dd{\varphi } = \frac{1}{1 + y^2 / x^2} \dd{\qty(y/x)}
  = \frac{x \dd{y} - y \dd{x}}{r^2 \sin^2\theta } \dd{\varphi }
\,,
\end{align}
%
so we get
%
\begin{align}
  \dd{s^2} = \eta_{\mu \nu } \dd{x^{\mu }} \dd{x^{\nu }}
  - \frac{4GJ}{r^2} \frac{x \dd{y} - y \dd{x}}{r} \dd{t}
\,,
\end{align}
%
since the \(\sin^2 \theta\) simplified.
So our metric is Minkowski plus a perturbation:
%
\begin{align}
  \delta g_{01} = \delta g_{10} = \frac{2GJy}{(x^2+ y^2+z^2)^{3/2}}
\,,
\end{align}
%
and
%
\begin{align}
    \delta g_{02} = \delta g_{20} = \frac{-2GJx}{(x^2+ y^2+z^2)^{3/2}}
  \,,
\end{align}
%
which means that our Christoffel symbols are
%
\begin{align}
  \Gamma^{\mu }_{\alpha \beta } = \frac{1}{2} \eta^{\mu \gamma } \qty(\delta g_{\gamma \beta, \alpha } + \delta  g_{\gamma \alpha , \beta } - \delta g_{\alpha \beta , \gamma })
  + O(J^2)
\,.
\end{align}

The 4-velocity is \(u^{ \alpha } = (u^{t}, 0, 0, u^{z})\), and the spin vector initially is \(s^{\alpha } _{\text{in}} = (0, s^{x} _{\text{in}}, 0,0)\).

We want to show that \(s^{z}\) is zero at all times.

The equation is
%
\begin{align}
  \dv{s^{z}}{\tau } + \Gamma^{z}_{\alpha \beta } u^{\alpha } s^{\beta }= 0
\,,
\end{align}
%
but the velocity has only two components: then
%
\begin{align}
  \dv{s^{z}}{\tau } + \Gamma^{3}_{0 \beta } u^{0} s^{\beta } + \Gamma^{3}_{3 \beta } u^{3} s^{\beta } = 0
\,,
\end{align}
%
but
%
\begin{align}
  \Gamma^{3}_{0 \beta } = \frac{1}{2} \eta^{3 3 } \qty(
      \delta g_{3 \beta , 0 }
      + \delta g_{03, \beta }
      - \delta g_{0 \beta , 3}
  )
\,,
\end{align}
%
but the first two are zero since they contain an index \(3\). The last is also zero:
%
\begin{align}
  \delta g_{0 \beta , 3} = \pdv{}{z} \qty(\frac{2GJy}{r^{3}}) \quad \text{or}\quad
  \pdv{}{z} \qty(-\frac{2GJx}{r^{3}})
\,,
\end{align}
%
and in either case we need to calculate the derivative on the \(x=y=0\) axis, therefore since the terms are proportional to \(x\) or \(y\) they will vanish.

For the other term we also have
%
\begin{align}
  \Gamma^{3}_{3 \beta } = \frac{1}{2} \eta^{33} \qty(\delta g_{3 \beta , 3} + \delta g_{33, \beta } - \delta g_{\beta 3, 3}) =0
\,,
\end{align}
%
so \(s^{z} = 0\) for all times.

Now let us write the evolution for the \(s^{t}\) component.
%
\begin{align}
  \dv{s^{t}}{\tau } + \Gamma^{0}_{\alpha \beta } u^{\alpha } s^{\beta } = 0
\,,
\end{align}
%
and then we must look at the Christoffels:
%
\begin{subequations}
\begin{align}
\Gamma^{0}_{0 \beta } &= \frac{1}{2} \eta^{00} \qty(
    \delta g_{0 0, \beta } +
    \delta g_{0 \beta , 0}
    - \delta g_{0 \beta ,0 }
) =0 \\
\Gamma^{0}_{3 \beta } &= \frac{1}{2} \eta^{00} \qty(
    \delta g_{0 \beta , 3 } +
    \delta g_{3 0 , \beta }
    - \delta g_{3 \beta ,0 }
) =0
\,,
\end{align}
\end{subequations}
%
so the only components which evolve are
%
\begin{subequations}
\begin{align}
  \dv{s^{1}}{\tau } + \Gamma^{1}_{\alpha \beta } u^{\alpha } s^{\beta } &= 0 \\
  \dv{s^{2}}{\tau } + \Gamma^{2}_{\alpha \beta } u^{\alpha } s^{\beta } &= 0
\,,
\end{align}
\end{subequations}
%
and now we notice this: since we need to form \(\delta g_{01}\) or \(\delta g_{02}\), one of the lowe two indices in the Christoffels must be zero: but it cannot be \(\beta \) since \(s^{t}=0\): so we set \(\alpha =0\), and consider only \(\Gamma^{1}_{0 \beta } u^{0} s^{\beta }\) and \(\Gamma^{2}_{0 \beta } u^{0} s^{\beta }\). Also, \(\beta \) can only be \(1\) or \(2\). Then the equations become:
%
\begin{subequations}
\begin{align}
    \dv{s^{1}}{\tau } + \Gamma^{1}_{0 1 } u^{0 } s^{1 } + \Gamma^{1}_{0 2 } u^{0 } s^{2 } &= 0 \\
  \dv{s^{2}}{\tau } + \Gamma^{2}_{0 1} u^{0 } s^{1}  + \Gamma^{2}_{0 2} u^{0 } s^{2} &= 0
\,,
\end{align}
\end{subequations}
%
and as yesterday we can use \(u^{0} = \dv*{t}{\tau }\) to get derivatives with respect to \(t\): we then get
%
\begin{subequations}
    \begin{align}
        \dv{s^{1}}{t } + \Gamma^{1}_{0 1 }  s^{1 } + \Gamma^{1}_{0 2 }  s^{2 } &= 0 \\
      \dv{s^{2}}{t } + \Gamma^{2}_{0 1}  s^{1}  + \Gamma^{2}_{0 2}  s^{2} &= 0
    \,.
    \end{align}
\end{subequations}

Now we can compute the Christoffels:
%
\begin{align}
  \Gamma^{1}_{01} = \frac{1}{2} \eta^{11} \qty(
      \delta g_{11, 0} + \delta g_{01, 1}
  ) = 0
\,,
\end{align}
%
and similarly for \(\Gamma^{2}_{02}=0\).

On the other hand:
%
\begin{subequations}
\begin{align}
  \Gamma^{1}_{02} &= \frac{1}{2} \eta^{11} \qty(
      \delta g_{12, 0} + \delta g_{01, 2}
      - \delta g_{02, 1}
  ) \\
  \Gamma^{2}_{01} &= \frac{1}{2} \eta^{22} \qty(
    \delta g_{21, 0} + \delta g_{02, 1}
    - \delta g_{01, 2}
) = - \Gamma^{1}_{02}
\,,
\end{align}
\end{subequations}
%
and the time derivative vanishes: so in the end we get
%
\begin{align}
  \Gamma^{1}_{02} = \frac{1}{2} \qty(\pdv{}{y} \qty(\frac{2GJy}{(x^2+y^2+z^2)^{3/2}}) - \pdv{}{x} \qty(- \frac{2GJx}{(x^2+y^2+z^2)^{3/2}}))
\,,
\end{align}
%
which we must calculate at \(x=y=0\): therefore the only relevant contribution to the derivative is the one in which the derivative acts on the numerator, since otherwise we get a term proportional to either \(x\) or \(y\). So in the end we get:
%
\begin{align}
  \Gamma^{1}_{02} = - \Gamma^{2}_{01} = \frac{2GJ}{z^3}
\,.
\end{align}

In the end then
%
\begin{subequations}
\begin{align}
  \dv{}{t} \left[\begin{array}{c}
  s^{x} \\
  s^{y}
  \end{array}\right] =
  \frac{2GJ}{z^3}
  \left[\begin{array}{cc}
  0 & 1 \\
  -1 & 0
  \end{array}\right]
  \left[\begin{array}{c}
    s^{x} \\
    s^{y}
\end{array}\right]
\,,
\end{align}
\end{subequations}
%
which, if \(z\) were constant, would be an angular precession with angular velocity \(\Omega = 2GJ/z^3\), called the Lense-Thirring precession.

\(z\) is not actually a constant: however this describes the instantaneous angular velocity.

We assumed that the spin vector and the angular momentum of the BH are orthogonal. In general, one finds the following expression in terms of the angular momentum vector \(\vec{J}\) and of the radial unit vector \(\hat{e}_{r}\):
%
\begin{align}
  \vec{\Omega}_{LT} = \frac{GJ}{c^2r^3} \qty(3 (\vec{J} \cdot e^{\hat{r}}) e^{\hat{r}} - \vec{J})
\,,
\end{align}
%
which notably has the same factor as the electric field of a dipole. It reduces to our formula if \(\vec{J}\) is parallel to \(e^{\hat{r}}\).

\subsection{The Kerr metric}

Now we treat Kerr geometry: from 1963, the vacuum solution of a rotating spherical mass. It is:
%

%
\boxalign{
\begin{subequations}
\begin{align}
\begin{split}
\dd{s^2} =
&- \qty(1 - \frac{2GMr}{\rho^2}) \dd{t^2} -
\frac{4GMar \sin^2\theta }{\rho^2} \dd{t} \dd{\varphi }
+ \frac{\rho^2}{\Delta } \dd{r^2} \\
&+ \rho^2 \dd{\theta^2} + \qty(r^2+a^2 + \frac{2GMr a^2 \sin^2\theta }{\rho^2}) \sin^2 \theta \dd{\varphi^2}
\end{split}
\,,
\end{align}
\end{subequations}}

%
where \(a = J/M\), \(\rho^2= r^2+a^2 \cos^2\theta  \) and \(\Delta = r^2 - 2GMr +a^2\).

Recall that in \(c=1\) velocities are dimensionless, so the units of angular momentum are mass times length, therefore the term \(a = J/M\) is a length.

At \(O(a^{0})\) we recover the Schwarzschild metric.

At \(O(a^{1})\) we have the slowly rotating geometry.

At \(r \gg GM\) we have
%
\begin{subequations}
\begin{align}
  \begin{split}
  \dd{s^2} = &- \qty(1 - \frac{2GM}{r}) \dd{t^2}
  - \frac{4GMa}{r} \sin^2 \theta \dd{t} \dd{\varphi } \\
  &+ \qty(1 + \frac{2GM}{r}) \dd{r^2}
  + r^2 \qty(\dd{\theta^2} + \sin^2 \theta \dd{\varphi })
  \end{split}
\,,
\end{align}
\end{subequations}
%
where we approximated \(r^2/(r^2-2GMr) \sim 1 + \frac{2GM}{r}\).

Then, we can orbit far away with a gyroscope and measure \(M\) and \(J\).

We still have the Killing vectors \(\xi^{\mu } = (1, \vec{0})\) and \(\xi^{\mu }= (0,0,0,1)\).

Notice that we have the symmetry \(\theta \rightarrow \pi - \theta \). We have a real singularity at \(\rho =0\): this means \(r =0 \) and \(\cos \theta =0\).

\(r=0\) is not a single point! This can be seen from the fact that to go from \((r, \theta ) = (0, \theta_0 )\) to \((0, \theta_1 )\) we have a positive distance since the \(\dd{\theta^2} \) metric element is nonzero if \(a^2 \cos^2\theta \neq 0\).

We have a coordinate singularity, an horizon, when \(\Delta = 0\): this means
%
\begin{align}
  r^2 - 2GMr + a^2 = 0 \implies r_{\pm} =GM \pm \sqrt{G^2M^2-a^2}
\,,
\end{align}
%
which means that we have an outer horizon and an inner horizon. As \(a=0\) we have \(r_+ = 2GM\), which is fine.
However if \(a> GM\) we have a \emph{naked singularity}: there is a postulate called the \emph{cosmic censorship} postulate which states that naked singularities do not exist.
This is not a theorem: it is just something which is suggested by black hole formation.

The case in which \(a = GM\) is called the extreme Kerr solution.

We are going to focus our attention on \(r_{+}\), the largest horizon.

First of all: \emph{$r_{+}$ is a null surface}.

It is a boundary which separates the region where light can go to \(r \rightarrow \infty \) from the one in which light eventually always goes to \(r \rightarrow 0\).

Then light is ``trapped inside this surface''.

The simplest case of a null surface  is a light cone in Minkowski: in polar coordinates, the set of vectors in the form \(x^{\mu } = (\alpha , \alpha , \beta , \gamma )\).

We can write this as
%
\begin{align}
  x^{\mu } = \alpha l^{\mu } + \beta m^{\mu } + \gamma n^{\mu }
\,,
\end{align}
%
and then we can see that \(l^{\mu } l_{\mu } = \alpha^2 - \alpha^2 = 0\). \(m^{\mu } \) and \(n^{\mu }\) are instead spacelike vectors.

The set \((l, m, n)\) is  a basis for the tangent space of the light cone.
So, not all the vectors tangent to the surface are null.

For the Schwarzschild horizon instead we can select
%
\begin{subequations}
\begin{align}
  l^{\mu } &= (1, 0, 0, 0)  \\
  m^{\mu } &= (0,0,1, 0)  \\
  n^{\mu } &= (0,0,0, 1)
\,,
\end{align}
\end{subequations}
%
and as before \(m\) and \(n\) are spacelike, while
%
\begin{align}
  l \cdot l = - \qty(1 - \frac{2GM}{r}) (l^{0})^2 = 0
\,,
\end{align}
%
since we are precisely at \(r = 2GM\).

At the Kerr horizon instead we have:
%
\begin{align}
  l^{\alpha } = (1, 0, 0, \Omega_{H})
\,,
\end{align}
%
where  \(\Omega_{H}= a / (2GMr_+)\): light will rotate along the Kerr black hole, while in the Schwarzschild case the light is stationary with respect to the spatial coordinates.

Now let us take a picture of the horizon at fixed \(t\):
it does not matter which time by stationarity. It will be homework to show that in that case on the horizon the metric will be
%
\begin{align}
\dd{s^2} = \rho_{+}^2 \dd{\theta^2} + \qty(\frac{2GMr_+}{\rho_{+}})^2 \sin^2 \theta \dd{\varphi^2}
\,.
\end{align}

This is not a spherical surface, since \(\rho_{+}\) depends on \(\theta \).

It can be shown that it looks like an ellipsoid. The equator is larger than the corresponding equator for a sphere. To show this we will compute the length of the equator: \(\theta = \pi /2\) and \(\varphi \in [0, 2 \pi ]\), and also we will compute the length of a north-south-north path: \(\varphi = \const\) and \(\theta \in [0, \pi ]\) and then \(\theta \in [\pi , 0]\).

For the equator we get
%
\begin{align}
  L _{\text{equator}} = \int_{0}^{2 \pi } \dd{\varphi } \sqrt{g_{33}}
\,,
\end{align}
%
calculated at \(\theta = \pi /2\). This becomes
%
\begin{align}
  L _{\text{equator}} = \int_{0}^{2 \pi } \dd{\varphi } 2GM = 4 \pi GM
\,,
\end{align}
%
since the \(\sin \theta = 1\) while \(\cos \theta = 0\). The harder one is the length of the circle: it comes out to be
%
\begin{align}
  L _{\text{NSN}} = 2 \int_{0}^{\pi }  \dd{\theta } \sqrt{g_{\theta \theta }}
  = 2 \int_{0}^{\pi } \dd{\theta } \sqrt{r^2 + a^2 \cos^2\theta }
\,,
\end{align}
%
which we will expand for small \(a\) since we do not like elliptic integrals. This becomes
%
\begin{align}
  L _{\text{NSN}} \approx 2 \int_{0}^{\pi } \dd{\theta } \qty(2GM + \frac{a^2}{4GM} \qty(-2 + \cos^2 \theta )  + O(a^2))
\,,
\end{align}
%
and since we are integrating over a period of \(\cos^2\theta\) we can substitute its average of \(1/2\): then the result is
%
\begin{align}
L _{\text{NSN}} \approx 4\pi GM - \frac{3a^2 \pi }{4GM}
\,,
\end{align}
%
so the length is less than the equatorial one.

\begin{bluebox}
Complement: what does the region \(r=0\) look like?

Inserting \(r=0\) into the Kerr metric we find
%
\begin{align}
  \dd{s^2} = - \dd{t^2} + \cos^2\theta \dd{r^2}
  + a^2 \qty(\cos^2 \theta \dd{\theta^2} + \sin^2 \theta \dd{\varphi^2})
\,,
\end{align}
%
and we can do the change of variable \(\psi = \sin \theta \): then the line element becomes
%
\begin{align}
  \dd{s^2} = - \dd{t^2} + \qty(1 - \psi^2) \dd{r^2} +a^2 \qty(\dd{\psi^2} + \psi^2 \dd{\varphi^2})
\,.
\end{align}

This is not a point anymore: it actually consists of two disjoint ``emispheres'', and to see what the geometry of these looks like we can notice that at constant \(\theta \) we have a circle of radius \(\psi =\sin \theta \).

How does this surface actually look like? We can notice that the metric \(\dd{s^2}_{(2)} = a^2 \qty(\dd{\psi^2}  + \psi^2 \dd{\varphi^2})\) is, in polar coordinates, the metric for the plane, which is intrinsically flat: we do not need to actually compute the Riemann tensor to already know that all of its components are zero.

Then, we have a bounded surface, made up of circles, which is flat so it can be made by folding a piece of paper; also it is axisymmetric and symmetric with respect to \(\theta  \rightarrow - \theta \): the surface is made up of two cones joined at the equator.

In a paper\footnote{\url{https://arxiv.org/abs/1205.5848v6}} I found the result that the aperture angle of the cone is
%
\begin{align}
  2 \arctan \frac{\abs{a}}{\abs{GM}}
\,,
\end{align}
%
but I have not figured out how to prove it myself yet.

\end{bluebox}

\end{document}


\section{Verification of stack axioms}
\Label{sec:stack-verification}

In this section we show that the stack functions defined in
\S\ref{sec:stack-functions}
satisfy the stack Axioms of \S\ref{sec:stack-axioms}.

The annotated code has been obtained from the axioms in a fully systematical way.
In order to transform a condition equation $p \rightarrow s = t$:
\begin{itemize}
\item Generate a clause \inl{requires p}.
\item Generate a clause \inl{requires x1 == ... == xn}
for each variable \inl{x} with $n$ occurrences
in $s$ and $t$.
\item Change the $i$-th occurrence of \inl{x} to \inl{xi}
in $s$ and $t$.
\item Translate both terms $s$ and $t$ to reversed polish notation.
\item Generate a clause \inl{ensures y1 == y2}, where \inl{y1} and
\inl{y2} denote the value corresponding to
the translated $s$ and $t$, respectively.
\end{itemize}

This makes it easy to implement a tool that does the translation automatically,
but yields a slightly longer contract in our example.

\subsection{Resetting a stack}
\Label{sec:axiomsizeofinit}

Our formulation in \acsl\slash\isoc  of the axiom in
Equation~\eqref{eq:stack-init-size} is shown in  the following
listing.

\input{Listings/axiom_size_of_init.c.tex}

\clearpage

\subsection{Adding an element to a stack}
\Label{sec:axiomsizeofpush}
\Label{sec:axiomtopofpush}

Axioms~\eqref{eq:stack-size-push} and~\eqref{eq:stack-top-push}
describe the behavior of a stack when an element is added.

\input{Listings/axiom_size_of_push.c.tex}

Except for the \inl{assigns} clauses, the \acsl specification refers
only to encapsulating logic functions and predicates defined in
\S\ref{sec:stack-invariants}.
If \acsl would provide a means to define encapsulating logic
functions returning also sets of memory locations,
the expressions in \inl{assigns} clauses would not need to refer to
the details of our
\stacktype implementation.\footnote{
In \cite[\S 2.3.4]{ACSLSpec}, a powerful sublanguage to
build memory location set expressions is defined.
We will explore its capabilities in a later version.
}
As an alternative, \inl{assigns} clauses could be omitted, as long as
the proofs are only used to convince a human reader.

\input{Listings/axiom_top_of_push.c.tex}

\clearpage

\subsection{Removing an element from a stack}
\Label{sec:axiompopofpush}
\Label{sec:axiomsizeofpop}
\Label{sec:axiompushofpoptop}

This section shows the Listings for Axioms~\ref{eq:stack-size-pop}, \ref{eq:stack-pop-push}
and~\ref{eq:stack-push-pop-top} which
describe the behavior of a stack when an element is removed.

\input{Listings/axiom_size_of_pop.c.tex}
\input{Listings/axiom_pop_of_push.c.tex}
\input{Listings/axiom_push_of_pop_top.c.tex}


\subsection{$||\delta_i||_1$ minimizer} \label{d1minimizer}
  We will now concentrate on finding a capacity configuration $C'$ such that $F' = F - V$ and that minimizes
  $\sum\limits_{i=1}^{n}\left(c_i-c'_i\right) = ||\delta_i||_1$ as well. We treat the flow problem as a linear programming
  problem. Next we see the formulation of the problem in this form, along with a breakdown of each relevant matrix and
  vector.

  \subimport{common/lp/}{lpprimal.tex}

  We would like to find a solution that, except for maximizing the flow, also minimizes $||\delta_i||_1$ at the same time.
  More precisely, we would like to optimize
  \begin{equation*}
    \min{\sum\limits_{v \in \mathcal{V}}\left(c_{Av} - c'_{Av}\right)}
  \end{equation*}
  as well. Since we wish to optimize with regards to two objective functions, we approach the problem as follows: Initially,
  we ignore the minimization and derive the dual of the previous problem with respect to the maximisation. We then substitute
  the two problems' optimisations with an additional constraint that equates the two objective functions. Due to the Strong
  Duality theorem of linear programming \cite{amo}, this equality can be achieved only by the common optimal solution of the
  two problems. Next we treat the combination of constraints and variables of the primal and the dual problem, along with the
  newly introduced constraint and the previously ignored $||\delta_i||_1$ minimisation as a new linear problem. For every $j
  \in \left[n\right]$, the solution to this problem contains a $c'_{1j}$. These capacities will comprise the new configuration
  that player $A$ requires.

  We will now describe the dual problem in detail.
  \subimport{common/lp/}{lpdual.tex}
  Everything is now in place to define the linear problem whose solution $C'$ yields $maxFlow\left(C'\right) = F - V$ and
  minimizes $||\delta_i||_1$. The constraints are (\ref{lp:primal:constraints}) and (\ref{lp:dual:constraints}) supplemented
  with a constraint that equates the two problems' optimality functions:
  \begin{equation*}
    \sum\limits_{j \in \left[n\right]}f'_{1j} = \sum\limits_{j \in \left[n\right]}c_{1j}y_{c1j} +
    \sum\limits_{i \in \left[n\right]}\sum\limits_{j \in \left[n\right]}c_{ij}y_{fcij} + \left(F - V\right)y_F \enspace.
  \end{equation*}
  The desired optimisation is
  \begin{equation*}
    \min{\sum\limits_{j \in \left[n\right]}\left(c_{1j} - c'_{1j}\right)} \enspace.
  \end{equation*}
  The final linear program consists of $2n^2 + 2n - 1$ variables and $2n^2 + 2n$ constraints. There exist a variety of
  algorithms that solve linear programs, such as simplex and ellipsoid algorithm. The ellipsoid algorithm requires polynomial
  time in the worst-case scenario, but for practical purposes algorithms of the simplex category seem to exhibit better
  behavior \cite{it}.


    \filetitle{trec}{Create new recursive time subscript object}{trec/trec}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
T = trec(Dates)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{Dates} {[} numeric {]} - Vector of dates or date range on
  which the final time-recursive expression will be evaluated.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{T} {[} trec {]} - New time-recursive subscript object.
\end{itemize}

\paragraph{Description}\label{description}

Time-recursive subscript objects are used to reference tseries objects
on both the left-hand side and the right-hand side of a time-recursive
assignment. The assignment is then evaluated for each date in
\texttt{Dates}, from the first to the last.

See more on time-recursive expressions in
\href{trec/Contents}{Contents}, including the description of instances
in which IRIS fails to evaluate the time-recursive expressions
correctly.

\paragraph{Example}\label{example}

Construct a first-order autoregressive process with normally distributed
residuals:

\begin{verbatim}
T = trec(qq(2010,1):qq(2020,4));
x = tseries(qq(2009,4),10);
e = tseries(qq(2010,1):qq(2020,4),@randn);
x(T) = 10 + 0.8*x(T-1) + e(T);
\end{verbatim}


\chapter{Repair}
Some heuristic could generate an invalid solution, like the merge or mutation phase in Genetic Algorithm. Here a repair heuristic is intended as a method that can return a valid solution from an invalid one. The input of the repair heuristic has peculiar property derived from the method that produce it.
\section{Patching} \label{section:patching}
The Patching heuristic is a way to merge together multiple tours and obtain a single one.
The algorithm proposed is an iteration of a \texttt{single\_patch} method which merge only one pair of tours. Note that an isolated node is the tour with minimum length therefore \texttt{patching} can also resolve a TSP problem. \\
The proposed algorithm work through the 6 steps represented in fig \ref{fig:single_patch}.
\begin{figure}[!h]
	\begin{subfigure}{.26\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching1.png}
		\caption{The \texttt{single\_patch} input instance.}
		\label{fig:patching1}
	\end{subfigure}
\hfill%
	\begin{subfigure}{.26\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching2.png}
		\caption{Node $ 3 $ is chosen randomly from a set of subtours.}
		\label{fig:patching2}
	\end{subfigure}
\hfill%
	\begin{subfigure}{.26\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching3.png}
		\caption{It is selected the closer node to $ 3 $ which is in a different tour (node $ 4 $).}
		\label{fig:patching3}
	\end{subfigure}
	\begin{subfigure}{.26\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching4.png}
		\caption{Iterating throw the first tour nodes, it is selected the node which minimize the next merge cost.}
		\label{fig:patching4}
	\end{subfigure}
\hfill%
	\begin{subfigure}{.26\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching5.png}
		\caption{Merge operation.}
		\label{fig:patching5}
	\end{subfigure}
\hfill%
	\begin{subfigure}{.26\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching6.png}
		\caption{The final result.}
		\label{fig:patching6}
	\end{subfigure}
\caption{Single patch move algorithm}
\label{fig:single_patch}
\end{figure}\\
A \texttt{succ} structure is use to represent the input tours, which for each  $ i = 1, .., n = |V|$ is defined as:
\begin{equation}
\texttt{succ[i] = } \begin{cases}
 \text{"\texttt{i} successor"} & \text{if \texttt{i} is in a tour,}\\
 \text{\texttt{-1}} & \text{otherwise.}
\end{cases}
\end{equation} \\
In the \texttt{succ} structure, each edge could be considered as oriented: if \texttt{succ[i] = j} than \texttt{i} $\rightarrow$ \texttt{j} is the considered edge orientation. \\
The merge phase is the core of the method. All the tour are initialize in clockwise orientation and the merge phase must keep the property in the output tour. When the solution is created from scratch by \texttt{patching}, it avoids a large number of intersection between edges, which is a sign of non optimal solution. \\
An example of merge is done in fig \ref{fig:patching_merge}. Note that once the two nodes are selected there are only two possible merge: one create the intersection and the other one does not. It is easy to prove that without intersection the tour is shorter. Moreover if each tour is clockwise, the property is naturally maintained
\begin{figure}[!h]
	\begin{subfigure}{.26\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching_merge1.png}
		\caption{}
		\label{fig:patching_merge1}
	\end{subfigure}
	\hfill%
	\begin{subfigure}{.26\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching_merge2.png}
		\caption{}
		\label{fig:patching_merge2}
	\end{subfigure}
	\hfill%
	\begin{subfigure}{.26\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching_merge3.png}
		\caption{}
		\label{fig:patching_merge3}
	\end{subfigure}

	\caption{Merge phase in detail. The orientation of the arcs is defined by the associated \texttt{succ} structure but, being an aTSP, the edges are undirected.}
	\label{fig:patching_merge}
\end{figure}\\

\begin{table}[h]
	\centering
	\caption{The \texttt{succ} structure for the tour in figure \ref{fig:patching_merge}}
	\begin{tabular}{clcccccc}
		\multirow{2}{*}{\ref{fig:patching_merge1})} 	& \texttt{i:}		& 1 & 2 & 3 & 4 & 5 & 6 \\
														& \texttt{succ[i]:}	& 2 & 3 & 1 & 5 & 6 & 4 \\
														&		   			&   &   &   &   &   &   \\
		\multirow{2}{*}{\ref{fig:patching_merge3})} 	& \texttt{i:}		& 1 & 2 & 3 & 4 & 5 & 6 \\
														& \texttt{succ[i]:}	& 5 & 3 & 1 & 2 & 6 & 4 \\
	\end{tabular}
\end{table}
To keep the clockwise orientation of the tour, there is a special consideration to be done to merge an isolated node with a tour of two nodes. In the final tour the clockwise property must is checked if the condition is satisfied:
$ (x_3 - x_1)(y_2 - y_1) < (y_3 - y_1)(x_2 - x_1) $,
otherwise the tour is reversed.
\begin{figure}[!h]
	\centering
	\begin{subfigure}{.2\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching_merge_clockwise1.png}
		\caption{Tours to be merged}
		\label{fig:patching_merge_clockwise1}
	\end{subfigure}
\hfill
	\begin{subfigure}{.46\columnwidth}
		\includegraphics[width=\columnwidth]{img/patching_merge_clockwise2.png}
		\caption{Two different orientation can result with the same cost. The clockwise orientation is accepted, otherwise the tour is reversed.}
		\label{fig:patching_merge_clockwise2}
	\end{subfigure}
	\caption{Special case of merge phase that create a tour with 3 edges. The orientation of the tour must be reverse if necessary.}
	\label{fig:patching_merge_clockwise}
\end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2003-2018 by The University of Queensland
% http://www.uq.edu.au
%
% Primary Business: Queensland, Australia
% Licensed under the Apache License, version 2.0
% http://www.apache.org/licenses/LICENSE-2.0
%
% Development until 2012 by Earth Systems Science Computational Center (ESSCC)
% Development 2012-2013 by School of Earth Sciences
% Development from 2014 by Centre for Geoscience Computing (GeoComp)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Darcy Flux}
\label{DARCY FLUX}
We want to calculate the flux $u$ and pressure $p$ on a domain $\Omega$
solving the Darcy flux problem\index{Darcy flux}\index{Darcy flow}
\begin{equation}\label{DARCY PROBLEM}
\begin{array}{rcl}
u_{i} + \kappa_{ij} p_{,j} & = & g_{i} \\
u_{k,k} & = & f
\end{array}
\end{equation}
with the boundary conditions
\begin{equation}\label{DARCY BOUNDARY}
\begin{array}{rcl}
u_{i} \; n_{i}  = u^{N}_{i}  \; n_{i} & \mbox{ on } & \Gamma_{N} \\
p = p^{D} &  \mbox{ on } & \Gamma_{D} \\
\end{array}
\end{equation}
where $\Gamma_{N}$ and $\Gamma_{D}$ are a partition of the boundary of
$\Omega$ with $\Gamma_{D}$ non-empty, $n_{i}$ is the outer normal field of the
boundary of $\Omega$, $u^{N}_{i}$ and $p^{D}$ are given functions on $\Omega$,
$g_{i}$ and $f$ are given source terms and $\kappa_{ij}$ is the given
permeability.
We assume that $\kappa_{ij}$ is symmetric (which is not really required) and
positive definite, i.e. there are positive constants $\alpha_{0}$ and
$\alpha_{1}$ which are independent from the location in $\Omega$ such that
\begin{equation}
\alpha_{0} \; x_{i} x_{i} \le \kappa_{ij} x_{i} x_{j} \le \alpha_{1} \; x_{i} x_{i}
\end{equation}
for all $x_{i}$.

\subsection{Solution Method \label{DARCY SOLVE}}
Unfortunate equation~\ref{DARCY PROBLEM} can not solved directly in an easy way and requires mixed FEM.
We consider a few options to solve equation~\ref{DARCY PROBLEM}
\subsubsection{Evaluation}\label{SEC DARCY SIMPLE}
The first equation of equation~\ref{DARCY PROBLEM} is inserted into the second one:
\begin{equation}\label{DARCY PROBLEM SIMPLE}
- (\kappa_{ij} p_{,j})_{,i}  =  f  - (g_{i})_{,i}
\end{equation}

with boundary conditions
\begin{equation}\label{DARCY BOUNDARY SIMPLE}
\begin{array}{rcl}
\kappa_{ij} p_{,j} \; n_{i}  = ( g_{i} - u^{N}_{i} )  \; n_{i} & \mbox{ on } & \Gamma_{N} \\
p = p^{D} &  \mbox{ on } & \Gamma_{D} \\
\end{array}
\end{equation}
Then the flux field is recovered by directly setting
\begin{equation}\label{DARCY PROBLEM SIMPLE FLUX}
 u_{j} = g_j -  \kappa_{ij} p_{,j}
\end{equation}
This simple recovery process will not ensure that the (numerically) calculated flux
meets the boundary conditions for flux or the incompressibility condition.
However this is a very fast way of calculating the flux.

\subsubsection{Global Postprocessing \label{SEC DARCY POST}}
An improved flux recovery can be achieved by solving a modified version of equation~\ref{DARCY PROBLEM SIMPLE FLUX}
adding the gradient of the divergence of the flux:
\begin{equation}\label{DARCY PROBLEM POST FLUX}
\kappa^{-1}_{ij} u_{j} -
(\lambda \cdot u_{k,k} )_{,i}=
\kappa^{-1}_{ij} g_j- p_{,i}
- (\lambda \cdot f )_{,i}
\end{equation}
where
\begin{equation}\label{DARCY PROBLEM POST FLUX A}
\lambda = \omega \cdot |\kappa^{-1}| \cdot vol(\Omega)^{1/d} \cdot h
\end{equation}
with a non-negative factor $\omega$, $d$ is the spatial dimension and $h$ is the local element size.
\begin{equation}\label{DARCY PROBLEM POST FLUX BOUNDARY}
\begin{array}{rcl}
u_{i} \; n_{i}  = u^{N}_{i}  \; n_{i} & \mbox{ on } & \Gamma_{N} \\
u_{k,k} = f & \mbox{ on } & \Gamma_{D} \\
\end{array}
\end{equation}
Notice that the second condition is a natural boundary condition.
Global post-processing is more expense than direct pressure evaluation
however the flux is more accurate and asymptotic incompressibility
for mesh size towards zero can be shown, if $\omega>0$.

\subsection{Functions}
\begin{classdesc}{DarcyFlow}{domain, \optional{w=1., \optional{solver=\member{DarcyFlow.POST},  \optional{
useReduced=\True, \optional{ verbose=\True} } }}}
opens the Darcy flux problem\index{Darcy flux} on the \Domain domain.
Reduced approximations for pressure and flux are used if \var{useReduced} is set.
Argument \var{solver} defines the solver method.
If \var{verbose} is set some information are printed.
\var{w} defines the weighting factor $\omega$ for global post-processing of the flux (see equation~\ref{DARCY PROBLEM POST FLUX A}.)
\end{classdesc}

\begin{memberdesc}[DarcyFlow]{EVAL}
flux is calculated directly from pressure evaluation, see section~\ref{SEC DARCY SIMPLE}.
\end{memberdesc}

\begin{memberdesc}[DarcyFlow]{SMOOTH}
solver using global post-processing of flux with weighting factor $\omega=0$, see section~\ref{SEC DARCY POST}.
\end{memberdesc}

\begin{memberdesc}[DarcyFlow]{POST}
solver using global post-processing of flux, see section~\ref{SEC DARCY POST}.
\end{memberdesc}

\begin{methoddesc}[DarcyFlow]{setValue}{\optional{f=None, \optional{g=None, \optional{location_of_fixed_pressure=None, \optional{location_of_fixed_flux=None,
\\\optional{permeability=None}}}}}}
assigns values to the model parameters. Values can be assigned using various
calls -- in particular in a time dependent problem only values that change
over time need to be reset. The permeability can be defined as a scalar
(isotropic), or a symmetric matrix (anisotropic).
\var{f} and \var{g} are the corresponding parameters in~\ref{DARCY PROBLEM}.
The locations and components where the flux is prescribed are set by positive
values in \var{location_of_fixed_flux}.
The locations where the pressure is prescribed are set by by positive values
of \var{location_of_fixed_pressure}.
The values of the pressure and flux are defined by the initial guess.
Notice that at any point on the boundary of the domain the pressure or the
normal component of the flux must be defined. There must be at least one point
where the pressure is prescribed.
The method will try to cast the given values to appropriate \Data class objects.
\end{methoddesc}

\begin{methoddesc}[DarcyFlow]{getSolverOptionsFlux}{}
returns the solver options used to solve the flux problems.
Use this \SolverOptions object to control the solution algorithms.
This option is only relevant if global postprocesing is used.
\end{methoddesc}

\begin{methoddesc}[DarcyFlow]{getSolverOptionsPressure}{}
returns a \SolverOptions object with the options used to solve the pressure
problems.
Use this object to control the solution algorithms.
\end{methoddesc}

\begin{methoddesc}[DarcyFlow]{solve}{u0,p0}
solves the problem and returns approximations for the flux $v$ and the pressure $p$.
\var{u0} and \var{p0} define initial guesses for flux and pressure.
Values marked by positive values \var{location_of_fixed_flux} and
\var{location_of_fixed_pressure}, respectively, are kept unchanged.
\end{methoddesc}

\begin{methoddesc}[DarcyFlow]{getFlux}{p, \optional{ u0 = None}}
returns the flux for a given pressure \var{p} where the flux is equal to \var{u0}
on locations where \var{location_of_fixed_flux} is positive,  see \member{setValue}.
Notice that \var{g} and \var{f} are used.
\end{methoddesc}

\begin{figure}
\centerline{\includegraphics[width=\figwidth]{darcy_result}}
\caption{Flux and pressure field of the Dary flow example.}
\label{DARCY FIG 1}
\end{figure}

\subsection{Example: Gravity Flow}
The following script \file{darcy.py}\index{scripts!\file{darcy.py.py}}\index{Darcy flow}
which is available in the \ExampleDirectory illustrates the usage of the\\
\class{DarcyFlow} class:
\begin{python}
   from esys.escript import *
   from esys.escript.models import DarcyFlow
   from esys.finley import Rectangle
   from esys.weipa import saveVTK
   mydomain = Rectangle(l0=2.,l1=1.,n0=40, n1=20)
   x = mydomain.getX()
   p_BC=whereZero(x[1]-1.)*wherePositive(x[0]-1.)
   u_BC=(whereZero(x[0])+whereZero(x[0]-2.)) * [1.,0.] + \
     (whereZero(x[1]) + whereZero(x[1]-1.)*whereNonPositive(x[0]-1.0)) * [0., 1.]
   mypde = DarcyFlow(domain=mydomain)
   mypde.setValue(g=[0., 2],
                  location_of_fixed_pressure=p_BC,
                  location_of_fixed_flux=u_BC,
                  permeability=100.)

   u,p=mypde.solve(u0=x[1]*[0., -1.], p0=0)
   saveVTK("u.vtu",flux=u, pressure=p)
\end{python}
In the example the pressure is fixed to the initial pressure \var{p0} on the right half of the top face.
The normal flux is set on all other faces. The corresponding values for the flux are set by the initial value
\var{u0}.