\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\DeclareMathOperator{\sgn}{sgn}
\usepackage{hyperref}
\usepackage{enumitem}
\usepackage{commath}
\usepackage[margin=0.5in]{geometry}
\title{Spline Math}
\author{James Kuszmaul}
\begin{document}

\maketitle

\section{Introductions}

This document describes some of the math going on in computing constraints on
the maximum velocities in \texttt{trajectory.*}. Some of the thoughts are
somewhat disorganized and represent a stream-of-consciousness more than actually
useful documentation.

For a given path, we will have:

\begin{tabular}{r|l}
Name & Description \\ \hline
$v_l$, $v_r$ & Left and right wheel velocities \\
$\theta$ & Robot heading/yaw at a position along the path \\
$r$ & Robot radius (assuming center of mass in center of robot) \\
$u_l$, $u_r$ & Left and right wheel voltages \\
$V_{max}$ & Maximum allowable voltage (typically 12V) \\
$v_{max}$ & User-specified maximum velocity \\
$u$ & $\begin{bmatrix} u_l \\ u_r \end{bmatrix}$ \\
$A$, $B$ & LTI, matrices such that $\begin{bmatrix} \dot{v}_l \\ \dot{v}_r
\end{bmatrix} = A \begin{bmatrix} v_l \\ v_r \end{bmatrix} + B u$ \\
$s$ & Distance along the path \\
\end{tabular}

We are interested in discovering the fastest way to traverse the path in
question, given constraints on:
\begin{itemize}
\item Friction limits of the wheels (assumed to static and time/velocity
      invariant).
\item Voltage limits on the motors.
\item Maximum velocity.
\end{itemize}

The voltage constraints are essentially provided by the linear dynamics shown
above:

\begin{align}
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
   A \begin{bmatrix} v_l \\ v_r \end{bmatrix} + B u
\end{align}

Also, at any given point we must maintain velocities/accelerations that actually
allow us to follow the path (as opposed to optimizing $v_l$/$v_r$ separately...).

Using the velocities of the individual wheels as a starting point, and
introducing dummy variables for the velocity/acceleration of the center of the
robot (corresponding to time derivatives of $s$):

\begin{align*}
v_c &= \frac{v_l  r_r + v_r  r_l}{r_r + r_l} \\
\begin{bmatrix} v_l \\ v_r \end{bmatrix} &=
    \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
\end{bmatrix} v_c \\
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
  \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} - r_l \frac{d^2\theta}{ds^2} v_c
  \\  r_r \frac{d^2\theta}{ds^2} v_c \end{bmatrix} v_c \\
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
  \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2 \\
\end{align*}

And finally, we need to know the lateral accelerations of the two wheels as
part of determining whether we would be breaking friction. Note that, because of
the general assumptions we are making about the setup of the robot, the lateral
acceleration on the two wheels will be identical (if they were not identical,
that would imply the robot was tearing itself apart side-to-side).

\begin{align*}
a_{lat} &= \frac{d\theta}{ds} v_c^2 \\
\end{align*}

Finally, we need to be combine the lateral/longitudinal accelerations of each
wheel to determine whether the wheel would break friction. While technically
there is only one expression for this, as a practical matter we have a bit of
leeway in defining exactly what the relationship should be (e.g., if we want
separate max longitudinal/lateral accelerations, we can create an ellipse; we
could also just create a raw rectangle and decouple the two, although that would
probably be a particularly poor approximation of the actual dynamics). For now
we will say that our friction is limited by some nonnegative convex function
$g(a_{lat}, a_{lng}) \le 1$ for any given wheel.

Summarizing our equations and constraints, we have:

\begin{align*}
v_c &= \frac{v_l r_r + v_r r_l}{r_r + r_l} \\
a_{lat} &= \frac{d\theta}{ds} v_c^2 \\
g(a_{lat}, \dot{v}_l) &\le 1 \\
g(a_{lat}, \dot{v}_r) &\le 1 \\
\abs{u_l} &\le V_{max} \\
\abs{u_r} &\le V_{max} \\
\begin{bmatrix} v_l \\ v_r \end{bmatrix} &= \begin{bmatrix} 1 - r_l
    \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
\end{bmatrix} v_c \\
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
   A \begin{bmatrix} v_l \\ v_r \end{bmatrix} + B \begin{bmatrix} u_l \\ u_r
   \end{bmatrix} \\
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
  \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2 \\
\end{align*}

With $v_l$, $v_r$, $u_l$, and $u_r$ free at all $s$ except the start and end, at
which the velocities must
be zero (technically, we could make do with allowing non-zero velocities, but
then we could end up in situations that would be impossible to solve--because
the splines have differentiable curvature, any of the splines we use will always
have some valid velocity profile, but depending on the curvature, that velocity
may be arbitrarily small).

We can provide an upper limit on the goal velocity at any given point with the
min of the specified max velocity and the lateral-acceleration limited velocity.
This also gives an initial pass of the longitudinal accelerations that would be
required to follow this profile.

\begin{align*}
v_{lat} &= \min (v_{max}, \sqrt{\frac{a_{lat,max}}{\frac{d\theta}{ds}}}) \\
\dot{v}_{lat} &= \begin{cases} 0 & v_{lat} = v_{max} \\
-\frac12 \sqrt{\frac{a_{lat,max}}{\frac{d\theta}{ds}^3}} \frac{d^2\theta}{ds^2}
v_{lat} & v_{lat} \ne v_{max} \\
\end{cases} \\
&= \begin{cases} 0 & v_{lat} = v_{max} \\
-\frac12 \frac{a_{lat,max}}{\frac{d\theta}{ds}^2} \frac{d^2\theta}{ds^2}
 & v_{lat} \ne v_{max} \\
\end{cases} \\
\end{align*}

If we start to consider accounting also for longitudinal accelerations, we can
start with:

\begin{align}
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
  \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2 \\
(r_r + r_l)
\left[ \begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
  \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \frac{\dot{v}_l r_r + \dot{v}_r r_l}{r_r + r_l} +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
  \right] \\
\begin{bmatrix}
r_l + r_l r_r \frac{d\theta}{ds} & -r_l + r_lr_l \frac{d\theta}{ds} \\
-r_r - r_r r_r \frac{d\theta}{ds} & r_r - r_r r_l \frac{d\theta}{ds} \\
\end{bmatrix}
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
  \begin{bmatrix} -r_l \\ r_r \end{bmatrix} (r_l + r_r) \frac{d^2\theta}{ds^2} v_c^2 \\
\begin{bmatrix}
1 + r_r \frac{d\theta}{ds} & -1 + r_l \frac{d\theta}{ds} \\
-1 - r_r \frac{d\theta}{ds} & 1 - r_l \frac{d\theta}{ds} \\
\end{bmatrix}
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
  \begin{bmatrix} -1 \\ 1 \end{bmatrix} (r_l + r_r) \frac{d^2\theta}{ds^2} v_c^2 \\
\begin{bmatrix}
1 + r_r \frac{d\theta}{ds} & -1 + r_l \frac{d\theta}{ds} \\
1 + r_r \frac{d\theta}{ds} & -1 + r_l \frac{d\theta}{ds} \\
\end{bmatrix}
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
   -(r_l + r_r) \frac{d^2\theta}{ds^2} v_c^2 \\
(1 + r_r \frac{d\theta}{ds}) \dot{v}_l + (-1 + r_l \frac{d\theta}{ds}) \dot{v}_r &=
   -(r_l + r_r) \frac{d^2\theta}{ds^2} v_c^2 \\
%(1 + r \frac{d\theta}{ds}) \dot{v}_l + (-1 + r \frac{d\theta}{ds}) (2\dot{v}_c -
%\dot{v}_l) &=
%   -2 r \frac{d^2\theta}{ds^2} v_c^2 \\
%2 \dot{v}_l + 2(-1 + r \frac{d\theta}{ds}) \dot{v}_c
% &= -2 r \frac{d^2\theta}{ds^2} v_c^2 \\
%\dot{v}_l &= (1 - r \frac{d\theta}{ds}) \dot{v}_c - 2 r \frac{d^2\theta}{ds^2} v_c^2 \\
%\dot{v}_r &= (1 + r \frac{d\theta}{ds}) \dot{v}_c + 2 r \frac{d^2\theta}{ds^2} v_c^2 \\
\begin{bmatrix}
1 & 0 \\
0 & 1 \\
-1 & 0 \\
0 & -1 \\
\end{bmatrix}
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &\le g'(v_c, s) \\
\begin{bmatrix}
1 & 0 \\
0 & 1 \\
-1 & 0 \\
0 & -1 \\
\end{bmatrix} \left(
\begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
\end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
\right) &\le g'(v_c, s)
\end{align}

Note how we have a linear relationship between $\dot{v}_l$ and $\dot{v}_r$,
where the slope of the line between the two is constant at any given point on
the path, but the y-intercept and the max/min bounds get tighter with speed.
Furthermore, the velocity along the path ($v_c$) is a linear function of the
wheel velocities $v_l$ and $v_r$. Thus, we can state, at any given point on the
path, a lower velocity along the path will not decrease our acceleration
bounds (if we are travelling in a straight line, then our speed does not affect
the straight-line acceleration limits).

We also can introduce the voltage constraints:

\begin{align}
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
   A \begin{bmatrix} v_l \\ v_r \end{bmatrix} + B \begin{bmatrix} u_l \\ u_r
   \end{bmatrix} \\
\begin{bmatrix} \dot{v}_l \\ \dot{v}_r \end{bmatrix} &=
   A \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
     \end{bmatrix} v_c + B \begin{bmatrix} u_l \\ u_r \end{bmatrix} \\
\begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
  &= A \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
     \end{bmatrix} v_c + B \begin{bmatrix} u_l \\ u_r \end{bmatrix} \\
\end{align}

With these constraints, we can no longer guarantee that the set of feasible
velocities at any given point along the path is contiguous. However, we can
guarantee that at any given point there will be at most 2 contiguous sets of
valid velocities---in the case where there are two separate sets of allowable
velocities, then one corresponds to the lower velocities which start at zero and
go up until we can no longer use our finite available voltage to track the path.
The second zone can occur at higher velocities where we are going so fast that
the deceleration from the Back-EMF of the motor is enough to slow us down, but
not so fast that we would lose friction.

This divergence does technically create a bit of a challenge--if it were to,
e.g., split into arbitrarily many possible paths than that would complicate the
optimization problem substantially. However, because:
\begin{enumerate}
\item We only split into two branches at once; and
\item we should only have a limited number of branchings for a given N-spline
\footnote{I'll leave this as an exercise to the reader/my future self. However,
logically, a spline with a given number of points can only have so many features
and so should be able to have a very finite number of branchings.}.
\end{enumerate}

It should not actually be an issue to exhaustively attempt to plan for all
possible branchings, knowing that the path of ``always follow the slower
branch'' will always provide a feasible path, and choose whatever feasible plan
is fastest.

Once we choose which branch to take, it is a relatively simple matter of
exercising the existing method. Essentially, taking the set of feasible
velocities/accelerations as a starting point we do a forwards and backwards pass
(order is unimportant)) where we start at our initial/terminal velocity, and
accelerate forwards/decelerate backwards as hard as possible for each pass. If
attempting to accelerate as hard as possible on the forwards pass (or,
conversely, decelerate on the backwards) results in us violating the feasible
velocity bounds, we can simply set the velocity at that point for that pass to
the highest feasible velocity. We can guarantee that the pass in the opposite
direction will \emph{not} get stuck at that same point because if that would
suggest we had different physics going backwards than forwards, which is not the
case\footnote{I'm too lazy to spell this logic out more carefully, and I don't
think it is wrong. That is not guaranteed, however.}.

\section{Specialization for Friction Circle}

As our particular friction circle, we will attempt to use a straight
ellipse\footnote{Note that this has numerical stability issues due to the
infinite slope near zero}.
With a $a_{lat,max}$ and $a_{lng,max}$ for the maximum lateral and longitudinal
accelerations, i.e.:

\begin{align*}
g(a_{lat}, a_{lng}) &= \left(\frac{a_{lat}}{a_{lat,max}}\right)^2
                       + \left(\frac{a_{lng}}{a_{lng,max}}\right)^2 \\
\end{align*}

We care about turning this into a function of the path geometry and the current
velocity that returns the max longitudinal acceleration and lets us calculate
the friction-limited maximum acceleration, $v_{c,fmax}$. In order to do so, we
note that, in order to be at the edge of a feasible velocity, we observe that,
at any given $v_c$, we will have two pairs of constraints on the center-of-mass
longitudinal acceleration that both must be met,
corresponding to the fact that the each wheel creates an upper/lower bound
on the valid accelerations of the center-of-mass of the robot, but at high
$\frac{d^2\theta}{ds^2}v_c^2$, the constraints from each wheel may not overlap.
Thus, we care about the $v_c$ such that the pairs of constraints \emph{just
barely} overlap. This will happen when one constraint from the right wheel and
one from the left are just on the edge of being violated; furthermore, of the
four possible left/right pairings (+/- of left/right), only two pairs correspond
to being on the edge of overlapping (the other two pairs correspond to, e.g.,
having the upper acceleration limit in the same spot, as occurs when driving
straight; in such a situation, there are plenty of valid accelerations, we just
happen to start slipping both wheels at the same time). Which pair to look at is
determined by the signs on $\dot{v}_c$--if they are the same (as would occur
when going nearly straight), then we care about when the opposite limits
overlap (e.g., upper limit on the left, lower on the right).

\begin{align*}
\dot{v}_l, \dot{v}_r &\le a_{lng, max} \sqrt{1 - \left(\frac{a_{lat}}{a_{lat,max}}\right)^2} \\
\dot{v}_l, \dot{v}_r &\le a_{lng, max} \sqrt{1 - \left(\frac{\frac{d\theta}{ds}v_c^2}{a_{lat,max}}\right)^2} \\
  \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds} \\
 -1 + r_l \frac{d\theta}{ds} \\ -1 - r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c  &\le  \begin{bmatrix} r_l \\ -r_r \\ -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
 + a_{lng, max} \sqrt{1 - \left(\frac{\frac{d\theta}{ds}v_c^2}{a_{lat,max}}\right)^2} \\
0 &= \pm \{r_l,r_r\} \frac{d^2\theta}{ds^2} v_{c,fmax}^2 +  a_{lng, max} \sqrt{1 -
  \left(\frac{\frac{d\theta}{ds}v_{c,fmax}^2}{a_{lat,max}}\right)^2} \\
\left(\frac{d^2\theta}{ds^2} \frac{\{r_l,r_r\}v_{c,fmax}^2}{a_{lng, max}}\right)^2 &=
  1 - \left(\frac{\frac{d\theta}{ds}v_{c,fmax}^2}{a_{lat,max}}\right)^2 \\
v_{c,fmax} &= \min_{r \in \{r_l, r_r\}}
\left(\left(\frac{d^2\theta}{ds^2} \frac{r}{a_{lng, max}}\right)^2 +
   \left(\frac{\frac{d\theta}{ds}}{a_{lat,max}}\right)^2\right)^{-\frac14} \\
\begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
\end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
&= \begin{bmatrix}\pm 1 \\ \pm 1\end{bmatrix}
   a_{lng, max} \sqrt{1 - \left(\frac{\frac{d\theta}{ds}v_c^2}{a_{lat,max}}\right)^2} \\
K_2 \dot{v}_c + K_1 v_c^2
&= \begin{bmatrix}\pm 1 \\ \pm 1\end{bmatrix}
   a_{lng, max} \sqrt{1 - \left(\frac{\frac{d\theta}{ds}v_c^2}{a_{lat,max}}\right)^2} \\
K_2' K_2 \dot{v}_c + K_2' K_1 v_c^2
&= K_2' \begin{bmatrix}\pm 1 \\ \pm 1\end{bmatrix}
   a_{lng, max} \sqrt{1 - \left(\frac{\frac{d\theta}{ds}v_c^2}{a_{lat,max}}\right)^2} \\
K_2' K_1 v_c^2
&= K_2' \begin{bmatrix}\pm 1 \\ \pm 1\end{bmatrix}
   a_{lng, max} \sqrt{1 - \left(\frac{\frac{d\theta}{ds}v_c^2}{a_{lat,max}}\right)^2} \\
\left(\frac{K_2' K_1 v_c^2}{K_2' \begin{bmatrix} 1 \\ \pm 1\end{bmatrix}a_{lng,max}}\right)^2
&= 1 - \left(\frac{\frac{d\theta}{ds}v_c^2}{a_{lat,max}}\right)^2 \\
v_c &= \min \left(
\left(\frac{K_2'K_1}{K_2' \begin{bmatrix} 1 \\ \pm 1\end{bmatrix}a_{lng, max}}\right)^2 +
   \left(\frac{\frac{d\theta}{ds}}{a_{lat,max}}\right)^2
  \right)^{-\frac14}
\end{align*}

Where the $\pm 1$ term will have opposite signs of the two coefficients of
$K_2$ have the same sign, and will have the same sign of the coefficients of
$K_2$ have opposite signs.

$K_2'$ refers to a transformed version of $K_2$ such that $K_2'K_2 = 0$ (see
below for exact definition).

\section{Calculating Feasible Voltage Velocities}

Given:

\begin{align*}
\begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
  &= A \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
     \end{bmatrix} v_c + B \begin{bmatrix} u_l \\ u_r \end{bmatrix} \\
\abs{u} &\le V_{max} \\
\norm{B^{-1} \left(\begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
  - A \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
     \end{bmatrix} v_c\right) }_\infty \le V_{max} \\
\end{align}

This is a linear program for fixed $v_c$. We wish to determine the limits of
$v_c$ for which the linear program will have a solution.

In order to analyze this linear program in $\dot{v}_c$, it is useful to know the
signs of the coefficients of $\dot{v}_c$, which in this case are
$B^{-1} \begin{bmatrix}1 - r_l\frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}\end{bmatrix}$

If either coefficient is zero, then for any given
$v_c$, then the linear program
becomes trivial, as the term with $\dot{v}_c$ will drop out of one of the rows
and so the other row will be guaranteed to be valid for some $\dot{v}_c$ while
the row with the dropped $\dot{v}_c$ will either be valid or not. If $B$ were
the identity matrix, then this would
correspond to situations where one wheel is stationary and the
acceleration of that wheel is not affected by the overall acceleration along the
path. Determining the limits of allowable velocities is then an issue of
calculating the point at which the stationary wheel goes from being controllable
to not, for a given overall velocity.

Otherwise, the edge of feasibility will occur
when the pair of constraints (where one pair is for the left voltage, one for
the right) cross and so we will have each voltage as $\pm V_{max}$; if
the signs of the coefficients differ, then we will have
$u_l = u_r$. Otherwise, $u_l = -u_r$.

For $u_l = u_r$, attempting to maximize $v_c$:
\begin{align*}
B^{-1} \left(\begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
  - A \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
     \end{bmatrix} v_c\right) &= \pm \begin{bmatrix} V_{max} \\ V_{max} \end{bmatrix} \\
\begin{pmatrix} 1 + r_r\frac{d\theta}{ds} \\ r_l\frac{d\theta}{ds} - 1
  \end{pmatrix}\left(
\begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
  - A \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
     \end{bmatrix} v_c &= \pm B \begin{bmatrix} V_{max} \\ V_{max} \end{bmatrix}
     \right) \\
(r_l \frac{d\theta}{ds} - 1) (1 + r_r \frac{d\theta}{ds}) \dot{v}_c +
  \begin{bmatrix} -r_l - r_lr_r\frac{d\theta}{ds} \\ -r_r +
  r_lr_r\frac{d\theta}{ds} \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
  - \begin{bmatrix} 1 + r_r \frac{d\theta}{ds} & 0 \\ 0 & r_l \frac{d\theta}{ds} - 1
  \end{bmatrix} A \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
     \end{bmatrix} v_c &=
   \begin{bmatrix} 1 + r_r \frac{d\theta}{ds} & 0 \\ 0 & r_l \frac{d\theta}{ds} - 1
   \end{bmatrix} B
     \begin{bmatrix} \pm V_{max} \\ \pm V_{max} \end{bmatrix}
\end{align*}

\begin{align*}
 -(r_l + r_r) \frac{d^2\theta}{ds^2} v_c^2
 - ((1 + r_r\frac{d\theta}{ds})(A_{11}(1 - r_l\frac{d\theta}{ds}) + A_{12}(1 + r_r\frac{d\theta}{ds}))
  + (r_l\frac{d\theta}{ds} - 1)(A_{12}(1 - r_l\frac{d\theta}{ds}) + A_{11}(1 + r_r\frac{d\theta}{ds}))
     ) v_c &= \\
   \pm V_{max}\frac{d\theta}{ds}(B_{12}r_l + B_{11}r_l + B_{11}r_r + B_{12}r_r) \\
 (r_l + r_r) \frac{d^2\theta}{ds^2} v_c^2 +
 \frac{d\theta}{ds}( A_{11} (-r_l - r_r) + A_{12}(r_r + r_l) +
  r_r(A_{11}(1 - r_l\frac{d\theta}{ds}) + A_{12}(1 + r_r\frac{d\theta}{ds}))  +
  r_l(A_{12}(1 - r_l\frac{d\theta}{ds}) + A_{11}(1 + r_r\frac{d\theta}{ds}))
     ) v_c &= \\
 (r_l + r_r) \frac{d^2\theta}{ds^2} v_c^2 +
 \frac{d\theta}{ds}( (r_l + r_r) (A_{12} - A_{11}) +
 A_{11}(r_l + r_r) + A_{12}(r_l + r_r) +
 A_{12}\frac{d\theta}{ds}(r_r^2 - r_l^2)) v_c &= \\
 (r_l + r_r) \left( \frac{d^2\theta}{ds^2} v_c^2 +
 \frac{d\theta}{ds}A_{12}\left(2 +
 \frac{d\theta}{ds}(r_r - r_l)\right) v_c\right) &= \\
     \pm V_{max}(r_l + r_r)\frac{d\theta}{ds}(B_{11} + B_{12}) \\
 \frac{d^2\theta}{ds^2} v_c^2 +
 \frac{d\theta}{ds}A_{12}\left(2 + \frac{d\theta}{ds}(r_r - r_l)\right) v_c =
     \pm V_{max}\frac{d\theta}{ds}(B_{11} + B_{12}) \\
\end{align*}

Note that when I eliminate parts of $A$ and $B$ I am taking advantage of the
fact that different rows of $A$ and $B$ are just reordered version of one
another (i.e., each is of the format $\begin{bmatrix} a & b \\ b & a
\end{bmatrix}$).

In the final equation, we then execute the quadratic formula for $v_c$.
Note that if you write
the quadratic formula out there will be up to 4 real solutions (the two
solutions to the quadratic formula by the two $\pm V_{max}$ results); because
of the placement of the $\pm V_{max}$, then
0, 1, or 3 of the solutions will be greater than zero \footnote{The basic format of
the quadratic formula
in this case will be $b \pm \sqrt{b^2 \pm k}$ or some equivalent, and
$0 < \abs{k} \le b^2$ ($\abs{k} \le b^$ is necessary to have 4 real solutions;
and $k = 0$ only if
$b$ also is zero, or if $V_{max} = 0$, which are not particularly
relevant corner cases). If $b$ is positive,
then 3 will be positive ($b +$ anything is positive, and since
$\sqrt{b^2 - k} \le b$, $b - \sqrt{b^2 - k} \ge 0$, while $b - \sqrt{b^2 + k}
\le 0$). A similar argument holds for negative $b$. If $b = 0$, then if there
are four real solutions, all four are zero. If there are not four real
solutions, then $\abs{k} > b^2$ and we have two real solutions. In this case,
because $\sqrt{b^2 + k} > b$, we will have one positive and one negative
solution.}. The zero positive solution case is not particularly interesting, as
it only applies at zero curvature (which cannot apply here, as we already stated
that $\abs{r\frac{d\theta}{ds}} > 1$ (or, if we have zero $V_{max}$). The one
or three provide us our cutoffs for where there may be a gap in allowable
accelerations.

For situations where the coefficients have the same sign, attempting to maximize $v_c$:

\begin{align*}
(r_l \frac{d\theta}{ds} - 1) (1 + r_r \frac{d\theta}{ds}) \dot{v}_c +
  \begin{bmatrix} -r_l - r_lr_r\frac{d\theta}{ds} \\ -r_r +
  r_lr_r\frac{d\theta}{ds} \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
  - \begin{bmatrix} 1 + r_r \frac{d\theta}{ds} & 0 \\ 0 & r_l \frac{d\theta}{ds} - 1
  \end{bmatrix} A \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
     \end{bmatrix} v_c &=
   \begin{bmatrix} 1 + r_r \frac{d\theta}{ds} & 0 \\ 0 & r_l \frac{d\theta}{ds} - 1
   \end{bmatrix} B
     \begin{bmatrix} \pm V_{max} \\ \mp V_{max} \end{bmatrix}
\end{align*}

\begin{align*}
 (r_l + r_r) \left( \frac{d^2\theta}{ds^2} v_c^2 +
 \frac{d\theta}{ds}A_{12}\left(2 +
 \frac{d\theta}{ds}(r_r - r_l)\right) v_c\right) &=
   \pm V_{max}(B_{11} - B_{12})(2 + \frac{d\theta}{ds}(r_r - r_l)) \\
 \frac{d^2\theta}{ds^2} v_c^2 +
 \frac{d\theta}{ds}A_{12}\left(2 +
 \frac{d\theta}{ds}(r_r - r_l)\right) v_c &=
   \pm V_{max}\frac{(B_{11} - B_{12})(2 + \frac{d\theta}{ds}(r_r - r_l))}{r_r + r_l}\\
\end{align*}

Where the same basic rules apply to the quadratic formula solutions.

Considering situations where one of the rows in the equations disappear,
we end up with the $\dot{v}_c$ term disappearing and
we must solve the following equation for whichever row of $B^{-1}$ $B^{-1}_N$
causes the $\dot{v}_c$ term to disappear:
\begin{align*}
B^{-1} &= \frac{1}{B_{11}^2 - B_{12}^2}\begin{bmatrix} B_{11} & -B_{12} \\
                                   -B_{12} & B_{11} \end{bmatrix} \\
B^{-1}_N\left(
  \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
  - A \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\
                      1 + r_r \frac{d\theta}{ds} \end{bmatrix} v_c \right)
                      &= \pm V_{max} \\
\end{align*}

Trying to be more general:
\begin{align*}
B^{-1} \left(\begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} -r_l \\ r_r \end{bmatrix} \frac{d^2\theta}{ds^2} v_c^2
  - A \begin{bmatrix} 1 - r_l \frac{d\theta}{ds} \\ 1 + r_r \frac{d\theta}{ds}
     \end{bmatrix} v_c\right) &= \pm \begin{bmatrix} V_{max} \\ V_{max} \end{bmatrix} \\
B^{-1} \left(K_2 \dot{v}_c +   K_1 v_c^2 - A K_2 v_c\right) &=
   \pm \begin{bmatrix} V_{max} \\ V_{max} \end{bmatrix} \\
K_2' &= K_2^T  \begin{bmatrix} 0 & 1 \\ -1 & 0 \end{bmatrix} \\
K_2' \left(K_2 \dot{v}_c +   K_1 v_c^2 - A K_2 v_c\right) &=
   K_2' B \begin{bmatrix} \pm 1 \\ \pm 1 \end{bmatrix} V_{max} \\
K_2' \left(K_2 \dot{v}_c +   K_1 v_c^2 - A K_2 v_c\right) &=
   K_2' B \begin{bmatrix} \pm 1 \\ \pm 1 \end{bmatrix} V_{max} \\
K_2' \left(K_1 v_c^2 - A K_2 v_c\right) &=
   K_2' B \begin{bmatrix} \pm 1 \\ \pm 1 \end{bmatrix} V_{max} \\
 \end{align*}

The quadratic formula coefficients:
\begin{align*}
a v_c^2 + b v_c + c &= 0 \\
a &= K_2'K_1 \\
b &= -K_2'AK_2 \\
c &= -K_2' B \begin{bmatrix} \pm 1 \\ \pm 1 \end{bmatrix} V_{max} \\
b^2 - 4 ac &= (K_2'AK_2)^2 + 4K_2'K_1 K_2'B \begin{bmatrix} \pm 1 \\ \pm 1 \end{bmatrix} V_{max} \\
\end{align*}

%Interestingly, the solutions for both basically devolve to:
%\begin{align*}
% \frac{d^2\theta}{ds^2} v_c^2 + 2 \frac{d\theta}{ds} A_{12} v_c
%  = \pm V_{max}\max(\abs{\frac{d\theta}{ds}}, 1)(B_{11} + B_{12}) \\
%\end{align*}
%
%While not overly interesting, the moments when we will have a split in
%feasibility will be at:
%
%\begin{align*}
%0 < \abs{4V_{max}\max(\abs{\frac{d\theta}{ds}}, 1)(B_{11} +
%B_{12})\frac{d^2\theta}{ds^2}} &\le \abs{2\frac{d\theta}{ds} A_{12}} \\
%\abs{\frac{d\theta}{ds}} &\le 1 : \\
%0 < \abs{2V_{max}(B_{11} + B_{12})\frac{d^2\theta}{ds^2}} &\le
%\abs{\frac{d\theta}{ds} A_{12}} \\
%\abs{\frac{d\theta}{ds}} &> 1 : \\
%0 < \abs{2V_{max}(B_{11} + B_{12})\frac{d^2\theta}{ds^2}} &\le \abs{A_{12}} \\
%\end{align*}
%
%Note that, in general, $A_{12}$ will be relatively small when compared to
%$2V_{max}(B_{11} + B_{12})$.

%\begin{align*}
%B_{12} = B_{21} &, B_{11} = B_{22} \\
%2 r \frac{d^2\theta}{ds^2} v_c^2 &= \pm 2V_{max}r\frac{d\theta}{ds} (B_{12} - B_{11}) \\
%v_c &= \sqrt{\pm\frac{V_{max}\frac{d\theta}{ds}}{\frac{d^2\theta}{ds^2}} (B_{12} - B_{11})} \\
%\end{align*}
%
%\begin{align*}
%\begin{bmatrix} 1 - r^2 \frac{d\theta}{ds}^2 \\ 1 - r^2 \frac{d\theta}{ds}^2
%  \end{bmatrix} \dot{v}_c +   \begin{bmatrix} -1 - r\frac{d\theta}{ds} \\ 1 -
%  r\frac{d\theta}{ds} \end{bmatrix} r \frac{d^2\theta}{ds^2} v_c^2
%  - A \begin{bmatrix} 1 - r^2 \frac{d\theta}{ds}^2 \\ 1 - r^2 \frac{d\theta}{ds}^2
%     \end{bmatrix} v_c &=  B
%     \begin{bmatrix} \mp V_{max} (1 + r\frac{d\theta}{ds}) \\ \pm V_{max} (1 -
%     r\frac{d\theta}{ds}) \end{bmatrix} \\
% 2 r \frac{d^2\theta}{ds^2} v_c^2
%      &= \pm V_{max}(B_{21} - B_{22} - B_{11} + B_{12}) \\
% r \frac{d^2\theta}{ds^2} v_c^2
%      &= \pm V_{max}(B_{12} - B_{11}) \\
%v_c &= \sqrt{\pm \frac{V_{max}}{r\frac{d^2\theta}{ds^2}} (B_{12} - B_{11})} \\
%\end{align*}

\end{document}


\subsection*{e)}
Here we will investigate the mixing and the posteriors in the proposal distribution. From the relation of $R$ in the definition. We note that $R$ depends both on $\rho$ and $t_{i+1}-t_{i-1}$. One can realize first that $R$ will be calculated for each breakpoint, hence an increase of breakpoints used, a decrease of the value of the differences will occur. We therefore need to increase $\rho$ accordingly. \\
\\
We know that $\rho$ will effect our sample space. Therefore if we choose $\rho$ for values
\[ || \rho || > 0 + C, \quad \text{for} \quad C>> 0 \]
indicates that we will consider a large portion of the sample space, in our case $X\in\mathbb{R}$. This in turn will only generate rejected samples, since we are almost 100\% outside of the our desired sample space. However, if $\rho=0$ or close to $0$,
\[ || \rho ||+ \epsilon \geq 0,  \quad \text{for} \quad \epsilon \geq 0 \]
we do not sample outside of the proposed distribution, meaning that we will only end up in the proposed values and again reject most of the sampled values. \\
\\
We know that $\rho$ must help to enclose the desired sample space. Since we know from the argumentation above, $\rho$ must be small and large enough but also depend on the breakpoints. Below in figures \ref{fig:fixrho1} and \ref{fig:fixrho4} are plots of the mixing for $\boldsymbol{t}$ with a fixed $\rho$. \\
\\
\begin{figure}[H]
  \centering
\begin{minipage}{1\textwidth}
    \includegraphics[scale=0.28]{./Figures/fixrho1.png}
  \label{fig:fixrho1}
\end{minipage}
%
\begin{minipage}{1\textwidth}
    \includegraphics[scale=0.28]{./Figures/fixrho4.png}
  \label{fig:fixrho4}
\end{minipage}

  \caption[An Electron]{Figure of t-posterior for one and four breakpoints for a fixed constant $\rho=0.05$}
\end{figure}

One can deduce from the figures \ref{fig:fixrho1},\ref{fig:fixrho4} that the mixing is good for one breakpoint. However, we will have more correlated values when using $\rho=0.05$ for four breakpoints. Instead of using a constant $\rho$ we introduce a varing $\rho$ which is determined by the number of breakpoints used.

\begin{figure}[H]

  \centering
\begin{minipage}{1\textwidth}
    \includegraphics[scale=0.28]{./Figures/varrho1.png}
  \label{fig:varrho1}
\end{minipage}

\begin{minipage}{1\textwidth}
    \includegraphics[scale=0.28]{./Figures/t4.png}
  \label{fig:varrho4}
\end{minipage}

  \caption[An Electron]{Figure of t-posterior for breakpoints$=1,4$ for a varing $\rho$}
\end{figure}

From the figures \ref{fig:varrho1},\ref{fig:varrho4} one can see that the mixing is more random and non-correlated compared with the fixed $\rho$. Indicating that our values should be better approximated using this relation.


\documentclass{article}
\usepackage[utf8]{inputenc}

\title{MAT257 Notes}
\author{Jad Elkhaleq Ghalayini}
\date{January 25 2019}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{mathtools}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{cancel}

\usepackage[margin=1in]{geometry}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{definition}{Definition}
\newtheorem*{corollary}{Corollary}
\newtheorem{exercise}{Exercise}
\newtheorem{claim}{Claim}

\DeclareMathOperator{\Int}{Int}
\DeclareMathOperator{\grad}{grad}
\DeclareMathOperator{\Ker}{Ker}
\DeclareMathOperator{\Ima}{Im}
\DeclareMathOperator{\Vol}{vol}

\newcommand{\reals}[0]{\mathbb{R}}
\newcommand{\nats}[0]{\mathbb{N}}
\newcommand{\ints}[0]{\mathbb{Z}}
\newcommand{\rationals}[0]{\mathbb{Q}}
\newcommand{\brac}[1]{\left(#1\right)}
\newcommand{\sbrac}[1]{\left[#1\right]}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\eval}[3]{\left.#3\right|_{#1}^{#2}}
\newcommand{\ip}[2]{\left\langle#1,#2\right\rangle}
\newcommand{\prt}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\mb}[1]{\mathbf{#1}}
\newcommand{\hlfspc}[0]{\mathbb{H}}
\newcommand{\loint}[0]{\operatorname{L}\int}
\newcommand{\hiint}[0]{\operatorname{U}\int}
\newcommand{\indic}[1]{\chi_{#1}}

\begin{document}

\maketitle

\section{Change of Variable Theorem}
We'll continue our reduction of the theorem from last time:
\begin{itemize}

  \item [4.] The theorem is true if \(g\) is an (invertible) linear transformation. We already showed that, for any \(g\), it's good enough to prove that the theorem is true on an open rectangle \(U\) with the constant function \(f = 1\), i.e. that if \(U\) is an open rectangle in \(A\), then
  \begin{equation}
    \int_{g(U)}1 = \int_U|\det g'| \iff \Vol(g(U)) = |\det g|\Vol(U)
  \end{equation}
  Note that we've already proved this in linear algebra, using the volume of parallelepipeds. But let's do it analytically.

  We will make use of the fact that any invertible linear transformation (i.e. invertible \(n \times n\) matrix) is a composite of elementary transformations (elementary matrices). Last time we showed that if the theorem is true for \(g, h\), then it is true for \(g \circ h\). Hence, all we need to do is show that the theorem is true for an arbitrary elementary transformation (matrix). We have 3 kinds of elementary matrices:
  \begin{itemize}

    \item A matrix which has all 1's on the diagonal, except for \(\lambda\) in one place. In this case, \(\det g = \lambda\), so we could use Fubini's theorem to scale by \(\lambda\), giving \(\Vol(g(U)) = \lambda \Vol(U) = |\det g|\Vol(U)\) as desired.

    \item A matrix which has all 1's on the diagonal, except for two adjacent rows which have \(\begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}\) on the diagonal. We have \(\det g = \pm 1\). Since we're just changing two coordinates, the integral remains unchanged.

    \item The identity matrix, plus \(1\) in a position \(i, j\) with \(i \neq j\). In this case, the determinant is just \(1\). Furthermore, we're just translating the top of our rectangle into a parallelogram. Fubini's theorem tells us that the volume is the same since the limits in terms of \(y\) are the same, and for fixed \(y\) we are integrating over a shifted integral of the same length, the shifting not having an effect since we are integrating a constant.

  \end{itemize}
  \label{p4}

\end{itemize}
We can now return to the proof of the theorem: by 3 and \ref{p4}, we can assume that, given \(a \in A, g'(a) = I\), where \(I\) is the identity. Why? Let \(T = g'(a)\) be an invertible linear transformation. Then we can write
\begin{equation}
  g = T \circ (T^{-1} \circ g)
\end{equation}
By \ref{p4}, this is true for \(T\). By 3, if this is true for \(T\) and \(T^{-1} \circ g\) then it's true for \(g\). So it's enough to prove for the case
\begin{equation}
    (T^{-1} \circ g)' = I
\end{equation}
To complete the proof, we use induction on \(n\):
\begin{itemize}

  \item \(n = 1\): by integration by substitution

  \item Assume it's true for \(n - 1\). It's enought to show that, for all \(a \in A\), we can find Jordan measurable open \(U \subset A\) containing \(a\), with closure in \(A\), so that the theorem is true on \(U\) when \(f = 1\).

  We can assume \(g'(A) = I\). We'll express \(g\) as the composition of 2 mappings, each of which changes at most \(n - 1\) coordinates. There are a number of ways to do this. One is defining \(h : A \to \reals^n\) by
  \begin{equation}
    h(x) = (g_1(x),...,g_n(x), x_n)
  \end{equation}
  Why is this a change of variable? \(h'(a) = I\), so there is an open neighborhood \(U'\) of \(a\) on which \(h\) is one to one and \(\det h'(a) \neq 0\). Now define
  \begin{equation}
    k: h(U') \to \reals^n, k(y) = (y_1,...,y_{n - 1}, g_n(h^{-1}(y_n)))
  \end{equation}
  We have \(g = k \circ h\),
  \begin{equation}
    g'(a) = k'(h(a))h'(a) = I \implies k'(h(a)) = I
  \end{equation}
  So there is an open neighborhood \(V\) of \(h(a)\) in \(U'\) on which \(k\) is one to one and
  \begin{equation}
    \det k'(g) = 0
  \end{equation}
  Let \(U = h^{-1}(V)\). On \(U\), \(g = k \circ h\), and \(h: U \to V\), \(k: V \to \reals^n\).

  So we have to prove the theorem for \(h\) and for \(k\). The proof is going to be the same for both cases, so let's do it for the harder case, \(h\), which changes \(n - 1\) coordinates (versus \(k\), which changes only one).

  We're going to use Fubini's theorem: we'll integrate with respect to \(x_n\). Next time...
\end{itemize}

\end{document}


\section{Implementation details}
\label{sec:Implementation details}

We use \Cplex~\cite{IBMILogCplex}~for solving the linear programs and performing the branch-and-bound tree search. To add the Benders cuts to the master problem during the search, we use {\tt CPXsetlazyconstraintcallbackfunc} and {\tt CPXsetusercutcallbackfunc} callback functions.

\subsection{Scenario partitioning}
\label{subsec:Scenario partitioning}

The hybrid-cut method partitions the scenarios into multiple sets and hence a partitioning method must be used. Different partitioning strategies have been studied such as random partitioning that randomly picks an aggregate for every scenario, or a round-robin partitioning that evenly distributes the scenarios on all aggregates~\cite{Wolf2013Advancedaccelerationtechniques}. In our implementation, we use two different way to for the scenario partitioning: round-robin partitioning and K-means clustering partitioning~\cite{Hartigan.Wong1979AlgorithmAS136:}. Below we give implementation details of the later approach using a K-means++ heuristic.

The purpose of K-means clustering is to partition a set of data into a given number of clusters such that each data point is included in the cluster whose mean is closet. The standard form of this method is depicted in Algorithm \ref{algorithm:K-menas}.

\begin{algorithm}[ht!]
\caption{\textit{K-means}}
\label{algorithm:K-menas}
\begin{algorithmic}[1]
    \REQUIRE $n$ data points and an initial set of $k$ means.
    \ENSURE $k$ clusters.
    \WHILE {Assignments no longer change}
        \STATE Assign each each point to the cluster with the closet mean.
        \STATE Calculate the new mean of each cluster.
    \ENDWHILE
\end{algorithmic}
\end{algorithm}

We determine the distance from a point (scenario) to another point or from a point to the mean of a group based on the Euclidean distance. The error associated with a group is then the total distance of the scenarios from the mean of the group to which they are assigned. However, the number of groups (or aggregates in our case), $k$, is not known a priori. To determine this value, we use
$C_{agg} \times \ln{| S |}^2$, where $C_{agg}$ is a positive constant number.

To create the initial set of $k$ groups for Algorithm~\ref{algorithm:K-menas}, we first identify $k$ core scenarios representing the the mean of a group using a K-means++ heuristic \cite{Arthur.Vassilvitskii2007Kmeans++:advantages}. This method iteratively chooses the core scenarios randomly from the set of scenarios with weighted probabilities that reflect the distance of each scenario from the previous core scenario chosen. Along with approximation guarantees relating to the error of the resulting clustering, the authors provided compelling computational evidence that this methodology is superior to simply choosing at random.

Let $d^s, s = 1,\ldots,|S|$ be denoted the vectors of statistics associated with each scenario. This method is depicted in Algorithm \ref{algorithm:K-menas++}.

\begin{algorithm}[ht!]
\caption{\textit{K-means++}}
\label{algorithm:K-menas++}
\begin{algorithmic}[1]
    \REQUIRE $n$ data points.
    \ENSURE An initial set of $k$ means.
    \STATE Let $W \coloneqq \llp 1, \ldots, S \rlp$.
    \STATE Take one scenario $c^1$ from $W$ randomly as a core scenario.
    \FOR{$p \coloneqq 2,\ldots,k$}
        \STATE Let $W \coloneqq W\backslash \{ p \}$.
        \STATE Define a new core scenario by randomly selecting among the scenarios net yet core with probabilities
        $$
            \frac{\| d^{j} - d^{p-1}  \|}{ \sum_{i \in W}\|d^i - d^{p-1} \| }, j \in W.
        $$
    \ENDFOR
\end{algorithmic}
\end{algorithm}

The computational complexity for the K-means is $\cO(kndi)$, where $n$ is the number of $d$-dimensional vectors, $k$ is the number of clusters, and $i$ the number of iterations needed until convergence, and the running time in practice can be significant. To reduce the computational burden, we may partition the whole data into $k_{seg}$ subsets through K-means method, and then perform the K-means method for each subset. This reduce the complexity to $\cO((k_{seg} + 1)ndi)$. Note that if the reduction of the complexity is not satisfactory, we may further partition subset, resulting in a recursive partitioning scheme.

\subsection{Advanced start}
\label{subsec:Advanced start}

Often time the initial iterations of cutting plane methods can be ineffective. This unfortunately make the cutting plane methods converge relatively slow. To overcome this difficulty, Infanger proposes to compute the expected value problem (EV) solution and then start the L-Shaped method on the original problem instance, but with the cuts generated during the computation of the EV solution. The hope is that the generated cuts make the algorithm prevent from generating ineffective cuts in the early L-Shaped iterations. In additional to the EV solution, other solutions such as an optimal solution for the worst-case scenario or the best-case scenario may also be used as an advanced start solutions~\cite{Wolf.Koberstein2013Dynamicsequencingand}. Such solutions can be obtained by solving the Wait-and-See problem, which is a relaxation to the original stochastic optimization problem obtained by dropping the nonanticipativity restriction.

%\subsection{Unbounded and infeasible status}
%\label{subsec:Unbounded and infeasible status}

%\subsection{Generation of a safe cut}
%\label{subsec:Generation of a safe cut}

%\subsection{Improvement on the initial lower bound}
%\label{subsec:Improvement on the initial lower bound}

\subsection{Parallelization}
\label{subsec:Parallelization}

As all subproblems are independent of one another, parallelization of the second stage computations can be done naturally. Modern processors have multiple cores that allow parallel execution at a single computer. To take advantage of that we parallelize the algorithm in a fashion similar to \cite{Ariyawansa.Hudson1991Performancebenchmarkparallel, Wolf2013Advancedaccelerationtechniques} on a symmetric-multiprocessing architecture. In particular, we solve the subproblems and perform the cut aggregation procedure concurrently. This is done though shared-memory parallel programming API {\tt OpenMp}~\cite{OpenMP(OpenMulti}.

In our implementation, we solve the subproblems concurrently while solving the relaxation problem and solving the root node problem. Once the branching starts, we solve the subproblems sequentially and let \Cplex~to perform the parallel tree search. We note that as \Cplex~callback functions is used, a run-to-run deterministic results cannot be guaranteed in multi-threaded mode.


The \eslmod{dmatrix} module implements 2D matrices and linear algebra
operations.

There are two objects. The main one is a \ccode{ESL\_DMATRIX}, a 2D
real-valued matrix of n rows and m columns. There is also
\ccode{ESL\_PERMUTATION}, a special matrix used in LU decompositions.

It is straightforward to call standard BLAS and LAPACK linear algebra
routines on the data in an \ccode{ESL\_DMATRIX}.

\begin{table}[hbp]
\begin{center}
{\small
\begin{tabular}{|ll|}\hline
\hyperlink{func:esl_dmatrix_Create()}{\ccode{esl\_dmatrix\_Create()}} & Create general matrix.\\
\hyperlink{func:esl_dmatrix_CreateUpper()}{\ccode{esl\_dmatrix\_CreateUpper()}} & Create packed upper triangular matrix.\\
\hyperlink{func:esl_dmatrix_Destroy()}{\ccode{esl\_dmatrix\_Destroy()}} & Free a matrix.\\
\hyperlink{func:esl_dmatrix_Dump()}{\ccode{esl\_dmatrix\_Dump()}} & Dump matrix internals to output stream.\\
\hyperlink{func:esl_dmatrix_Copy()}{\ccode{esl\_dmatrix\_Copy()}} & Make a copy of a matrix (no new allocation).\\
\hyperlink{func:esl_dmatrix_Clone()}{\ccode{esl\_dmatrix\_Clone()}} & Duplicate a matrix (allocate new storage).\\
\hyperlink{func:esl_dmatrix_Compare()}{\ccode{esl\_dmatrix\_Compare()}} & Compare two matrices for equality.\\
\hyperlink{func:esl_dmatrix_Set()}{\ccode{esl\_dmatrix\_Set()}} & Set all cells in matrix to same scalar value.\\
\hyperlink{func:esl_dmatrix_SetZero()}{\ccode{esl\_dmatrix\_SetZero()}} & Set all cells in matrix to zero.\\
\hyperlink{func:esl_dmatrix_SetIdentity()}{\ccode{esl\_dmatrix\_SetIdentity()}} & Set diagonal elements to 1, all others to zero.\\
\hyperlink{func:esl_dmx_Max()}{\ccode{esl\_dmx\_Max()}} &Returns maximum element value.\\
\hyperlink{func:esl_dmx_Min()}{\ccode{esl\_dmx\_Min()}} &Returns maximum element value.\\
\hyperlink{func:esl_dmx_Sum()}{\ccode{esl\_dmx\_Sum()}} &Returns sum of all elements.\\
\hyperlink{func:esl_permutation_Create()}{\ccode{esl\_permutation\_Create()}} & Create a permutation matrix.\\
\hyperlink{func:esl_permutation_Destroy()}{\ccode{esl\_permutation\_Destroy()}} & Free a permutation matrix.\\
\hyperlink{func:esl_permutation_Reuse()}{\ccode{esl\_permutation\_Reuse()}} & Reuse a permutation matrix.\\
\hyperlink{func:esl_permutation_Dump()}{\ccode{esl\_permutation\_Dump()}} & Dump permutation matrix internals to output stream.\\
\hyperlink{func:esl_dmx_Multiply()}{\ccode{esl\_dmx\_Multiply()}} & Matrix multiplication.\\
\hyperlink{func:esl_dmx_Transpose()}{\ccode{esl\_dmx\_Transpose()}} & Matrix transpostion.\\
\hyperlink{func:esl_dmx_Add()}{\ccode{esl\_dmx\_Add()}} & Matrix addition.\\
\hyperlink{func:esl_dmx_Scale()}{\ccode{esl\_dmx\_Scale()}} & Multiply a matrix by a scalar.\\
\hyperlink{func:esl_dmx_AddScale()}{\ccode{esl\_dmx\_AddScale()}} & $A + kB$ \\
\hyperlink{func:esl_dmx_Permute_PA()}{\ccode{esl\_dmx\_Permute\_PA()}} & $B = PA$, a row-wise permutation of $A$.\\
\hyperlink{func:esl_dmx_LUP_decompose()}{\ccode{esl\_dmx\_LUP\_decompose()}} & Permuted LU decomposition.\\
\hyperlink{func:esl_dmx_LU_separate()}{\ccode{esl\_dmx\_LU\_separate()}} & Get answers from a LU decomposition.\\
\hyperlink{func:esl_dmx_Invert()}{\ccode{esl\_dmx\_Invert()}} & Matrix inversion.\\
\hline
\end{tabular}
}
\end{center}
\caption{The \eslmod{dmatrix} API.}
\label{tbl:dmatrix_api}
\end{table}

\subsection{Example of using the dmatrix API}

A toy example that demonstrates the syntax of creating three 4x4
square matrices and doing some simple operations on them:

\input{cexcerpts/dmatrix_example}

\subsection{Accessing matrix values}

The accessible internals of the \ccode{ESL\_DMATRIX} structure are:

\input{cexcerpts/dmatrix_obj}

The matrix is stored in row-major orientation: the value in cell
$(i,j)$ in row $i$ and column $j$ is in \ccode{mx->mx[i][j]}.

Elements are stored in a single array \ccode{mx->mx[0]}. This is
important for interoperability with BLAS and LAPACK; see below. The
row pointers \ccode{mx->mx[i]} are initialized so that elements may be
accessed simply as \ccode{mx->mx[i][j]}, rather than by pointer
arithmetic \ccode{mx->mx[0] + i*mx->m + j}.

\subsection{Specialized matrix types}

Normally matrices are created with \ccode{esl\_dmatrix\_Create()},
which allocates storage for all $n \times m$ cells. Easel calls this a
matrix of type \ccode{eslGENERAL}.

Matrices may have more restricted forms, which may constrain certain
values and may allow packed storage. For example, an upper triangular
matrix is one in which all elements $i>j$ have a value of zero. When
we calculate the minimum in such a matrix with
\ccode{esl\_dmatrix\_Min()}, we probably don't want to consider the
$i>j$ elements. We also can save almost two-fold in storage by not
storing the $i>j$ elements at all. Other types include square, lower
triagonal, and symmetric matrices.

We expect to need to expand Easel's implementation of different matrix
types in the future, but right now, Easel has just one other matrix
type, \ccode{eslUPPER}, for packed upper triangular matrices.

\subsubsection{\ccode{eslUPPER}: packed upper triangular matrices}

An \ccode{eslUPPER} matrix is created with
\ccode{esl\_dmatrix\_CreateUpper(int n)}. It is necessarily square $n
\times n$, so only one dimension argument is passed. Most but not all
functions in \eslmod{dmatrix} can operate on \ccode{eslUPPER} matrix
types in addition to the usual \ccode{eslGENERAL} type.

The caller must not access any cell $i>j$ in an \ccode{eslUPPER}
matrix. Setting a cell $i>j$ will corrupt the matrix. Accessing cell
$i>j$ will return an incorrect value, not zero.

The $n (n+1) / 2$ elements of the upper triagonal matrix are packed
into an array \ccode{mx->mx[0]}. You can access element $i,j$ by
pointer arithmetic at \ccode{mx->mx[j + i(2*mx->m-i-1)/2]} if you
like, but it is easier to access element $i,j$ by the usual
\ccode{mx->mx[i][j]}. This is made possible because the row pointers
\ccode{mx->mx[i]} in an \ccode{eslUPPER} matrix are tricksily
initialized in an overlapping fashion so that \ccode{mx->mx[i][j]}
does the right thing for $i \leq j$. This overlapping is also the
reason why \ccode{mx->mx[i][j]} accesses the wrong element when $i>j$.

\subsubsection{Notes on the current implementation of matrix types}

Easel matrix types conflate packing and element validity together. For
example, an upper triangular matrix may be stored either in an
\ccode{eslGENERAL} matrix type (in which case elements $i>j$ are set
to zero) or the packed \ccode{eslUPPER} matrix type (in which case
elements $i>j$ aren't even stored). Using the \ccode{eslUPPER} matrix
type is 2x more space efficient, and also, operations like
\ccode{esl\_dmatrix\_Min()} and \ccode{esl\_dmatrix\_Max()} will
examine all elements in an \ccode{eslGENERAL} matrix (including the
zeros), but only the elements $i \leq j$ in a \ccode{eslUPPER} matrix.

This design is provisional. We may adopt a system more closely akin to
BLAS/LAPACK in the future, which distinguish between matrix type and
matrix storage. For example, BLAS has matrices of form \ccode{TR} and
\ccode{TP} for triangular and packed triangular. Easel's
implementation seems sufficient for the moment, and should also extend
to lower diagonal and symmetric matrices without difficulty when and
if they become needed. In any future development, look to BLAS and
LAPACK for guidance.

\subsection{Interoperability with BLAS and LAPACK}

The BLAS and LAPACK libraries provide optimized, standardized linear
algebra routines. The storage in \ccode{ESL\_DMATRIX} is designed so
you can call routines in these libraries. The \ccode{mx->mx[0]} array
is a valid matrix for BLAS and LAPACK so long as you know the right
incantations. These are summarized here:

{\small
\begin{tabular}{llllll}
Easel type          & \ccode{CBLAS\_ORDER}   & stride  & \ccode{CBLAS\_UPLO} & type &  code \\ \hline
\ccode{eslGENERAL}  & \ccode{CblasRowMajor}  & \ccode{mx->m} &        n/a          &     double & \ccode{GE} (GEneral) \\
\ccode{eslUPPER}    & \ccode{CBlasRowMajor}  & \ccode{mx->m} & \ccode{CblasUpper}  &     double & \ccode{TP} (Triangular Packed) \\
\end{tabular}
}

For example, to call the CBLAS (C implementation of BLAS) for an
operation on an Easel matrix of type \ccode{eslGENERAL}, you look for
a routine that starts with prefix \ccode{cblas\_dge*} (\ccode{d} for
double, \ccode{ge} for general).  An example is
\ccode{cblas\_dgemm()}, the matrix multiplication (\ccode{mm})
routine, which computes $C = \alpha \mathit{op}(A) \mathit{op}(B) +
\beta C$ for matrices $A,B,C$ and scalars $\alpha,\beta$, where
$\mathit{op}(A)$ means $A$, $A^T$ (the transpose), or $A^H$ (the
conjugate transpose). $\mathit{op}(A)$ is an $M \times K$ matrix,
$\mathit{op}(B)$ is $K \times N$ matrix, and the result $C$ is $M
\times N$. The prototype for \ccode{cblas\_dgemm} is:

\begin{cchunk}
    void
    cblas_dgemm (const enum CBLAS_ORDER Order, const enum CBLAS_TRANSPOSE TransA,
                 const enum CBLAS_TRANSPOSE TransB, const int M, const int N,
                 const int K, const double alpha, const double *A, const int lda,
                 const double *B, const int ldb, const double beta, double *C,
                 const int ldc)
\end{cchunk}

The \ccode{Order} argument is always \ccode{CblasRowMajor} for Easel
matrices.  The \ccode{TransA} and \ccode{TransB} arguments specify
$\mathit{op}()$: \ccode{CblasNoTrans} means just the matrix
itself. The \ccode{ld*} arguments are the major strides for each
matrix: the number of elements in each row, for our row-major
matrices. So, we could call:

\begin{cchunk}
    cblas_dgemm (CblasRowMajor, CblasNoTrans, CblasNoTrans,
                 A->n, B->m, A->m,
		 1.0, A->mx[0], A->m,
		 B->mx[0], B->m,
		 1.0, C->mx[0], C->m);
\end{cchunk}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{physics}
\usepackage{amsmath}
\usepackage{showlabels}
\title{Statistical Computing for Scientists and Engineers\\[1em] Homework 2}
\author{Jiale Shi}
\date{September/21/2018}

\usepackage{natbib}
\usepackage{graphicx}
\usepackage{array}
\begin{document}
\maketitle

\newpage
\section{Problem 1}
(a) Obtain analytic forms of:
The posterior distribution of Eq. (3) and the marginal posterior distribution over $\alpha$ and $\beta$:$p(\alpha,\beta|y)$ by using Eq. (4), Eq.(5) and the hint provided.

Answer:
\begin{equation}
\begin{aligned}
    p(\theta,\alpha,\beta|y) & \propto p(\alpha,\beta) p(\theta|\alpha,\beta)p(y|\theta,\alpha,\beta) \\
    & \propto (\alpha+\beta)^{-5/2}\prod_{j=1}^{J} \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\theta_{j}^{\alpha+y_{j}-1}(a-\theta_{j})^{\beta+n_{j}-y_{j}-1}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    p(\alpha,\beta|y) & = \frac{ p(\theta,\alpha,\beta|y)}{ p(\theta|\alpha,\beta,y)} \\
    & \propto \frac{(\alpha+\beta)^{-5/2}\prod_{j=1}^{J} \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\theta_{j}^{\alpha+y_{j}-1}(a-\theta_{j})^{\beta+n_{j}-y_{j}-1}}{ \prod_{j=1}^{J} \frac{\Gamma(\alpha+\beta+n_{j})}{\Gamma(\alpha+y_{j})\Gamma(\beta+n_{j}-y_{j})}\theta_{j}^{\alpha+y_{j}-1}(a-\theta_{j})^{\beta+n_{j}-y_{j}-1} } \\
     & \propto \frac{(\alpha+\beta)^{-5/2}\prod_{j=1}^{J} \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}}{ \prod_{j=1}^{J} \frac{\Gamma(\alpha+\beta+n_{j})}{\Gamma(\alpha+y_{j})\Gamma(\beta+n_{j}-y_{j})}} \\
\end{aligned}
\end{equation}

(b) Plot the marginal posterior density $p(\alpha,\beta|y)$ as a function of the transformed variables $\log(\frac{\alpha}{\beta})$ and $\log(\alpha+\beta) \in [(-1.3,-2.3);(1,5)]$. Obtain the corresponding value of $(\alpha,\beta)$.

Answer:
Let $X = \log{\frac{\alpha}{\beta}}, Y = \log(\alpha+\beta)$. Then $\beta = \frac{\exp(Y)}{1+\exp(X)}, \alpha = \exp(X)\beta$.
Using the python code that provided to us.
\begin{figure}[ht]
\centering
\includegraphics[scale=0.5]{hw2_1b.jpg}
\caption{the marginal posterior density $p(\alpha,\beta|y)$ as a function of the transformed variables $\log(\frac{\alpha}{\beta})$ and $\log(\alpha+\beta)$}
%\label{fig:universe}
\end{figure}

$X = -1.79, Y = 2.99$,then the corresponding value of $(\alpha,\beta) = (2.85,17.04)$

\newpage
\section{Problem 2}
Jeffrey's prior and maximum entropy prior: Consider a random variable $x$ described by a Possion distribution:

\begin{equation}
    x \sim p(x;\theta) = \frac{\theta^x e^{-\theta}}{x!}
\end{equation}
(a) Determine the Jeffrey prior $\pi^{J}$ for $\theta$. Is the scale invariant prior $\pi_{0}(\theta) = \frac{1}{\theta}$ preferable to $\pi^{J}$? Why?

Answer:
\begin{equation}
   p(x|\theta) = \frac{\theta^x e^{-\theta}}{x!}
\end{equation}
Therefore,
\begin{equation}
   I(\theta) = -E\left[\frac{\partial^2}{\partial \theta^2} \ln p(x|\theta)\right] = \frac{\theta}{\theta^2} = \frac{1}{\theta}
\end{equation}

Therefore the Jeffreys' prior is given by:
\begin{equation}
   \pi^J = \left[I(\theta)\right]^{1/2} = \theta^{-1/2}
\end{equation}

The scale invariant prior $\pi_{0}(\theta) = \frac{1}{\theta}$ is not preferable to $\pi^{J}$ because they are not the same function.

(b) Find the maximum entropy prior for $\theta$ for the reference measure $\pi^J$ subject to the constraints $E^{\pi}[\theta] = 1$, $Var^{\pi}[\theta] = 1$.

Answer: considering the reference measure as $\pi_{ref}=\pi^J \propto \theta^{-1/2}$.

The maximum entropy prior under the constraints that the prior mean and variance of $\theta$ are both 1:

Two constrains, therefore,$K=2$.

$E^{\pi}[\theta] = 1$, $g_{1}(\theta)=\theta$.

$Var^{\pi}[\theta] = 1= E[(\theta-1)^2]$, $g_{2}(\theta)=(\theta-1)^2$.

\begin{equation}
\hat{\pi} = \frac{\pi_{ref}(\theta)\exp(\sum_{k=1}^{K} \lambda_{k}g_{k}(\theta))}{\int \pi_{ref}(\theta)\exp(\sum_{k=1}^{K} \lambda_{k}g_{k}(\theta))}
\end{equation}

In this problem,
\begin{equation}
\hat{\pi} \propto \theta^{-1/2} \exp(\lambda_{1}\theta+\lambda_{2}(\theta-1)^2)
\end{equation}

(c) Find the maximum entropy prior for $\theta$ for the reference measure $\pi_0$ subject to the constraints $E^{\pi}[\theta] = 1$, $Var^{\pi}[\theta] = 1$.

Answer: Considering the reference measure as $\pi_{ref}=\pi_{0} \propto \theta^{-1}$.

The maximum entropy prior under the constraints that the prior mean and variance of $\theta$ are both 1:

Two constrains, therefore,$K=2$.

$E^{\pi}[\theta] = 1$, $g_{1}(\theta)=\theta$.

$Var^{\pi}[\theta] = 1= E[(\theta-1)^2]$, $g_{2}(\theta)=(\theta-1)^2$.

\begin{equation}
\hat{\pi} = \frac{\pi_{ref}(\theta)\exp(\sum_{k=1}^{K} \lambda_{k}g_{k}(\theta))}{\int \pi_{ref}(\theta)\exp(\sum_{k=1}^{K} \lambda_{k}g_{k}(\theta))}
\end{equation}

In this problem,
\begin{equation}
\hat{\pi} \propto \theta^{-1} \exp(\lambda_{1}\theta+\lambda_{2}(\theta-1)^2)
\end{equation}

\newpage
\section{Problem 3}
Laplace approximation: the data set $X= (X_1,...,X_n)$ presents the number of the wins of a football team in the past n home games. We can model this using
\begin{equation}
    X_{i} \sim g(x_{i}|\theta) = \theta (\theta+1)x_{i}^{\theta -1}(1-x_{i}), x_{i} = (0,1)
\end{equation}
with parameter $\theta > 0$. Unfortunately, this model does not have any corresponding, useful, conjugate prior. But it is acceptable to impose a prior model on $\theta$ with Gamma distribution.

(a) Derive the posterior PDF of $\theta$.

Answer:
\begin{equation}
\begin{aligned}
    p(\theta | x) &= Gamma(\theta;a,b) \prod_{i=1}^{n}p(x_{i}|\theta) \\
    &= \frac{b^{a}\theta^{a-1}\exp{-b\theta}}{\Gamma(a)}\theta^{n}(\theta+1)^{n}\prod_{i=1}^{n}x_{i}^{\theta-1}(1-x_{i})
\end{aligned}
\end{equation}

(b) Using Laplace approximation, find a normal distribution but approximates the posterior distribution using n = 20.
\begin{equation}
    \sum_{x=i} \ln X_{i} = -4.59
\end{equation}
and a = b = 1 where a and b are the hyperparameters of the gamma distribution $Gamma(a,b)$.

Answer:

\begin{math}
n = 20; a=b=1
\end{math}

\begin{equation}
\begin{aligned}
    p(\theta | x)
    &= \frac{\exp{-\theta}}{\Gamma(1)}\theta^{20}(\theta+1)^{20}\prod_{i=1}^{20}x_{i}^{\theta-1}(1-x_{i})
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    \log p(\theta | x)
    &= -\theta + 20 \log(\theta(\theta+1))+(\theta-1)\sum_{i}^{20}\log(x_{i})+C
\end{aligned}
\end{equation}

the first derivative
\begin{equation}
\begin{aligned}
    \frac{d \log p(\theta | x) }{d \theta}
    &= -1 + \frac{20}{\theta} +\frac{20}{\theta+1}+\sum_{i}^{20}\log(x_{i}) = 0
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    \theta^{MAP} \approx 6.69
\end{aligned}
\end{equation}

the second derivative
\begin{equation}
\begin{aligned}
    A = -\frac{d^2 \log p(\theta | x) }{d \theta^2}{\theta=\theta^{MAP}=6.69}
    &= - \frac{20}{\theta^2} -\frac{20}{(\theta+1)^2} \approx -(-0.785) = 0.785 = \frac{1}{\sigma^2}
\end{aligned}
\end{equation}

Therefore,

\begin{equation}
\begin{aligned}
    p(\theta | x)
    &\approx (2\pi)^{-10} |A|^{1/2}\exp{-\frac{1}{2}(\theta - \theta^{MAP})^{2} A} \\
    &\approx (2\pi)^{-10} (0.785)^{1/2}\exp{-\frac{1}{2}(\theta - 6.69)^{2} 0.785} \\
    &\approx (2\pi)^{-10} (\frac{1}{\sigma^2})^{1/2}\exp{-\frac{1}{2\sigma^2}(\theta - \theta^{MAP})^{2} }
\end{aligned}
\end{equation}

where $\frac{1}{\sigma^2}= 0.785,\theta^{MAP}=6.69 $

\newpage
\section{Problem 4}
Monte Carlo integration: Consider the following function,
\begin{equation}
    f(x) = x^{3} + 5x \cos{x}
\end{equation}

(a) Calculate the integral $I = \int_{a}^{b} f(x) d x$ with $a = 3$ and $b = 4$ using Monte Carlo integration with $N = 10000$ samples. Compare this value with the exact solution.

Answer:
\begin{equation}
\begin{aligned}
I_{exact} &= \int_{a}^{b} f(x) d x \\
&= \int_{a}^{b} (x^{3} + 5x \cos{x}) d x \\
&=  \left[\frac{x^{4}}{4} +5x\sin{x}+5\cos{x}\right]\mid_{a}^{b}  \\
& = \left[\frac{x^{4}}{4} +5x\sin{x}+5\cos{x}\right]\mid_{3}^{4}\\
& \approx 28.178894351627594
\end{aligned}
\end{equation}

Using Monte Carlo integration with $N=10,000$ samples.
using the python's function numpy.random.uniform(3,4,10000).

The integral $I$ from Monte Carlo integration
\begin{equation}
I_{MC} = 28.192521342751455 \approx  I_{exact}
\end{equation}
the error is about 0.05$\%$.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.6]{HW2_P4_a.png}
\caption{Monte Carlo integration for P4a with N= 10,000}
%\label{fig:universe}
\end{figure}

(b) Check the relation between the number of samples $N$ and solution accuracy by plotting the error for $N = [10,1000]$.

Answer:

%See Figure 3
\begin{figure}[ht]
\centering
\includegraphics[scale=0.5]{HW2_P4_b.png}
\caption{a. Integral I for N = [10,1000]
b. Error for N = [10,1000]}
%\label{fig:universe}
\end{figure}

From the plot, we find that when N is small, the error is very large, but when N increases, the error becomes smaller.

\newpage
(c) For $N = 100, 1000, 10000$ and $100000$ repeat the MC integration for  m = 10000 times. Plot the histogram of the results of MC integration for each $N$. Use the law of large numbers to justify the trend in the histograms.

Answer:

\begin{figure}[ht]
\centering
\includegraphics[scale=0.5]{HW2_P4_c.png}
\caption{Histogram of the results of MC for each N}
%\label{fig:universe}
\end{figure}

Using the law of large numbers to justify the trend in the histogram.
$Var[\Bar{X_{N}}] = \frac{\sigma^2}{N}$, therefore when N increases, $Var[\Bar{X_{N}}]=\frac{\sigma^2}{N}$ decreases, the histogram becomes more like a Gaussian distribution.

\newpage
\section{Problem 5}
Bayesian Information Criterion (BIC): suppose we toss a biased coin where probability of heads ($x=1$) is $\theta_{1}$. However, we only know about the outcome through an unreliable friend of ours, Joey, who can be trusted with a probability $\theta_{2}$. Let us call this report $y$. This means that we can write down $p(y|x,\theta_{2})$ as

(a) What is the joint probability distribution $p(x,y|\theta_{1},\theta_{2}$? Write your name in a table.

Answer:

\begin{center}
\begin{tabular}{ c c c }
   & y=0 & y=1 \\
x=0 & $(1-\theta_{1})\theta_{2}$ & $(1-\theta_{1})(1-\theta_{2})$ \\
x=1 & $\theta_{1}(1-\theta_{2})$ & $\theta_{1}\theta_{2}$
\end{tabular}
\end{center}

(b) Consider we have the outcomes
\begin{equation}
\begin{aligned}
    x = (1,1,0,1,1,0,0) \\
    x = (1,0,0,0,1,0,1)
\end{aligned}
\end{equation}

Find the maximum likelihood estimate for $\theta_{1}$ and $\theta_{2}$.

\begin{equation}
    p(\theta_{1},\theta_{2}|X,Y) = \theta_{1}^{4}(1-\theta_{1})^{3}\theta_{2}^{4}(1-\theta_{2})^{3}
\end{equation}

\begin{equation}
    \log p(\theta_{1},\theta_{2}|X,Y) = 4\log{\theta_{1}}+3\log{(1-\theta_{1})}+4\log{\theta_{2}}+3\log{(1-\theta_{2})}
\end{equation}

\begin{equation}
    \frac{\log p}{d \theta_{2}} = \frac{4}{\theta_{2}}-\frac{3}{1-\theta_{2}} = 0; \theta_{2}= \frac{4}{7}
\end{equation}

\begin{equation}
    \frac{\log p}{d \theta_{1}} = \frac{4}{\theta_{1}}-\frac{3}{1-\theta_{1}} = 0; \theta_{1}= \frac{4}{7}
\end{equation}

(c)We denote this model with $M_2$, where index 2 stands for the number of parameters in the model. Find $p(D|\hat{\theta_{1}},\hat{\theta_{2}},M_2)$ where $\hat{\theta}$ denotes the MLE solution for parameter $\theta$.

Answer:

From b, $\theta_{1}= \frac{4}{7}$ and $\theta_{2}= \frac{4}{7}$
\begin{center}
\begin{tabular}{ c c c }
   & y=0 & y=1 \\
x=0 & $\frac{12}{49}$ & $\frac{9}{49}$ \\
x=1 & $\frac{12}{49}$ & $\frac{16}{49}$
\end{tabular}
\end{center}

\begin{equation}
    p(X,Y|\theta_{1},\theta_{2},M_{2})= (\frac{4}{7})^{8}(\frac{3}{7})^{6} \approx 7.044 \times 10^{-5}
\end{equation}

(d) If we also denote a model with 4 parameters $\Bar{\theta} = (\theta_{0,0},\theta_{0,1},\theta_{1,0},\theta_{1,1})$ that represents $p(x,y|\Bar{\theta}) = \theta_{x,y}$. Find the MLE of $\Bar{\theta}$.

Answer:

\begin{center}
\begin{tabular}{ c c c }
   & y=0 & y=1 \\
x=0 & $\theta_{0,0}$ & $\theta_{0,1}$ \\
x=1 & $\theta_{1,0}$ & $\theta_{1,1}$
\end{tabular}
\end{center}

\begin{equation}
\begin{aligned}
    &p(\Bar{\theta}|X,Y) = \theta_{0,0}^{2} \theta_{0,1} \theta_{1,0}^{2} \theta_{1,1}^{2}\\
    &\theta_{0,1} = 1- \theta_{0,0}-\theta_{1,0}-\theta_{1,1}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    p(\Bar{\theta}|X,Y) = \theta_{0,0}^{2} (1- \theta_{0,0}-\theta_{1,0}-\theta_{1,1}) \theta_{1,0}^{2} \theta_{1,1}^{2}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    \log p = 2\log\theta_{0,0} +\log(1- \theta_{0,0}-\theta_{1,0}-\theta_{1,1}) +2\log \theta_{1,0} +2\log \theta_{1,1}
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    \frac{d \log p}{d \theta_{0,0}} = \frac{2}{\theta_{0,0}} - \frac{1}{1- \theta_{0,0}-\theta_{1,0}-\theta_{1,1}}=0
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    \frac{d \log p}{d \theta_{1,0}} = \frac{2}{\theta_{1,0}} - \frac{1}{1- \theta_{0,0}-\theta_{1,0}-\theta_{1,1}}=0
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    \frac{d \log p}{d \theta_{1,1}} = \frac{2}{\theta_{1,1}} - \frac{1}{1- \theta_{0,0}-\theta_{1,0}-\theta_{1,1}}=0
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
    & \theta_{0,0}=\frac{2}{7} \\
    & \theta_{0,1}=\frac{1}{7} \\
    & \theta_{1,0}=\frac{2}{7} \\
    & \theta_{1,1}=\frac{2}{7}
\end{aligned}
\end{equation}
(e) Find $p = (D|\hat{\theta}, M_4)$ where $\hat{\theta}$ denotes the MLE solution for parameters $\Bar{\theta}$.

Answer:

\begin{center}
\begin{tabular}{ c c c }
   & y=0 & y=1 \\
x=0 & $\frac{2}{7}$ & $\frac{1}{7}$ \\
x=1 & $\frac{2}{7}$ & $\frac{2}{7}$
\end{tabular}
\end{center}

\begin{equation}
    p(X,Y|\theta_{0,0},\theta_{0,1},\theta_{1,0},\theta_{1,1},M_{2})= (\frac{2}{7})^{6}(\frac{1}{7}) \approx 7.771 \times 10^{-5}
\end{equation}

(f) Find the Bayesian Information Criterion for $M_2$ and $M_4$. Which model is preferred by this criterion?

Answer:
From the Bayesian Information Criterion for $M_2$ and $M_4$,

For $M_2$,$k=2, N=7, L = p(X,Y|\theta_{1},\theta_{2},M_{2})= (\frac{4}{7})^{8}(\frac{3}{7})^{6} \approx 7.044 \times 10^{-5} $
\begin{equation}
    BIC =  k\log(N) -2 \log (L) \approx 23.01
\end{equation}

For $M_4$,$k=3$(there is a constrain for the four parameters, therefore only three parameters are free parameters, and k should be 3), $N=7$, $L =  p(X,Y|\theta_{0,0},\theta_{0,1},\theta_{1,0},\theta_{1,1},M_{2})= (\frac{2}{7})^{6}(\frac{1}{7}) \approx 7.771 \times 10^{-5} $
\begin{equation}
    BIC =  k\log(N) -2 \log (L) \approx 24.76
\end{equation}

Since $BIC_{M_4} > BIC_{M_2} $, $M_2$ is more preferred by this criterion.

\newpage
\section{Problem 6}
Maximum Likelihood Estimation (MLE) and Maximum A Posterior (MAP):
Consider a random variable $x$ described by

(a) Derive the maximum likelihood estimate (MLE) ($\lambda_{MLE}$)

Answer:
The likelihood is given by
\begin{equation}
p(x|\lambda) = \prod_{i=1}^{n} \lambda \exp{-\lambda x_{i}}
\end{equation}
The log likelihood:
\begin{equation}
    \ln{p} = n \ln{\lambda} -\lambda \sum{x_i}
\end{equation}
Now set set derivative w.r.t $\lambda$ to $0$:
\begin{equation}
    \frac{d \ln{p}}{d \lambda} = \frac{n}{\lambda} + \sum{x_i} =0
\end{equation}
Therefore,
\begin{equation}
    \lambda_{MLE} = \frac{1}{\Bar{x}}. \left( \Bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_{i} \right)
\end{equation}

(b)Obtain an analytic form of the posterior distribution of Eq.(11) and Derive the maximum a posterior estimator (MAP) $\lambda_{MAP}$ as a function of $\alpha,\beta$.

Answer:
Let us consider the data $X = x_{1},x_{2},...,x_{n}$. The posterior distribution $p(\lambda | X)$ is given by:
\begin{equation}
\begin{aligned}
p(\lambda | X) &= \frac{p(\lambda | X) p(\lambda)}{\int p(\lambda | X) p(\lambda)} \\
& \propto p(\lambda | X) p(\lambda) \\
& \lambda^{n} \exp{-\lambda \sum_{i=1}^{N} x_{i}} Gamma(\alpha,\beta) \\
& \lambda^{n} \exp{-\lambda \sum_{i=1}^{N} x_{i}} \lambda^{\alpha-1}\exp{-\beta \lambda} \\
& e^{-\lambda(\sum_{i=1}^{N} x_{i}+\beta)} \lambda^{n+\alpha-1}
\end{aligned}
\end{equation}

\begin{equation}
p(\lambda | X) \propto Gamma(\alpha+n,\sum_{i=1}^{N} x_{i}+\beta )
\end{equation}

The log posterior:
\begin{equation}
\log p(\lambda | X) \propto -\lambda(\sum_{i=1}^{N} x_{i}+\beta)+(n+\alpha-1)\log \lambda
\end{equation}

\begin{equation}
0 = \frac{d \log p(\lambda | X) }{d \lambda} = -(\sum_{i=1}^{N} x_{i}+\beta)+ \frac{n+\alpha-1}{\lambda}
\end{equation}

\begin{equation}
\lambda_{MAP}= \frac{n+\alpha-1}{\sum_{i=1}^{N} x_{i}+\beta}
\end{equation}

(c) Generate $N=20$ samples drawn from an exponential distribution with parameter $\lambda = 0.2$. Fix $\beta = 100$ and vary $\alpha$ over the range(1,40) using a step-size of 1.

Compute the corresponding MLE and MAP estimates for $\lambda$

For each $\alpha$, compute the mean squared $error^2$ of both estimates compared against the true value and then plot the mean squared error as a function of $\alpha$.

Now, fix $\alpha= 30$, $\theta=100$ and vary N over the range(1,500) using a step-size of 1. Plot the mean squared error for each N of the corresponding estimates and explain under what condition is the MAP estimator better.

\begin{figure}[ht]
\centering
\includegraphics[scale=0.5]{HW2_P6_c1.png}
\caption{MSE as a function of $\alpha$}
%\label{fig:universe}
\end{figure}

\begin{figure}[ht]
\centering
\includegraphics[scale=0.5]{HW2_P6_c2.png}
\caption{MSE as a function of N}
%\label{fig:universe}
\end{figure}

From Figure 6, when N is small, then the MAP estimator would be better than the MLE estimator.
%\bibliographystyle{plain}
%\bibliography{references}
\end{document}


\section{Variance of OLS estimators}


% ------------------------------------------------------------------
\chapter{Geometry}\label{s:geometry}
% ------------------------------------------------------------------

This chapter looks at the geometry of the CNN input-output mapping.

% ------------------------------------------------------------------
\section{Preliminaries}\label{s:preliminaries}
% ------------------------------------------------------------------

In this section we are interested in understanding how components in a CNN depend on components in the layers before it, and in particular on components of the input.  Since CNNs can incorporate blocks that perform complex operations, such as for example cropping their inputs based on data-dependent terms (e.g. Fast R-CNN), this information is generally available only at ``run time'' and cannot be uniquely determined given only the structure of the network. Furthermore, blocks can implement complex operations that are difficult to characterise in simple terms. Therefore, the analysis will be necessarily limited in scope.

We consider blocks such as convolutions for which one can deterministically establish dependency chains between network components. We also assume that all the inputs $\bx$ and outputs $\by$ are in the usual form of spatial maps, and therefore indexed as $x_{i,j,d,k}$ where $i,j$ are spatial coordinates.

Consider a layer $\by = f(\bx)$. We are interested in establishing which components of $\bx$ influence which components of $\by$. We also assume that this relation can be expressed in terms of a sliding rectangular window field, called \emph{receptive field}. This means that the output component  $y_{i'', j''}$ depends only on the input components $x_{i,j}$ where $(i,j) \in \Omega(i'', j'') $ (note that feature channels are implicitly coalesced in this discussion). The set $\Omega(i'',j'')$ is a rectangle defined as follows:
\begin{align}\label{e:receptive}
     i &\in \alpha_h (i'' -1) + \beta_h + \left[- \frac{\Delta_h-1}{2}, \frac{\Delta_h-1}{2}\right] \\
     j &\in \alpha_v (j'' -1) + \beta_v + \left[- \frac{\Delta_v-1}{2}, \frac{\Delta_v-1}{2}\right]
\end{align}
where $(\alpha_h,\alpha_v)$ is the \emph{stride}, $(\beta_h,\beta_v)$ the offset, and $(\Delta_h, \Delta_v)$ the \emph{receptive field size}.

% ------------------------------------------------------------------
\section{Simple filters}\label{s:receptive-simple-filters}
% ------------------------------------------------------------------

We now compute the receptive field geometry $(\alpha_h,\alpha_v,\beta_h,\beta_v,\Delta_h,\Delta_v)$ for the most common operators, namely filters. We consider in particular \emph{simple filters} that are characterised by an integer size, stride, and padding.

It suffices to reason in 1D.  Let $H'$ bet the vertical filter dimension, $S_h$ the subampling stride, and $P_h^-$ and $P_h^+$ the amount of zero padding applied to the top and the bottom of the input $\bx$. Here the value $y_{i''}$ depends on the samples:
\begin{align*}
 x_i : i
 &\in
 [1, H'] + S_h (i'' - 1) - P_h^-
=
\left[-\frac{H'-1}{2}, \frac{H'-1}{2}\right] + S_h (i''-1) - P_h^- + \frac{H'+1}{2}.
\end{align*}
Hence
\[
\alpha_h = S_h,
\qquad
\beta _h = \frac{H'+1}{2} - P_h^-,
\qquad
\Delta_h = H'.
\]
A similar relation holds for the horizontal direction.

Note that many blocks (e.g. max pooling, LNR, ReLU, most loss functions etc.) have a filter-like receptive field geometry. For example, ReLU can be considered a $1 \times 1$ filter, such that $H'=S_h=1$ and $P_h^-=P_h^+ =0$. Note that in this case $\alpha_h=1$, $\beta_h=1$ and $\Delta_h=1$.

In addition to computing the receptive field geometry, we are often interested in determining the sizes of the arrays $\bx$ and $\by$ throughout the architecture. In the case of filters, and once more reasoning for a 1D slice, we notice that $y_i''$ can be obtained for $i''=1,2,\dots,H''$ where $H''$ is the largest value of $i''$ before the receptive fields falls outside $\bx$ (including padding). If $H$ is the height of the input array $\bx$, we get the condition
\[
   H' + S_h (H'' - 1) - P_h^- \leq H + P_h^+.
\]
Hence
\begin{equation}\label{e:filtered-height}
   H'' = \left\lfloor \frac{H - H' + P_h^- + P_h^+}{S_h} \right\rfloor + 1.
\end{equation}

% ------------------------------------------------------------------
\subsection{Pooling in Caffe}
% ------------------------------------------------------------------

MatConvNet treats pooling operators like filters, using the rules above. In the library Caffe, this is done slightly differently, creating some incompatibilities. In their case, the pooling window is allowed to shift enough such that the last application always includes the last pixel of the input. If the stride is greater than one, this means that the last application of the pooling window can be partially outside the input boundaries even if padding is ``officially'' zero.

More formally, if $H'$ is the pool size and $H$ the size of the signal, the last application of the pooling window has index $i'' = H''$ such that
\[
  S_h(i''-1) + H' \big|_{i''= H''} \geq H
  \qquad
  \Leftrightarrow
  \qquad
  H'' = \left\lceil
  \frac{H - H'}{S_h}
  \right\rceil
  + 1.
\]
If there is padding, the same logic applies after padding the input image, such that the output has height:
\[
H'' = \left\lceil
  \frac{H - H' + P_h^- + P_h^+}{S_h}
  \right\rceil
  + 1.
\]
This is the same formula as for above filters, but with the ceil instead of floor operator. Note that in practice $P_h^- = P_h^+ = P_h$ since Caffe does not support asymmetric padding.

Unfortunately, it gets more complicated. Using the formula above, it can happen that the last padding application is completely outside the input image and Caffe tries to avoid it. This requires
\begin{equation}\label{e:pooling-caffe-constr}
  S_h(i'' - 1) - P_h^- + 1 \big|_{i''= H''} \leq H
  \qquad
  \Leftrightarrow
  \qquad
  H'' \leq \frac{H - 1 + P_h^-}{S_h} + 1.
\end{equation}

Using the fact that for integers $a,b$, one has $\lceil a/b \rceil = \lfloor (a+b-1)/b \rfloor$, we can rewrite the expression for $H''$ as follows
\begin{align*}
H'' = \left\lceil
  \frac{H - H' + P_h^- + P_h^+}{S_h}
  \right\rceil
  + 1
  =
  \left\lfloor
  \frac{H - 1 +P_h^-}{S_h}
  +
  \frac{P^+_h + S_h - H'}{S_h}
  \right\rfloor
  +1.
 \end{align*}
Hence if $P_h^+ +  S_h \leq H' $ then the second term is less than zero and \eqref{e:pooling-caffe-constr} is satisfied. In practice, Caffe assumes that $P_h^+, P_h^- \leq H' -1$, as otherwise the first filter application falls entirely in the padded region.  Hence, we can upper bound the second term:
\[
\frac{P^+_h + S_h - H'}{S_h}
\leq
\frac{S_h - 1}{S_h}
\leq
1.
\]
We conclude that, for any choices of $P_h^+$ and $S_h$ allowed by Caffe, the formula above may violate constraint \eqref{e:pooling-caffe-constr} by at most one unit. Caffe has a special provision for that and lowers $H''$ by one when needed. Furthermore, we see that if $P_h^+=0$ \emph{and} $S_h \leq H'$ (which is often the case and may be assumed by Caffe), then the equation is also satisfied and Caffe skips the check.

Next, we find MatConvNet equivalents for these parameters. Assume that Caffe applies a symmetric padding $P_h$. Then in MatConvNet $P_h^-=P_h$ to align the top part of the output signal. To match Caffe, the last sample of the last filter application has to be on or to the right of the last Caffe-padded pixel:
\[
\underbrace{
S_h
\left(
\underbrace
{
\left\lfloor
\frac{H - H' + P_h^- + P_h^+}{S_h}  + 1
\right\rfloor
}_{\text{MatConvNet rightmost pooling index}}
- 1
\right)
+ H'
}_{\text{MatConvNet rightmost pooled input sample}}
\geq
\underbrace{
H + 2P_h^-
}_{\text{Caffe rightmost input sample with padding}}.
\]
Rearranging
\[
\left\lfloor
\frac{H - H' + P_h^- + P_h^+}{S_h}
\right\rfloor
\geq
\frac{H - H' + 2P_h^{-}}{S_h}
\]
Using $\lfloor a/b \rfloor = \lceil (a - b + 1)/b\rceil$ we get the \emph{equivalent} condition:
\[
\left\lceil
\frac{H - H' + 2P_h^-}{S_h} + \frac{P_h^+ - P_h^- - S_h + 1}{S_h}
\right\rceil
\geq
\frac{H - H' + 2P_h^-}{S_h}
\]
Removing the ceil operator lower bounds the left-hand side of the equation and produces the \emph{sufficient} condition
\[
 P_h^+ \geq P_h^- + S_h - 1.
\]
As before, this may still be too much padding, causing the last pool window application to be entirely in the rightmost padded area. MatConvNet places the restriction $P_h^+ \leq H' -1$, so that
\[
  P_h^+ = \min\{ P_h^- + S_h - 1 , H' - 1\}.
\]
For example, a pooling region of width $H'=3$ samples with  a stride of $S_h=1$ samples and null Caffe padding $P_h^-=0$, would result in a right MatConvNet padding of $P_h^+ = 1$.

% ------------------------------------------------------------------
\section{Convolution transpose}\label{s:receptive-convolution-transpose}
% ------------------------------------------------------------------

The convolution transpose block is similar to a simple filter, but somewhat more complex. Recall that convolution transpose (\cref{s:impl-convolution-transpose}) is the transpose of the convolution operator, which in turn is a filter. Reasoning for a 1D slice, let $x_i$ be the input to the convolution transpose block and $y_{i''}$ its output. Furthermore let $U_h$, $C_h^-$, $C_h^+$ and $H'$ be the upsampling factor, top and bottom crops, and filter height, respectively.

If we look at the convolution transpose backward, from the output to the input (see also \cref{f:convt}), the data dependencies are the same as for the convolution operator, studied in \cref{s:receptive-simple-filters}. Hence there is an interaction between $x_i$ and $y_{i''}$ only if
\begin{equation}\label{e:convt-bounds}
   1 + U_h(i - 1) - C_h^- \leq i'' \leq H' + U_h(i - 1) - C_h^-
\end{equation}
where cropping becomes padding and upsampling becomes downsampling. Turning this relation around, we find that
\[
 \left\lceil \frac{i'' + C_h^- -H'}{U_h} \right\rceil + 1
 \leq
 i
 \leq
 \left\lfloor \frac{i'' + C_h^- - 1}{U_h} \right\rfloor + 1 .
\]
Note that, due to rounding, it is not possible to express this set tightly in the form outlined above. We can however relax these two relations (hence obtaining a slightly larger receptive field) and conclude that
\[
\alpha_h = \frac{1}{U_h},
\qquad
\beta_h = \frac{2C_h^- - H' + 1}{2 U_h} + 1,
\qquad
\Delta_h = \frac{H' -1}{U_h} + 1.
\]

Next, we want to determine the height $H''$ of the output $\by$ of convolution transpose as a function of the heigh $H$ of the input $\bx$ and the other parameters. Swapping input and output in  \eqref{e:filtered-height} results in the constraint:
\[
H = 1+ \left\lfloor \frac{H'' - H' + C_h^- + C_h^+}{U_h} \right\rfloor.
\]
If $H$ is now given as input, it is not possible to recover $H''$ uniquely from this expression; instead, all the following values are possible
\[
   U_h (H-1) +H' -  C_h^- - C_h^+ \leq H'' < U_h H +H' -  C_h^- - C_h^+.
\]
This is due to the fact that $U_h$ acts as a downsampling factor in the standard convolution direction and some of the samples to the right of the convolution input $\by$ may be ignored by the filter (see also \cref{f:conv} and \cref{f:convt}).

Since the height of $\by$ is then determined up to $S_h$ samples, and since the extra samples would be ignored by the computation and stay zero, we choose the tighter definition and set
\[
H'' =  U_h (H-1) +H' -  C_h^- - C_h^+.
\]

% ------------------------------------------------------------------
\section{Transposing receptive fields}\label{s:receptive-transposing}
% ------------------------------------------------------------------

Suppose we have determined that a later $\by = f(\bx)$ has a receptive field transformation $(\alpha_h,\beta_h,\Delta_h)$ (along one spatial slice). Now suppose we are given a block $\bx = g(\by)$ which is the ``transpose'' of $f$, just like the convolution transpose layer is the transpose of the convolution layer. By this, we mean that, if $y_{i''}$ depends on $x_{i}$ due to $f$, then $x_{i}$ depends on $y_{i''}$ due to $g$.

Note that, by definition of receptive fields, $f$ relates the  inputs and outputs index pairs $(i,i'')$ given by \eqref{e:receptive}, which can be rewritten as
\[
- \frac{\Delta_h-1}{2} \leq  i - \alpha_h (i'' -1) - \beta_h \leq\frac{\Delta_h-1}{2}.
\]
A simple manipulation of this expression results in the equivalent expression:
\[
- \frac{(\Delta_h + \alpha_h - 1)/\alpha_h-1}{2} \leq  i'' - \frac{1}{\alpha_h} (i - 1) - \frac{1 + \alpha_h - \beta_h }{\alpha_h} \leq\frac{(\Delta_h + \alpha_h - 1)/\alpha_h-1}{2\alpha_h}.
\]
Hence, in the reverse direction, this corresponds to a RF transformation
\[
\hat \alpha_h = \frac{1}{\alpha_h},
\qquad
\hat \beta_h = \frac{1 + \alpha_h - \beta_h}{\alpha_h},
\qquad
\hat \Delta_h = \frac{\Delta_h + \alpha_h -1}{\alpha_h}.
\]

\begin{example}
For convolution, we have found the parameters:
\[
\alpha_h = S_h,
\qquad
\beta_h = \frac{H'+1}{2} - P_h^-,
\qquad
\Delta_h = H'.
\]
Using the formulas just found, we can obtain the RF transformation for convolution transpose:
\begin{align*}
\hat \alpha_h &= \frac{1}{\alpha_h} = \frac{1}{S_h},
\\
\hat \beta_h &= \frac{1 + S_h - (H'+1)/2 + P_h^-}{S_h}
= \frac{P_h^- -H'/2 +1/2}{S_h} + 1
= \frac{2P_h^- -H' + 1}{S_h} + 1,
\\
\hat \Delta_h &= \frac{H' + S_h - 1}{S_h} = \frac{H' -1}{S_h} + 1.
\end{align*}
Hence we find again the formulas obtained in \cref{s:receptive-convolution-transpose}.
\end{example}

% ------------------------------------------------------------------
\section{Composing receptive fields}\label{s:receptive-composing}
% ------------------------------------------------------------------

Consider now the composition of two layers $h = g \circ f$ with receptive fields $(\alpha_f, \beta_f, \Delta_f)$ and $(\alpha_g, \beta_g, \Delta_g)$ (once again we consider only a 1D slice in the vertical direction, the horizontal one being the same). The goal is to compute the receptive field of $h$.

To do so, pick a sample $i_g$ in the domain of $g$. The first and last sample $i_f$ in the domain of $f$ to affect $i_g$ are given by:
\[
  i_f = \alpha_f (i_g- 1) + \beta_f \pm \frac{\Delta_f - 1}{2}.
\]
Likewise, the first and last sample $i_g$ to affect a given output sample $i_h$ are given by
\[
  i_g = \alpha_g (i_h- 1) + \beta_g \pm \frac{\Delta_g - 1}{2}.
\]
Substituting one relation into the other, we see that the first and last sample $i_f$ in the domain of $g \circ f$ to affect $i_h$ are:
\begin{align*}\
 i_f &= \alpha_f \left(\alpha_g (i_h- 1) + \beta_g \pm \frac{\Delta_g - 1}{2} - 1\right) + \beta_f \pm \frac{\Delta_f - 1}{2}
 \\
&= \alpha_f\alpha_g (i_h-1)
 + \alpha_f (\beta_g - 1) + \beta_f
 \pm \frac{\alpha_f (\Delta_g - 1) + \Delta_f -1}{2}.
\end{align*}
We conclude that
\[
\alpha_h = \alpha_f \alpha_g,
\qquad
\beta_h =  \alpha_f (\beta_g - 1) + \beta_f,
\qquad
\Delta_h = \alpha_f (\Delta_g - 1) + \Delta_f.
\]

% ------------------------------------------------------------------
\section{Overlaying receptive fields}\label{s:receptive-overlying}
% ------------------------------------------------------------------

Consider now the combination $h(f(\bx_1), g(\bx_2))$ where the domains of $f$ and $g$ are the same. Given the rule above, it is possible to compute how each output sample $i_h$ depends on each input sample $i_f$ through $f$ and on each input sample $i_g$ through $g$. Suppose that this gives receptive fields $(\alpha_{hf}, \beta_{hf}, \Delta_{hf})$ and $(\alpha_{hg}, \beta_{hg}, \Delta_{hg})$ respectively. Now assume that the domain of $f$ and $g$ coincide, i.e. $\bx = \bx_1 = \bx_2$. The goal is to determine the combined receptive field.

This is only possible if, and only if, $\alpha = \alpha_{hg} = \alpha_{hf}$. Only in this case, in fact, it is possible to find a sliding window receptive field that tightly encloses the receptive field due to $g$ and $f$ at all points according to formulas~\eqref{e:receptive}. We say that these two receptive fields are \emph{compatible}. The range of input samples $i = i_f = i_g$ that affect any output sample $i_h$ is then given by
\begin{align*}
	  i_\text{max}&=
  \alpha (i_h- 1) + a, & a = \min
  \left\{\beta_{hf}- \frac{\Delta_{hf} - 1}{2}, \beta_{hg} - \frac{\Delta_{hg} - 1}{2}\right\},
  \\
  	  i_\text{min} &=
  \alpha (i_h- 1) + b, & b = \max
  \left\{\beta_{hf}+ \frac{\Delta_{hf} - 1}{2}, \beta_{hg} + \frac{\Delta_{hg} - 1}{2}\right\}.
\end{align*}
We conclude that the combined receptive field is
\[
\alpha = \alpha_{hg} = \alpha_{hf},
\qquad
\beta = \frac{a+b}{2},
\qquad
\Delta = b - a + 1.
\]


\subsection{Vertical velocity closure}
The hydrostatic momentum equations are written in terms of horizontal velocities, combined with mass conservation over the water column. Often it is necessary to recover the compatible conservative 3D velocity field, either for diagnostic purposes or to feed into the transport equations.

SELFE infers vertical velocities by integrating the 3D mass conservation equation (\ref{3Dcon})
from the bed to the free surface. The process is illustrated in Figure \ref{fig:vel_close}. Starting at the bed, a suitable boundary condition (e.g. zero flow) is assumed. This is combined with horizontal flows across the three vertical faces of the prism as well as the component of flow flows directed across the (very slightly tilted) top face of the
prism. Given that water is incompressible, what comes out must be the vertical component of flow up out of the top face. The
calculation then proceeds to the prism above.

\begin{figure}
	\centering
		\includegraphics[scale=1.0]{image/vel_closure}
	\caption{First closure calculation on bottom prism. The horizontal velocities at the three vertical faces and the horzizontal component of velocity normal to the top face are shown and designated $(u,v)$. The vertical velocity $w$ is obtained from the
	net flux. Movement of the S grid (typically a small fraction of a millimeter per time step) is not considered.}
	\label{fig:vel_close}
\end{figure}