\documentclass[a4paper, 12pt]{article}

\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage[]{amsfonts}
\usepackage[]{graphicx}

\title{CS231A Course Notes 3: Epipolar Geometry}
\author{Kenji Hata and Silvio Savarese}
\date{}

\renewcommand\emph{\textbf}

\begin{document}

\maketitle

\section{Introduction}
Previously, we have seen how to compute the intrinsic and extrinsic parameters of a camera using one or more views using a typical camera calibration procedure or single view metrology. This process culminated in deriving properties about the 3D world from one image. However, in general, it is not possible to recover the entire structure of the 3D world from just one image. This is due to the intrinsic ambiguity of the 3D to the 2D mapping: some information is simply lost.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/pisa.jpg}
\caption{A single picture such as this picture of a man holding up the Leaning Tower of Pisa can result in ambiguous scenarios. Multiple views of the same scene help us resolve these potential ambiguities.}
\label{fig:pisa}
\end{figure}

For example, in Figure~\ref{fig:pisa}, we may be initially fooled to believe that the man is holding up the Leaning Tower of Pisa. Only by careful inspection can we tell that this is not the case and merely an illusion based on the projection of different depths onto the image plane. However, if we were able to view this scene from a completely different angle, this illusion immediately disappears and we would instantly figure out the correct scene layout.

The focus of these lecture notes is to show how having knowledge of geometry when multiple cameras are present can be extremely helpful. Specifically, we will first focus on defining the geometry involved in two viewpoints and then present how this geometry can aid in further understanding the world around us.

\section{Epipolar Geometry}
\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/epipolar.png}
\caption{The general setup of epipolar geometry. The gray region is the epipolar plane. The orange line is the baseline, while the two blue lines are the epipolar lines.}
\label{fig:epipolar}
\end{figure}
Often in multiple view geometry, there are interesting relationships between the multiple cameras, a 3D point, and that point's projections in each of the camera's image plane. The geometry that relates the cameras, points in 3D, and the the corresponding observations is referred to as the \emph{epipolar geometry} of a stereo pair.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/epipolar_example.png}
\caption{An example of epipolar lines and their corresponding points drawn on an image pair.}
\label{fig:epipolar_example}
\end{figure}
As illustrated in Figure~\ref{fig:epipolar}, the standard epipolar geometry setup involves two cameras observing the same 3D point $P$, whose projection in each of the image planes is located at $p$ and $p'$ respectively. The camera centers are located at $O_1$ and $O_2$, and the line between them is referred to as the \emph{baseline}. We call the plane defined by the two camera centers and $P$ the \emph{epipolar plane}. The locations of where the baseline intersects the two image planes are known as the the \emph{epipoles} $e$ and $e'$. Finally, the lines defined by the intersection of the epipolar plane and the two image planes are known as the \emph{epipolar lines}. The epipolar lines have the property that they intersect the baseline at the respective epipoles in the image  plane.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/epipolar_parallel.png}
\caption{When the two image planes are parallel, then the epipoles $e$ and $e'$ are located at infinity. Notice that the epipolar lines are parallel to the $u$ axis of each image plane.}
\label{fig:epipolar_parallel}
\end{figure}

An interesting case of epipolar geometry is shown in Figure~\ref{fig:epipolar_parallel}, which occurs when the image planes are parallel to each other. When the image planes are parallel to each other, then the epipoles $e$ and $e'$ will be located at infinity since the baseline joining the centers $O_1,O_2$ is parallel to the image planes. Another important byproduct of this case is that the epipolar lines are parallel to an axis of each image plane. This case is especially useful and will be covered in greater detail in the subsequent section on image rectification.

In real world situations, however, we are not given the exact location of the 3D location $P$, but can determine its projection in one of the image planes $p$. We also should be able to know the cameras locations, orientations, and camera matrices. What can we do with this knowledge? With the knowledge of camera locations $O_1, O_2$ and the image point $p$, we can define the epipolar plane. With this epipolar plane, we can then determine the epipolar lines\footnote{This means that epipolar lines can be determined by just knowing the camera centers $O_1, O_2$ and a point in one of the images $p$}. By definition, $P$'s projection into the second image $p'$ must be located on the epipolar line of the second image. Thus, a  basic understanding of epipolar geometry allows us to create a strong constraint between image pairs without knowing the 3D structure of the scene.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/essential_fundamental_setup.png}
\caption{The setup for determining the essential and fundamental matrices, which help map points and epipolar lines across views.}
\label{fig:essential_fundamental_setup}
\end{figure}

We will now try to develop seamless ways to map points and epipolar lines across views. If we take the setup given in the original epipolar geometry framework (Figure~\ref{fig:essential_fundamental_setup}), then we shall further define $M$ and $M'$ to be the camera projection matrices that map 3D points into their respective 2D image plane locations. Let us assume that the world reference system is associated to the first camera with the second camera offset first by a rotation $R$ and then by a translation $T$. This specifies the camera projection matrices to be:
\begin{equation}
    M = K\begin{bmatrix} I & 0\end{bmatrix} \ \ \ \ \ M' = K'\begin{bmatrix}R^T & -R^TT\end{bmatrix}
    \label{eq:projection_matrices}
\end{equation}

\section{The Essential Matrix}
In the simplest case, let us assume that we have \emph{canonical cameras}, in which $K = K' = I$. This reduces Equation~\ref{eq:projection_matrices} to

\begin{equation}
    M = \begin{bmatrix} I & 0\end{bmatrix} \ \ \ \ \ M' = \begin{bmatrix}R^T & -R^TT\end{bmatrix}
    \label{eq:projection_matrices_canonical}
\end{equation}

Furthermore, this means that the location of $p'$ in the first camera's reference system is $Rp' + T$. Since the vectors $Rp' + T$ and $T$ lie in the epipolar plane, then if we take the cross product of $T\times(Rp' + T) = T \times(Rp')$, we will get a vector normal to the epipolar plane. This also means that $p$, which lies in the epipolar plane is normal to $T \times(Rp')$, giving us the constraint that their dot product is zero:
\begin{equation}
    \begin{split}
      p^T\cdot[T \times(Rp')] = 0\\
    \end{split}
    \label{eq:essential1}
\end{equation}

From linear algebra, we can introduce a different and compact expression for the cross product: we can represent the cross product between any two vectors $a$ and $b$ as a matrix-vector multiplication:
\begin{equation}
    \mathbf{a}\times \mathbf{b} = \begin{bmatrix} 0 & -\mathbf{a}_z & \mathbf{a}_y \\ \mathbf{a}_z & 0 & -\mathbf{a}_x \\ -\mathbf{a}_y & \mathbf{a}_x & 0\end{bmatrix}\begin{bmatrix} \mathbf{b}_x \\ \mathbf{b}_y \\ \mathbf{b}_z\end{bmatrix} = [\mathbf{a}_\times] \mathbf{b}
\end{equation}

Combining this expression with Equation~\ref{eq:essential1}, we can convert the cross product term into matrix multiplication, giving
\begin{equation}
    \begin{split}
      p^T\cdot[T_\times] (Rp') = 0\\
    	  p^T [T_\times] R p' = 0 \\
    \end{split}
\end{equation}
The matrix $E = [T_\times]R$ is known as the \emph{Essential Matrix}, creating a compact expression for the epipolar constraint:
\begin{equation}
    p^T E p' = 0
\end{equation}
The Essential matrix is a $3\times3$ matrix that contains 5 degrees of freedom. It has rank 2 and is singular.

The Essential matrix is useful for computing the epipolar lines associated with $p$ and $p'$. For instance, $\ell' = E^Tp$ gives the epipolar line in the image plane of camera 2. Similarly $\ell = Ep'$ gives the epipolar line in the image plane of camera 1. Other interesting properties of the essential matrix is that its dot product with the epipoles equate to zero: $E^Te = Ee' = 0$.
%TODO: WHY???
Because for any point $x$ (other than $e$) in the image of camera 1, the corresponding epipolar line in the image of camera 2, $l'=E^Tx$, contains the epipole $e'$. Thus $e'$ satisfies $e'^T(E^Tx) = (e'^TE^T)x=0$ for all the $x$, so $e'^TE=0$. Similarly $E^Te=0$.

%TODO: Section on deriving the Rotation and Translation from E

\section{The Fundamental Matrix}
Although we derived a relationship between $p$ and $p'$ when we have canonical cameras, we should be able to find a more general expression when the cameras are no longer canonical. Recall that gives us the projection matrices:

\begin{equation}
    M = K\begin{bmatrix} I & 0\end{bmatrix} \ \ \ \ \ M' = K'\begin{bmatrix}R^T & -R^TT\end{bmatrix}
    \label{eq:projection_matrices2}
\end{equation}

First, we must define $p_c = K^{-1}p$ and $p_c' = K'^{-1}p'$ to be the projections of $P$ to the corresponding camera images if the cameras were canonical. Recall that in the canonical case:
\begin{equation}
    p^T_c [T_\times]Rp'_c = 0
    \label{eq:canonical_constraint}
\end{equation}
By substituting in the values of $p_c$ and $p_c'$, we get
\begin{equation}
    p^TK'^{-T} [T_\times]RK^{-1}p' = 0
    \label{eq:canonical_constraint_substitute}
\end{equation}
The matrix $F = K'^{-T} [T_\times]RK^{-1}$ is known as the \emph{Fundamental Matrix}, which acts similar to the Essential matrix from the previous section but also encodes information about the camera matrices $K, K'$ and the relative translation $T$ and rotation $R$ between the cameras. Therefore, it is also useful in computing the epipolar lines associated with $p$ and $p'$, even when the camera matrices $K, K'$ and the transformation $R,T$ are unknown. Similar to the Essential matrix, we can compute the epipolar lines $\ell' = F^Tp$ and $\ell = F p'$ from just the Fundamental matrix and the corresponding points. One main difference between the Fundamental matrix and the Essential matrix is that the Fundamental matrix contains 7 degrees of freedom, compared to the Essential matrix's 5 degrees of freedom.

But how is the Fundamental matrix useful? Like the Essential matrix, if we know the Fundamental matrix, then simply knowing a point in an image gives us an easy constraint (the epipolar line) of the corresponding point in the other image. Therefore, without knowing the actual position of $P$ in 3D space, or any of the extrinsic or intrinsic characteristics of the cameras, we can establish a relationship between any $p$ and $p'$.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/correspondence.png}
\caption{Corresponding points are drawn in the same color on each of the respective images.}
\label{fig:correspondence}
\end{figure}

\subsection{The Eight-Point Algorithm}
Still, the assumption that we can have the Fundamental matrix, which is defined by a matrix product of the camera parameters, seems rather large. However, it is possible to estimate the Fundamental matrix given two images of the same scene and without knowing the extrinsic or intrinsic parameters of the camera. The method we discuss for doing so is known as the \emph{Eight-Point Algorithm}, which was proposed by Longuet-Higgins in 1981 and extended by Hartley in 1995. As the title suggests, the Eight-Point Algorithm assumes that a set of at least 8 pairs of corresponding points between two images is available.

Each correspondence $p_i = (u_i, v_i, 1)$ and $p'_i = (u_i', v_i', 1)$ gives us the epipolar constraint $p_i'^T F p_i = 0$. We can reformulate the constraint as follows:
\begin{equation}
    \begin{bmatrix}u_iu_i' & v_iu_i' & u'_i & u_iv'_i &  v_iv_i' & v_i' & u_i & v_i & 1\end{bmatrix}
    \begin{bmatrix}F_{11}\\F_{12}\\F_{13}\\F_{21}\\F_{22}\\F_{23}\\F_{31}\\F_{32}\\F_{33}\end{bmatrix} = 0
\end{equation}

Since this constraint is a scalar equation, it only constrains one degree of freedom. Since we can only know the Fundamental matrix up to scale, we require eight of these constraints to determine the Fundamental matrix:

\begin{equation}
    \begin{bmatrix}
    u_1u_1' & v_1u_1' & u'_1 & u_1v'_1 &  v_1v_1' & v_1' & u_1 & v_1 & 1 \\
    u_2u_2' & v_2u_2' & u'_2 & u_2v'_2 &  v_2v_2' & v_2' & u_2 & v_2 & 1 \\
    u_3u_3' & v_3u_3' & u'_3 & u_3v'_3 &  v_3v_3' & v_3' & u_3 & v_3 & 1 \\
    u_4u_4' & v_4u_4' & u'_4 & u_4v'_4 &  v_4v_4' & v_4' & u_4 & v_4 & 1\\
    u_5u_5' & v_5u_5' & u'_5 & u_5v'_5 &  v_5v_5' & v_5' & u_5 & v_5 & 1\\
    u_6u_6' & v_6u_6' & u'_6 & u_6v'_6 &  v_6v_6' & v_6' & u_6 & v_6 & 1 \\
    u_7u_7' & v_7u_7' & u'_7 & u_7v'_7 &  v_7v_7' & v_7' & u_7 & v_7 & 1 \\
    u_8u_8' & v_8u_8' & u'_8 & u_8v'_8 &  v_8v_8' & v_8' & u_8 & v_8 & 1
    \end{bmatrix}
    \begin{bmatrix}F_{11}\\F_{12}\\F_{13}\\F_{21}\\F_{22}\\F_{23}\\F_{31}\\F_{32}\\F_{33}\end{bmatrix} = 0
\end{equation}
This can be compactly written as
\begin{equation}
    W\mathbf{f} = 0
    \label{eq:compact_8}
\end{equation}
where $W$ is an $N\times 9$ matrix derived from $N \geq 8$ correspondences and $\mathbf{f}$ is the values of the Fundamental matrix we desire.

In practice, it often is better to use more than eight correspondences and create a larger $W$ matrix because it reduces the effects of noisy measurements. The solution to this system of homogeneous equations can be found in the least-squares sense by Singular Value Decomposition (SVD), as $W$ is rank-deficient. SVD will give us a estimate of the Fundamental matrix $\hat{F}$, which may have full rank. However, we know that the true Fundamental matrix has rank 2. Therefore, we should look for a solution that is the best rank-2 approximation of $\hat{F}$. To do so, we solve the following optimization problem:
\begin{equation}
\begin{aligned}
& \underset{F}{\text{minimize}}
& & \|F-\hat{F}\|_\mathrm{F}  \\
& \text{subject to}
& & \det F = 0
\end{aligned}
\end{equation}
This problem is solved again by SVD, where $\hat{F} = U\Sigma V^T$, then the best rank-2 approximation is found by
\begin{equation}
F = U\begin{bmatrix}\Sigma_1&0&0\\0&\Sigma_2&0\\0&0&0\end{bmatrix}V^T
\end{equation}

\subsection{The Normalized Eight-Point Algorithm}
In practice, the standard least-squares approach to the Eight-Point Algorithm is not precise. Often, the distance between a point $p_i$ and its corresponding epipolar line $\ell_i = F^Tp'$ will be very large, usually on the scale of 10+ pixels. To reduce this error, we can consider a modified version of the Eight-Point Algorithm called the \emph{Normalized Eight-Point Algorithm}.

The main problem of the standard Eight-Point Algorithm stems from the fact that $W$ is ill-conditioned for SVD. For SVD to work properly, $W$ should have one singular value equal to (or near) zero, with the other singular values being nonzero. However, the correspondences $p_i = (u_i, v_i, 1)$ will often have extremely large values in the first and second coordinates due to the pixel range of a modern camera (i.e. $p_i = (1832, 1023, 1))$. If the image points used to construct $W$ are in a relatively small region of the image, then each of the vectors for $p_i$ and $p_i'$ will generally be very similar. Consequently, the constructed $W$ matrix will have one very large singular value, with the rest relatively small.

To solve this problem, we will normalize the points in the image before constructing $W$. This means we \emph{pre-condition} $W$ by applying both a translation and scaling on the image coordinates such that two requirements are satisfied. First, the origin of the new coordinate system should be located at the centroid of the image points (translation). Second, the mean square distance of the transformed image points from the origin should be 2 pixels (scaling). We can compactly represent this process by a transformation matrices $T, T'$ that translate by the centroid and scale by the scaling factor $\sqrt{\frac{2}{\mathrm{mean\ distance}}}$ for each respective image.

Afterwards, we normalize the coordinates:
\begin{equation}
q_i = Tp_i \ \ \ \ \ q_i' = T'p_i'
\end{equation}
Using the new, normalized coordinates, we can compute the new $F_q$ using the regular least-squares Eight Point Algorithm. However, the matrix $F_q$ is the fundamental matrix for the normalized coordinates. For it to be usable on regular coordinate space, we need to de-normalize it, giving
\begin{equation}
F = T'^TF_qT
\end{equation}
Ultimately, this new Fundamental matrix $F$ gives good results in real-world applications.

\section{Image Rectification}

Recall that an interesting case for epipolar geometry occurs when two images are parallel to each other. Let us first compute the Essential matrix $E$ in the case of parallel image planes. We can assume that the two cameras have the same $K$ and that there is no relative rotation between the cameras ($R= I$). In this case, let us assume that there is only a translation along the $x$ axis, giving $T = (T_x, 0, 0)$. This gives
\begin{equation}
E = [T_\times] R = \begin{bmatrix} 0 & 0 & 0\\ 0 & 0 & -T_x \\ 0 & T_x  & 0 \end{bmatrix}
\end{equation}

Once $E$ is known, we can find the directions of the epipolar lines associated with points in the image planes. Let us compute the direction of the epipolar line $\ell$ associated with point $p'$:
\begin{equation}
\ell = E^Tp' = \begin{bmatrix} 0 & 0 & 0\\ 0 & 0 & T_x \\ 0 & -T_x  & 0 \end{bmatrix} \begin{bmatrix}u'\\v'\\1\end{bmatrix} = \begin{bmatrix} 0\\ T_x \\ -T_xv'\end{bmatrix}
\end{equation}
We can see that the direction of $\ell$ is horizontal, as is the direction of $\ell'$, which is computed in a similar manner.
\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/rectification_example.png}
\caption{The process of image rectification involves computing two homographies that we can apply to a pair of images to make them parallel.}
\label{fig:rectification_example}
\end{figure}

If we use the epipolar constraint $p'^T Ep=0$, then we arrive at the fact that $v=v'$, demonstrating that $p$ and $p'$ share the same $v$-coordinate. Consequently, there exists a very straightforward relationship between the corresponding points. Therefore, \emph{rectification}, or the process of making any two given images parallel, becomes useful when discerning the relationships between corresponding points in images.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/rectification_setup.jpg}
\caption{The rectification problem setup: we compute two homographies that we can apply to the image planes to make the resulting planes parallel.}
\label{fig:rectification_setup}
\end{figure}

Rectifying a pair of images does not require knowledge of the two camera matrices $K, K'$ or the relative transformation $R,T$ between them. Instead, we can use the Fundamental matrix estimated by the Normalized Eight Point Algorithm. Upon getting the Fundamental matrix, we can compute the epipolar lines $\ell_i$ and $\ell_i'$ for each correspondence $p_i$ and $p_i'$.

From the set of epipolar lines, we can then estimate the epipoles $e$ and $e'$ of each image. This is because we know that the epipole lies in the intersection of all the epipolar lines. In the real world, due to noisy measurements, all the epipolar lines will not intersect in a single point. Therefore, computing the epipole can be found by minimizing the least squared error of fitting a point to all the epipolar lines. Recall that each epipolar line can be represented as a vector $\ell$ such that all points on the line (represented in homogeneous coordinates) are in the set $\{x | \ell^Tx = 0\}$. If we define each epipolar line as $\ell_i = \begin{bmatrix}\ell_{i,1} & \ell_{i,2} & \ell_{i,3} \end{bmatrix}^T$, then we can we formulate a linear system of equations and solve using SVD to find the epipole $e$:
\begin{equation}
    \begin{bmatrix}
    \ell_{1}^T\\
    \vdots\\
    \ell_{n}^T
    \end{bmatrix}e = 0
\end{equation}

After finding the epipoles $e$ and $e'$, we will most likely notice that they are not points at infinity along the horizontal axis. If they were, then, by definition, the images would already be parallel. Thus, we gain some insight into how to make the images parallel: can we find a homography to map an epipole $e$ to infinity along the horizontal axis? Specifically, this means that we want to find a pair of homographies $H_1, H_2$ that we can apply to the images to map the epipoles to infinity. Let us start by finding a homography $H_2$ that maps the second epipole $e'$ to a  point on the horizontal axis at infinity $(f,0,0)$. Since there are many possible choices for this homography, we should try to choose something reasonable. One condition that leads to good results in practice is to insist that the homography acts like a transformation that applies a translation and rotation on points near the center of the image.

The first step in achieving such a transformation is to translate the second image such that the center is at $(0,0,1)$ in homogeneous coordinates. We can do so by applying the translation matrix
\begin{equation}
    T = \begin{bmatrix}
    1&0&-\frac{\mathrm{width}}{2}\\0&1&-\frac{\mathrm{height}}{2}\\ 0 &0 &1
    \end{bmatrix}
\end{equation}
After applying the translation, we apply a rotation to place the epipole on the horizontal axis at some point $(f,0,1)$. If the translated epipole $Te'$ is located at homogeneous coordinates $(e'_1, e'_2, 1)$, then the rotation applied is
\begin{equation}
    R =
    \begin{bmatrix}
        \alpha\frac{e'_1}{\sqrt{e_1'^2+e_2'^2}} & \alpha\frac{e'_2}{\sqrt{e_1'^2+e_2'^2}} & 0 \\ -\alpha\frac{e'_2}{\sqrt{e_1'^2+e_2'^2}} &\alpha \frac{e'_1}{\sqrt{e_1'^2+e_2'^2}} & 0\\ 0&0&1
    \end{bmatrix}
\end{equation}
where $\alpha=1$ if $e'_1\geq 0$ and $\alpha=-1$ otherwise. Value of $f$ can be computed by applying the calculated rotation matrix $R$ to the translated epipole $Te'$. This point will always of the form $(f,0,1)$. After applying this rotation, notice that given any point at $(f,0,1)$, bringing it to a point at infinity on the horizontal axis $(f,0,0)$ only requires applying a transformation
\begin{equation}
    G =
    \begin{bmatrix}
    1 & 0 &0 \\ 0 & 1 & 0 \\ -\frac{1}{f} & 0 & 1
    \end{bmatrix}
\end{equation}
After applying this transformation, we finally have an epipole at infinity, so we can translate back to the regular image space. Thus, the homography $H_2$ that we apply on the second image to rectify it is
\begin{equation}
    H_2 = T^{-1}GRT
\end{equation}

Now that a valid $H_2$ is found, we need to find a matching homography $H_1$ for the first image. We do so by finding a transformation $H_1$ that minimizes the sum of square distances between the corresponding points of the images
\begin{equation}
    \arg \min_{H_1} \sum_i \|H_1p_i - H_2 p_i'\|^2
    \label{eq:rectification_minimization}
\end{equation}
Although the derivation\footnote{If you are interested in the details, please see Chapter 11 of Hartley \& Zisserman's textbook \textit{Multiple View Geometry}} is outside the scope of this class, we can actually prove that the matching $H_1$ is of the form:
\begin{equation}
    H_1 = H_A H_2 M
\label{eq:matching_homography}
\end{equation}
where $F= [e]_\times M$ and
\begin{equation}
    H_A = \begin{bmatrix} a_1 & a_2 & a_3\\ 0 & 1 & 0 \\ 0 & 0 &1\end{bmatrix}
\end{equation}
with $(a_1, a_2, a_3)$ composing the elements of a certain vector $\mathbf{a}$ that will be computed later.

First, we need to know what $M$ is. An interesting property of any $3\times3$ skew-symmetric matrix $A$ is $A = A^3$ up to scale. Because any cross product matrix $[e]_\times$ is skew-symmetric and that we can only know the Fundamental matrix $F$ up to scale, then
\begin{equation}
    F = [e]_\times M = [e]_\times [e]_\times [e]_\times M = [e]_\times [e]_\times F
\end{equation}
By grouping the right terms, we can find that
\begin{equation}
    M = [e]_\times F
\end{equation}
Notice that if the columns of $M$ were added by any scalar multiple of $e$, then the $F = [e]_\times M$ still holds up to scale. Therefore, the more general case of defining $M$ is
\begin{equation}
    M = [e]_\times F + e v^T
\end{equation}
for some vector $v$. In practice, defining $M$ by setting $v^T=\begin{bmatrix}1 & 1 & 1\end{bmatrix}$ works very well.

To finally solve for $H_1$, we need to compute the $\mathbf{a}$ values of $H_A$. Recall that we want to find a $H_1, H_2$ to minimize the problem posed in Equation~\ref{eq:rectification_minimization}. Since we already know the value of $H_2$ and $M$, then we can substitute $\hat{p}_i = H_2Mp_i$ and $\hat{p}_i' = H_2p_i'$ and the minimization problem becomes
\begin{equation}
\arg \min_{H_A} \sum_i \|H_A\hat{p}_i - \hat{p}_i'\|^2
\end{equation}

In particular, if we let $\hat{p}_i = (\hat{x}_i, \hat{y}_i, 1)$ and $\hat{p}_i' = (\hat{x}_i', \hat{y}_i', 1)$, then the minimization problem can be replaced by:
\begin{equation}
\arg \min_{\mathbf{a}} \sum_i (a_1\hat{x}_i + a_2\hat{y}_i + a_3 - \hat{x}_i')^2 + (\hat{y}_i -\hat{y}_i')^2
\end{equation}
Since $\hat{y}_i -\hat{y}_i'$ is a constant value, the minimization problem further reduces to
\begin{equation}
\arg \min_{\mathbf{a}} \sum_i (a_1\hat{x}_i + a_2\hat{y}_i + a_3 - \hat{x}_i')^2
\end{equation}

Ultimately, this breaks down into solving a least-squares problem $W\mathbf{a} = b$ for $\mathbf{a}$ where
\begin{equation}
W = \begin{bmatrix} \hat{x}_1 & \hat{y}_1 & 1 \\ &\vdots& \\ \hat{x}_n & \hat{y}_n & 1 \end{bmatrix} \ \ \ \ \ \ b = \begin{bmatrix}\hat{x}_1' \\ \vdots \\ \hat{x}_n'\end{bmatrix}
\end{equation}

After computing $\mathbf{a}$, we can compute $H_A$ and finally $H_1$. Thus, we generated the homographies $H_1,H_2$ to rectify any image pair given a few correspondences.
\end{document}


\chapter{Boolean funcoids}

\section{One-element boolean lattice}

Let $\mathfrak{A}$ be a boolean lattice and $\mathfrak{B}= \subsets 0$.
It's sole element is $\bot$.

$f \in \mathsf{pFCD} (\mathfrak{A}; \mathfrak{B}) \Leftrightarrow \forall X \in
\mathfrak{A}: (\supfun{f} X \nasymp \bot \Leftrightarrow \langle f^{-
1} \rangle \bot \nasymp X) \Leftrightarrow \forall X \in \mathfrak{A}: (0
\Leftrightarrow \supfun{f^{- 1}} \bot \nasymp X) \Leftrightarrow
\forall X \in \mathfrak{A}: \supfun{f^{- 1}} \bot \asymp X
\Leftrightarrow \forall X \in \mathfrak{A}: \supfun{f^{- 1}} \bot =
\bot^{\mathfrak{A}} \Leftrightarrow \supfun{f^{- 1}} \bot =
\bot^{\mathfrak{A}} \Leftrightarrow \supfun{f^{- 1}} = \{ (\bot ;
\bot^{\mathfrak{A}}) \}$.

Thus $\card \mathsf{pFCD} (\mathfrak{A}; \subsets 0) = 1$.

\section{Two-element boolean lattice}

Consider the two-element boolean lattice $\mathfrak{B}= \subsets 1$.

Let $f$ be a pointfree protofuncoid from $\mathfrak{A}$ to $\mathfrak{B}$
(that is $(\mathfrak{A};\mathfrak{B};\alpha;\beta)$ where $\alpha\in\mathfrak{B}^{\mathfrak{A}}$, $\beta\in\mathfrak{A}^{\mathfrak{B}}$).

$f \in \mathsf{pFCD} (\mathfrak{A}; \mathfrak{B}) \Leftrightarrow \forall X \in
\mathfrak{A}, Y \in \mathfrak{B}: (\supfun{f} X \nasymp Y
\Leftrightarrow \supfun{f^{- 1}} Y \nasymp X) \Leftrightarrow \forall X
\in \mathfrak{A}, Y \in \mathfrak{B}: ((0 \in \supfun{f} X \wedge 0 \in
Y) \vee (1 \in \supfun{f} X \wedge 1 \in Y) \Leftrightarrow \langle
f^{- 1} \rangle Y \nasymp X)$.

$T = \setcond{ X \in \mathfrak{A} }{ 0 \in \langle f
\rangle X }$ is an ideal. Really: That it's an upper set is obvious.
Let $P \cup Q \in \setcond{ X \in \mathfrak{A} }{ 0 \in
\supfun{f} X }$. Then $0 \in \supfun{f} (P \cup Q) =
\supfun{f} P \cup \supfun{f} Q$; $0 \in \supfun{f} P \vee
0 \in \supfun{f} Q$.

Similarly $S = \setcond{ X \in \mathfrak{A} }{ 1 \in
\supfun{f} X }$ is an ideal.

Let now $T, S \in \subsets \mathfrak{A}$ be ideals. Can we restore $\langle
f \rangle$? Yes, because we know $0 \in \supfun{f} X$ and $1 \in
\supfun{f} X$ for every $X \in \mathfrak{A}$.

So it is equivalent to $\forall X \in \mathfrak{A}, Y \in \mathfrak{B}: ((X
\in T \wedge 0 \in Y) \vee (X \in S \wedge 1 \in Y) \Leftrightarrow \langle
f^{- 1} \rangle Y \nasymp X)$.

$f \in \mathsf{pFCD} (\mathfrak{A}; \mathfrak{B})$ is equivalent to conjunction
of all rows of this table:

\begin{tabular}{|l|l|}
  \hline
  $Y$ & equality \\ \hline\hline
  $\emptyset$ & $\supfun{f^{- 1}} \emptyset = \emptyset$\\
  \{0\} & $X \in T \Leftrightarrow \supfun{f^{- 1}} \{ 0 \} \nasymp
  X$\\ \hline
  \{1\} & $X \in S \Leftrightarrow \supfun{f^{- 1}} \{ 1 \} \nasymp
  X$\\ \hline
  \{0,1\} & $X \in T \vee X \in S \Leftrightarrow \supfun{f^{- 1}} \{
  0, 1 \} \nasymp X$ \\
  \hline
\end{tabular}

Simplified:

\begin{tabular}{|l|l|}
  \hline
  $Y$ & equality \\ \hline\hline
  $\emptyset$ & $\supfun{f^{- 1}} \emptyset = \emptyset$\\ \hline
  \{0\} & $T = \corestar \supfun{f^{- 1}} \{ 0 \}$\\ \hline
  \{1\} & $S = \corestar \supfun{f^{- 1}} \{ 1 \}$\\ \hline
  \{0,1\} & $T \cup S = \corestar \supfun{f^{- 1}} \{ 0, 1 \}$ \\
  \hline
\end{tabular}

From the last table it follows that $T$ and $S$ are principal ideals.

So we can take arbitrary either $\supfun{f^{- 1}} \{ 0 \}$, $\langle
f^{- 1} \rangle \{ 1 \}$ or principal ideals $T$ and $S$.

In other words, we take $\supfun{f^{- 1}} \{ 0 \}$, $\langle f^{- 1}
\rangle \{ 1 \}$ arbitrary and independently. So we have $\mathsf{pFCD}
(\mathfrak{A}; \mathfrak{B})$ equivalent to product of two instances of
$\mathfrak{A}$. So it a boolean lattice. \fxerror{I messed product with disjoint union below.)}

\section{Finite boolean lattices}

We can assume $\mathfrak{B}= \subsets B$ for a set $B$, $\card B =
n$. Then

$f \in \mathsf{pFCD} (\mathfrak{A}; \mathfrak{B}) \Leftrightarrow \forall X \in
\mathfrak{A}, Y \in \mathfrak{B}: (\supfun{f} X \nasymp Y
\Leftrightarrow \supfun{f^{- 1}} Y \nasymp X) \Leftrightarrow \forall X
\in \mathfrak{A}, Y \in \mathfrak{B}: (\exists i \in Y : i \in \langle f
\rangle X \Leftrightarrow \supfun{f^{- 1}} Y \nasymp X)$.

Having values of $\supfun{f^{- 1}} \{ i \}$ we can restore all $\langle
f^{- 1} \rangle Y$. [need this paragraph?]

Let $T_i = \setcond{ X \in \mathfrak{A} }{ i \in
\supfun{f} X }$.

Let now $T_i \in \subsets \mathfrak{A}$ be ideals. Can we restore $\langle
f \rangle$? Yes, because we know $i \in \supfun{f} X$ for every $X \in
\mathfrak{A}$.

So, it is equivalent to:
\begin{equation}
  \label{pfcond-x} \forall X \in \mathfrak{A}, Y \in \mathfrak{B}: (\exists i
  \in Y : X \in T_i \Leftrightarrow \supfun{f^{- 1}} Y \nasymp X) .
\end{equation}
\begin{lem}
  The formula (\ref{pfcond-x}) is equivalent to:
  \begin{equation}
    \label{pfcond-i} \forall X \in \mathfrak{A}, i \in B : (X \in T_i
    \Leftrightarrow \supfun{f^{- 1}} \{ i \} \nasymp X) .
  \end{equation}
\end{lem}

\begin{proof}

  \begin{description}
    \item[(\ref{pfcond-x})$\Rightarrow$(\ref{pfcond-i})] Just take $Y = \{ i
    \}$.

    \item[(\ref{pfcond-i})$\Rightarrow$(\ref{pfcond-x})] Let (\ref{pfcond-i})
    holds. Let also $X \in \mathfrak{A}, Y \in \mathfrak{B}$. Then $\langle
    f^{- 1} \rangle Y \nasymp X \Leftrightarrow \bigcup_{i \in Y} \langle f^{-
    1} \rangle \{ i \} \nasymp X \Leftrightarrow \exists i \in Y : \langle
    f^{- 1} \rangle \{ i \} \nasymp X \Leftrightarrow \exists i \in Y : X \in
    T_i$.
  \end{description}
\end{proof}

Further transforming: $\forall i \in B : T_i = \corestar \langle f^{- 1}
\rangle \{ i \}$.

So $\supfun{f^{- 1}} \{ i \}$ are arbitary elements of $\mathfrak{B}$
and $T_i$ are corresponding arbitrary principal ideals.

In other words, $\mathsf{pFCD} (\mathfrak{A}; \mathfrak{B}) \cong \mathfrak{A}
\Pi \ldots \Pi \mathfrak{A}$ ($\card B$ times). Thus $\mathsf{pFCD}
(\mathfrak{A}; \mathfrak{B})$ is a boolean lattice.

\section{About infinite case}

Let $\mathfrak{A}$ be a complete boolean lattice, $\mathfrak{B}$ be an
atomistic boolean lattice.

$f \in \mathsf{pFCD} (\mathfrak{A}; \mathfrak{B}) \Leftrightarrow \forall X \in
\mathfrak{A}, Y \in \mathfrak{B}: (\supfun{f} X \nasymp Y
\Leftrightarrow \supfun{f^{- 1}} Y \nasymp X) \Leftrightarrow \forall X
\in \mathfrak{A}, Y \in \mathfrak{B}: (\exists i \in \atoms Y : i \in
\atoms \supfun{f} X \Leftrightarrow \supfun{f^{- 1}} Y
\nasymp X)$.

Let $T_i = \setcond{ X \in \mathfrak{A} }{ i \in
\atoms \supfun{f} X }$.

$T_i$ is an ideal: Really: That it's an upper set is obvious. Let $P \cup Q
\in \setcond{ X \in \mathfrak{A} }{ i \in \atoms
\supfun{f} X }$. Then $i \in \atoms \supfun{f} (P
\cup Q) = \atoms \supfun{f} P \cup \atoms \supfun{f}
Q$; $i \in \supfun{f} P \vee i \in \supfun{f} Q$.

Let now $T_i \in \subsets \mathfrak{A}$ be ideals. Can we restore $\langle
f \rangle$? Yes, because we know $i \in \atoms \supfun{f} X$ for
every $X \in \mathfrak{A}$ and $\mathfrak{B}$ is atomistic.

So, it is equivalent to:
\begin{equation}
  \label{mpfcond-x} \forall X \in \mathfrak{A}, Y \in \mathfrak{B}: (\exists i
  \in \atoms Y : X \in T_i \Leftrightarrow \supfun{f^{- 1}} Y
  \nasymp X) .
\end{equation}
\begin{lem}
  The formula (\ref{mpfcond-x}) is equivalent to:
  \begin{equation}
    \label{mpfcond-i} \forall X \in \mathfrak{A}, i \in
    \atoms^{\mathfrak{B}} : (X \in T_i \Leftrightarrow \langle f^{- 1}
    \rangle i \nasymp X) .
  \end{equation}
\end{lem}

\begin{proof}

  \begin{description}
    \item[(\ref{mpfcond-x})$\Rightarrow$(\ref{mpfcond-i})] Let
    (\ref{mpfcond-x}) holds. Take $Y = i$. Then $\atoms Y = \{ i \}$ and
    thus $X \in T_i \Leftrightarrow \exists i \in \atoms Y : X \in T_i
    \Leftrightarrow \supfun{f^{- 1}} Y \nasymp X \Leftrightarrow
    \supfun{f^{- 1}} i \nasymp X$.

    \item[(\ref{mpfcond-i})$\Rightarrow$(\ref{mpfcond-x})] Let
    (\ref{pfcond-i}) holds. Let also $X \in \mathfrak{A}$, $Y \in \mathfrak{B}$.
    Then $\supfun{f^{- 1}} Y \nasymp X \Leftrightarrow \langle f^{- 1}
    \rangle \bigsqcup \atoms Y \nasymp X \Leftrightarrow \bigsqcup_{i
    \in \atoms Y} \supfun{f^{- 1}} i \nasymp X \Leftrightarrow
    \exists i \in \atoms Y : \supfun{f^{- 1}} i \nasymp X
    \Leftrightarrow \exists i \in \atoms Y : X \in T_i$.
  \end{description}
\end{proof}

Further equivalently transforming: $\forall i \in \atoms^{\mathfrak{B}}
: T_i = \corestar \supfun{f^{- 1}} i$.

So $\supfun{f^{- 1}} i$ are arbitary elements of $\mathfrak{B}$ and
$T_i$ are corresponding arbitrary principal ideals.

In other words, $\mathsf{pFCD} (\mathfrak{A}; \mathfrak{B}) \cong \prod_{i \in
\card \atoms^{\mathfrak{B}}} \mathfrak{A}$. Thus $\mathsf{pFCD}
(\mathfrak{A}; \mathfrak{B})$ is a boolean lattice.

So finally we have a very weird theorem, which is a partial solution for the
above open problem (The weirdness is in its partiality and asymmetry):

\begin{thm}
  If $\mathfrak{A}$ is a complete boolean lattice and $\mathfrak{B}$ is an
  atomistic boolean lattice (or vice versa), then $\mathsf{pFCD} (\mathfrak{A};
  \mathfrak{B})$ is a boolean lattice.
\end{thm}

\cite{shmuely1974}~proves
``THEOREM 4.6. Let $A$, $B$ be bounded posets. $A \otimes B$ is a completely
distributive complete Boolean lattice iff $A$ and $B$ are completely distributive
Boolean lattices.'' (where $A \otimes B$ is equivalent to the set of Galois connections between~$A$ and~$B$) and other interesting results.


\documentclass[11pt,a4paper]{report}
\usepackage{amsmath,amsfonts,amssymb,amsthm,epsfig,epstopdf,titling,url,array}
\usepackage{enumitem}
\usepackage{changepage}
\usepackage{graphicx}
\usepackage{caption}
\theoremstyle{plain}
\newtheorem{thm}{Theorem}[section]
\newtheorem{lem}[thm]{Lemma}
\newtheorem{prop}[thm]{Proposition}
\newtheorem*{cor}{Corollary}
\theoremstyle{definition}
\newtheorem{defn}{Definition}[section]
\newtheorem{conj}{Conjecture}[section]
\newtheorem{exmp}{Example}[section]
\newtheorem{exercise}{Exercise}[section]
\theoremstyle{remark}
\newtheorem*{rem}{Remark}
\newtheorem*{note}{Note}
\def\changemargin#1#2{\list{}{\rightmargin#2\leftmargin#1}\item[]}
\let\endchangemargin=\endlist
\begin{document}

\section*{Problem} Given a circle of radius $r$, write a function $f(r)$ that
gives the radius of a circle whose area is the square of the area of the
original circle.

\section*{Bonus} Start with a circle of radius $10$ and draw another circle of
radius $r$ inside it.  What value of $r$ makes the product of the of the areas
of the inner and outer circles largest?

\section*{Solution} The square of the area of a circle of radius r is
$( \pi r^2)^2 = \pi^2r^4$.   So if we make $f(r) = \sqrt{\pi}r^2$ then a circle
of this radius will have area $\pi ( \sqrt{\pi}r^2)^2 = \pi^2r^4$ as desired.

\section*{Bonus Solution}

\end{document}


\chapter{Regression}
Regression is a very important topic in statistics that is applied extremely
frequently. There are many different kinds of regression, but in STATS 331 we will
mostly focus on linear regression. This gives us a nice familiar example
example to demonstrate how Bayesian statistics works and how it is different
from classical or frequentist statistics. Here we will study an example of a
simple linear regression problem taken from STATS 20X.

\section{A Simple Linear Regression Problem}
Data were collected from a sample of 30 drivers. The age of the driver and the
maximum distance at which they could read a newly designed road sign were
recorded. It is of interest to build a simple model that can be used to predict the
maximum distance at which the sign is legible, using the age of the driver.
Figure~\ref{fig:road} shows the data.
\begin{figure}
\begin{center}
\includegraphics[scale=0.5]{Figures/road.pdf}
\caption{\it The maximum distance at which a person can read a road sign vs. the age of
the person. There are $N=30$ data points. You can clearly see that older people
have, on average, worse eyesight. Simple linear regression can be thought of as
``fitting a straight line'' to the data.
\label{fig:road}}
\end{center}
\end{figure}
The purpose of simple linear regression is to find a straight line that goes
throught the data points. The slope and intercept of the straight line are then
helpful for understanding the relationship between the variables. Also, the straight line can be used
to predict future data, such as the maximum distance at which a 90-year-old person
could read the sign. The most common method used to obtain the straight
line is to find the line (i.e. the slope and intercept values) which fits best
by the criterion of ``least squares''.

\section{Interpretation as a Bayesian Question}
From what you now know about Bayesian statistics, you might be able to come up
with some reasons why the standard least squares
solution is unsatisfactory. One glaring issue is that the data are hardly ever
going to be good enough to tell us with certainty that a particular slope and intercept
are correct (the exception would be if three or more points lie perfectly on a straight
line, with no scatter).
In principle, we will almost always have uncertainty about the slope and
the intercept. From a Bayesian perspective, our goal is not to find a point
estimate for the slope and the intercept. Instead we should calculate the
{\it posterior distribution for the slope and the intercept, given the data}.
The posterior distribution will tell us exactly how much uncertainty we have.
If we do want summaries for convenience, we can use the posterior distribution
to create the summaries, as discussed in Chapter~\ref{chapter:summaries}.

The equation for a straight line is usually written as $y = mx + b$ where $m$
is the gradient/slope and $b$ is the intercept. However,
for consistency with later, more complex regression models, we will write the
equation as:
\begin{eqnarray}
y = \beta_0 + \beta_1 x.
\end{eqnarray}
Here, $\beta_0$ is the $y$-intercept and $\beta_1$ is the slope. Our goal is
to calculate the posterior distribution for $\beta_0$ and $\beta_1$ given the
data.

\section{Analytical Solution With Known Variance}
Bayes' rule (parameter estimation version) tells us how to calculate the
posterior distribution:
\begin{eqnarray}
p(\theta|x) \propto p(\theta)p(x|\theta)
\end{eqnarray}
This is the generic form for parameters $\theta$ and data
$x$. In our particular case, the
unknown parameters are $\beta_0$ and $\beta_1$, and the data are the
$y$ values of the data points. The data also consist of a number $N$ of points
and the $x$-values,
but we shall assume that these on their own provide no information about the
slope and intercept (it would be a bit strange if they did). So the $x$-values
and the number of points $N$ act like prior information that lurks ``in the
background'' of this entire analysis. The $y$-values are our data in the sense
that we will obtain our likelihood by writing down a probability distribution
for the $y$-values given the parameters.

Therefore, Bayes' rule {\it for this problem} (i.e. with the actual names
of our parameters and data, rather than generic names) reads:
\begin{eqnarray}
p(\beta_0, \beta_1 | y_1, y_2, ..., y_N) \propto
p(\beta_0, \beta_1)p(y_1, y_2, ..., y_N | \beta_0, \beta_1)
\end{eqnarray}

We can now say some things about Bayesian linear regression by working
analytically. For starters,
let's assume uniform priors for both $\beta_0$ and $\beta_1$, and that the
prior for these two parameters are independent. The probability density for
a uniform prior distribution can be written simply as:
\begin{eqnarray}
p(\beta_0, \beta_1) \propto 1.
\end{eqnarray}
Note that we have written proportional instead of equals. If we decided to
place the limits at -500 and 500 (say) then the actual value of the density
would be $10^{-6}$. But this is just a number and in the end, when we normalise
the posterior distribution, it won't matter. We can even imagine making our
prior ``infinitely wide'', which is called an improper prior. In many cases
simply writing $p(\beta_0, \beta_1) \propto 1$ will not cause any problems. We
are assuming the prior probability density is uniform over a very wide range
which we will not specify.

Now, on to the likelihood. There are $N$ data points and so there are $N$
$y$-values in the dataset, called $\{y_1, y_2, ..., y_N\}$. We can obtain the
likelihood by writing down a probability distribution for the data given the
parameters, sometimes called a ``sampling distribution''. This describes our
beliefs about the connection between the data and the parameters, without which
it would be impossible to learn anything from data.
If we
knew the true values of $\beta_0$ and $\beta_1$, then we would predict the
$y$-values to be scattered around the straight line. Specifically we
will assume that each point departs from the straight line by an amount
$\epsilon_i$ which has a $\mathcal{N}(0, \sigma^2)$ probability distribution.
For now, we will assume $\sigma$, the standard deviation of the scatter, is known.
In ``$\sim$'' notation, this can be written as:
\begin{eqnarray}
y_i \sim \mathcal{N}(\beta_0 + \beta_1 x_i, \sigma^2).
\end{eqnarray}
It is implied that all of the data values are independent (given the
parameters). Therefore the likelihood can be written as a product of $N$
normal densities, one for each data point:
\begin{eqnarray}
p(\{y_1, y_2, ..., y_N\}|\beta_0, \beta_1) &=& \prod_{i=1}^N \frac{1}{\sigma\sqrt{2\pi}}
\exp\left[-\frac{1}{2\sigma^2}\left(y_i - (\beta_0 + \beta_1 x_i)\right)^2\right].
\end{eqnarray}
Remember, when we combine the likelihood with the prior using Bayes' rule,
we can usually ignore any constant factors which do not depend on
the parameters. This allows us to ignore the first part of the product, outside
the exponential (since we are assuming $\sigma$ is known).
\begin{eqnarray}
p(\beta_0, \beta_1 | y_1, y_2, ..., y_N) &\propto& p(\beta_0, \beta_1)
p(y_1, y_2, ..., y_N|\beta_0, \beta_1)\\
&\propto& 1 \times \prod_{i=1}^N \exp\left[-\frac{1}{2\sigma^2}\left(y_i - (\beta_0 + \beta_1 x_i)\right)^2\right]\\
&\propto& \exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^N\left(y_i - (\beta_0 + \beta_1 x_i)\right)^2\right].
\label{eq:leastsquares}
\end{eqnarray}
We have just found the expression for the posterior distribution for $\beta_0$
and $\beta_1$. This is a distribution for two parameters (i.e. it is bivariate).
It is not easy to interpret this equation just by
looking at it, but we could use it to work out the value of the posterior
probability density for any possible values of $\beta_0$ and $\beta_1$

There are a few things you may notice about the posterior distribution in
Equation~\ref{eq:leastsquares}. Firstly, the way it depends on the parameters
is an exponential of something involving $\beta_0$
and $\beta_1$ in linear and second-order ways (if you were to expand the square,
you would get terms like $\beta_0\beta_1$ and $\beta_0^2$). Mathematicians would
call the expression inside the exponential a {\it quadratic form}. When a
probability density can be written as the exponential of a quadratic form, it
is a normal density. Therefore, the posterior distribution for $\beta_0$ and
$\beta_1$ is a (bivariate) normal distribution.

We can obtain some more insight about this problem by inspecting the
sum term inside the exponential in the posterior distribution
(Equation~\ref{eq:leastsquares}). The sum is over all the data points, and
what is being summed is the difference between the data value $y_i$ and the
straight line prediction $\beta_0 + \beta_1 x_i$, all squared.
The sum term is just the sum of squared
residuals that is minimised when solving this problem by ``least squares''.
In classical least squares linear regression, $\beta_0$ and $\beta_1$ are
estimated by minimising this sum of squared residuals. Because of the exp and
the minus sign, the posterior distribution is telling us that the choice of
$\beta_0$ and $\beta_1$ that minimises the sum of squared residuals, maximises
the posterior probability density. Other values of the parameters that
don't quite minimise the sum of squared residuals are somewhat plausible, and
the form of the posterior density tells us exactly how much less plausible they
are. The take home message is summarised below.

\begin{framed}
{\bf Doing a linear regression by least squares is equivalent to having a
uniform prior and a normal likelihood, and finding the posterior mode.
If you think this is appropriate in a particular application, and you are happy
with just a point estimate, then classical
least squares fitting will be fine. Otherwise, you'd better do a Bayesian analysis.}
\end{framed}

While classical regression results may come with ``standard errors'', these are
not the same as a posterior distribution. A posterior distribution describes the
uncertainty about the parameters given the specific data set you actually have.
Standard errors describe how different your point estimate would be if your data
set was different.

\section{Solution With JAGS}
The above analytical results made the unrealistic assumption that the standard
deviation $\sigma$, of the scatter, was known. In practice, $\sigma$ usually needs to be
estimated from the data as well. Therefore, in the Bayesian framework, we should
include it as an extra unknown parameter. Now we have three unknown parameters instead
of two. Our parameters are now $\beta_0$, $\beta_1$, and $\sigma$.
One major advantage of MCMC is that we can increase the number of unknown parameters
without having to worry about the fact that the posterior distribution might
be hard to interpret or plot.

The data is the same as before, $\{y_1, y_2, ..., y_N\}$. The likelihood is
also the same as before:
\begin{eqnarray}
y_i \sim \mathcal{N}(\beta_0 + \beta_1 x_i, \sigma^2).
\end{eqnarray}

Our three parameters will need priors. JAGS requires proper priors (i.e. we
can't have a uniform prior over an infinite range), but we can still make our
priors very wide. Instead of using uniform distributions this time,
we will use normal distributions with a mean of 0 and a large standard deviation
of 1000.

For the standard deviation parameter $\sigma$, we know firstly that this cannot
be negative. We will use a ``log-uniform'' prior with generous lower and upper limits,
to express uncertainty about the order of magnitude of $\sigma$. This prior implies
things like $P(1 < \sigma < 10) = P(10 < \sigma < 100) = P(100 < \sigma < 1000)$,
which is sometimes a good description of a large amount of uncertainty about
a positive parameter. The easiest way to implement this in JAGS is to actually
use $\log(\sigma)$ as the parameter, with a uniform prior, and then define
$\sigma = \exp\left(\log(\sigma)\right)$. Notice that ``deterministic nodes''
(quantities that are defined in terms of other variables) in JAGS are defined
using ``{\tt <-}'' instead of ``{\tt \~{ }}''.

In JAGS, the model looks like this:
\begin{minted}[mathescape,
               numbersep=5pt,
               gobble=0,
               frame=single,
               framesep=2mm, fontsize=\small]{r}
model
{
  # Prior for all the parameters
  beta0 ~ dnorm(0, 1/1000^2)
  beta1 ~ dnorm(0, 1/1000^2)
  log_sigma ~ dunif(-10, 10)
  sigma <- exp(log_sigma)

  # Likelihood
  for(i in 1:N)
  {
    y[i] ~ dnorm(beta0 + beta1*x[i], 1/sigma^2)
  }
}
\end{minted}

The first part defines the priors for the parameters. For $\beta_0$ and $\beta_1$,
we have just chosen very broad priors that describe vague prior knowledge. Note
that the standard deviations of the priors are 1000, so we should be careful
to only apply this code in situtations where we don't expect the intercept or
slope to have an extreme value (either positive or negative).

For the standard deviation parameter $\sigma$, which describes how much we
expect the data points to be scattered around the straight line, we have assigned
a log-uniform prior. The limits of $-10$ and $10$ for {\tt log\_sigma} imply
limits of $4.5 \times 10^{-5}$ to 22,000 for {\tt sigma}, a generous range.
If we thought $\sigma$
might actually be outside this range, we should change the prior to something
else or risk getting strange answers.

The likelihood part of the code involves a {\tt for} loop, because our data
is more than just a single number. The code inside the loop is effectively
duplicated $N$ times (with $i=1$, then with $i=2$, etc), once for each data
point. Since the loop refers to a quantity {\tt N}, this must be specified in
the {\tt data} list if you are using my {\tt use\_jags.r} template code.

Another new feature of this JAGS model is the normal distribution, which is
called {\tt dnorm} in JAGS. Usually a normal distribution is written
$\mathcal{N}(\mu, \sigma^2)$ where $\mu$ is the mean and $\sigma$ is the standard
deviation (and $\sigma^2$ is the variance). Unfortunately, in JAGS there is a
quirk: the first argument to {\tt dnorm} is indeed the mean, but the second argument
must be one over the variance, or one over the standard deviation squared.

\section{Results for ``Road'' Data}
Our JAGS output will contain samples from the posterior distribution for
$\beta_0$, $\beta_1$ and $\sigma$. The first thing we should do is make
trace plots and check that everything converged properly. Then we can make
histograms of each parameter to visually inspect the (marginal) posterior
distribution for each parameter. We can also plot one parameter vs. another
to look at the {\it joint} posterior distribution for the parameters. R code
for all of these is given below.

\begin{minted}[mathescape,
               numbersep=5pt,
               gobble=0,
               frame=single,
               framesep=2mm, fontsize=\small]{r}
# Plot trace plots
plot(results$beta0, type='l', xlab='Iteration', ylab='beta0')
plot(results$beta1, type='l', xlab='Iteration', ylab='beta1')
plot(results$sigma, type='l', xlab='Iteration', ylab='sigma')

# Plot histograms
hist(results$beta0, breaks=20, xlab='beta0')
hist(results$beta1, breaks=20, xlab='beta1')
hist(results$sigma, breaks=20, xlab='sigma')

# Plot joint posterior distribution of beta0 and beta1
plot(results$beta0, results$beta1, cex=0.1, xlab='beta0', ylab='beta1')
\end{minted}

All of the plots for the road data are shown in Figure~\ref{fig:road_results}.
\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.35]{Figures/road_trace.pdf}
\includegraphics[scale=0.35]{Figures/road_hist.pdf}
\includegraphics[scale=0.35]{Figures/road_joint.pdf}
\caption{\it Results (posterior samples) from the simple linear regression model
applied to the road data.
{\bf Top Left}: Trace plots showing the parameters moving around over time (as the
MCMC progressed).
{\bf Top Right}: Histograms of the posterior samples, showing the marginal
posterior distributions for the parameters.
{\bf Bottom}: A plot of $\beta_1$ vs. $\beta_0$, showing samples from the joint
posterior for these two parameters. These parameters had independent priors, but
the posterior shows a correlation. All this means is that if $\beta_0$ is a
high value then $\beta_1$ must be low, and vice versa.
\label{fig:road_results}}
\end{center}
\end{figure}

With classical linear regression it is usually helpful to plot the best fitting
line through the data. In Bayesian linear regression our output is posterior
samples for what the parameters might be (and therefore what the line might be).
A common way of displaying the posterior is to plot many lines through the data,
with the lines produced using the posterior samples. Some R code for doing this
is given below. The plot produced looks like the one in
Figure~\ref{fig:road_lines}.

\begin{minted}[mathescape,
               numbersep=5pt,
               gobble=0,
               frame=single,
               framesep=2mm, fontsize=\small]{r}
# Plot the data
plot(data$x, data$y)

# Make some x-values for plotting lines
x = c(0, 100)
# Plot the first 30 lines from the posterior distribution
for(i in 1:30)
{
    lines(x, results$beta0[i] + results$beta1[i]*x)
}
\end{minted}

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.5]{Figures/road_lines.pdf}
\caption{\it The road data with credible regression lines (sampled from the
posterior distribution) overplotted.\label{fig:road_lines}}
\end{center}
\end{figure}

\section{Predicting New Data}
One of the most important uses of regression models is for prediction. Given
this data, what can we say about the value of the output variable $y$ at some
new value of the input variable, $x_{\rm new}$? It's unknown, but let's call it
$y_{\rm new}$. With our road example, we will try to predict the maximum reading
distance of a person who is $x_{\rm new}=90$ years old.
If we knew the parameters of the straight line (or had a good point estimate),
we could extend it out to $x=90$, and compute our predicted value:

\begin{eqnarray}
y_{\rm new} &=& \beta_0 + \beta_1 x_{\rm new}\\
&=& \beta_0 + \beta_1\times 90
\end{eqnarray}
but that's not quite in the Bayesian spirit. Firstly, we don't know the value
of the parameters, we only have the posterior distribution. Secondly, we would
like not just a point estimate for $y_{\rm new}$ but a whole probability distribution,
describing all of the uncertainty in the prediction. One simple improvement
would be to state that our uncertainty about $y_{\rm new}$ should be described
by a normal distribution around the straight line, with standard deviation $\sigma$.
This still isn't quite right though, because we'd need to know the true values
of the parameters to actually do it.

In general, Bayesian prediction works like so. With parameters
$\theta$ and data $x$, we can predict ``new data'' $x'$ by calculating the
``posterior predictive distribution'' which is just the probability distribution
for $x'$ given $x$:
\begin{eqnarray}
p(x' | x) &=& \int p(\theta, x' | x) \, d\theta\\
&=& \int p(\theta|x) p(x' | x, \theta)\, d\theta
\end{eqnarray}
We first saw this in Section~\ref{sec:prediction_bus_problem}.
The first term inside the integral is the posterior, and therefore the
whole integral is an expectation value, with respect to the posterior distribution.
The second term is
the probability distribution for $x'$ given the parameters, i.e. imagining that
we knew the parameters.

This equation is telling us to imagine that we know the
true parameter values, make a prediction (in terms of a probability distribution),
repeat this for all possible parameter values
and then average together all the probability distributions into one ``final''
distribution. In the averaging process we should give more weight
to the probability distributions that were based on plausible values of $\theta$.

Thankfully, actually doing this is much easier than it sounds, thanks to MCMC.
Remarkably, we can accomplish all this, and obtain our probability distribution
for $y_{\rm new}$, by adding just a single line to the JAGS model:
\begin{minted}[mathescape,
               numbersep=5pt,
               gobble=0,
               frame=single,
               framesep=2mm, fontsize=\small]{r}
y_new ~ dnorm(beta0 + beta1*90, 1/sigma^2)
\end{minted}
Of course, to look at the posterior samples for {\tt y\_new}, you'll need to
monitor it. The samples of {\tt y\_new}
will be drawn from the posterior predictive distribution for the new data.
Internally, JAGS will simulate a new value of {\tt y\_new} at every iteration,
from the specified distribution, and using its current estimates of the parameters.
Since the parameters are not fixed, but explore the posterior distribution, the
distribution of {\tt y\_new} values will take into account all of the uncertainty
we have about the parameter values.

As a general rule for predicting new data in JAGS, the extra line(s) you'll add to the JAGS
model will usually resemble the likelihood, but the variable will have a different name.
The results for the road data prediction are shown
in Figure~\ref{fig:road_prediction}.
\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.5]{Figures/road_prediction.pdf}
\caption{\it Samples from the posterior predictive distribution, answering the
question ``what do we know about $y$ at $x=90$?''. Summaries are shown in
the title. If we simply assumed the best fit line was true and applied a
point estimate of $\sigma$ to get our uncertainty, we would have obtained a
prediction of $306.07 \pm 49.76$.\label{fig:road_prediction}}
\end{center}
\end{figure}

\section{Simple Linear Regression With Outliers}
One complication that is common in linear regression (or other model-fitting)
problems is the existence of outliers. These are points that do not fit in with
the general trend assumed by the model. Many methods exist for deciding how to
``detect'' and ``remove''
outliers. From a Bayesian point of view, there is not a one-size-fits-all approach
to outliers. If you really believe in your model assumptions, an ``outlier''
might be your most important data point. If you aren't really sure of your
model assumptions and are just using a convenient default model, then your results
may be misleading.
In lectures and labs we will study an extension to the simple linear
regression model that allows for outliers by using a heavier-tailed sampling
distribution, the Student-$t$ distribution.

\section{Multiple Linear Regression and Logistic Regression}
We will study an example of multiple linear regression, and nonlinear regression
(involving an ``interaction term'') in lectures.
We will not study a logistic regression explicitly, but the special
lecture on predicting sports matches is very closely related to logistic
regression. The main difference between linear regression is that the output
variable can only be 0 or 1, so the likelihood will usually involve the
``Bernoulli'' distribution (a Binomial distribution with one trial).


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{tcolorbox}
\date{}

\title{\textbf{Quantum Computing: An Applied Approach}\\\vspace*{1cm}
Chapter 2 Problems: \\
A Brief History of Quantum Computing
}

\begin{document}

\maketitle

\section{}

(a) No. This system is not necessarily scalable to higher numbers of qubits (1), may not maintain coherent states for sufficient periods to perform gate operations (3), and does not necessarily facilitate two-qubit gate operations (4).\newline
(b) No. This system does not clearly define distinctly demarcated qubits (1), does not clearly define whether the ground state is a $|0\rangle$ basis state (2), has no coherence guarantees (3), has not demonstrated single- or double-qubit gate operations (4), and has not been shown to facilitate strong measurements (5).

\section{}

Unitary gate operations do not operate on 3-vectors, they involve operation of a unitary 2x2 complex matrix in the space $\mathbf{C}^{2\times 2}$ on a 2x1 complex vector $\mathbf{C}^{2\times 1}$, yielding another element in the space $\mathbf{C}^{2\times 1}$.

No, no ``dimensional shenanigans'' are occuring here. A state vector can be represented as a point on a sphere because pure states are normalized to a magnitude of 1 via the relation $\langle q|q\rangle=1$. This constraint limits possible states to a subspace of $\mathbf{C}^2$ that is instead isomorphic to $\mathbf{R}^3$, so it can be geometrically represented with three-dimensional geometry. These are still elements of the higher four-dimensional space though, so the restriction of the operation of elements of $\mathbf{C}^{2\times 2}$ on qubit states in this subspace is still well-defined.

\section{}

No, the $CZ$ gate is symmetric. In the standard basis $\{|00\rangle,|01\rangle,|10\rangle,|11\rangle\}$,
$$
CZ\coloneqq \begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & -1
\end{pmatrix}
$$

Flipping the control and target qubits effectively rotates the second and third basis elements into each other, so that the basis ordering instead becomes $\{|00\rangle,|10\rangle,|01\rangle,|11\rangle\}$.

Since the $CZ$ matrix is invariant under the exchange of indices $2\leftrightarrow 3$, the matrix $CZ$ is the same regardless of which qubit is used as target and which as control.

\end{document}


\SecDef{truncated}{Truncated Differential Analysis of SPARKLE}

We performed exhaustive search of all structural truncated trails in \aCipher{}, i.e. when each branch can be either active or inactive. Our approach consists of two steps. The first step is to generate the matrix of probabilities of all truncated transitions through the linear layer. I propose a new generic and precise method for this step. The second step is a simple iterative search, where for each round and for each truncated pattern at this round we keep the best truncated trail leading tho this pattern.

\subsection{Generating Truncated Trail Matrix of a Linear Layer}
I describe a generic method to generate the matrix of probabilities of truncated transitions from the binary matrix of the analyzed linear layer.

\newcommand\alphabeta{\alpha \xrightarrow{L} \beta}
\newcommand\exact{+}
\newcommand\loose{*}

\newcommand\VA{p(\alpha)}
\newcommand\VB{p(\beta)}

\begin{definition}
Let $L: (\field{m})^t \to (\field{m})^t$ be a linear bijective mapping. An \emph{exact truncated transition} over $L$ is a pair of vectors from $\{0,\exact\}^t$. A \emph{loose truncated transition} over $L$ is a pair of vectors from $\{0, \loose\}^t$. A truncated transition $\alpha, \beta$ over $L$ is denoted $\alphabeta$.
\end{definition}

\begin{definition}[Support Sets]
$ $\newline
The \emph{support} of a symbol $\gamma_i \in \pset{0, \loose, \exact}$ is defined as the set
$$
p(\gamma_i) \eqdef
\begin{cases}
\{0\}, &\text{if}~ \gamma_i = 0, \\
\field{m}, &\text{if}~ \gamma_i = \loose, \\
\field{m}\setminus\{0\}, &\text{if}~ \gamma_i = \exact.
\end{cases}
$$
The \emph{support} of a vector $\gamma \in \{0, \loose, \exact\}^t$ is defined as the set
$$
p(\gamma) \eqdef p(\gamma_0) \times \ldots \times p(\gamma_{t-1}).
$$
The \emph{support} of a truncated transition $\alphabeta$ is defined as the set
$$
p(\alphabeta) \eqdef \pset{(x,L(x)) | \mid x \in \VA, L(x) \in \VB} \subseteq (\field{mt})^2.
$$
\end{definition}

\begin{definition}
The \emph{cardinality} of a truncated transition $\alphabeta$ is defined as the cardinality of its support:
$$
|\alphabeta| \eqdef |p(\alphabeta)| = |L(\VA)\cap\VB|.
$$
The \emph{probability} of a truncated transition $\alphabeta$ is defined as
$$
\Pr[\alphabeta] \eqdef \Pr_{x \in \VA}[L(x) \in \VB] = \frac{|\alphabeta|}{|\VA|}.
$$
\end{definition}

\begin{remark}
The cardinality is a property of the \emph{graph} of $L$, whereas the probability differentiates the input from the output. In particular, the problem of computing cardinalities of truncated transitions is equivalent to the problem of finding the number of codewords fitting the truncated mask $(\alpha, \beta) \in \field{2mt}$ in the linear code with the generator matrix $\matb{\idmat{mt} ~\mid~ L}$. The matrix of exact truncated transitions also trivially allows to compute the branching number of the matrix.
\end{remark}

Finally, the table of truncated transition probabilities can be formally defined.

\newcommand\TTT[1]{\mathsf{TTT}_{#1}}
\begin{definition}[TTT]
Let $L\colon (\field{m})^t \to (\field{m})^t$ be a linear bijective mapping. The \emph{table of truncated transitions} (TTT) of $L$ is the $2^{t}\times 2^{t}$ matrix $\TTT{L}$ given by
$$
\TTT{L}[a,b] = \Pr{\alphabeta},
$$
where $a,b \in \field{t}$ are mapped naturally to $\alpha',\beta' \in \pset{0,\exact}^t$ respectively.
\end{definition}

Since $|\VA|$ is easy to compute, we focus on computing the cardinalities of loose and exact truncated transitions.

% =============================================

\paragraph{Cardinalities of Loose Transitions.}
The first step is to compute the cardinalities of \emph{all} possible \emph{loose} truncated transitions over $L$. Let $\alphabeta$ be a loose transition over $L$. Observe that $p(\alpha),p(\beta)$ are linear subspaces of $\field{mt}$.
The cardinality $|\alphabeta| = L(\VA) \cap \VB$ can be computed as follows.

A vector $b \in \ffield{m}{t}$ belongs to $\VB$ if and only if $\beta_i = 0$ implies $b_i = 0$ for all $i$. Let
$$
\pi_{\beta}(b)\colon \ffield{m}{t} \to \VB^{\bot}
$$
be the linear map returning the part of the vector $b$ consisting of all elements $b_i$ for which $\beta_i=0$. The dimension of the space $L(\VA)\cap \VB$ can be computed as the nullity of the linear map
$$
\phi\colon \VA \to \VB^{\top},~~~~\phi \eqdef \pi_{\beta} \circ L.
$$

Let $L$ be given as a block matrix $(L_{i,j})_{i,j\in\seg{1}{t}}$ with blocks of size $m \times m$. The vector space $L(\VA)$ is spanned by bit-columns of the submatrix $$
(L_{i,j}), ~\text{where}~ i,j\in\seg{1}{t},\alpha_j=\loose.
$$
\newcommand\LSUB{L\big|_{\alpha=\loose,\beta=0}}
Furthermore, the map $\pi_{\beta}$ simply truncates its input vector to positions where $\beta_i = 0$. It follows that $\phi \eqdef \pi_{\beta} \circ L$ can be given by the submatrix
$$
\LSUB \eqdef (L_{i,j})~\text{where}~ i,j\in\seg{1}{t},\beta_i=0,\alpha_j=\loose.
$$
Given the binary rank of this matrix, the nullity of $\phi$ is computed as
$$
\dim{\VA}-\rank{\LSUB}.
$$
We conclude that $|\alphabeta|$ can be computed as
$$
|\alphabeta| = 2^{\dim{p(\alpha)} - \rank{\LSUB}}.
$$
% \Todo{Grid image with example}

\paragraph{Cardinalities of Exact Transitions.}
The second step is to compute the probabilities of all \emph{exact} truncated transitions over $L$. Observe that a loose truncated truncated trail can be seen as a union of precise truncated trails. For example,
\begin{align*}
    &(\loose, 0) \xrightarrow{L} (0, \loose)
\end{align*}
is equivalent to the following union of disjoint transitions:
\begin{align*}
        \pset{
        (0, 0) \xrightarrow{L} (0, 0),~~~~
        (0, 0) \xrightarrow{L} (0, \exact),~~~~
        (\exact, 0) \xrightarrow{L} (0, 0),~~~~
        (\exact, 0) \xrightarrow{L} (0, \exact)},
\end{align*}
i.e. the cardinalities are summed.

\newcommand\alphabetap{\alpha' \xrightarrow{L} \beta'}
\newcommand\alphabetaex{\alpha_{\exact} \xrightarrow{L} \beta_{\exact}}
\newcommand\alphabetaexp{\alpha_{\exact}' \xrightarrow{L} \beta_{\exact}'}
\newcommand\alphabetalo{\alpha_{\loose} \xrightarrow{L} \beta_{\loose}}

\begin{lemma}
\LemLabel{trunc-disjoint}
Let $\alphabeta\ne\alphabetap$ be exact truncated transitions. Then their supports are disjoint.
\end{lemma}
\begin{proof}
The lemma follows from the fact that $p(\exact)$ and $p(0)$ are disjoint.
\end{proof}

\begin{definition}
For any symbols/vectors/truncated transitions $\gamma,\gamma'$ let
$$
\gamma \preceq \gamma' ~\text{if and only if}~ p(\gamma) \subseteq p(\gamma').
$$
\end{definition}

\begin{lemma}
Let $\alphabetalo$ be a loose truncated transition. Then the set
$$
\mathbf{P}(\alphabetalo) \eqdef \pset{
p(\alphabetaexp) \mid
\alpha_{\exact}',\beta_{\exact}' \in \pset{0,\exact}^t,
(\alphabetaexp) \preceq (\alphabetalo)
}
$$
is a partition of $p(\alphabeta)$.
\end{lemma}
\begin{proof}
It follows from \LemRef{trunc-disjoint} and the fact that $
p(\loose) = p(\exact) \sqcup p(0),
$
where $\sqcup$ denotes the disjoint union.
\end{proof}

\begin{corollary}
Let $\alphabetaex$ be an exact truncated transition and let $\alphabetalo$ be the same transition, but with all symbols $\exact$ replaced by $\loose$. Then
$$
|\alphabetaex| = |\alphabetalo| - \sum_{
\alpha_{\exact}',\beta_{\exact}' \in \pset{0,\exact}^t,
(\alphabetaexp) \preceq (\alphabetalo)
}
|p(\alphabetaexp)|.
$$
\end{corollary}

%Note that $|\alphabetaex|$ can be also expressed in terms of cardinalities of loose ``subtransitions'' of $\alphabetalo$, which were computed in the first step, by using the inclusion-exclusion principle. However, since we need to compute all cardinalities of all exact transitions, we can simply use the \emph{partition} into exact ``subtransitions'' given by Equation~\ref{eq:partition-trunc}. Indeed,
This corollary gives an efficient way to compute cardinalities of all exact truncated transitions. By computing the cardinalities in the lexicographic order of transitions, we can ensure that all sub-transitions are processed before processing the current transition.

Given the cardinalities of exact transitions, it is easy to compute the probabilities of exact transitions, and thus, the matrix $\TTT{L}$.

\paragraph{Complexity.}
The time complexity of the naive implementation is $\OO(4^t(tm)^3 + 4^t4^t))$, where the first term corresponds to the complexity of the rank computation for all block-aligned submatrices of $L$, and the second term corresponds to the complexity of the summing over ``subtransitions''. The latter step can be done in one extra pass in time $\OO(t \cdot 4^t)$ by an algorithm similar to the well-known algorithm for the \Mobius{} transform. Then the complexity becomes fully dominated by the rank computations: $\OO(4^t\cdot (tm)^3)$.
The algorithm can be directly improved if a better algorithm for computing the ranks of all block-aligned submatrices exists.

\subsection{Iterative Algorithm for Truncated Trail Search}

The trails of truncated transitions that are the most useful for cryptanalysis, should have probabilities significantly higher than the probability of sampling the final truncated difference uniformly at random. Such trails may be used to distinguish the analyzed structure from an ideal primitive.

\begin{definition}
A truncated trail $\alpha_0 \xrightarrow{L} \ldots \xrightarrow{L} \alpha_r$ over $L: \ffield{m}{t} \to \ffield{m}{t}$ is said to be \emph{effective} if all $\alpha_i \ne 0$ and
$$
\Pr\psquare{\alpha_0 \xrightarrow{L} \ldots \xrightarrow{L} \alpha_r} >
\Pr_{\delta \in \ffield{m}{t}\setminus\pset{0}} \psquare{\delta \in p(\alpha_r)} = p(\alpha_r)/(2^{mt}-1).
$$
where the trail probability is equal to the
\end{definition}

Given the $\TTT{}$ a linear layer, it is easy to compute the best effective truncated trails in an iterative way. The method is based on dynamic programming. For each round $r$, we keep for each truncated output mask the best probability of reaching this mask from arbitrary input mask over $r$ rounds. Extension to $r+1$ rounds is done by enumerating best output masks for $r$ rounds and extending them using the $\TTT{}$. The algorithm sketch is given in \AlgRef{truncated-trail-search}.

\begin{algorithm}
    \AlgDef{truncated-trail-search}{Search for best truncated trails.}
    \begin{algorithmic}[1]
    \Require a binary matrix of $L: (\field{m})^t \to (\field{m})^t$, $\TTT{L}\colon \field{t} \times \field{t} \to \mathbb{Q}$;
    \Ensure the map $\alpha_r \mapsto q$ for the best effective truncated trails $\alpha_0 \xrightarrow{L} \ldots \xrightarrow{L} \alpha_r$

    \State $d_0 \gets \pset{\alpha \mapsto 1 \mid \alpha \in \pset{0,\exact}^t\setminus(0,\ldots,0)}$
    \ForAll{$r \in \seg{1}{}$}
        \State $d_r \gets \pset{\alpha \mapsto 0 \mid \alpha \in \pset{0,\exact}^t}$
        \State $effective \gets false$
        \ForAll{$\alphabeta \in \TTT{L}$}
            \State $d_r(\beta) \gets \max\pround{ d_r(\beta), d_{r-1}(\alpha) \cdot \Pr\psquare{\alphabeta}}$
            \If{$d_r(\beta) > p(\beta)/(2^{mt}-1)$} \Comment{$p(\beta)$ is the support of $\beta$}
                \State $effective \gets true$
            \EndIf
        \EndFor
        \If{not $effective$}
            \State \Return $d_r$ \Comment{Trail recovery can be added}
        \EndIf
    \EndFor
    \end{algorithmic}
\end{algorithm}

Using precise arithmetics over rationals, the precise $\TTT{}$ can be computed. For example, consider the AES MixColumn matrix $L\colon \ffield{8}{4} \to \ffield{8}{4}$.
The algorithm finds the following two-round effective trail:
$$
(+, 0, 0, 0) \xrightarrow{L} (+, +, +, +) \xrightarrow{L} (+, 0, 0, 0),
$$
which has probability
$$
1/16581375 \approx 2^{-23.983}
$$
which is greater than the probability of sampling the difference fitting $(+, 0, 0, 0)$ uniformly at random from all non-zero differences, which is equal to $$
(2^8-1)/(2^{32}-1)=1/16843009\approx 2^{-24.006}.
$$
The position of active word in the initial and output masks does not matter. The first transition has probability one due to the fact that $L$ has branching number 5 (note that the algorithm was given only the binary matrix of $L$). The second transition is possible due to uneven distribution of weights of differences in the image of $p((+,+,+,+))$ under $L$. This is an interesting observation, though the difference between the probabilities is too small for exploiting it for the cryptanalysis purpose.

\subsection{Truncated Trails in \aCipher{}}

For most linear layers used in practice, the probabilities of truncated transitions over the linear layer are usually close to powers of 2 raised to the word size. The error term is ignored as insignificant. Indeed, since the S-Boxes are fixed, the assumed independence between sequential truncated transitions does not hold.

Consider the linear layer of \aCipher{} as a mapping of $\ffield{64}{\branches}$ to itself. For the analysis of \aCipher{}, we also utilize the assumption, as all the transition probabilities over the linear layer are very close to $2^{-64k}$ for some k. We strengthen the definition of an effective trail by requiring that the trail probability is higher than $2^{-64k+0.01}$, where $k$ is the number of inactive words in the output mask.

For $\aCipher{256}$, the longest effective truncated differential trail covers two steps and has probability 1. It can be described as follows, where $\exact$ indicates an active branch and $0$ indicates an inactive branch:
$$
\begin{array}{ccccc}
\text{input } : & 0 & 0 & 0 & \exact\\
\text{step } 1: & \exact & 0 & 0 & 0\\
\text{step } 2: & \exact & \exact & \exact & 0
\end{array}.
$$

Another similar one can be obtained using the input $00\exact0$. When restricting the input difference to be only in the left branches (i.e., for the setting in \aeadversion{128}{128}), the longest effective truncated differential trail covers only one step (and probability 1):
$$
\begin{array}{ccccc}
\text{input } : & \exact & 0 & 0 & 0\\
\text{step } 1: & \exact & \exact & \exact & 0
\end{array}.
$$

For $\aCipher{384}$, the longest effective truncated differential trail also covers two steps and has probability 1:
$$
\begin{array}{ccccccc}
\text{input } : & 0 & 0 & 0 & \exact & 0 & 0\\
\text{step } 1: & 0 & 0 & \exact & 0 & 0 & 0\\
\text{step } 2: & \exact & \exact & \exact & 0 & 0 & \exact
\end{array}.
$$

Two similar ones can be obtained using inputs $0000\exact0$ and $00000\exact$.

For $\aCipher{512}$, the longest effective truncated differential trail covers three steps and has probability close to $2^{-64}$:
$$
\begin{array}{ccccccccc}
\text{input } : & 0 & 0 & 0 & 0 & 0 & \exact & 0 & \exact\\
\text{step } 1: & \exact & 0 & \exact & 0 & 0 & 0 & 0 & 0\\
\text{step } 2: & 0 & \exact & 0 & \exact & \exact & 0 & \exact & 0\\
\text{step } 3: & \exact & \exact & \exact & \exact & 0 & \exact & 0 & \exact\\
\end{array},
$$
where we associate a probability of $2^{-64}$ for the transition between step 1 and step 2.

% Such truncated trails where two branches are active and activate only two outputs of the Feistel functions exist for all number of branches strictly greater than 4 and, in particular, for both \aCipher{384} and \aCipher{512}. The purpose of the rotation of the branches after the addition of the Feistel function in the linear layer is to prevent the iteration of such truncated trails. While they would pose no threat as such, they could serve as the template for high probability not-truncated differential trails.


% Created 2021-10-05 Tue 08:37
% Intended LaTeX compiler: pdflatex
\documentclass[presentation,aspectratio=169, usenames, dvipsnames]{beamer}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{grffile}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{textcomp}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage{khpreamble}
\usepackage{amssymb}
\usepgfplotslibrary{groupplots}
\newcommand*{\shift}{\operatorname{q}}
\definecolor{ppc}{rgb}{0.1,0.1,0.6}
\definecolor{iic}{rgb}{0.6,0.1,0.1}
\definecolor{ddc}{rgb}{0.1,0.6,0.1}
\usetheme{default}
\author{Kjartan Halvorsen}
\date{\today}
\title{}
\hypersetup{
 pdfauthor={Kjartan Halvorsen},
 pdftitle={},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 26.3 (Org mode 9.4.6)},
 pdflang={English}}
\begin{document}

\section{Fitting first-order model with delay}
\label{sec:org6bf3618}

\begin{frame}[label={sec:org723cf14}]{First-order model with delay - example}
\[  \quad Y(s) = \frac{K\mathrm{e}^{-s\tau}}{sT + 1}U(s) \quad \overset{U(s) = \frac{u_f}{s}}{\Longrightarrow} \quad y(t) = u_f K\big( 1 - \mathrm{e}^{-\frac{t-\tau}{T}}\big)u_s(t-\tau)\]
\def\Tcnst{1.8}
\def\tdelay{1.2}
\def\ggain{1.5}
\def\uampl{0.9}
\pgfmathsetmacro{\yfinal}{\uampl*\ggain}
\pgfmathsetmacro{\yone}{0.283*\yfinal}
\pgfmathsetmacro{\ytwo}{0.632*\yfinal}
\pgfmathsetmacro{\tone}{\tdelay + \Tcnst/3}
\pgfmathsetmacro{\two}{\tdelay + \Tcnst}

\begin{center}
  \begin{tikzpicture}
    \begin{axis}[
    width=12cm,
    height=4cm,
    grid = both,
    %xtick = {0, \tdelay, \tone, \two},
    %xticklabels = {0, $\tau$, $\tau+\frac{T}{3}$, $\tau + T$},
    %ytick = {0, \yone, \ytwo, \uampl, \yfinal},
    %yticklabels = {0, $0.283y_{f}$, $0.632y_f$, $u_f$, $y_f$},
    xmin = -0.2,
    minor y tick num=9,
    minor x tick num=9,
    every major grid/.style={red, opacity=0.5},
    %xlabel = {$t$},
    clip = false,
    ]
      \addplot [thick, green!50!black, smooth, no marks, domain=0:10, samples=16] {\uampl*\ggain*(x>\tdelay)*(1 - exp(-(x-\tdelay)/\Tcnst)} node [coordinate, pos=0.9, pin=-90:{$y(t)$}] {};
      \addplot [const plot, thick, blue!80!black, no marks, domain=-1:10, samples=100] coordinates {(-1,0) (0,0) (0,\uampl) (10,\uampl)} node [coordinate, pos=0.9, pin=-90:{$u(t)$}] {};
      \draw[thick, green!70!black, dashed] (axis cs: 10, \yfinal) -- (axis cs: -1, \yfinal, -0.9) node[left, anchor=east] {$y_f = \yfinal$};
      \draw[blue!70!black, dashed] (axis cs: 0, \uampl) -- (axis cs: -1, \uampl, -0.9) node[left, anchor=east] {$u_f = \uampl$};
    \end{axis}
  \end{tikzpicture}
\end{center}
\end{frame}

\begin{frame}[label={sec:org2546d65}]{First-order model with delay - example}
\[  \quad Y(s) = \frac{K\mathrm{e}^{-s\tau}}{sT + 1}U(s) \quad \overset{U(s) = \frac{u_f}{s}}{\Longrightarrow} \quad y(t) = u_f K\big( 1 - \mathrm{e}^{-\frac{t-\tau}{T}}\big)u_s(t-\tau)\]
\def\Tcnst{1.8}
\def\tdelay{1.2}
\def\ggain{1.5}
\def\uampl{0.9}
\pgfmathsetmacro{\yfinal}{\uampl*\ggain}
\pgfmathsetmacro{\yone}{0.283*\yfinal}
\pgfmathsetmacro{\ytwo}{0.632*\yfinal}
\pgfmathsetmacro{\tone}{\tdelay + \Tcnst/3}
\pgfmathsetmacro{\two}{\tdelay + \Tcnst}

\begin{center}
  \begin{tikzpicture}
    \begin{axis}[
    width=12cm,
    height=4cm,
    grid = both,
    %xtick = {0, \tdelay, \tone, \two},
    %xticklabels = {0, $\tau$, $\tau+\frac{T}{3}$, $\tau + T$},
    %ytick = {0, \yone, \ytwo, \uampl, \yfinal},
    %yticklabels = {0, $0.283y_{f}$, $0.632y_f$, $u_f$, $y_f$},
    xmin = -0.2,
    minor y tick num=9,
    minor x tick num=9,
    every major grid/.style={red, opacity=0.5},
    %xlabel = {$t$},
    clip = false,
    ]
      \addplot [thick, green!50!black, smooth, no marks, domain=0:10, samples=16] {\uampl*\ggain*(x>\tdelay)*(1 - exp(-(x-\tdelay)/\Tcnst)} node [coordinate, pos=0.9, pin=-90:{$y(t)$}] {};
      \addplot [const plot, thick, blue!80!black, no marks, domain=-1:10, samples=100] coordinates {(-1,0) (0,0) (0,\uampl) (10,\uampl)} node [coordinate, pos=0.9, pin=-90:{$u(t)$}] {};
      \draw[thick, red, dashed] (axis cs: \tone, \yone) -- (axis cs: \tone, -0.45) node[below] {$t_1 = \tone = \tau + \frac{T}{3}$};
      \draw[thick, red, dashed] (axis cs: \tone, \yone) -- (axis cs: -1,\yone) node[left, anchor=east] {$0.283y_f = \yone$};
      \draw[thick, orange, dashed] (axis cs: \two, \ytwo) -- (axis cs: \two, -0.9) node[below] {$t_2 = \two = \tau + T$};
      \draw[thick, orange, dashed] (axis cs: \two, \ytwo) -- (axis cs: -1, \ytwo, -0.9) node[left, anchor=east] {$0.632y_f = \ytwo$};
      \draw[thick, green!70!black, dashed] (axis cs: 10, \yfinal) -- (axis cs: -1, \yfinal, -0.9) node[left, anchor=east] {$y_f = \yfinal$};
      \draw[blue!70!black, dashed] (axis cs: 0, \uampl) -- (axis cs: -1, \uampl, -0.9) node[left, anchor=east] {$u_f = \uampl$};
    \end{axis}
  \end{tikzpicture}
\end{center}
\[ \begin{cases} \tone = \tau + \frac{T}{3}\\ \two = \tau + T \end{cases} \quad \Rightarrow \quad \begin{cases} \tau = \tdelay \\ T = \Tcnst \end{cases}, \qquad  K = \frac{y_f}{u_f} = \frac{\yfinal}{\uampl} = \ggain \]
\end{frame}
\end{document}


\section{Notes on quantum mechanics}

\subsection{$\widehat{x}$ and $\widehat{p}$ acting on $\Ket{x}$}

\begin{namedthm}{Linear hermitian functions}
\begin{equation}
	\widehat{p} \Ket{x} = \rbr{+\ii \frpa{}{x}} \Ket{x},\qquad
	\Bra{x} \widehat{p} = \rbr{-\ii \frpa{}{x}} \Bra{x}.
\end{equation}

\end{namedthm} % Linear hermitian functions
\begin{proof}
\begin{align}
\Bra{x} \widehat{p} &= \int_{-\infty}^{+\infty}\dif p\,\Braket{x | p}
\Bra{p} \widehat{p}
= \int_{-\infty}^{+\infty}\dif p\,\rbr{2\pp}^{-1/2}\ee^{\ii p x}
\Bra{p} p \nonumber \\
&= \int_{-\infty}^{+\infty}\dif p\,
\rbr{-\ii \frpa{}{x}} \rbr{2\pp}^{-1/2} \ee^{\ii p x} \Bra{p}
= \rbr{-\ii \frpa{}{x}} \int_{-\infty}^{+\infty}\dif p\,
\Braket{x | p} \Bra{p} \nonumber \\
&= \rbr{-\ii \frpa{}{x}} \Bra{x}.
\end{align}
\end{proof}

\begin{namedthm}{Linear unitary exponentials}

\begin{equation}
\ee^{\ii p_0 \widehat{x}} \Ket{x} = \ee^{\ii p_0 x} \Ket{x},
\qquad
\Bra{x} \ee^{\ii p_0 \widehat{x}} = \Bra{x} \ee^{\ii p_0 x};
\end{equation}
\begin{equation}
\ee^{\ii x_0 \widehat{p}} \Ket{x} = \Ket{x-x_0},\qquad
\Bra{x} \ee^{\ii x_0 \widehat{p}} = \Bra{x+x_0}.
\end{equation}

\end{namedthm} % Linear unitary exponentials

\begin{proof}
\begin{align}
\ee^{\ii x_0 \widehat{p}} \Ket{x} &= \int_{-\infty}^{+\infty} \dif p\,
\ee^{\ii x_0 \widehat{p}}\Ket{p}\Braket{p | x} \nonumber \\
&= \int_{-\infty}^{+\infty} \dif p\,
\ee^{\ii x_0 p}\Ket{p}\rbr{2\pp}^{-1/2}\ee^{-\ii x p}
= \int_{-\infty}^{+\infty} \dif p\,
\rbr{2\pp}^{-1/2}\ee^{-\ii \rbr{x-x_0} p}\Ket{p} \nonumber \\
&= \int_{-\infty}^{+\infty} \dif p\,
\Ket{p}\Braket{p | x-x_0} = \Ket{x-x_0};
\label{eq:displace-x-ket}
\end{align}

\begin{equation}
\Bra{x} \ee^{\ii x_0 \widehat{p}} = \int_{-\infty}^{+\infty} \dif p\,
\ee^{\ii x_0 p}\Bra{p}\rbr{2\pp}^{-1/2}\ee^{+\ii x p}
= \Bra{x+x_0}.
\label{eq:displace-x-bra}
\end{equation}

\end{proof}

\begin{namedthm}{Quadratic hermitian functions}
\begin{equation}
\widehat{x}^2 \Ket{x} = x^2 \Ket{x}, \qquad
\Bra{x} \widehat{x}^2 = \Bra{x} x^2;
\end{equation}
\begin{equation}
	\widehat{p}^2 \Ket{x} = -\frpa{^2}{x^2} \Ket{x}, \qquad
	\Bra{x} \widehat{p}^2 = -\frpa{^2}{x^2} \Bra{x};
\end{equation}
\begin{equation}
	\frac{\widehat{x}\widehat{p} + \widehat{p}\widehat{x}}{2} \Ket{x}
	= +\ii \rbr{\frac{1}{2}+x\frpa{}{x}} \Ket{x}, \qquad
	\Bra{x} \frac{\widehat{x}\widehat{p} + \widehat{p}\widehat{x}}{2}
	= -\ii \rbr{\frac{1}{2}+x\frpa{}{x}} \Bra{x}.
\end{equation}

\end{namedthm} % Quadratic hermitian functions
\begin{proof}
456
\end{proof}

\begin{namedthm}{Quadratic unitary exponentials}
\begin{equation}
\ee^{\ii \sigma \widehat{x}^2} \Ket{x} = \ee^{\ii \sigma x^2} \Ket{x},
\qquad
\Bra{x} \ee^{\ii \sigma \widehat{x}^2} = \Bra{x} \ee^{\ii \sigma x^2};
\end{equation}
\begin{align}
	\ee^{\ii \sigma \widehat{p}^2} \Ket{x}
	&= \rbr{-4\pp\ii\sigma}^{-1/2}\int_{-\infty}^{+\infty}\dif x'\,\Ket{x'}
	\cfun{\expi}{-\frac{\rbr{x-x'}^2}{4\sigma}}, \nonumber \\
	\Bra{x} \ee^{\ii \sigma \widehat{p}^2}
	&= \rbr{-4\pp\ii\sigma}^{-1/2}\int_{-\infty}^{+\infty}\dif x'\,\Bra{x'}
	\cfun{\expi}{-\frac{\rbr{x-x'}^2}{4\sigma}};
\end{align}
\begin{align}
	\ee^{\ii \lambda \rbr{\widehat{x}\widehat{p}
		+ \widehat{p}\widehat{x}}/2} \Ket{x}
	&= \ee^{-\lambda/2} \Ket{x\ee^{-\lambda}}, \nonumber \\
	\Bra{x} \ee^{\ii \lambda \rbr{\widehat{x}\widehat{p}
		+ \widehat{p}\widehat{x}}/2}
	&= \ee^{+\lambda/2} \bra{x\ee^{+\lambda}}.
\end{align}

\end{namedthm} % Quadratic unitary exponentials
\begin{proof}
789
\end{proof}

\subsection{Time evolution}

\begin{equation}
\rfun{\widehat{\mscrU}}{t;t_0} = \ee^{-\ii \widehat{H}\rbr{t-t_0}}
\end{equation}
is the time-evolution operator in the Schrödinger picture.

\subsection{Wigner transformation}

\emph{Wigner transformation}
\begin{equation}
\rfun{g}{x,p} \coloneqq \frac{1}{2\pp}\int_{-\infty}^{+\infty}\dif s\,
\ee^{\ii p s} \Braket{ x-\frac{s}{2} | \widehat{G} | x + \frac{s}{2} }.
\end{equation}

\begin{equation}
\int_{\BbbR^2}\dif x\diff p\,\rfun{g}{x,p} \equiv \tr \widehat{G}.
\end{equation}

\


\chapter{3D geometry}
In this chapter we cover the basic objects and concepts of 3D geometry, and the operations we can do on them. Although the additional dimension adds some complexity which can make things more difficult and interesting, a lot of things work the same way as they do in 2D. Therefore, we will make frequent references to chapter \ref{ch:basics}, which we consider as a prerequisite.

\input{3d/p3}
\input{3d/plane}
\input{3d/line3d}
\input{3d/para-perp}
\input{3d/polyhedron}
\input{3d/sph/s-sph}


\subsection{Initial and final topologies}\label{subsec:initial_final_topologies}

\begin{definition}\label{def:category_of_small_topological_spaces}
  Since the topology \( \mscrT \) of a \hyperref[def:topological_space]{topological space} \( (\mscrX, \mscrT) \) consists of subsets of \( \mscrX \), we cannot build a \hyperref[def:first_order_theory]{first-order theory} from \fullref{def:topological_space/O1}-\fullref{def:topological_space/O3}. We can, however, explicitly describe the \hyperref[def:category]{category} \( \cat{Top} \) of topological spaces as
  \begin{refenum}
    \refitem{def:category/objects} The \hyperref[def:set]{class} of objects is the class of all topological space.
    \refitem{def:category/morphisms} The morphisms between two topological spaces are the \hyperref[def:global_continuity]{continuous functions} between them.
    \refitem{def:category/composition} Composition of morphisms is the usual \hyperref[def:multi_valued_function/composition]{function composition}.
  \end{refenum}
\end{definition}

\begin{theorem}\label{thm:top_complete_cocomplete}
  The category \( \cat{Top} \) of is both \hyperref[def:category_of_cones/limit]{complete} and \hyperref[def:category_of_cones/colimit]{cocomplete}.
\end{theorem}

\begin{definition}\label{def:initial_topology}\mcite{nLab:top}
  Let \( \{ (X_k, \mscrT_k) \}_{k \in \mscrK} \) be a \hyperref[def:cartesian_product/indexed_family]{family} of topological spaces. Let \( X \) be a bare set and let
  \begin{equation*}
    \{ f_k: X \to X_k \}_{k \in \mscrK}
  \end{equation*}
  be a family of functions.

  The topology on \( X \) generated by the subbase
  \begin{equation*}
    \mathcal{P} \coloneqq \{ f_k^{-1}(U) \colon k \in \mscrK, U \in T_k \}
  \end{equation*}
  is called the \term{initial} (or \term{weak}) topology on \( X \) generated by the family \( \{ f_k \}_{k \in \mscrK} \).

  It is the weakest topology that makes all functions in the family \( \{ f_k \}_{k \in \mscrK} \) continuous.
\end{definition}

\begin{definition}\label{def:final_topology}\mcite{nLab:top}
  Dually, if the family of functions is of the type
  \begin{equation*}
    \{ f_k: X_k \to X \}_{k \in \mscrK},
  \end{equation*}
  then we define the \term{final} (or \term{strong}) topology on \( X \) generated by the family \( \{ f_k \}_{k \in \mscrK} \) as the topology
  \begin{equation*}
    \mscrT \coloneqq \{ U \subseteq X \colon \forall k \in \mscrK, f_k^{-1}(U) \in T_k \}.
  \end{equation*}

  It is the strongest topology that makes all functions in the family \( \{ f_k \}_{k \in \mscrK} \) continuous.
\end{definition}

\begin{proposition}\label{thm:initial_final_topology_limit}\mcite{nLab:top}
  Let \( D: \boldop I \to \cat{Top} \) be a small \hyperref[def:categorical_diagram]{diagram}. For each space in the image \( D(\boldop I) \), denote the set corresponding by \( X_k \) and the corresponding topology by \( \mscrT_k \).

  The limit (resp. colimit) \( (X, \mscrT) \) of \( D \) can then be described as
  \begin{thmenum}
    \item \( (X, \{ f_k \}_{k \in \cat{I}}) = \varprojlim UD \) (resp. \( \varinjlim UD \)) is the limit (resp. colimit) in \( \cat{Set} \) of \( U \circ D \), where \( U: \boldop{Top} \to \cat{Set} \) is the forgetful functor.
    \item \( \mscrT \) is the \hyperref[def:initial_topology]{initial} (resp. \hyperref[def:final_topology]{final}) topology on \( X \) generated by the family of functions \( \{ f_k \}_{k \in \cat{I}} \).
  \end{thmenum}
\end{proposition}

\begin{definition}\label{def:topological_subspace}
  Let \( (X, \mscrT) \) be a topological space and let \( M \subseteq X \) be a subset of \( X \). The \term{topological subspace} \( (M, \mscrT_M) \) is obtained by endowing \( M \) with the topology
  \begin{equation*}
    \mscrT_M \coloneqq \{ U \cap M \colon U \in T \}.
  \end{equation*}

  The topology \( \mscrT_M \) is called the \term{subspace topology} or \term{induced topology}.

  It is the initial topology generated by the canonical embedding \( \iota: M \to X \).
\end{definition}

\begin{definition}\label{def:topological_product}
  The \term{topological product} or \term{Tychonoff product}
  \begin{equation*}
    \left( \prod_{k \in \mscrK} X_k, \prod_{k \in \mscrK} \mscrT_k \right)
  \end{equation*}
  of the family \( { (X_k, \mscrT_k) }_{k \in \mscrK} \) is simply the categorical product in the category \( \cat{Top} \) (see \fullref{def:discrete_category_limits}). The underlying set \( \prod_{k \in \mscrK} X_k \) is the \hyperref[thm:discrete_category_limits_in_set]{Cartesian product} and the topology \( \prod_{k \in \mscrK} \mscrT_k \) is called the \term{product topology}.

  Let \( { (X_k, \mscrT_k) }_{k \in \mscrK} \) and \( { (Y_k, \mathcal{O}_k) }_{k \in \mscrK} \) be two families of topological spaces and let
  \begin{equation*}
    \{ f_k: X_k \to Y_k \}_{k \in \mscrK}
  \end{equation*}
  be a family of arbitrary functions between them.

  We define the \term{product \( \prod_{k \in \mscrK} f_k \) of \( \{ f_k \}_{k \in \mscrK} \)} as the function
  \begin{balign*}
     & \left(\prod_{k \in \mscrK} f_k \right): \prod_{k \in \mscrK} X_k \to \prod_{k \in \mscrK} Y_k              \\
     & \left(\prod_{k \in \mscrK} f_k \right)(\{ x_k \}_{k \in \mscrK}) \coloneqq \{ f_k (x_k) \}_{k \in \mscrK}.
  \end{balign*}

  If all of the spaces \( (X_k, \mscrT_k) \) are equal to some space \( (X, \mscrT) \), we call the product of \( \{ f_k \}_{k \in \mscrK} \) the \term{diagonal product} and denote it by
  \begin{equation*}
    \Delta_{k \in \mscrK} f_k: X \to \prod_{k \in \mscrK} Y_k.
  \end{equation*}
\end{definition}

\begin{definition}\label{def:topological_quotient}\mcite[90]{Engelking1989}
  Let \( X \) be a topological space and let \( \cong \) be an \hyperref[def:equivalence_relation]{equivalence relation} on \( X \). The \term{quotient space} \( (X, \mscrT) / \sim \) is obtained by endowing the quotient set \( X / \cong \) with the final \hyperref[def:final_topology]{topology} given by the canonical projection map \( x \mapsto [x] \).
\end{definition}

\begin{definition}\label{def:topological_sum}\mcite[74]{Engelking1989}
  The \term{topological direct sum}
  \begin{equation*}
    (\oplus_{k \in \mscrK} X_k, \oplus_{k \in \mscrK} \mscrT_k)
  \end{equation*}
  of the family \( { (X_k, \mscrT_k) }_{k \in \mscrK} \) is simply the categorical coproduct in the category \( \cat{Top} \) (see \fullref{def:discrete_category_limits}). The underlying set \( \oplus_{k \in \mscrK} X_k \) is the \hyperref[thm:discrete_category_limits_in_set]{disjoint union} and the topology \( \oplus_{k \in \mscrK} \mscrT_k \) is called the \term{direct sum topology}.

  Let \( { (X_k, \mscrT_k) }_{k \in \mscrK} \) and \( { (Y_k, \mathcal{O}_k) }_{k \in \mscrK} \) be two families of topological spaces and let
  \begin{equation*}
    \{ f_k: X_k \to Y_k \}_{k \in \mscrK}
  \end{equation*}
  be a family of arbitrary functions between them. Let \( \iota_{X_k}: X_k \to \oplus_{k \in \mscrK} X_k \) and \( \iota_{Y_k}: Y_k \to \oplus_{k \in \mscrK} Y_k \) be the corresponding canonical embeddings.

  We define the \term{direct sum \( \oplus_{k \in \mscrK} f_k \) of \( \{ f_k \}_{k \in \mscrK} \)} as the function
  \begin{balign*}
     & (\oplus_{k \in \mscrK} f_k): \oplus_{k \in \mscrK} X_k \to \oplus_{k \in \mscrK} Y_k   \\
     & (\oplus_{k \in \mscrK} f_k){\rvert}_{X_k} \coloneqq \iota_{Y_k} \circ f_k.
  \end{balign*}

  Obviously \( \oplus_{k \in \mscrK} f_k \) is continuous whenever all \( f_k \) are continuous.

  If all of the spaces \( (Y_k, \mathcal{O}_k) \) are equal to some space \( (Y, \mathcal{O}) \), we call the direct sum of \( \{ f_k \}_{k \in \mscrK} \) simply a \term{sum} and denote it by
  \begin{equation*}
    \sum_{k \in \mscrK} f_k: \oplus_{k \in \mscrK} X_k \to Y.
  \end{equation*}
\end{definition}


\documentclass{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath, amssymb, amsfonts}
% change equation numbering to section.eq_num
\numberwithin{equation}{section}
% set paragraph indent
\setlength\parindent{0pt}
% makes clickable links to sections
\usepackage{hyperref}
% make the link colors blue, as well as cite colors
\hypersetup{
    colorlinks, linkcolor = blue, citecolor = blue, urlcolor = magenta
}

\title{A normal cdf expectation}
\author{Derek Huang}
\date{January 2, 2021\thanks{Original published June 2019.}}

\begin{document}

\maketitle

\section{Introduction}

A couple years ago, I had a problem that involved the computation of the
expectation of a function of a normal random variable, where the function in
question was the cumulative distribution function of a different, independent
normal random variable. At the time, I wrote a 3-page article that mostly
consisted of me rewriting normal random variables as affine transformations of
other normal random variables, which consisted of much rote derivation and
frankly was not too interesting.

\medskip

This article is a much shorter version of the original. The problem has also
been trivially modified to be more general by dropping the assumption that the
normal random variables $ X, Y $ are independent.

\section{A solution}

Suppose we have $ X \sim \mathcal{N}(\mu, \sigma^2) $,
$ Y \sim \mathcal{N}(\nu, \tau^2) $, with correlation
$ \rho_{X, Y} \in [-1, 1] $. We want to compute the quantity
$ \mathbb{E}[F_X(Y)] $, where $ F_X $ is the cdf of $ X $, i.e. for
$ x \in \mathbb{R} $,
\begin{equation*}
    F_X(x) = \Phi\left(\frac{x - \mu}{\sigma}\right)
\end{equation*}

Here $ \Phi $ is the standard normal cdf. By the definition of expectation for a
continous random variable and by the law of total probability, we can
conveniently rewrite $ \mathbb{E}[F_X(Y)] $ as
\begin{equation*}
    \mathbb{E}[F_X(Y)] = \int_\mathbb{R}F_X(y)\phi_Y(y)\,dy =
    \int_\mathbb{R}\mathbb{P}\{X \le Y \mid Y = y\}\phi_Y(y)\,dy =
    \mathbb{P}\{X \le Y\}
\end{equation*}

Since $ \mathbb{P}\{X \le Y\} = \mathbb{P}\{X - Y \le 0\} $, we just need the
distribution of $ X - Y $ and we can compute $ \mathbb{E}[F_X(Y)] $. Since
$ X, Y $ are both normal random variables, we know
$ \mathbb{E}[X - Y] = \gamma \triangleq \mu - \nu $. However, since $ X, Y $ are
not necessarily independent, using properties of
variance\footnotemark\footnotetext{
    We can verify the first equality manually at the cost of some uninteresting
    work. Using the definition of variance,
    \begin{equation*}
        \begin{split}
            \mathrm{Var}(X - Y) & = \mathbb{E}\big[(X - Y - \gamma)^2\big] =
            \mathbb{E}\big[(X - Y)^2 - 2\gamma(X - Y) + \gamma^2\big] =
            \mathbb{E}\big[(X - Y)^2\big] - \gamma^2 \\
            & = \mathbb{E}\big[X^2 - 2XY + Y^2\big] - \mu^2 + 2\mu\nu - \nu^2 =
            \mathbb{E}\big[X^2] - \mu^2 + \mathbb{E}\big[Y^2\big] - \nu^2 -
            2(\mathbb{E}[XY] - \mu\nu) \\
            & = \mathrm{Var}(X) + \mathrm{Var}(Y) - 2\mathrm{cov}(X, Y)
        \end{split}
    \end{equation*}
},
\begin{equation*}
    \mathrm{Var}(X - Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) -
        2\mathrm{cov}(X, Y) =
    \sigma^2 + \tau^2 - 2\rho_{X, Y}\sigma\tau
\end{equation*}

We can then rewrite $ F_{X - Y} $, the cdf of $ X - Y $, using the standard
normal cdf. Therefore,
\begin{equation} \label{solution}
    \mathbb{E}[F_X(Y)] = F_{X - Y}(0) =
    \Phi\left(-\frac{\mu - \nu}{
        \sqrt{\sigma^2 + \tau^2 - 2\rho_{X, Y}\sigma\tau}
    }\right)
\end{equation}

In the typical scenario, $ Y $ is usually independent of $ X $, so the
correlation term in (\ref{solution}) may be omitted.

\end{document}