\documentclass[12pt,a4paper]{article}

\usepackage{fullpage}
\usepackage{parskip}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{hyperref}
\usepackage{bm}
\usepackage{enumerate}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{booktabs}

\newcommand{\pluseq}{\mathrel{+}=}
\newcommand{\asteq}{\mathrel{*}=}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

\title{Pattern Recognition Assignment 3: K-Means}
\author{Yu Hao\\[10pt] Student ID: 1831607}

\date{\today}

\begin{document}

\maketitle

\section{Introduction}
In this assignment, we are expected to implement a clustering algorithm. There are many kinds of methods for clustering, for example, K-Means, hierarchical clustering, DBSCAN, etc. I choose standard K-Means as the basic method and there will be some extensions to be mentioned later. As we confront a real problem from Huawei project which needs to partition dataset equally under the condition of minimizing within-cluster sum of squares (WCSS), we propose an extension of K-means to accomplish the objective.

Hence, the assignment is organized by two parts: the standard K-Means implementation and its extension. Furthermore, the second part will be filled and plump at our team's final project.
\section{Algorithm Formulation}
Given a set of data points $\mathbf{H}=\left \{ \bm{h}_{1},\ldots,\bm{h}_{N} \right \}$, where each point is a d-dimensional real-valued vector.
The aim of K-means algorithm is to divide $N$ data points into $K\ (\leqslant N)$ sets $\mathbf{C}=\left \{ \mathit{C}_{1},\ldots,\mathit{C}_{K} \right \}$
so as to minimize the within-cluster sum of squares (WCSS) function. Formally, the objective is to find
\begin{equation}
\begin{aligned}
\underset{\mathbf{C}}{\rm argmin}~&\sum\nolimits_{k=1}^{K}\sum\nolimits_{i\in C_{k}} \norm{ \bm{h}_{i}-\bm{c}_{k}}^{2}\\[6pt]
&=\sum\nolimits_i{ \norm{\bm{h}_i}^{2}} -\sum\nolimits_{k=1}^{K}\frac{1}{n_k}\sum\nolimits_{i,j\in C_{k}}{\bm{h}_{i}}^{T}\bm{h}_{j},
\end{aligned}
\end{equation}
where $\bm{c}_{k}$ represents the centroid of cluster $C_{k}$ for $k=1,2,\ldots,K$ and $\bm{n}={[n_{1},\ldots,n_{K}]}^{T}\in \mathbb{R}^{K}$ records the number of data points for $K$ clusters.
The optimization process is known as a NP-hard problem~\cite{garey1982complexity}, and the standard K-means algorithm  \cite{hartigan1975clustering} was proposed to provide an approximate solution. The algorithm uses an iterative refinement technique. Given an initial set of $K$ centroids $\{ \bm{c}_{1}^{(0)},\bm{c}_{2}^{(0)},\ldots,\bm{c}_{K}^{(0)}\}$, the algorithm proceeds with alternating strategy between an assignment step and an update step as follows.

During the assignment step, each data point is assigned to the cluster characterized by nearest centroid, that is
\begin{equation}
\begin{aligned}
C_{k}^{\left(t\right)}&=\left \{{ \mathbf {h}_{p}:\left \|{\mathbf {h}_{p} - \mathbf {c}_{k}^{\left(t\right)}}\right \|^{2}\le \left \|{ \mathbf {h}_{p} - \mathbf {c}_{j}^{\left(t\right)} }\right \|^{2}, 1\leq j\leq K }\right \}, \\[6pt]
n_{k}^{\left(t\right)}&=|C_{k}^{\left(t\right)}|, \\[6pt]
k&=1,\ldots , K,
\end{aligned}
\end{equation}
where each data point is assigned to exactly one cluster, and this step can minimize the WCSS function.

During the update step, the new centroids can be computed with current clusters:
\begin{equation}
\mathbf c_{k} ^{\left ({t+1}\right )} = \frac {1}{n_{k}^{(t)}} \sum _{\mathbf h_{j} \in C_{k}^{\left ({t}\right )}} \mathbf h_{j},~k=1,\ldots ,K.
\end{equation}
Since the arithmetic mean is a least-squares estimator, this step also minimizes the WCSS function.

The algorithm converges when the centroids no longer change, in other words, the data points are no longer assigned to new cluster. The K-means algorithm can converge to a local optimum. Since for a given $K$, the result of above clustering algorithm depends solely on the initial centroids.

The computational complexity of the K-means algorithm is $O(NdKM)$, where $N$ is the number of the d-dimensional vectors, $K$ is the number of clusters and $M$ is the number of iterations before reaching convergence.

\section{K-Means Implementation}
Three parts will be mentioned there. Similar with former assignments, I choose two datasets: The mobile prices classifications and glasses types classifications. Algorithm implementation consists of three parts, which are centroids initialization, update data points assignments, and recalculate centroids. After acquiring the result, we will compare clustering results with true labels to examine the clustering quality. Visualization is the last implementation to see the result clearly.

\subsection{Data Preprocessing}
We have two datasets and both of them are numerical, so I just need to normalize the features. The purpose of normalization is to avoid the influences of features' order of magnitudes, rather focus on their distributions and relative distances. Standardizing is very simple:
\begin{equation}
z=\frac{x-\mu}{\sigma},
\end{equation}
where $x$ is the number to be standardized, $\mu$ is the feature's mean, and $\sigma$ is the standard deviation.

\subsection{Fit}
This module is to do the clustering. Firstly we need to choose $K$ centroids as the initial centroids. I choose the corresponding data points from the dataset as the centroids. There are some input checks. The algorithm cannot handle the circumstance that the number of clusters bigger than data points'. And obviously, the clusters number needs to be a positive integer.

Then we iterate the next two steps until the \textbf{centroids} don't change anymore:
\begin{itemize}
	\item Update assignments. I calculate the Euclidean distances from all data points to all centroids and choose the closest centroid to be the representative of each point. After accomplishing the calculation, I record the result to an array: \textbf{clustering\_results}.

	\item Recalculate centroids. For the data points belong to the same cluster, they need a new representative. For simplicity, the mean of these data points is a suitable choice. I denote the result as \textbf{centroids}.
\end{itemize}

\subsection{Result Comparison}
As the dataset has true labels, we can examine the quality of clustering using Adjusted Rand index. The Rand index in data clustering is a measure for calculating the similarity between two data clusterings. From a mathematical standpoint, the Rand index is related to the accuracy but is applicable even when class labels are not used.
\subsubsection{Rand index}
Given a set of $N$ elements $\mathbf{S}=\left\{s_1,\ldots,s_N\right\}$ and two partitions of $\mathbf{S}$ to compare, $\mathbf{X}=\left\{X_1,\ldots,X_K\right\}$, a partition of $\mathbf{S}$ into $K$ subsets, and $\mathbf{Y}=\left\{Y_1,\ldots,Y_Q\right\}$, a partition of $\mathbf{S}$ into $Q$ subsets. define the following:
\begin{itemize}
	\item $a$, the number of pairs of elements in $\mathbf{S}$ that are in the same subset in $\mathbf{X}$ and in the same subset in $\mathbf{Y}$;
	\item $b$, the number of pairs of elements in $\mathbf{S}$ that are in different subsets in $\mathbf{X}$ and in different subsets in $\mathbf{Y}$;
	\item $c$, the number of pairs of elements in $\mathbf{S}$ that are in the same subset in $\mathbf{X}$ and in different subsets in $\mathbf{Y}$;
	\item $b$, the number of pairs of elements in $\mathbf{S}$ that are in different subsets in $\mathbf{X}$ and in the same subset in $\mathbf{Y}$.
\end{itemize}
The Rand index, $R$, is:
\begin{equation}
R=\frac{a+b}{a+b+c+d}=\frac{a+b}{\binom{N}{2}}.
\end{equation}

Intuitively, $a+b$ can be considered as the number of agreements between $\mathbf{X}$ and $\mathbf{Y}$, and $c+d$ as the number of disagreement between them. Since the denominator is the total number of pairs, the Rand index represents the frequency of occurrence of agreement over the total pairs or the probability that $\mathbf{X}$ and $\mathbf{Y}$ will agree on a randomly chosen pair.

\subsubsection{Adjusted Rand index}
The adjusted Rand index is the corrected-for-chance version of the Rand index. Such a correction for chance establishes a baseline by using the expected similarity of all pair-wise comparisons between clusterings specified by a random model. Traditionally, the Rand Index was corrected using the Permutation Model for clusterings (the number and size of clusters within a clustering are fixed, and all random clusterings are generated by shuffling the elements between the fixed clusters). However, the premises of the permutation model are frequently violated; in many clustering scenarios, either the number of clusters or the size distribution of those clusters vary drastically. For example, consider that in K-Means the number of clusters is fixed by the practitioner, but the sizes of those clusters are inferred from the data. Variations of the adjusted Rand Index account for different models of random clusterings.

Though the Rand Index may only yield a value between 0 and +1, the adjusted Rand index can yield negative values if the index is less than the expected index.

Given a set S of $N$ elements, and two groupings or partitions (e.g. clusterings) of these elements, namely $\mathbf{X}=\{X_{1},X_{2},\ldots ,X_{K}\}$ and $\mathbf{Y}=\{Y_{1},Y_{2},\ldots ,Y_{Q}\}$, the overlap between X and Y can be summarized in a contingency table $\left[n_{ij}\right]$ where each entry $n_{ij}$ denotes the number of objects in common between $X_i$ and $Y_j$: $n_{ij}=|X_{i} \cap Y_{j}|$.

\begin{table}
\centering
\begin{tabular}{c c c c c c}
\hline
\hline
~ & $Y_1$ &	$Y_2$ &	$\cdots$ & $Y_Q$  & Sums \\[6pt]
\hline
$X_1$ &	$n_{11}$ & 	$n_{12}$ & 	$\cdots$  &	$n_{1Q}$  &	$a_1$ \\[6pt]
\hline
$X_2$ & $n_{21}$ 	& $n_{22} $	& $\cdots$	& $n_{2Q}$	& $a_2$ \\[6pt]
\hline
$\vdots$ & $\vdots$ & $\vdots$ & $\ddots$ & $\vdots$ & $\vdots$ \\[6pt]
\hline
$X_K$ & $n_{K1}$ & $n_{K2}$ & $\cdots$ & $n_{KQ}$ & $a_K$ \\[6pt]
\hline
$Sums$ & $b_1$ & $b_2$ & $\cdots$ &$b_Q$ & ~\\[6pt]
\hline
\hline
\end{tabular}
\caption{contingency table}
\end{table}

The original Adjusted Rand Index using the Permutation Model is
\begin{equation}
\begin{aligned}
ARI &= \frac{Index-ExpectedIndex}{MaxIndex-ExpectedIndex} \\[6pt]
Index &= \sum_{ij}\binom{n_{ij}}{2} \\[6pt]
ExpectedIndex &= \frac{\sum_{i}\binom{a_i}{2}\sum_{j}\binom{b_j}{2}}{\binom{N}{2}} \\[6pt]
MaxIndex &= \frac{\sum_{i}\binom{a_i}{2}+\sum_{j}\binom{b_j}{2}}{2}
\end{aligned}
\end{equation}

Hence, the $ARI$ can be an effective index to examine the similarity between the clustering result and true classification. For the two datasets I mentioned, table \ref{ARI result} shows simulation result.

\begin{table}
\centering
\begin{tabular}{l l l l}
\hline
\hline
Dataset & Number of dataset & Number of clusters & ARI \\[6pt]
\hline
mobile prices classes & 2000 & 4 & 0.466 \\[6pt]
\hline
glasses types & 142 & 7 & 0.598 \\[6pt]
\hline
\hline
\end{tabular}
\caption{ARI of two datasets \label{ARI result}}
\end{table}

\subsection{Visualization}
Here I draw the true classifications and clustering (predicting) results in pictures using different colors to classify corresponding clusters.

\begin{figure}
\centering
\includegraphics[width=11cm]{mobile-true}
\caption{Mobile prices classes: True result}
\label{mobile-true}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=11cm]{mobile-predict}
\caption{Mobile prices classes: Predicting result}
\label{mobile-predict}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=11cm]{glasses-true}
\caption{Glasses types: True result}
\label{glasses-true}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=11cm]{glasses-predict}
\caption{Glasses types: Predicting result}
\label{glasses-predict}
\end{figure}

Compare figure \ref{mobile-true} and \ref{mobile-predict}, we can see that the true result is a parallel-shaped figure with overlaps whereas K-Means cannot partition like that. In fact, I think it's a challenge for most clustering methods. For the sparser dataset, glasses, it has some obvious centers in distribution. Intuitively, it is easier to be partitioned, and the $ARI$ proves the assumption.

\section{Improvement of K-Means}
K-Means has been proposed for many years and it has a lot of perfection versions now. We just list some representative works here.

\subsection{Advantages and Disadvantages of K-Means}
\subsubsection{Advantages}
\begin{itemize}
\item Easy to implement.
\item With a large number of variables, K-Means may be computationally faster than hierarchical clustering (if $K$ is small).
\item k-Means may produce tighter clusters than hierarchical clustering.
\item An instance can change cluster (move to another cluster) when the centroids are recomputed.
\end{itemize}
\subsubsection{Disadvantages}
\begin{itemize}
\item Difficult to predict the number of clusters (K-Value).
\item Initial seeds have a strong impact on the final results.
\item The order of the data has an impact on the final results.
\item Sensitive to scale: rescaling your datasets (normalization or standardization) will completely change results. While this itself is not bad, not realizing that you have to spend extra attention (on to scaling your data might be bad).
\end{itemize}

\subsection{Variations of the K-Means Algorithm}
The K-Means algorithm can be classified into center-based clustering algorithms. These clustering algorithms were developed to improve the performance of the standard K-Means algorithm\cite{gan2007data}. We will address some of these algorithms in subsequent sections.
\subsubsection{The Continuous K-Means Algorithm}
The continuous K-Means algorithm, proposed by Faber, is faster than the standard K-Means algorithm. It is different from the standard K-Means algorithm in the following aspects. Firstly, in the continuous K-Means algorithm, the prototypes (or reference points) are chosen as a random sample from the whole database, while in the standard K-Means algorithm the initial points are chosen arbitrarily. Secondly, the data points are treated differently. During each complete iteration, the continuous K-Means algorithm examines only a sample of the data points, while the standard K-Means algorithm examines all the data points in sequence.

\subsubsection{The Compare-means Algorithm}
In order to accelerate the K-Means algorithm, the algorithm compare-means uses a simple approach to avoid many unnecessary comparisons. Since the number of clusters $K$ is usually small, distances of all pairs of means are precomputed before each iteration. Then, before comparing a point x to a mean $\mu_j$ , the above test is performed using the closest known mean to x.

\subsubsection{The Sort-means Algorithm}
The algorithm sort-means is an extension of compare-means. In this algorithm, the means are sorted in order of increasing distance from each mean in order to obtain a further speedup.

\subsubsection{Acceleration of the K-Means Algorithm with the kd-tree}
Pelleg and Moore proposed an algorithm for the K-Means clustering problem using the $kd$-tree data structure. The $kd$-tree data structure, can be used to reduce the large number of nearest-neighbor queries issued by the traditional K-Means algorithm. Hence, an analysis of the geometry of the current cluster centers can lead to a great reduction in the work needed to update the cluster centers. In addition, the initial centers of the K-Means algorithm can be chosen by the $kd$-tree efficiently. One way to use the $kd$-tree in the inner loop of the K-Means algorithm is to store the centers in the tree; another way is to store the whole data set in the tree. To describe the application of the $kd$-tree in the K-Means algorithm, let us start with an iteration of the K-Means algorithm. Let $C^{\left(i\right)}$ denote the set of centroids after the $i-th$ iteration. Before the first iteration,
$C^{\left(0\right)}$ is initialized to a set of random values. The stop criterion of the algorithm is that $C^{\left(i\right)}$ and $C^{\left(i-1\right)}$ are identical. In each iteration of the algorithm, the following two steps are performed:
\begin{enumerate}
\item For each data point $x$, find the center in $C^{\left(i\right)}$ that is closest to $x$ and associate $x$ with this center.
\item Update $C^{\left(i\right)}$ to $C^{\left(i+1\right)}$ by taking, for each center, the center of mass of all the data points associated with this center.
\end{enumerate}

\subsection{Our project's requirement}
The variations for K-Means above are mostly performing in the speed or the quality. We consider that K-Means cannot obtain the results with some constraints, like sizes. However, we need to partition our project's large-scale dataset into some uniform clusters so that we could solve the subproblems with similar scales concurrently. We cannot partition our original dataset directly as the elements in the set are relational. Hence, we have to adjust current methods to meet our requirement. K-Means has good expandability, which can be our foundation.

\section{Looking Forward: Clustering with Size Constraints}
This section will be our team's final project and the outline is disclosed in this assignment. Look forward to the main body at the final report and presentation.

\begin{itemize}
\item Matrix form of K-Means
\item Constraints description
\item Problem derivation
\item Problem optimization
\item Parallel solving
\item Simulation and comparison
\item Perfection under large-scale data
\end{itemize}

\clearpage

\bibliographystyle{IEEEtran}
\bibliography{ref}

\end{document}


\chapter{Causal Inference}

\section{Bivariate causal model}

\subsection{ANM}

We will now introduce\footnote{Note that this introduction follows closely that of \cite{Mooij2016jmlr}, and
we encourage the reader to have a look to fill in details that have been omitted here.}
the bivariate causal model and define the particular subset of such models that
we will work on. For the bivariate case, if we have $Y \in \R$ as a direct cause of $X \in \R$, then we can model
the relationship as follows:

\begin{equation}
    \begin{cases}
        & Y = f(X, Z) \\
        & X \bigCI Z,\quad X \thicksim P_X,\quad Z \thicksim P_{Z}
     \end{cases}
     \label{causal_model}
\end{equation}

where $P_X$ is the density of the cause and $P_Z$ that of the latent variable; $f: \R \times \R \rightarrow \R$
is a borel measurable function (w.r.t. to the Borel sets of $\R \times \R$ and $\R$).

If we assume that there is no confounder, no sampling bias\footnote{If you consider the
example given in the introduction, then if Bob always sampled when Alice was exercising, then this
would have lead to sampling bias, as during exercise both heart rate and blood pressure tend to increase.}
and no cycles then it is natural to assume that $X \bigCI Z$.

A confounder is a third variable say $Z$, that influences both $X$ and $Y$; for example in figure \ref{fig:conditionalIndep},
if the edges are directed from $X$ to $Z$ and also from $Z$ to $Y$, then we would say that $Z$ is a confounder.

While we can model $Z$ as either a scalar or a vector, we can without loss of generality assume it to be a scalar.
It can be shown that if $Z$ is a vector, then one can construct a simpler model with scalar noise, which has the
same observational and interventional distribution \cite{Mooij2016jmlr}.

An important remark is that given the direct model in equation \ref{causal_model}, we can find some
$\tilde{f}$ and $\tilde{Z}$ such that

\begin{equation}
    \begin{cases}
        & X = \tilde{f}(Y, \tilde{Z}) \\
        & Y \bigCI \tilde{Z},\quad Y \thicksim P_Y,\quad \tilde{Z} \thicksim P_{\tilde{Z}}
     \end{cases}
\end{equation}

with the important property that it induces an equivalent observational distribution $p_{X, Y}$ as that of
equation \ref{causal_model}. However in general the interventional distribution will differ. In particular
this means that with observational data alone we are not able identify the right causal direction.

We must therefore make further assumptions on $f$ that break this symmetry and allow us to make causal inference
using observational data alone.
In particular we will consider the following class of models. Note that these models are a subset of those
that we just introduced (equation \ref{causal_model}).

\begin{definition} Given a triplet $(P_X, P_Z, f)$, consisting of two finite mean densities and a Borel-measurable function
$f: \R \rightarrow \R$, we can define a \textbf{bivariate Additive Noise Model (ANM)} $X \rightarrow Y$
\[
    \begin{cases}
        & Y = f(X) + Z \\
        & X \bigCI Z,\quad X \thicksim P_X,\quad Z \thicksim P_{Z}
     \end{cases}
\]
If the induced joint density of $X$ and $Y$, $P_{X, Y}$ has a density with respect to Lebesgue measure,
we say that $P_{X, Y}$ satisfies the ANM $X \rightarrow Y$.
\end{definition}

Given such a model, we are interested in the cases when the observational distribution $P_{X, Y}$ can only
lead to only one causal explanation; this motivates the following definition:

\begin{definition} If the joint density $P_{X, Y}$ satisfies an ANM $X \rightarrow Y$, but does not satisfy
ANM $Y \rightarrow X$, then we call the ANM $X \rightarrow Y$ \textbf{identifiable}.
\end{definition}

Intuitively, non-linearities due to $f$ will break the symmetry needed to make a reverse ANM. This is what
\cite{hoyer2009nonlinear} explore; they show that for the
a triplet $(P_X, P_Z, f)$ to generate a non-identifiable ANM, it needs to satisfy a particular
differential equation. Loosely speaking this cannot happen in the generic case: in other words the forward
model $X \rightarrow Y$ cannot be inverted.

\newpage
If $f$ is linear, then one can give a much more precise statement about identifiability:

\begin{theorem} Let $X$ and $Y$ be random variables, such that
$$
    Y = aX + Z, \quad X \bigCI Z, \; a \neq 0
$$
Then we can reverse the process, i.e. there exists $\tilde{a} \in \R$ and a noise $\tilde{Z}$ such that
$$
    X = \tilde{a}Y + \tilde{Z}, \quad Y \bigCI \tilde{Z}
$$
if and only if $X, Y, Z, \tilde{Z}$ are Gaussian distributed.
\label{linear_indentifiability}
\end{theorem}

The proof is a simple application of the Darmois-Skitovich Theorem\footnote{ The theorem also plays an important role
in independent component analysis (ICA), in short it deals with source separation. It turns out that if we have the
linear multivariate causal setting, then we can cast it as a source separation problem and use ICA to solve it.
Note that there too Gaussianty makes or breaks the method.} (\cite{darmois}); which we state without proof:

\begin{theorem}[Darmois-Skitovich] Let $X_i$, $i \in [n]$ be independent random variables, and let
    $\alpha_i$, $\beta_j$ be non zero constants. Then, if the random variables
    $$
        L_1 = \sum_{i \in [n]} \alpha_i X_i
    $$
    $$
        L_2 = \sum_{i \in [n]} \beta_i X_i
    $$
    are independent, i.e. $L_1 \bigCI L_2$; then all the random variables $X_i$ are Gaussian.
\end{theorem}

We now prove Theorem \ref{linear_indentifiability} using the Darmois-Skitovich Theorem.

\begin{proof}
    ~
For the "only if" part, note that by simple manipulation, we have the following:

\[
    \begin{bmatrix}
        Y \\
        \tilde{Z}
    \end{bmatrix}
    =
    \begin{bmatrix}
    a & 1 \\
    1 - \tilde{a}a & -\tilde{a}
    \end{bmatrix}
    \cdot
    \begin{bmatrix}
        X \\
        Z
    \end{bmatrix}
\]

If $\tilde{a} \neq 0$ and $1 - \tilde{a}a \neq 0$ then by Darmois-Skitovich, the result follows.

We next show that both of these conditions must be true for the process to be reversible:

\begin{enumerate}
    \item If $\tilde{a} = 0$ then $\tilde{Z} = X$, but then $X \bigCI Y$, a contradiction\footnote{
        Note that $X \bigCI aX + Z$ is trivially false
        in the discrete case, but if both are continuous then we need to be a bit more careful as is the
        case with degenerate random variables; but essentially the same holds (see \cite{Peters2008diploma}).
    }.
    \item Finally, if $1 - \tilde{a}a = 0$ then $\tilde{Z} = -\tilde{a}Z$, and thus
    $-\tilde{a}Z \bigCI Y$, a contradiction.
\end{enumerate}

We have thus show the "only if" part.

We next show the "if" part; first assume that $X$ and $Z$ are Gaussian random variables.
It is easy to verify that:
$\operatorname{Cov}(Y, \tilde{Z}) = a(1 - a\tilde{a})\operatorname{Var}(X) - \tilde{a}\operatorname{Var}(Z)$.
Thus if we set $\tilde{a} = \frac{a \operatorname{Var}(X)}{a^2 \operatorname{Var}(X) + \operatorname{Var}(Z)}$
we get that $\operatorname{Cov}(Y, \tilde{Z}) = 0$, and since they are Gaussian random variables we get
that they are also independent.

\end{proof}

These results show that indeed, in most cases, we should be able to perform causal inference as most
additive models should be identifiable. Interestingly, the Gaussian setting provides difficulty  due
to the symmetry of the Gaussian (in the linear case). In general the Gaussian is our friend, but not today.
We will next explore some methods for causal inference.

\section{ANM Methods}

Additive noise model (ANM) methods are a very simple score based family of methods for causal inference.
The following lemma\footnote{A simple proof can be found here \cite{Mooij2016jmlr}} motivates the method:

\begin{lemma}
    Given a joint density $P_{X, Y}$ of two random variables $X$, $Y$ s.t. the conditional expectation
    $\E(Y |X = x)$ is well-defined for all x and measurable. Then, $P_{X, Y}$ satisfies a bivariate
    Additive Noise Model $X \rightarrow Y$ if and only if $E_Y := Y - \E(Y|X)$ has
    finite mean and is independent of $X$.
\end{lemma}

In practice we get some data from $P_{X, Y}$ ; say
$\mathcal{D}_N = \{ (x_i, y_i )\}_{i \in [N]}$. We can then either split it into a test/train in order to
first fit a regression which we then evaluate using the test set. If the data is scarce we may alternatively
recycle the data --- i.e. reuse it for both training and evaluation.

First we estimate through regression the function $x \mapsto \E(Y | X=x)$, say $\hat{f}$; we
then compute the estimated residual $\hat{e} = \hat{f}(X) - Y$.
Next, we estimate the dependence between $\hat{e}$ and $X$ using a score function $C$; e.g. $C$ could be
the empirical mutual information between them. Thus a low score would be evidence for an ANM in that direction;
we can compute the score for the reverse model by switching the roles of $X$ and $Y$. We can then compare
the scores and use this as a criteria for inference. We write down this idea more explicitly
in Algorithm \ref{alg:anm_general}.

In order to show that such a procedure is consistent\footnote{By consistent we mean that the probability of
predicting the true causal direction goes to 1 as the number of samples $n \rightarrow \infty$.
} we need 3 things:

\begin{enumerate}
    \item $P_{X, Y}$ satisfies either $X \rightarrow Y$ or $Y \rightarrow X$, but not both.
    \item The regression method should be \textbf{suitable} for regressing $Y$ on $X$.
    \item If $X \rightarrow Y$, then asymptotically $\hat{C}_{X \rightarrow Y} < \hat{C}_{Y \rightarrow X}$
\end{enumerate}

We take point 1 as an assumption as there is currently no theoretical result that allows for a consistent
test to check if $P_{X, Y}$ satisfies an ANM $X \rightarrow Y$.

In point 2, we will say that a regression is \textbf{suitable} if the following holds:
Given two real-valued random variable $X$, $Y$, with joint distribution $P_{X, Y}$. If
we are given two sequences --- for training and test --- say $D_N = X_1, ..., X_N$ and
$D_N^\prime = X^\prime_1, ..., X^\prime_N$. We say that a regression method is \textbf{suitable} for
regressing $Y$ on $X$ satisfies

$$
\lim _{N \rightarrow \infty} \frac{1}{N}
\sum_{n=1}^{N} \E_{\mathcal{D}_{N}, \mathcal{D}_{N}^{\prime}}\left(\left|\hat{f}_{Y}\left(X_{n}^{\prime} ;
\mathcal{D}_{N}\right)-\E\left(Y \mid X=X_{n}^{\prime}\right)\right|^{2}\right)=0
$$

\begin{algorithm}%[H]%[tbp]

    \caption{General procedure to decide whether $P_{X, Y}$ satisfies and ANM $X \rightarrow Y$
        or $Y \rightarrow X$}

    \textbf{Input}:

    \begin{enumerate}
        \item i.i.d samples $\mathcal{D}_N = \{ (x_i, y_i )\}_{i \in [N]}$ of $X$ and $Y$
        \item Regression method
        \item Score estimator $\hat{C}: \R^n \times \R^n \rightarrow \R$
    \end{enumerate}

    \textbf{Output}: $\hat{C}_{X \rightarrow Y}$, $\hat{C}_{Y \rightarrow X}$, dir

    \begin{enumerate}

        \item Split the $\mathcal{D}_N$ in half randomly to obtain  $\mathcal{D}_{train}$
        and $\mathcal{D}_{test}$

        \item Use the regression method on the training data $\mathcal{D}_{train}$:
        \begin{itemize}
            \item[--] $\hat{f}_X$ of the regression function $x \mapsto \E(Y | X=x)$
            \item[--] $\hat{f}_Y$ of the regression function $y \mapsto \E(X | Y=y)$
        \end{itemize}

        \item Estimate residuals using the predicted regressions on the test data $\mathcal{D}_{test}$:
        \begin{itemize}
            \item[--] $ \mathbf{\hat{e}_Y} := \mathbf{y} - \hat{f}_Y(\mathbf{x})$
            \item[--] $ \mathbf{\hat{e}_X} := \mathbf{x} - \hat{f}_X(\mathbf{y})$
        \end{itemize}

        \item Compute scores to measure dependence between inputs and estimated residuals based on the
        test data $\mathcal{D}_{test}$
        \begin{itemize}
            \item[--] $\hat{C}_{X \rightarrow Y}: = \hat{C}(\mathbf{x}, \mathbf{\hat{e}_Y})$
            \item[--] $\hat{C}_{Y \rightarrow X}: = \hat{C}(\mathbf{y}, \mathbf{\hat{e}_X})$
        \end{itemize}

        \item Output $\hat{C}_{X \rightarrow Y}$, $\hat{C}_{Y \rightarrow X}$, and

        \[
        \text{dir} :=
         \begin{cases}
            & X \rightarrow Y \quad \text{if} \; \hat{C}_{X \rightarrow Y} \leq \hat{C}_{Y \rightarrow X}\\
            & Y \rightarrow X \quad \text{otherwise}
         \end{cases}
        \]

    \end{enumerate}

  \label{alg:anm_general}
  \end{algorithm}

  We will now overview some score functions, note that the list is far from exhaustive and other methods can be found
  in \cite{Mooij2016jmlr}. We end the chapter by presenting some alternative methods.

\subsection{HSIC score}

First considered by \cite{hoyer2009nonlinear}, is the Hilbert-Schmidt independence Criterion (HSIC) for
testing the independence between the residuals and the input.

$$
  \hat{C}(\mathbf{u}, \mathbf{v}) := \widehat{\operatorname{HSIC}}_{k, l} (\mathbf{u}, \mathbf{v})
$$

We will give a formal description of the HSIC in the next chapter, where we will develop some of the background
to get an intuition on this measure. For now, one can think of the HSIC as a metric that computes the distance
between the the product distribution and the joint (similar to the Mutual information); the key difference is that
we provide two kernels, $k$ and $l$ which will transform the distributions to a different space. For specific
kernel choices the HSIC indeed becomes a metric.

Note that one can approximate the distribution of $\widehat{\operatorname{HSIC}}$, and use hypothesis testing
for the independence test.

\subsection{Entropy score}

Another type of score function looks at differential entropies instead of directly testing for independence.
These ideas stem from \cite{kpotufe2014consistency} and \cite{nowzohour2016score}; The following lemma shows how this might be used in practice:

\begin{lemma} Consider random variables $X$ and $Y$, with joint density $P_{X, Y}$. For any functions
$f, g : \R \rightarrow \R$ we have:

$$
    H(X) + H(Y - f(X)) = H(Y) + H(X - g(X)) - I(X - g(X), Y) + I(Y - f(X), X)
$$

where $H(\cdot)$ denotes the differential entropy and $I(\cdot, \cdot)$ denotes the differential mutual information
(\cite{cover1999elements}).
\end{lemma}

The proof is a simple application of the chain rule. Note that if $X \rightarrow Y$ then $I(Y - f(X), X) = 0$;
since $I(X, Y) \geq 0$ for any $X$, $Y$ it follows
that:

$$
    H(X) + H(Y - f(X)) \leq  H(Y) + H(X - g(X))
$$

Which motivates the score function

$$
    C(U, V) = H(U) + H(V)
$$

This approach to estimate the direction of the ANM is consistent under certain assumptions as is shown by
\cite{kpotufe2014consistency} and \cite{nowzohour2016score}. One of the main drawbacks of using differential
entropy is that we need to go through discretization, which can lead to undesired effects --- and can also be
inefficient in higher dimensions.

\subsection{CGNN: Causal Generative Neural Nets}

In the work of \cite{goudet2017causal}, the authors estimate a \textit{generative} model by approximating
the FCM structure using neural networks given some data from $P_{X, Y}$. Using the same notation
as in the introduction on FCM, the idea is to estimate each $f_i$ by a neural network, and the
search through the DAG space. Since the DAG space is super-exponential in the number of variables,
they apply a greedy procedure to decide whether or not to include an edge $X_i \rightarrow X_j$.
In some sense this is similar to greedy methods used in model selection. The caveat here is that
using the generative model they can backpropagate to learn all the $f_i$ simultaneously.

More specifically, given the current graph estimate $\mathcal{G}$, they can generate some $\hat{P}$
from the current $f_i$ and some noise $\mathcal{E}$. Then they can train the model by using
the MMD\footnote{We present and give some background on the MMD in the next chapter}
as a loss function between $\hat{P}$ and $P$ (the data distribution).

So far the CGNN appears to have the best performance on the various benchmarks.

\subsection{Other methods}

\textbf{IGIC} (Information Geometric Causal Inference), the method stems from \cite{janzing2010causal}.
The idea is based on the following observation: If $X \rightarrow Y$, then a reasonable assumption is that
$P_X$ contains no information about $P_{Y | X}$. If one thinks about $X$ as as the distribution of
elevations in a country, and $Y$ the temperature, then this assumption is likely to hold. The difficulty
lies in formalizing the notion of "information" in a good way.

There are many other methods some of which can be found in the review of \cite{Mooij2016jmlr}.


\chapter{Fourier Series}
\begin{enumerate}
	\item The Fourier series of the function $f(x)$ as shown in figure is:
	\begin{figure}[H]
		\centering
		\includegraphics[height=2cm,width=8cm]{FS-assignment-01}
	\end{figure}
	 \begin{tasks}(1)
		\task[\textbf{a.}]$\frac{2 k}{\pi}\left(\cos \frac{\pi}{2} x-\frac{1}{3} \cos \frac{3 \pi}{2} x+\frac{1}{5} \cos \frac{5 \pi}{2} x-+\ldots\right)$
		\task[\textbf{b.}]$\frac{k}{\pi}\left(\cos \frac{\pi}{2} x-\frac{1}{3} \cos \frac{3 \pi}{2} x+\frac{1}{5} \cos \frac{5 \pi}{2} x-+\ldots\right)$
		\task[\textbf{c.}]$\frac{k}{2}+\frac{k}{\pi}\left(\cos \frac{\pi}{2} x-\frac{1}{3} \cos \frac{3 \pi}{2} x+\frac{1}{5} \cos \frac{5 \pi}{2} x-+\ldots\right)$
		\task[\textbf{d.}] $\frac{k}{2}+\frac{2 k}{\pi}\left(\cos \frac{\pi}{2} x-\frac{1}{3} \cos \frac{3 \pi}{2} x+\frac{1}{5} \cos \frac{5 \pi}{2} x-+\ldots\right)$
	\end{tasks}
\begin{answer}
	\begin{align*}
	f(x)&=\left\{\begin{array}{clc}0 & \text { if } & -2<x<-1 \\ k & \text { if } & -1<x<1 \\ 0 & \text { if } & 1<x<2\end{array} \quad p=2 L=4, L=2\right.\\
	\text{Let }f(x)&=a_{0}+\sum_{n=1}^{\infty}\left(a_{n} \cos \frac{n \pi}{L} x+b_{n} \sin \frac{n \pi}{L} x\right)\\
	\because a_{0}&=\frac{1}{2 L} \int_{-L}^{L} f(x) d x\\
	\Rightarrow a_{0}&=\frac{1}{4} \int_{-2}^{2} f(x) d x=\frac{1}{4} \int_{-1}^{1} k d x=\frac{k}{2}\\
	\because a_{n}&=\frac{1}{L} \int_{-L}^{L} f(x) \cos \frac{n \pi x}{L} d x\\
	\Rightarrow a_{n}&=\frac{1}{2} \int_{-2}^{2} f(x) \cos \frac{n \pi x}{2} d x=\frac{1}{2} \int_{-1}^{1} k \cos \frac{n \pi x}{2} d x\\
	\Rightarrow a_{n}&=\frac{k}{2}\left[\frac{2}{n \pi} \sin \frac{n \pi x}{2}\right]_{-1}^{1}=\frac{2 k}{n \pi} \sin \frac{n \pi}{2}=\left\{\begin{array}{cl}0, & n=2,4,6 \ldots . \\ \frac{2 k}{n \pi}, & n=1,5,9 \ldots . . \\ -\frac{2 k}{n \pi}, & n=3,7,11 \ldots . .\end{array}\right.\\
	\because b_{n}&=\frac{1}{L} \int_{-L}^{l} f(x) \sin \frac{n \pi x}{L} d x\\
	\Rightarrow b_{n}&=\frac{1}{2} \int_{-2}^{2} f(x) \sin \frac{n \pi x}{2} d x=\frac{1}{2} \int_{-1}^{1} k \sin \frac{n \pi x}{2} d x=\frac{k}{2}\left[-\frac{2}{n \pi} \cos \frac{n \pi x}{2}\right]_{-1}^{1}=0\\
	\text{Thus Fourier series is }f(x)&=\frac{k}{2}+\frac{2 k}{\pi}\left(\cos \frac{\pi}{2} x-\frac{1}{3} \cos \frac{3 \pi}{2} x+\frac{1}{5} \cos \frac{5 \pi}{2} x-+\ldots\right)
	\end{align*}
	So the correct option is \textbf{Option (d)}
\end{answer}
	\item The Fourier series of the periodic function
	$$
	f(x)=e^{x} ; \quad(-\pi<x<\pi), \quad \text { having period } 2 \pi \text { is }
	$$
	 \begin{tasks}(1)
		\task[\textbf{a.}]$\frac{1}{\pi}\left[\frac{1}{2}-\frac{1}{1^{2}}(\cos x-\sin x)+\frac{1}{2^{2}}(\cos 2 x-2 \sin 2 x)-+\ldots\right]$
		\task[\textbf{b.}]$\frac{2}{\pi}\left[\frac{1}{2}-\frac{1}{1^{2}}(\cos x-\sin x)+\frac{1}{2^{2}}(\cos 2 x-2 \sin 2 x)-+\ldots\right]$
		\task[\textbf{c.}] $\frac{\sinh \pi}{\pi}\left[\frac{1}{2}-\frac{1}{1+1^{2}} \cdot(\cos x-\sin x)+\frac{1}{1+2^{2}}(\cos 2 x-2 \sin 2 x)-+\ldots\right]$
		\task[\textbf{d.}] $\frac{2 \sinh \pi}{\pi}\left[\frac{1}{2}-\frac{1}{1+1^{2}}(\cos x-\sin x)+\frac{1}{1+2^{2}}(\cos 2 x-2 \sin 2 x)-+\ldots\right]$
	\end{tasks}
\begin{answer}
	\begin{align*}
&\text{	Let Fourier series is }f(x)=\sum_{n=\infty}^{\infty} c_{n} e^{i n x}\\
\because c_{n}&=\frac{1}{2 \pi} \int_{-\pi}^{\pi} f(x) e^{-i n x} d x\\
\Rightarrow c_{n}&=\frac{1}{2 \pi} \int_{-\pi}^{\pi} e^{x} e^{-i n x} d x=\frac{1}{2 \pi} \int_{-\pi}^{\pi} e^{(1-i n) x} d x=\frac{1}{2 \pi} \frac{1}{(1-i n)}\left[e^{(1-i n) x}\right]_{-\pi}^{\pi}=\frac{1}{2 \pi} \frac{1}{(1-i n)}\left[e^{(1-i n) \pi}-e^{-(1-i n) \pi}\right]\\
\Rightarrow c_{n}&=\frac{1}{2 \pi} \frac{1}{(1-i n)}\left[e^{\pi} e^{-i n \pi}-e^{-\pi} e^{i n \pi}\right]=\frac{1}{2 \pi}\left(\frac{1+i n}{1+n^{2}}\right)\left[e^{\pi}-e^{-\pi}\right](-1)^{n} \quad \because e^{\pm i n \pi}=(-1)^{n}\\
\Rightarrow c_{n}&=\left(\frac{1+i n}{1+n^{2}}\right) \frac{\sinh \pi}{\pi}(-1)^{n} \quad \quad \because \sinh \pi=\frac{e^{\pi}-e^{-\pi}}{2}
	\end{align*}
	\begin{align}
	\text{Thus Fourier series is }f(x)&=\frac{\sinh \pi}{\pi} \sum_{n=-\infty}^{\infty}(-1)^{n}\left(\frac{1+i n}{1+n^{2}}\right) e^{i n x}\label{FS-01}
	\intertext{Let us derive the real Fourier series}\notag
	\because(1+i n) e^{i n x}&=(1+i n)(\cos n x+i \sin n x)=(\cos n x-n \sin n x)+i(\cos n x+\sin n x)\notag\\
	\because n\text{ varies from }&\text{ $-\infty$ to $+\infty$, equation (\ref{FS-01}) has corresponding term with $-n$ instead of $n$.}\notag
	\intertext{Thus}
	f(x)&=\frac{\sinh \pi}{\pi}+2 \frac{\sinh \pi}{\pi} \sum_{n=1}^{\infty}(-1)^{n} \frac{(\cos n x-n \sin n x)}{1+n^{2}}\notag\\
	f(x)&=\frac{2 \sinh \pi}{\pi}\left[\frac{1}{2}-\frac{1}{1+1^{2}}(\cos x-\sin x)+\frac{1}{1+2^{2}}(\cos 2 x-2 \sin 2 x)-+\ldots\right]\notag
	\end{align}
		So the correct option is \textbf{Option (d)}
\end{answer}
	\item  A sinusoidal voltage $E_{0} \sin \omega t$ where $t$ is time is passed through half-wave rectifier that clips the negative portion of the wave.
$f(t)=\left\{\begin{array}{ccc}0 & \text { if } & -\frac{\pi}{\omega}<t<0 \\ E_{0} \sin \omega t & \text { if } & 0<t<\frac{\pi}{\omega}\end{array}\right.$
	Then the Fourier series of the resulting periodic function is
	\begin{figure}[H]
		\centering
		\includegraphics[height=2.5cm,width=8cm]{FS-assignment-02}
	\end{figure}
	 \begin{tasks}(1)
		\task[\textbf{a.}]$f(t)=a_{0}+b_{1} \sin \omega t+\sum_{n=2.4}^{\infty} \frac{2 E_{0}}{\pi\left(1-n^{2}\right)} \cos n \omega t$
		\task[\textbf{b.}] $f(t)=a_{0}+b_{1} \sin \omega t+\sum_{n=1,2,3_{1}, .}^{\infty} \frac{2 E_{0}}{\pi\left(1-n^{2}\right)} \cos n \omega t$
		\task[\textbf{c.}]$f(t)=b_{1} \sin \omega t+\sum_{n=2,4 \ldots}^{\infty} \frac{2 E_{0}}{\pi\left(1-n^{2}\right)} \cos n \omega t$
		\task[\textbf{d.}] $f(t)=\sum_{n=2,4 \ldots}^{\infty} \frac{2 E_{0}}{\pi\left(1-n^{2}\right)} \cos n \omega t$
	\end{tasks}
\begin{answer}
	\begin{align*}
	f(t)&=\left\{\begin{array}{cl}0 & \text { if } \quad-\frac{\pi}{\omega}<t<0 \\ E_{0} \sin \omega t & \text { if } \quad 0<t<\frac{\pi}{\omega}\end{array} \quad p=2 L=2 \frac{\pi}{\omega}, L=\frac{\pi}{\omega}\right.\\
	\text{Let }f(x)&=a_{0}+\sum_{n=1}^{\infty}\left(a_{n} \cos \frac{n \pi}{L} x+b_{n} \sin \frac{n \pi}{L} x\right)\\
	\because a_{0}&=\frac{1}{2 L} \int_{-L}^{L} f(t) d t \Rightarrow a_{0}=\frac{\omega}{2 \pi} \int_{-\pi / \omega}^{\pi / \omega} f(t) d t=\frac{\omega}{2 \pi} \int_{0}^{\pi / \omega} E_{0} \sin \omega t d t\\
	\Rightarrow a_{0}&=\frac{\omega}{2 \pi}\left[-\frac{E_{0}}{\omega} \cos \omega t\right]_{0}^{\pi / \omega}=\frac{E_{0}}{\pi}\\
	\because a_{n}&=\frac{1}{L} \int_{-L}^{L} f(t) \cos \frac{n \pi t}{L} d t\\
	\Rightarrow a_{n}&=\frac{\omega}{\pi} \int_{-\pi / \omega}^{\pi / \omega} f(t) \cos n \omega t d t=\frac{\omega}{\pi} \int_{0}^{\pi / \omega} E_{0} \sin \omega t \cos n \omega t d t=\frac{E_{0} \omega}{2 \pi} \int_{0}^{\pi / \omega} 2 \sin \omega t \cos n \omega t d t\\
	\Rightarrow a_{n}&=\frac{E_{0} \omega}{2 \pi} \int_{0}^{\pi / \omega}[\sin (1+n) \omega t+\sin (1-n) \omega t] d t\\
	\text { For } n&=1 ; \quad a_{1}=\frac{E_{0} \omega}{2 \pi} \int_{0}^{\pi / \omega} \sin 2 \omega t d t=\frac{E_{0} \omega}{2 \pi}\left[\frac{-\cos 2 \omega t}{2 \omega}\right]_{0}^{\pi / \omega}=0\\
	\text{For }n&=2,3,4 \ldots\\
	\Rightarrow a_{n}&=\frac{E_{0} \omega}{2 \pi}\left[-\frac{\cos (1+n) \omega t}{(1+n) \omega}-\frac{\cos (1-n) \omega t}{(1-n) \omega}\right]_{0}^{\pi / \omega}\\
	\Rightarrow a_{n}&=\frac{E_{0}}{2 \pi}\left[\frac{-\cos (1+n) \pi+1}{(1+n)}+\frac{-\cos (1-n) \pi+1}{(1-n)}\right]=\left\{\begin{array}{cc}0 & n=3,5,7 \ldots \\ \frac{2 E_{0}}{\pi\left(d-n^{2}\right)} & n=2,4,6 \ldots\end{array}\right.\\
	\because b_{n}&=\frac{1}{L} \int_{-L}^{L} f(x) \sin \frac{n \pi x}{L} d x\\
	\Rightarrow b_{n}&=\frac{\omega}{\pi} \int_{-\pi / \omega}^{\pi / \omega} f(t) \sin n \omega t d t=\frac{\omega}{\pi} \int_{0}^{\pi / \omega} E_{0} \sin \omega t \sin n \omega t d t=\frac{E_{0} \omega}{2 \pi} \int_{0}^{\pi / \omega} 2 \sin \omega t \sin n \omega t d t\\
	\Rightarrow b_{n}&=\frac{E_{0} \omega}{2 \pi} \int_{0}^{\pi / \omega}[-\cos (1+n) \omega t+\cos (1-n) \omega t] d t\\
	\text{For }n&=1\\
	b_{1}&=\frac{E_{0} \omega}{2 \pi} \int_{0}^{\pi / \omega}[1-\cos 2 \omega t] d t=\frac{E_{0} \omega}{2 \pi} \int_{0}^{\pi / \omega}\left[t-\frac{\sin 2 \omega t}{2 \omega}\right]_{0}^{\pi / \omega}=\frac{E_{0} \omega}{2 \pi} \frac{\pi}{\omega}=\frac{E_{0}}{2}\\
	\text{For }n&=2,3,4 \ldots .\\
	\Rightarrow b_{n}&=\frac{E_{0} \omega}{2 \pi}\left[-\frac{\sin (1+n) \omega t}{(1+n) \omega}+\frac{\sin (1-n) \omega t}{(1-n) \omega}\right]_{0}^{\pi / \omega}=\frac{E_{0}}{2 \pi}\left[\frac{-\sin (1+n) \pi}{(1+n)}+\frac{\sin (1-n) \pi}{(1-n)}\right]=0
	\intertext{Thus Fourier series}
	f(t)&=a_{0}+b_{1} \sin \omega t+\sum_{n=2,4}^{\infty} a_{n} \cos n \omega t=a_{0}+b_{1} \sin \omega t+\sum_{n=2,4 \ldots}^{\infty} \frac{2 E_{0}}{\pi\left(1-n^{2}\right)} \cos n \omega t
	\end{align*}
		So the correct option is \textbf{Option (a)}
\end{answer}
	\item  The Fourier series of even periodic expansion of the function $f(x)$ as shown in figure below is
	\begin{figure}[H]
		\centering
		\includegraphics[height=3cm,width=5cm]{FS-assignment-03}
	\end{figure}
	$f(x)=\left\{\begin{array}{clc}\frac{2 k}{L} x & \text { if } & 0<x<\frac{L}{2} \\ \frac{2 k}{L}(L-x) & \text { if } & \frac{L}{2}<x<L\end{array}\right.$
	 \begin{tasks}(1)
		\task[\textbf{a.}]$f(x)=\frac{4 k}{\pi^{2}}\left(\frac{1}{2^{2}} \cos \frac{2 \pi x}{L}+\frac{1}{6^{2}} \cos \frac{6 \pi x}{L}+\frac{1}{10^{2}} \cos \frac{10 \pi x}{L}+\ldots \ldots\right)$
		\task[\textbf{b.}]$f(x)=\frac{k}{2}-\frac{4 k}{\pi^{2}}\left(\frac{1}{2^{2}} \cos \frac{2 \pi x}{L}+\frac{1}{6^{2}} \cos \frac{6 \pi x}{L}+\frac{1}{10^{2}} \cos \frac{10 \pi x}{L}+\ldots .\right)$
		\task[\textbf{c.}]$f(x)=\frac{k}{2}-\frac{16 k}{\pi^{2}}\left(\frac{1}{2^{2}} \sin \frac{2 \pi x}{L}+\frac{1}{6^{2}} \sin \frac{6 \pi x}{L}+\frac{1}{10^{2}} \sin \frac{10 \pi x}{L}+\ldots . .\right)$
		\task[\textbf{d.}] $f(x)=\frac{k}{2}-\frac{16 k}{\pi^{2}}\left(\frac{1}{2^{2}} \cos \frac{2 \pi x}{L}+\frac{1}{6^{2}} \cos \frac{6 \pi x}{L}+\frac{1}{10^{2}} \cos \frac{10 \pi x}{L}+\ldots .\right)$
	\end{tasks}
\begin{answer}
	Even periodic extension
	\begin{figure}[H]
		\centering
		\includegraphics[height=2.5cm,width=8cm]{FS-assignment-04}
	\end{figure}
	\begin{align*}
	\because a_{0}&=\frac{1}{L} \int_{0}^{L} f(x) d x\\
	\Rightarrow a_{0}&=\frac{1}{L}\left[\frac{2 k}{L} \int_{0}^{\frac{L}{2}} x d x+\frac{2 k}{L} \int_{\frac{L}{2}}^{L}(L-x) d x\right]=\frac{k}{2}\\
	\because a_{n}&=\frac{2}{L} \int_{0}^{L} f(x) \cos \frac{n \pi x}{L} d x\\
	\Rightarrow a_{n}&=\frac{1}{L}\left[\frac{2 k}{L} \int_{0}^{\frac{L}{2}} x \cos \frac{n \pi x}{L} d x+\frac{2 k}{L} \int_{\frac{L}{2}}^{L}(L-x) \cos \frac{n \pi x}{L} d x\right]
	\intertext{Let us calculate the integral}
	\int_{0}^{\frac{L}{2}} x \cos \frac{n \pi x}{L} d x&=\frac{L}{n \pi}\left[x \sin \frac{n \pi x}{L}\right]_{0}^{L / 2}+\left(\frac{L}{n \pi}\right)^{2}\left[\cos \frac{n \pi x}{L}\right]_{0}^{L / 2}\\
	\Rightarrow \int_{0}^{\frac{L}{2}} x \cos \frac{n \pi x}{L} d x&=\frac{L^{2}}{2 n \pi} \sin \frac{n \pi}{2}+\frac{L^{2}}{n^{2} \pi^{2}}\left(\cos \frac{n \pi}{2}-1\right)\\
	\text{and }\int_{L / 2}^{L}(L-x) \cos \frac{n \pi x}{L} d x&=\frac{L}{n \pi}\left[(L-x) \sin \frac{n \pi x}{L}\right]_{L / 2}^{L}-\left(\frac{L}{n \pi}\right)^{2}\left[\cos \frac{n \pi x}{L}\right]_{L / 2}^{L}\\
	\Rightarrow \int_{L / 2}^{L}(L-x) \cos \frac{n \pi x}{L} d x&=-\frac{L^{2}}{2 n \pi} \sin \frac{n \pi}{2}-\frac{L^{2}}{n^{2} \pi^{2}}\left(\cos n \pi-\cos \frac{n \pi}{2}\right)\\
	\Rightarrow a_{n}&=\frac{1}{L}\left[\frac{2 k}{L} \int_{0}^{\frac{L}{2}} x \cos \frac{n \pi x}{L} d x+\frac{2 k}{L} \int_{\frac{L}{2}}^{L}(L-x) \cos \frac{n \pi x}{L} d x\right]=\frac{4 k}{n^{2} \pi^{2}}\left(2 \cos \frac{n \pi}{2}-\cos n \pi-1\right)\\
	\Rightarrow a_{2}&=-\frac{16 k}{2^{2} \pi^{2}}, a_{6}=-\frac{16 k}{6^{2} \pi^{2}}, a_{10}=-\frac{16 k}{10^{2} \pi^{2}}, \ldots \ldots \ldots . . \text{and }a_{n}=0\text{ if }n \neq 2,6,10 \ldots .
	\intertext{Hence the first half-range expansion of $f(x)$ is}
	f(x)&=\frac{k}{2}-\frac{16 k}{\pi^{2}}\left(\frac{1}{2^{2}} \cos \frac{2 \pi x}{L}+\frac{1}{6^{2}} \cos \frac{6 \pi x}{L}+\frac{1}{10^{2}} \cos \frac{10 \pi x}{L}+\ldots . .\right)
	\intertext{This Fourier cosine series represents the even periodic extension of the given function $f(x)$, of period $2 L$ as shown in figure.}
	\end{align*}
	So the correct option is \textbf{Option (d)}
\end{answer}
\item  The total square error of $F$ with $N=3$ relative to
$$
f(x)=x+\pi \quad(-\pi<x<\pi)
$$
on the interval $-\pi \leq x \leq \pi$ is
	 \begin{tasks}(2)
		\task[\textbf{a.}]$E^{*}=\frac{8}{3} \pi^{3}-\pi\left[2 \pi^{2}+\frac{49}{9}\right]$
		\task[\textbf{b.}]$E^{*}=\frac{4}{3} \pi^{3}-\pi\left[2 \pi^{2}+\frac{49}{9}\right]$
		\task[\textbf{c.}]$E^{*}=\frac{8}{3} \pi^{3}-\pi\left[\pi^{2}+\frac{49}{9}\right]$
		\task[\textbf{d.}] $E^{*}=\frac{8}{3} \pi^{3}-\left[2 \pi^{2}+\frac{49}{9}\right]$
	\end{tasks}
	\begin{answer}
		\begin{align*}
		\text{Fourier coefficients are }a_{0}&=\pi, a_{n}=0\text{ and } b_{n}=-\frac{2}{n} \cos n \pi.\\
		\text{Its Fourier series is given by }F(x)&=\pi+2 \sin x-\sin 2 x+\frac{2}{3} \sin 3 x.
		\intertext{Hence}
		E^{*}=\int_{-\pi}^{\pi} f^{2} d x-\pi\left[2 a_{0}^{2}+\sum_{n=1}^{N}\left(a_{n}^{2}+b_{n}^{2}\right)\right]&=\int_{-\pi}^{\pi}(x+\pi)^{2} d \dot{x}-\pi\left[2 \pi^{2}+2^{2}+1^{2}+\left(\frac{2}{3}\right)^{2}\right]\\
		E^{*}&=\frac{8}{3} \pi^{3}-\pi\left[2 \pi^{2}+\frac{49}{9}\right]
		\end{align*}
		So the correct option is \textbf{Option (a)}
	\end{answer}

\end{enumerate}

CSIR- Mathematical Physics/chapter/Assignments/Assignment-Fourier Series.tex


%!TEX root = ../CombinatoricsNotes.tex

\section{Algebraic methods}
% \marginnote{Lecture X: Wednesday, April 6, 2016.}
\lect{2}{6}

\subsection{Combinatorial Nullstellensatz (\cite{AlonTarsi89})}
\newthought{For background} we will first consider the following result about zeros of multivariate polynomials.
\begin{theorem*}[Hilbert Nullstellensatz (\cite{HilbertNull})]
Let $\F$ be an algebraically closed field. Let $g_1,g_2,\dotsc,g_m \in \F[x_1,x_2,\dotsc,x_n]$ be polynomials of $n$ variables over $\F$. Suppose $f\in \F[x_1,\dotsc,x_n]$ is such that $f=0$ whenever $g_1=g_2=\dotsm = g_m = 0$.
\marginnote{In one variable: $g = (x-a_1)\cdot (x-a_2)\dotsm (x-a_z)$. Assume all roots are distinct. Then $f=0$ at zeros of $g$ iff $f = gh$ for some $h$. If not all roots are distinct, then $f=0$ at zeros of $g$ iff $f^k  =gh$ for some $k\in \N$.}
Then there exists $n\in \N$ and polynomials $h_1,\dotsc,h_m \in \F[x_1,\dotsc,x_n]$ such that $f^n = h_1 g_1 + h_2 g_2 + \dotsm + h_m g_m$.
\end{theorem*}

\newthought{Let us proceed} to \defn{Combinatorial Nullenstellensatz}.
Recall the degree of a multivariable monomial is the sum of degrees in each variable (e.g. $\deg (x_1^2 x_2^3) = 5$), and the degree of a polynomial is as usual the maximum degree of its monomials.
\begin{theorem} \label{thm:comb_nullstell}
Let $\F$ be a field, $S_1,S_2,\dotsc,S_n\subset \F$,
\[
g_i = \prod_{s\in S_i}(x_i - s)
\]
for $i=1,2,\dotsc, n$.
If $f\in \F[x_1,\dotsc,x_n]$ such that $f$ vanishes on all common zeros of $g_1,g_2,\dotsc,g_n$ (i.e., $f(x_1,\dotsc,x_n) = 0$ if $x_i\in S_i$ for all $i$), then there exist polynomials $h_1,h_2,\dotsc,h_n\in \F[x_1,\dotsc,x_n]$ with $\deg h_i \leq \deg f - \deg g_i$ and $f = h_1g_1 + h_2g_2 + \dotsm + h_n g_n$.
\end{theorem}
\begin{remark}
The $g_i$ are monic, single variable, with no double roots. This is very restrictive compared to Hilbert's Nullstellensatz, but we do obtain a stronger conclusion.
\end{remark}
\marginnote{The degree bound in \cref{thm:comb_nullstell}  means that each $h_ig_i$ has degree at most that of $f$.}
Given a multivariable polynomial $p(x_1,\dotsc,x_n)$, let us denote $\deg_{x_i} p$ as the ``degree of $p$ over $x_i$'', i.e. the degree of $p$ when considered as a polynomial in the variable $x_i$ only, treating $x_1,\dotsc,x_{i-1},x_{i+1},\dotsc,x_n$ as constants. To prove \cref{thm:comb_nullstell}, we'll need the following lemma.
% \marginnote{}
\begin{lemma} \label{lem:comb_null_p_vanish}
Let $P\in \F[x_1,\dotsc,x_n]$ with $\deg_{x_i} P \leq t_i$ for $i=1,\dotsc,n$. Let $S_1,\dotsc, S_n \subset \F$ , $|S_i|\geq t_i+1$ for $i\in[n]$. If $P (x_1,\dotsc,x_n) = 0$ when  $x_i\in S_i$ for each $i\in [n]$, then $P \equiv 0$.
\end{lemma}
\begin{proof}[Proof by induction on $n$.]
Base case $(n=1)$: a degree $t$ polynomial has at most $t$ roots.
Induction step: we write
\begin{fullwidth}
\begin{align*}
P(x_1,\dotsc,x_n) &= P_{t_n}(x_1,\dotsc,x_{n-1}) x_n^{t_n} + \dotsm + P_1 (x_1,\dotsc,x_{n-1}) x_n + P_0 (x_1,\dotsc,x_{n-1}) \\
&= \sum_{i=1}^{t_n} P_{t_i}(x_1,\dotsc,x_{n-1}) x_n^i.
\end{align*}
\end{fullwidth}
for some polynomials $P_{t_n},\dotsc,P_0$ in variables $x_1,\dotsc,x_{n-1}$. \marginnote{I.e., the $P_{t_i}$ are the coefficients of $x_n^i$, and do not depend on $x_n$.}
Now, fix $x_1\in S_1$, \ldots, $x_{n-1} \in S_{n-1}$. Then $P(x_1,\dotsc,x_n)$ becomes a one-variable polynomial in $x_n$ of degree at most $t_n$ with at least $t_n+1$ roots (one for each $x_n \in S_n$). So by the base case, each $P_i(x_1,\dotsc,x_{n-1}) = 0$ for $i\in[t_n]$. Since we chose $x_1\in S_1$, \ldots, $x_{n-1}\in S_{n-1}$ arbitrarily, by the induction hypothesis, $P_i \equiv 0$. Hence $P\equiv 0$.
\end{proof}
We may proceed to prove the theorem.
\begin{proof}[Proof of \cref{thm:comb_nullstell}]

Let $t_i = |S_i| - 1$.  For $i\in [n]$, we write
\[
g_i = x_i^{t_i+1} - \sum_{j=0}^{t_i} a_{ij} x_i^j
\]
for some coefficients $a_{ij}$.
Given a polynomial $f$ in variables $x_1,\dotsc, x_n$, we ``divide with remainder'' by each $g_i$, yielding
\[
f = \left( \sum_{i=1}^n h_i g_i \right) + P.
\]
with $\deg(h_i) \leq \deg(f) - \deg (g_i)$ and $\deg_{x_i} (P)\leq t_i$. Now, if $f$ vanishes when all $x_i\in S_i$, then $P$ does too, so \cref{lem:comb_null_p_vanish} we have $P\equiv 0$, giving the result.
\end{proof}

We may reformulate this result into a perhaps more helpful form.

\begin{theorem}[Combinatorial Nullstellensatz] \label{thm:comb_nullstell_nonzero}
Let $\F$ be a field, $f\in \F[x_1,\dotsc,x_n]$, $\deg(f) = t_1 + t_2 + \dotsm + t_n$ and the coefficient of $x_1^{t_1}x_2^{t_2}\dotsm x_n^{t_n}$ in $f$  non-zero. Let $S_1,S_2,\dotsc,S_n\subset \F$ with $|S_i| \geq t_i+1$. Then there exists $s_1\in S_1$, $s_2\in S_2$,\ldots, $s_n \in S_n$ such that $f(s_1,s_2,\dotsc,s_n) \neq 0$.
\end{theorem}
\begin{proof}
Suppose not. Then $f$ vanishes whenever $x_i \in S_i$, so by \cref{thm:comb_nullstell}, there exist polynomials $h_1,\dotsc,h_n$ with $\deg(h_i) \leq \deg (f) - |S_i| \leq \deg (f) - t_i -1$ such that
\[
f = \sum_{i=1}^n h_i g_i
\]
where $g_i = \prod_{s\in S_i}(x_i-s)$. Then the monomial $x_1^{t_1}x_2^{t_2}\dotsm x_n^{t_n}$ must appear with non-zero coefficient in some $h_ig_i$. Since $\deg f \geq \deg h_i + \deg g_i$, this term must be obtained by multiplying a  monomial in $h_i$ by the highest degree monomial in $g_i$, namely $x_i^{|S_i|}$. Since $|S_i| > t_i$, this is a contradiction.
\end{proof}

\begin{theorem*}[Cauchy-Davenport theorem]
Let $A,B \subset \Z/ p\Z =: \Z_p$ for $p$ prime. Then $|A+B| \geq \min\{ p, |A|+|B|-1\}$.
\end{theorem*}
% \begin{remark}
% \missing{put in ref to old C-D thoerem}
% \end{remark}
\begin{proof}
Suppose $|A|+|B| \leq p+1$ without loss of generality\sidenote{by throwing away elements of $A$ and $B$. If the minimum $m$ is acheived by $p$, we throw away elements until $|A|+|B|-1 = p = m$. Then we show that the now-smaller $|A+B|$ has $|A+B| \geq m$. Then the original $|A+B|$ must be at least $m$ as well.}. Suppose, for contradiction, that $|A+B| \leq |A|+|B|- 2$. Then select $C\supseteq A+B$ such that $|C| = |A| + |B| - 2$.

Let $a= |A|$ and $b=|B|$. Set $f(x,y) = \prod_{c\in C} (x+y-c)$. Then
\[
\deg f = |C| = a+b-2 = \overbrace{(a-1)}^{t_1} + \overbrace{(b-1)}^{t_2}
\]
The coefficient of $x_1^{t_1}x_2^{t_2}$ in $f$ is ${a+b-2\choose a-1}$. Note since $a+b-2 \leq p+1-2 = p-1 < p$ so the binomial coefficient is not zero. Set $S_1= A$, $S_2=B$. By \cref{thm:comb_nullstell_nonzero}, there exists $s_1\in A$, $s_2\in B$ such that $f(s_1,s_2) \neq 0$. Then $s_1+s_2\not\in C$ which is a contradiction.
\end{proof}

Let us consider a conjecture by \cite{ErdosHeilbronn}, proven in 1994.
\begin{theorem}[\cite{SilvaHamidoune1994}]
Let $A,B\subset \Z_p$, with $A\oplus B = \{a+b: a\in A, b\in B, a\neq b\}$. Then
\[
|A\oplus B| \geq \min \{p, |A| +|B|-2 \}
\]
if $|A|\neq |B|$.
\end{theorem}
\begin{remark}
The assumption $|A|\neq|B|$ is necessary. Say $A= \{0,1,\dotsc,k\}$ with $k< p/2$, then $|A| = k+1$ and
\[
A\oplus A = \{1,2,\dotsc,2k-1 \}
\]
so $|A\oplus A| = 2k-1 = 2|A|-3$.
\end{remark}
\begin{proof}
Let $a=|A|$, $b=|B|$. WLOG, assume $a+b\leq p+2$. Suppose for a contradiction that  there exists $C \supset A\oplus B$ such that $|C| = a+b-3$. Let
\[
f(x,y) = (x-y)\prod_{c\in C} (x+y-c).
\]
Then $\deg f = |C|+1 = a+b-2 = (a-1) + (b-1)$. Assume for now that the coefficient of $x^{a-1}y^{b-1}$ is non-zero. Then by \cref{thm:comb_nullstell_nonzero} there exists $s_1\in A$, $s_2\in B$ such that $f(s_1,s_2) \neq 0$. But then $s_1\neq s_2$ and $s_1+s_2\not \in C$. This is a contradiction.

It remains to find the coefficient of $x^{a-1}y^{b-1}$ in $f(x,y)$. The coefficient must be the same as in the polynomial $(x-y)(x+y)^{a+b-3}$. We use binomial expansion and only keep the terms of the right degree: $(x-y) (\alpha_1 x^{a-2}y^{b-1} +  \alpha_2  x^{a-1} y^{b-2})$. Then the coefficient is
\[
\alpha_1 - \alpha_2 = {a+b-3 \choose a-2} - {a+b-3 \choose a-1} \neq 0 \pmod{p}
\]
if and only if
\[
\frac{(a+b-3)!}{(a-2)!(b-1)!} \neq \frac{(a+b-3)!}{(a-1)!(b-2)!} \pmod{p}
\]
which we may simplify to obtain
\[
a-1 \neq b-1 \pmod{p}.\qedhere
\]
\end{proof}

\paragraph{$n$-dimensional hypercube}
$Q_n:=\{0,1\}^n = \{(x_1,\dotsc,x_n) \in \R^n: x_i \in \{0,1\}\}$. $2^n$ points. See \cref{fig:cube} for $Q_3$.
We'd like to cover $Q_n$ by affine hyperplanes; this can be done with $2$: one on top and one on bottom.
Suppose we'd like to cover exactly all verticies except the origin.

\begin{marginfigure}
\begin{center}
\begin{tikzpicture}
\newcommand{\Depth}{2}
\newcommand{\Height}{2}
\newcommand{\Width}{2}
\def\radius{2pt}
\coordinate (O) at (0,0,0);
\coordinate (A) at (0,\Width,0);
\coordinate (B) at (0,\Width,\Height);
\coordinate (C) at (0,0,\Height);
\coordinate (D) at (\Depth,0,0);
\coordinate (E) at (\Depth,\Width,0);
\coordinate (F) at (\Depth,\Width,\Height);
\coordinate (G) at (\Depth,0,\Height);

\draw[black!50] (O) -- (C) -- (G) -- (D) -- cycle;% Bottom Face
\draw[black!50] (O) -- (A) -- (E) -- (D) -- cycle;% Back Face
\draw[black!50] (O) -- (A) -- (B) -- (C) -- cycle;% Left Face
\draw[black!50] (D) -- (E) -- (F) -- (G) -- cycle;% Right Face
\draw[black!50] (C) -- (B) -- (F) -- (G) -- cycle;% Front Face
\draw[black!50] (A) -- (B) -- (F) -- (E) -- cycle;% Top Face

\filldraw (O) circle(\radius);
\filldraw (A) circle(\radius);
\filldraw (B) circle(\radius);
\filldraw (C) circle(\radius);
\filldraw (D) circle(\radius);
\filldraw (E) circle(\radius);
\filldraw (F) circle(\radius);
\filldraw (G) circle(\radius);

%% Following is for debugging purposes so you can see where the points are
%% These are last so that they show up on top
%\foreach \xy in {O, A, B, C, D, E, F, G}{
%    \node at (\xy) {\xy};
%}
\end{tikzpicture}
\end{center}
\caption{The cube $Q_3$.}\label{fig:cube}
\end{marginfigure}
\begin{theorem}[\cite{AlonFuredi1993}]
Let $H_1,H_2,\dotsc,H_m$ be affine hyperplanes in $\R^n$ with cover exactly $2^n-1$ vertices of the $n$-dimensinal hypercube $Q_n$. Then $m\geq n$.
\end{theorem}
\lect{4}{11}
\begin{proof}
Let $H_i = \{x: \braket{a_i,x} = b_i\}$ for some $a_i = (a_{i_1},\dotsc,a_{i_n})\in \R^n$ and $b_i\in \R$. That is, if $x=(x_1,\dotsc,x_n)\in H_i$,
\[
a_{i_1}x_1 + a_{i_2}x_2 + \dotsm + a_{i_n}x_n = b_i.
\]
Assume for contradiction that $m<n$, and $H_1,\dotsc,H_m$ do not cover the vertex $(0,0,\dotsc,0)$\marginnote{The vertex $\vec 0$ was chosen without loss of generality, of course.}, but cover every $x = (x_1,\dotsc,x_n)$ for $x_i\in \{0,1\}$ with not all $x_i = 0$. Define
\[
h(x_1,\dotsc,x_n) = \prod_{i=1}^m (\braket{a_i,x} - b_i).
\]
Then by assumption, $h=0$ for every vertex of $Q_n$ except $0$.\marginnote{Note then $h(0,\dotsc,0) = \prod_{i=1}^m (-b_i) \neq 0$.} Using the language of \cref{thm:comb_nullstell_nonzero}, set $S_i = \{0,1\}$, $t_i=1$. We want to choose a polynomial $f$ with $\deg(f)=n$, the coefficient of $x_1\dotsm x_n$ non-zero, and $f(x)=0$ for every vertex of $Q_n$. Our polynomial $h$ will not suffice because $\deg h =m < n$ and in particular the coefficient of $x_1\dotsm x_n$ is zero. Instead, set
\[
f(x_1,\dotsc,x_n) := (-1)^{m+n+1}  \prod_{i=1}^m b_i \prod_{i=1}^n (x_i - 1) + h(x_1,\dotsc,x_n).
\]
Then $\deg f = n$ and the coefficient of $x_1\dotsm x_n$ is $(-1)^{m+n+1} \prod_{i=1}^m b_i \neq 0$.
\Cref{thm:comb_nullstell_nonzero} then yields some $x_1\in S_1$, \ldots, $x_n \in S_n$ such that $f(x_1,\dotsc,x_n)\neq 0$. Since  if $x_i \in \{0,1\}$ not all zero, the first term of $f$ vanishes (because at least one $x_i=1$), and $h$ vanishes by assumption, we must have $x_i\equiv 0$. But then
\[
0 \neq f(0,\dotsc,0) = -(-1)^{m}  \prod_{i=1}^m b_i + h(0,\dotsc,0) = -(-1)^m \prod_{i=1}^m b_i + (-1)^m \prod_{i=1}^m b_i
\]which is a contradiction.
\end{proof}

% \marginnote{Lecture X,  Monday April 11, 2016}

%Lemma 10.6
\begin{lemma} \label{lem:permanent}
Let $A$ be an $n\times n$ matrix over a field $\field$. Suppose $\per(A)\neq 0$.
% , where  if $A = (a_{ij})$,
\marginnote{
The \emph{permanent} of $A$, if $A = (a_{ij})$, is
\[
\per (A) = \sum_{\pi \in S_n} \prod_{i=1}^n a_{i \pi(i)},
\]
i.e. is the determinant without the signs. For example, $\per \begin{pmatrix}
a & b\\ c & d
\end{pmatrix} = ad+bc$. Unlike the determinant, it is not known to be efficiently computable.}
Let $S_1,S_2,\dotsc,S_n \subset \F$, $|S_i| = 2$ for every $i$. Let $b\in \F^n$. Then there exist $x_1\in S_1$, $x_2\in S_2$, \ldots, $x_n\in S_n$ such that if $x = (x_1,\dotsc,x_n)$, then $(Ax)_j \neq b_j$ for every $j=1,\dotsc, n$.
\end{lemma}
\begin{proof}
Let $f(x_1,\dotsc,x_n) = \prod_{i=1}^n (a_{i1}x_1 + a_{i2}x_2 + \dotsm + a_{in}x_n - b_i)$. By \cref{thm:comb_nullstell_nonzero}, we can find $x_i\in S_i$ such that $f(x_1,\dotsc,x_n)\neq 0$, as desired, if the coefficient of $x_1,\dotsc,x_n$ in $f$ is non-zero. But this coefficient is exactly the permanent of $A$.
\end{proof}

Let $a_1,a_2,\dotsc,a_k$ be integers. We'd like to find nonempty $S\subset[k]$ such that $\sum_{i\in S} a_i$ is divisible by $n$. How large must $k$ be (as a function of $n$)?

One needs at least $n$, as shown by taking $a_i= 1$ for every $i$.

In fact, $k=n$ suffices. Consider $\sigma_i = \sum_{j=1}^i a_j$ for $i=1,\dotsc,n$, i.e., $\sigma_1 = a_1$, $\sigma_2 = a_1 + a_2$, etc. If $\sigma_i \neq 0 \pmod{n}$ for every $n$, then there must exist $i,j$ such that $\sigma_i = \sigma_j \pmod{n}$. But then $\{i+1,\dotsc,j\}$ is the required set.\marginnote{If $\sigma_i = 0 \pmod{n}$ for some $i$, $\{1,2,\dotsc,i\}$ suffices. }

Now, let $a_1,\dotsc,a_k$ be integers. We'd like to find $S\subset [k]$ with $|S| = m$ such that $n$ divides $\sum_{i\in S} a_i$. If $m< n$, we can never guarantee the existence of $S$, as the example $a_i\equiv 1$ shows. In fact, we need at least $m$ divisible by $n$. Let us take $m=n$.

Suppose $n=2$. How many integers must we have such that two of them sum to an even number? $k=3$; then there is always at least two evens or two odds.

Consider $n=3$. We note $\{0,0,1,1\}$ has no 3 numbers summing to a number divisible by 3. But $k=5$ suffices.

In general, $k=2n-1$. For $2n-2$, we may take $n-1$ $0$'s and $n-1$ $1$'s as a counterexample.

\begin{theorem}[\cite{ErdosGinzburgZiv1961}]
Let $a_1,a_2,\dotsc,a_{2n-1}\in \Z$ be integers. Then there exists $S\subset [2n-1]$ with $|S| = n$ such that $\sum_{i\in S} a_i$ is divisible by $n$.
\end{theorem}
\begin{proof}
Assume that $n=p$ is prime. We may reduce modulo $p$ to assume $0\leq a_i \leq p-1$ and an ordering $a_1\leq a_2 \leq \dotsm \leq a_{2p-1}$. Let $S_i = \{a_i,a_{i+p-1}\}$ for $i=1,\dotsc,p-1$. Let $A$ be the all $1$'s matrix: $a_{ij} = 1$ for all $1\leq i,j\leq p-1$.

Let $\{b_1,\dotsc,b_{p-1}\}$ be all elements of $\Z_p$ except $-a_{2p-1}$.

\begin{example}
$p=3$, $a_1,a_2,\dotsc,a_5$. $S_1 = \{a_1,a_3\}$, $S_2 = \{a_2,a_4\}$. Suppose $a_5=2$. Then $\{b_1,b_2\} = \{0,2\}$.
\[
\begin{pmatrix}
1 & 1\\
1 & 1
\end{pmatrix} \begin{pmatrix}
a_1 \text{ or } a_3 \\ a_2 \text{ or } a_4
\end{pmatrix} \neq \begin{pmatrix}
0 \\ 2
\end{pmatrix}
\]
where the inequality in fact occurs componentwise.
\end{example}

Suppose there exists $s_1\in S_1$, \ldots, $s_{p-1} \in S_{p-1}$ such that for $s = (s_1,\dotsc,s_{p-1})$,
\[
As \neq \begin{pmatrix}
b_1 \\ \vdots \\ b_{p-1}
\end{pmatrix}
\]
where the inequality occurs for each coordinate. Equivalently, $s_1 + s_2 + \dotsm + s_{p-1} = - a_{2p-1} \pmod{p}$. That is, $s_1 + \dotsm +  s_{p-1} + a_{2p-1} =  0 \pmod{p}$.

We may find such $s_1,\dotsc,s_{p-1}$ using \cref{lem:permanent}, using that
\[
\per \begin{pmatrix}
1 & \dotsm & 1\\
\vdots & & \vdots \\
1 & \dotsm & 1
\end{pmatrix} = (p-1)! \neq 0 \pmod{p}.
\]
However, we still need to check the condition $|S_i| = 2$ for each $S_i$. If for some $i$ we have $|S_i|=1$, then, $a_i = a_{i+p-1}$. In such a case, by our ordering, $a_i = a_{i+1} = \dotsm = a_{i+p-1}$,  hence
\[
\sum_{j=1}^{i + p-1} a_j = 0 \pmod{p}
\]
as desired.
The composite case will be left as an exercise by induction.
\end{proof}

\subsection{Kakeya needle problem (\cite{Kakeya})}
What is the minimum area of a region in the plane in which one can continuously rotate by 360\textdegree  a needle of length 1? \Cref{fig:needle_circ_tri} shows two attempts.
In fact, the answer is zero.
\begin{marginfigure}
\hfill
\begin{tikzpicture}[scale=2]
\draw[opacity=.5] (0,0) circle (0.5);
\draw[<->] (0,-.5) -- (0,.5);
\end{tikzpicture}
\hfill
\begin{tikzpicture}[scale=2]
\def\x{2/1.732}
\coordinate (a) at (0,0);
\coordinate (b) at (\x/2,1);
\coordinate (c) at (\x,0);
\draw[opacity=.5] (a)--(b) -- (c) -- cycle;
\draw[<->] (\x/2,0) node[label=left:$b$]{} -- (\x/2,.95)node[label=left:$a$]{};
\end{tikzpicture}
\hfill
\caption{Left: we may simply rotate our needle of length about its middle, using a circle area of $\pi/4$. Right: we could rotate our needle by pulling end $b$ to the bottom left corner of the triangle, then end $a$ down the right side of the triangle. Then we could pull $b$ up the left side of the triangle, and finally $a$ to the middle, therby completing the rotation. This uses an equilateral triangle of altitude $1$, which then has area $\frac{1}{\sqrt{3}}$.} \label{fig:needle_circ_tri}
\end{marginfigure}

Let us try to restate this in terms of finite fields. First, we have the equivalent\sidenote{It turns out these are equivalent, by adding regions of area zero connecting the segments in each direction.} question: What is the minimum area of a region in the plane in which contains a segment of length 1 in every direction?

Now, we have the finite field Kakeya problem: What is the minimum $|E|$ such that $E\subset \field^d$ and $E$ contains a \emph{line} in every direction. That is, for every $v\in \field^d\setminus\{0\}$ there exists $x_v$ such that $x_v + tv\in E$ for every $t\in \field$.
In particular, does there exist $c_d>0$ (depending on $d$ but not $\field$) such that
\[
|E| \geq c_d |\field|^d
\]
for all finite fields $\field$?
% This is in a dual sense to Combinatorial Nullenstallsatz, which says roughly that if $f$ is a low-degree polynomial is cannot vanish everywhere on a set of a certain structure. Here, we will show that with a set of a different st
\begin{lemma}\label{lem10p8:polynomial_vanishes_on_E}
Let $E\subset \F^d$ such that $|E|< {k+d \choose k}$. Then there exists non-zero polynomial $f$ of $d$ variables such  $f(x)=0$ for every $x\in E$, and $\deg f \leq k$.
\end{lemma}
\begin{proof}
Let $V$ be the vector space of polynomials in $d$ variables over $\F$ of degree at most $k$. Then $\dim V$ is the number of monomials $x_1^{k_1}x_2^{k_2}\dotsm x_d^{k_d}$ such that $k_1+k_2+\dotsc+k_d \leq k$. This is the same as the number of decompositions $k_1 + k_2 + \dotsm + k_d + k_{d+1} = k$ such that $k_i\geq 0$, $k_i \in \Z$. This is ${k+d \choose k}$ as follows:
 if we write $k$ stars
\[
\star \star \star \star \star \dotsm \star\]
then our task is to put $d$ bars between the stars and thus partition the elements into $k_1,k_2,\dotsc,k_{d+1}$
\[
 \operatorname*{\star}_{k_1}|\operatorname*{\star \star \star}_{k_2} | \star \dotsm \star
\]
sowe must choose $k$ stars from the $k+d$ total number of symbols.

Let $E = \{x_1,\dotsc,x_m\}\subset\field^d$. For $p\in V$, the map
\[
p\mapsto ( p(x^1),p(x^2),\dotsc,p(x^m))
\]
is a linear map from $V$ to $|E|$-dimensional space.
If $\dim(V) > |E|$, then there exists $p\in V$, $p\neq 0$ such that $p$ is mapped to zero, i.e. $p$ is zero everywhere on $E$.
\end{proof}
\lect{4}{13}
% \marginnote{Lecture X: Wednesday, April 13, 2016}
Now, given $\field$ a finite field, we will say $E\subset \field^d$ is a \defn{Kakeya set} if for every $v\in \field^d \setminus \{0\}$ there exists $x\in \field^d$ such that $x+tv \in E$ for every $t$.
\begin{lemma}
If $E\subset \field^d$ is Kakeya, then there exists no $p\in \field[x_1,\dotsc,x_d]$, $p\neq 0$, $\deg p\leq |F| - 1$ such that $p(x) = 0$ for every $x\in E$.
\end{lemma}
\begin{proof}
Suppose such $p$ exists. Then
\[
p(x) = p_k(x) + p_{k-1}(x) + \dotsm + p_0(x)
\]
where $p_i(x)$ is homogenous polynomial of degree $i$ and $p_k(x)$ not identically $0$ ($\deg p = k$).
By definition, for every $v\in \field^d$ there exists $x$ such that $p(x+tv) =0$ for every $t\in \field$. Fix $x$ and $v$, $p(x+tv)$ is a polynomial in $t$ of degree $\leq |F| - 1$. If $x= ( x_1,\dotsc,x_d)$ and $v = ( v_1,\dotsc,v_d)$, then $x+tv = (x_1+tv_1,\dotsc,x_d+ tv_d)$. The coefficient of $t^k$ in $p$ is $p_k(v)$.

On the other hand, $p$ has $|F|$ roots so $p$ is identically zero as a polynomial in $t$. In particular, $p_k(v) = 0$. But this is true for every $v$, contradicting the assumption that $p_k(v) $ is not identically zero.
\end{proof}

\begin{corollary}[\cite{dvir2009size}]
If $E\subset \field^d$ is Kakeya, then
\[
|E| \geq {|F| - 1 + d \choose d} = \frac{|\field|^d}{d!} + o(|\field|^d).
\]
\end{corollary}

\subsection{Shannon capacity (\cite{shannonGraph})}
We are transmitting messages in alphabet $V$ over a noisy channel. Some symbols can be confused with each other during transmission.

Let $G$ be a graph with vertex set $V$ with a pair of symbols joined by an edge if they can be confounded.

\begin{example}
$V = \{a,b,d,p,q\}$.

\begin{figure}
\begin{center}
\begin{tikzpicture}%[color=DarkBlue]
\def\n{5}
\def\radius{1}
\def\rotation{-30}
\def\circrad{1pt}
\def\deg{360/\n}
\pgfmathsetmacro\nminusone{\n-1}

% \foreach \x in {1,...,\n}
% {
	\def\x{1}
	\pgfmathsetmacro\myangle{\x*\deg+\rotation};
	\filldraw (\myangle:\radius cm) circle (\circrad) node[label=$a$](a\x){};
	\def\x{2}
	\pgfmathsetmacro\myangle{\x*\deg+\rotation};
	\filldraw (\myangle:\radius cm) circle (\circrad) node[label=$d$](a\x){};
	\def\x{3}
	\pgfmathsetmacro\myangle{\x*\deg+\rotation};
	\filldraw (\myangle:\radius cm) circle (\circrad) node[label=left:$q$](a\x){};
	\def\x{4}
	\pgfmathsetmacro\myangle{\x*\deg+\rotation};
	\filldraw (\myangle:\radius cm) circle (\circrad) node[label=below:$p$](a\x){};
\def\x{5}
	\pgfmathsetmacro\myangle{\x*\deg+\rotation};
	\filldraw (\myangle:\radius cm) circle (\circrad) node[label=right:$b$](a\x){};

\foreach \y in {1,...,\n}
	{

	\pgfmathsetmacro\z{\y+1}
% 	{
		% \ifthenelse{\y < \n+1}
		% {
		\pgfmathsetmacro\myangley{\y*\deg+\rotation}
		\pgfmathsetmacro\myanglez{\z*\deg+\rotation}
		\draw (\myangley:\radius cm) -- (\myanglez:\radius cm);
% 		}{}

	}

\end{tikzpicture}
\end{center}
\caption{A graph representing transmitting the alphabet $V = \{a,b,d,p,q\}$. Adjacency indicates symbols which could be confused for each other. This graph is $C_5$, the cycle of length five.}
\end{figure}
% Write $a,d,q,p,b$ clockwise, connect each subsequent one.
We'd like to send a set $S_n$ of $n$ letter messages such that no two messages in $S_n$ can be confused with each other. One solution: we can send $2^n$ messages which can't be confused just by using sympols $a$ and $p$.
\end{example}

A \emph{strong product} of graphs $G$ and $H$ denoted $G\boxtimes H$ is a graph with vertex set $V(G)\times V(H)$ and $(u_1,v_1)\sim (u_2,v_2)$ if
\begin{itemize}
	\item[\phantom{and}] $u_1=u_2$ or $u_1$ is adjacent to $u_2$
	\item[and] $v_1 = v_2$ or $v_1$ is adjacent to $v_2$.
\end{itemize}
Let $\boxtimes^n G$ denote $\underbrace{G\boxtimes G\boxtimes \dotsm \boxtimes G}_{n \text{ times}}$.

We are interested in the maximum size of an independent set in $\boxtimes^n G$, which we will call $\alpha(\boxtimes^n G)$. Let $\Theta(G) = \lim_{n\to\infty} \sqrt[n]{\alpha(\boxtimes^n G)}$ be the \emph{Shannon capacity}. This exists because $\alpha(G\boxtimes H) \geq \alpha(G) \alpha(H)$, as the product of two independent sets produces an independent set in the product graph.

\begin{lemma} \label{lem:chrom_num_shannon_capacity}
For any graph, $\alpha(G) \leq \Theta(G) \leq \chi(\bar G)$.\marginnote{$\bar G$ denotes the complement of $G$ in $V(G)^{(2)}$, i.e. two vertices are adjacent in $\bar G$ iff they are not adjacent in $G$.}
\end{lemma}
For brevity, let us write $G^n:=\boxtimes^n G$.
\begin{proof}
By the product inequality for $\alpha$, we have $\alpha(G^n) \geq (\alpha(G))^n$, hence $\alpha(G) \leq \Theta(G)$.

Now, note $\alpha(G)\leq \chi(\bar G)$, since $\alpha(G)$ is the size of the largest complete subgraph of $\bar G$. Further, $\chi(\overline{G\boxtimes H}) \leq \chi(\bar G)\chi(\bar H)$.

We may think of coloring $\bar G$ as coloring $G$ such that any two vertices which are the same color are connected. To see $\chi(\overline{G\boxtimes H}) \leq \chi(\bar G)\chi(\bar H)$,  color each vertex $(u,v)$ of $\overline{G\boxtimes H}$ by the pair of color $(c_1(u),c_2(v))$ where $c_1$ is a coloring of $\bar G$ and $c_2$ as a coloring of $\bar H$.

Now every color class is used on $K_l\boxtimes K_s$ for some $l$ and $s$, but the strong product of complete graphs is complete. Therefore
\[
\alpha(G^n) \leq \chi(\bar G^n) \leq (\chi(\bar G))^n
\]
and we have the result as derised.
\end{proof}

In fact, $\alpha(G) = \chi(\bar G)$ for many graphs, including perfect graphs (hence all graphs on $4$ vertices). The smallest graph in which  $\alpha(G) \neq \chi(\bar G)$ is the cycle of length five, $C_5$, considered earlier. There, we saw we could achieve $2$, and $\chi(\bar C_5) = 3$, so $2\leq \Theta(C_5)\leq 3$.
\begin{marginfigure}
\begin{center}
\begin{tikzpicture}%[color=DarkBlue]
\def\n{5}
\def\radius{1}
\def\rotation{-30}
\def\circrad{1pt}
\def\deg{360/\n}
\pgfmathsetmacro\nminusone{\n-1}

% \foreach \x in {1,...,\n}
% {
	\def\x{1}
	\pgfmathsetmacro\myangle{\x*\deg+\rotation};
	\filldraw (\myangle:\radius cm) circle (\circrad) node[label=$1$](a\x){};
	\def\x{2}
	\pgfmathsetmacro\myangle{\x*\deg+\rotation};
	\filldraw (\myangle:\radius cm) circle (\circrad) node[label=$2$](a\x){};
	\def\x{3}
	\pgfmathsetmacro\myangle{\x*\deg+\rotation};
	\filldraw (\myangle:\radius cm) circle (\circrad) node[label=left:$3$](a\x){};
	\def\x{4}
	\pgfmathsetmacro\myangle{\x*\deg+\rotation};
	\filldraw (\myangle:\radius cm) circle (\circrad) node[label=below:$4$](a\x){};
\def\x{5}
	\pgfmathsetmacro\myangle{\x*\deg+\rotation};
	\filldraw (\myangle:\radius cm) circle (\circrad) node[label=right:$5$](a\x){};

\foreach \y in {1,...,\n}
	{

	\pgfmathsetmacro\z{\y+1}
% 	{
		% \ifthenelse{\y < \n+1}
		% {
		\pgfmathsetmacro\myangley{\y*\deg+\rotation}
		\pgfmathsetmacro\myanglez{\z*\deg+\rotation}
		\draw (\myangley:\radius cm) -- (\myanglez:\radius cm);
% 		}{}

	}

\end{tikzpicture}
\end{center}
\caption{$C_5$, the cycle of length 5.} \label{fig:C5}
\end{marginfigure}

We also have $\alpha(C_5^2)\geq 5$, that is, you can send 5 different 2 symbol messages. These messages are: $\{(1,1),(2,3)(3,5),(4,2),(5,4)\}$, using the labelling of \cref{fig:C5}.
In particular, $2<\sqrt{5} \leq \Theta(C_5)$, so our naive bound of $2$ is not optimal.

To prove an upper bound on $\Theta$, we'd like a function $
\theta(G)$ such that
\begin{enumerate}
	\item $\alpha(G) \leq \theta(G)$ \label{enum:theta1}
	\item $\theta(G\boxtimes H)\leq \theta(G) \theta(H)$.  \label{enum:theat2}
\end{enumerate}
Then, as in \cref{lem:chrom_num_shannon_capacity}, any such function will upper bound $\Theta$, i.e., $\Theta(G) \leq \theta(G)$. Ideally, $\theta(C_5)= \sqrt{5}$, to prove the lower bound of $\sqrt{5}$ is optimal. This would confirm that we've found a sharper upper bound than $\chi (\bar G)$.

One function satisfying all of these is the \defn{Lov\'asz theta}\sidenote{\cite{Lovasztheta}}, defined as follows.
Let $G$ be a graph with $V(G) = [n]$. A collection $v_1,v_2,\dotsc,v_n\in \R^d$ is an \defn{orthonormal representation} of $G$ if
\begin{itemize}
	\item $\|v_i\| = 1$,
	\item $v_i \perp v_j$ if $i$ and $j$ are not adjacent.
\end{itemize}
\marginnote{Every graph has an orthonormal representation in dimension $n$.}
Let the \defn{value}[orthonormal representation!value] of an orthonormal representation be
\[
\min_{\|c\|=1} \max_{v_i} \frac{1}{\braket{v_i,c}^2}
\]
where $\braket{\cdot,\cdot}$ is the Euclidean inner product (dot product).
The vector $c$ acheiving the minimum is called a \emph{handle}.
Finally, define $\theta(G)$ as the minimal value over orthonormal representations.

That $\theta(G)$ satisfies points \labelcref{enum:theta1,enum:theat2} is the content of \cref{lem:alpha_less_than_theta,lem:shannon_cap_theta_submult}. That $\theta(C_5) \leq \sqrt{5}$ is the content of  \cref{lem:theta_C5}. This will conclude our discussion of Shannon capacity.

\begin{lemma} \label{lem:alpha_less_than_theta}
$\alpha(G)\leq \theta(G)$
\end{lemma}
\begin{proof}
Let $v_1,\dotsc,v_n$ be the optimal orthonormal representation of $G$ with handle $c$. Let $\alpha(G) = k$, and suppose $\{1,2,\dotsc,k\}$ form an independent set. Then $v_1,\dotsc,v_k$ can be completed to an orthonormal basis, so
\[
	1=\|c\|^2 = \braket{c,c} \geq \sum_{i=1}^k (\braket{c,v_i})^2  \geq \frac{k}{\theta(G)}
\]
since
% \[
$\theta(G) \geq \frac{1}{\braket{c,v_i}^2}$.
 % \]
\end{proof}

\begin{lemma}\label{lem:shannon_cap_theta_submult}
\[
\theta(G\boxtimes H)\leq \theta(G)\theta(H).
\]
\end{lemma}
\begin{proof}
Observation: if $u,v\in U$ and $x,y \in V$, then
\[
\braket{u\otimes x, v \otimes y} = \braket{u,v}\braket{x,y}
\]
\marginnote{We can see this by working in components: if $u = (u_i)$, $v = (v_i)$, $x = (x_i)$, $y = (y_i)$, then $\braket{u\otimes x, v\otimes y} = \sum_{i,j} u_i x_j v_i y_j$ which we may regroup into the product of the two inner products.}

% \begin{remark}
If $u_1,\dotsc,u_n$ is an o.r. of $G$ with handle $c$, and $v_1,\dotsc,v_n$ is an o.r. of $H$ with handle $d$, then $(u_i\otimes v_j)$ is an o.r. of $G\boxtimes H$, by the above identity.

Let $c\otimes d$ be a handle. Then
\begin{align*}
\theta(G\otimes H) &\leq \max_{u_i,v_j}\frac{1}{ (\braket{c\otimes d, u_i\otimes v_j})^2}\\ &=\max_{u_i,v_j} \frac{1}{(\braket{c,u_i})^2 (\braket{d,v_j})^2}\leq \theta(G)\theta(H).
\end{align*}
% \end{remark}
\end{proof}

\begin{lemma} \label{lem:theta_C5}
$\theta(C_5)\leq\sqrt{5}$.
\end{lemma}
\begin{proof}
Start with
\begin{align*}
v_{i+1} = \frac{(\cos( \tfrac{2\pi}{5}i),(\sin( \tfrac{2\pi}{5}i), p)}{\sqrt{1+p^2}}
\end{align*}
and increase $p$ from zero until each $v_i \perp v_{i+2},v_{i+3}$.
\begin{figure*}[ht]
% \includegraphics[width=.25\linewidth]{graphics/C5p=0.pdf}\includegraphics[width=.25\linewidth]{graphics/C5p=p3.pdf}\includegraphics[width=.25\linewidth]{graphics/C5p=p6.pdf}\includegraphics[width=.25\linewidth]{graphics/C5p=p0.pdf}
\includegraphics[width=.33\linewidth]{graphics/C5p=0.pdf}\includegraphics[width=.33\linewidth]{graphics/C5p=p45.pdf}\includegraphics[width=.33\linewidth]{graphics/C5p=p0.pdf}
\caption{From left to right, the vectors $v_i$ with $p=0,0.45,\frac{1+\sqrt{5}}{4}\approx 0.89$. The intuition is that we are starting with our five vectors equally spaced on the unit circle. Then we ``close the umbrella'', increasing $p$, and pointing the vectors towards the $z$-axis, until each vector is orthogonal to the two vectors across from it.}
\end{figure*}

We have $(1,0,p) \perp (\cos \tfrac{4\pi}{5}, \sin \tfrac{4\pi}{5},p)$ if $-p^2 = \cos \tfrac{4\pi}{5}$. Then $p^2 = \frac{1+\sqrt{5}}{4}$. Now, we may check that at this value of $p$, each $v_i \perp v_{i+2},v_{i+3}$. That is, the $v_i$ form an orthonormal representation of $C_5$. Take $c = (0,0,1)$. Then $\braket{c,v_i} = \frac{p}{\sqrt{1+p^2}}$ for all $i$, and $\frac{1}{\big( \frac{p}{\sqrt{1+p^2}} \big)^2} = \dotsm = \sqrt{5}$.
\end{proof}


\chapter*{Appendices}
\addcontentsline{toc}{chapter}{Appendices}

\appendix

\chapter{The celestial sphere and astronomical coordinates}
\label{app:coord}

\section{A place on Earth}

The terrestrial equator is defined as the great circle halfway between
the North and the South Poles.  The ``Prime Meridian'' was defined in
1884 as the half circle passing through the poles and the old Royal
Observatory in Greenwich, England (see Fig.~\ref{figearth}).

A location on the surface of the Earth is defined by three numbers:
the longitude $\lambda$, the latitude $\phi$, and the height above sea
level, $h$.

The longitude of a given point on Earth is measured westward from
Prime Meridian to the intersection with the equator of the circle of
longitude that passes through the point.

The latitude is the angle measured northward (positive) or southward
(negative) along the circle of longitude from the equator to the
point.

Onsala Space Observatory is located just a few meters above sea level,
at the following longitude and latitude:

\begin{center}
${\bf{\boxed{\lambda=12^{\circ}01'00'' \text{E}
\hspace{1cm}\phi=57^{\circ}25'00''\text{N.} }}}$
\end{center}

\begin{figure}[ht]
\begin{center}
\includegraphics[width=8cm]{../figures/longlat.pdf}
\end{center}
\caption{Illustration of the terrestrial system, with longitude
  ($\lambda$) and latitude ($\phi$).  Great circles passing through
  the poles are circles of longitude.  Circles parallel to the Earth
  equator are circles of latitude. }
\label{figearth}
\end{figure}

\section{The celestial sphere}

\subsection{Equatorial coordinates}

\begin{figure}[ht]
\begin{center}
\includegraphics[width=8cm]{../figures/celestial.pdf}
\end{center}
\caption{Illustration of the celestial coordinate system, with right ascension $\alpha$
and declination $\delta$.
Earth is at the center. The plane of the ecliptic is inclined by 23.5$^\circ$ with
respect to the celestial equator. }
\label{figcelest}
\end{figure}

The celestial sphere is an imaginary sphere concentric with the Earth, on which
astronomical objects appear (see Fig.~\ref{figcelest}).
The celestial equator is the
natural extension of Earth's equator. Because of the inclination of
Earth's axis of rotation, the apparent annual path of the Sun
on the celestial sphere (the {\em ecliptic}) doesn't  coincide with the
celestial equator; the ecliptic makes an angle of 23.5$^\circ$
with the celestial equator. The point where the Sun crosses the
celestial
equator going northward in spring is called {\bf the Vernal
Equinox}.
On the equinox, the Sun lies
in the Earth's equatorial plane, and the day and night are equally
long. The solstices mark the dates when the Sun is farthest from
the celestial equator and occur around June~21 and December~21.

\medskip
{\bf{$\rightarrow$ Mark the north and south celestial poles,
the celestial equator, the ecliptic,
the solstices and the equinoxes on Fig.~\ref{figcelest}.}}
\medskip

Positions on the celestial sphere are defined by angles along
great circles. By analogy with terrestrial longitude,
right ascension (RA), $\alpha$, is the celestial longitude of an
astronomical object,
but it is measured {\em eastward} from the Vernal Equinox
along the celestial equator. RA is expressed
in hours and minutes of time, with 24 hours corresponding
to 360$^\circ$.

In analogy with terrestrial latitude, declination (DEC), $\delta$,
is the angular distance of an object from the celestial equator.
Right ascension and declination $(\alpha,\delta)$ specify completely
a position on the celestial sphere.

Imagine now that we find ourselves at location $P$ on Earth at a
latitude $\phi$, and we want to observe the sky.
An astronomical object
of declination $\delta$ reaches a maximum altitude above
the horizon, $h_{\rm max}$, and a minimum altitude, $h_{\rm min}$,
given by
\begin{equation}
\begin{array}{l}
h_{\rm max} = 90^\circ -|\phi-\delta| \\
h_{\rm min} = -90^\circ +|\phi+\delta|.
\end{array}
\end{equation}

{\bf{$\rightarrow$ In Onsala, astronomical objects with $\delta > 33^\circ$
always remain above the horizon (they are circumpolar); those with $\delta <
-33^\circ$ never rise above the horizon.}}

\subsection{The Local Sidereal Time}

\begin{figure}[ht]
\begin{center}
 \includegraphics[width=8cm]{../figures/lst.pdf}
\end{center}
\caption{The Local Sidereal Time is the time relative to the stars, whereas the
solar time is relative to the Sun. 24~hours of LST time pass in only
23~hours 56~minutes 05~seconds of solar time, because Earth, having
moved on its orbit around the Sun, finds itself the next day at the same position
relative the stars {\em a bit earlier} than at the same position relative
to the Sun.
In one year (365 days), Earth has moved 360$^\circ$ around the Sun and lost one
turn (24 hours) relative to the stars. So the difference in one month is 2 hours, and in one day it is 24~hours/365 =
3m56s.
}
\label{figlst}
\end{figure}

The convention of measuring RA eastward was chosen
because it makes the celestial sphere into the face of a clock.
The hand of the clock is {\bf the local meridian},
the north-south line passing through the observer's zenith
(the zenith is the upward prolongation of the observer's plumb line --
right overhead!).

When the Vernal Equinox is on the local meridian, the Local Sidereal
Time (LST), or star time, is said to be 0 hours.
On the Spring equinox (around March 21), this happens at noon (solar clock).

{\bf$\rightarrow$ To remember: On the Spring equinox (around March~21), LST= 0~h at noon (solar time).}

As time passes,
astronomical objects on the local meridian will have a larger RA.
At any moment and any location on Earth,
the LST equals the RA of astronomical objects on the
local meridian.

The celestial clock moves ahead of the solar clock every day. This
is because Earth rotates both around its own axis, and also around the
Sun, as illustrated in Fig.~\ref{figlst}.
After 24~hours of solar time, a given location on Earth
finds itself at the same solar time (which is with respect to the Sun).
For instance, every day the Sun reaches its highest elevation at the same
solar time.
But relative to the stars, it finds itself on
the same position a little bit earlier (corresponding
to only one rotation around its axis). 24~hours of LST time
pass in only
23 hours 56 minutes 05 seconds of solar time. A certain LST time
occurs about 4 minutes earlier than it had the day before.

We have said that on the Spring equinox, around March 21, the LST is 0~h
at 12~h solar time (noon). The next day at noon,
the LST will be 0 h 3 min and 56 sec; inversely, 0~h of LST time
correspond to 11~h 46~m 05~s solar time.
With each passing month, a given LST time occurs 2~hours earlier.

\subsection{How can I know whether my source is up?}
\label{app:visiblecoords}

Let's imagine that we want to observe in a certain direction in the
Galactic plane (a certain Galactic longitude $l$, and a Galactic
latitude $b=0$) at a certain time.

First of all, we need to convert the Galactic coordinates into
celestial equatorial coordinates: RA and DEC. For this, we may use
the table to find $\alpha$ and $\delta$.

As we have shown above, some sources will never rise above the horizon in Onsala
(the ones with $\delta < -33^\circ$).

It is best to study examples
to learn how to find out when a source is visible.

\bigskip
{\bf Example~1.} I have been allocated time to observe with the radio telescope
on May~5, starting at 15~hours local time in Onsala.
This corresponds to 13~hours solar time.
What is the corresponding LST time? What part of the Galaxy can I observe?

\smallskip
On March 21, LST=0~h at noon.

$\Rightarrow$ LST= 1~h at 13~h.

May~5 occurs 1.5 months after March~21. The LST time shifts by
24 hours per year, or 2 hours per month. So, 1.5~months after
March~21, LST=1~h will occur at $=13 -(1.5\times 2) = 10$~h.
And at 13~h, LST= 4~h. This means that sources will $\alpha=4$~h
will cross their meridian (be highest above the horizon)
at that time.

Looking at the table, we see that $l\simeq 150^\circ$ corresponds to $\alpha\simeq 4$~h.
Since that Galactic longitude doesn't reach a very high elevation in Onsala, it is a
good idea to start by observing it because it will be setting quickly.

\bigskip
{\bf Example~2.} Today is Christmas Eve (December 24), and I have been offered a small optical
telescope. I would like to use it to observe the beautiful Whirlpool galaxy M~51.
Is it possible?

\smallskip
The coordinates of M~51 are $\alpha\simeq13$~h~30m, $\delta\simeq+47^\circ$.
This means that M~51 will be at its highest elevation above the horizon at
LST=13~h~30.

LST= 0~h at noon = 12~h around March~21.

LST= 0~h at $12-(2\times 9) = -6$~h (or $24-6=18$~h) around Dec. 21 (9 months after March~21,
since the LST time shifts ahead by 2 hours every month).

LST=13~h~30 at 18+13~h~30= 7~h~30.

M~51 will be highest up on the sky in the morning at 7~h30, and it will be rising during the
second part of the night.

\begin{table}
\label{tabcoord}
\begin{center}
\begin{tabular}{rrr}
\hline
\medskip
$l$ & $\alpha(J2000)$ &$\delta(J2000)$\\
$^\circ$ &h m &$^\circ\,\,\,\,\,$$'$\\
\hline
0 & 17h45 & --28:56 \\
20 & 18h27 & --11:29 \\
40 & 19h04 & 06:17 \\
60 & 19h43 & 23:53 \\
{\bf\green 80}  & 20h35 & 40:39\\
{\bf\green 100} & 22h00 & 55:02\\
{\bf\green 120} & 00h25 & 62:43 \\
{\bf\green 140} & 03h07 & 58:17 \\
{\bf\green 160} & 04h46 & 45:14\\
180 & 05h45 & 28:56\\
200 & 06h27 & 11:29\\
220 & 07h04 & --06:17\\
240 & 07h43 & --23:53\\
\red 260 & 08h35 & --40:39\\
\red 280 & 10h00 & --55:02\\
\red 300 & 12h25 & --62:43\\
\red 320 & 15h07 & --58:17\\
\red 340 & 16h46 & --45:14\\
\hline
\end{tabular}
\caption{Conversion from Galactic coordinates to right ascension and
  declination for different values of $l$, with $b=0$.  The Galactic
  longitudes in bold face (and green) are circumpolar at the latitude
  of Onsala ($\delta > 33^\circ$).  The Galactic longitudes in red are
  always below the horizon at the latitude of Onsala.  }
\end{center}
\end{table}