\documentclass{article}

% TODO: please download neurips_2020.sty
% from https://media.neurips.cc/Conferences/NeurIPS2020/Styles/neurips_2020.sty
% before building this document
\usepackage[preprint]{neurips_2020}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{graphicx}
\usepackage{amsthm}
\usepackage{amsmath}
\usepackage{bbm}
\usepackage{subfigure}

\newtheorem{theorem}{Theorem}

\title{Spectral clustering}

% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.

\author{%
  Davide Riva \\
  \texttt{driva95@protonmail.com}
}

\begin{document}

\maketitle

\begin{abstract}
    The term "spectral clustering" refers to a series of unsupervised techniques to perform community detection in graphs using eigenvalues and eigenvectors.
    Even though there are some review articles about it, they usually have a lack of empirical analyses linked to the theoretical proofs.
    In this document, we are going to cover that part through experiments.
\end{abstract}

\section{Introduction}
Spectral clustering identifies communities inside a graph.
In Section \ref{section:twocom} we analyse how to extract communities in a noiseless scenario
in which there is only one connected component, identified as the only one community in the graph.
Then, Section \ref{section:multiplecom} considers multiple connected components and
Section \ref{section:noisecom} extends the previous ideas to a graph in which
the number of clusters is greater than the number of connected components.
Finally, Section \ref{section:nograph} explains how to use these techniques with data
which is not structured as a graph.

Note that every time we use the term "graph" we refer to undirected, weighted graphs
to avoid useless redundancy. All weights must be positive.

The experiments are written in Python in the form of Jupyter Notebooks
and they are provided together with this document as appendices.
The name of the used libraries and their version are also supplied through a \path{requirements.txt} file.

\section{Graphs with 1 connected component} \label{section:twocom}
We start by looking at a graph with a single connected component, as the one in Figure \ref{figure:plotga}.
We denote its weight matrix as $W \in \mathbb{R}^{n \times n}$.

\begin{figure}[hb]
    \hfill
    \subfigure[A graph with one connected component.]{\includegraphics[width=0.5\linewidth]{figures/one-component.eps}}%
    \hfill
    \subfigure[A graph with three connected components.]{\includegraphics[width=0.5\linewidth]{figures/multiple-component.eps}}%
    \hfill
    \caption{Examples of graphs with various connected components.}%
    \label{figure:plotga}%
\end{figure}

The degree matrix $D$ of a graph is defined as a diagonal matrix whose diagonal elements are:
\begin{equation*}
    D_{ii} = \sum_{j=1}^n W_{ij}
\end{equation*}

The goal is to obtain an indicator vector $f = (f_1, \dots, f_n)$ which can identify which
node belongs to which connected component.
To do so, we build a matrix that we call Laplacian matrix:
\begin{equation*}
    L = D - W
\end{equation*}

\begin{theorem}
    Minimizing $\frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n W_{ij} (f_i - f_j)$
    is equivalent to minimizing $f^T L f$.
\end{theorem}

\begin{proof}
    \begin{align*}
        f^T L f & = f^T D f - f^T W f                                                                                \\
                & = \sum_{i=1}^n D_{ii} f_i^2 - \sum_{i=1}^n \sum_{j=1}^n f_i f_j W_{ij}                                              \\
                & = \frac{1}{2} ( 2 \sum_{i=1}^n D_{ii} f_i^2 - 2 \sum_{i=1}^n \sum_{j=1}^n f_i f_j W_{ij})                           \\
                & = \frac{1}{2} ( \sum_{i=1}^n D_{ii} f_i^2 + \sum_{j=1}^n D_{jj} f_j^2 - 2 \sum_{i=1}^n \sum_{j=1}^n f_i f_j W_{ij}) \\
                & = \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n W_{ij} (f_i - f_j)^2 \tag*{\qedhere}
    \end{align*}
\end{proof}

We also know that $L$ is symmetric by definition. Furthermore, it is positive semi-definite since from the previous theorem
$f^T L f \geq 0 \; \forall f \in \mathbb{R}^n$.

It follows that the smallest eigenvalue of $L$ is 0. Its corresponding eigenvector is the vector $a [1, 1, \dots, 1], a \in \mathit{R}$
due to the fact that the sum of the elements of each row or column is always zero.
Note that if the graph has one connected component, the eigenvector corresponding to the eigenvalue with value 0 is the global minimum of the
minimization problem and a perfect solution for our task.

We can observe in an empirical way what we have just proved.
First, we generate a pre-defined number of graphs with one connected component.
In particular, each graph is a Watts–Strogatz model \cite{wc} which has the desired propriety of having a non-small clustering coefficient.
Then, we produce a box plot of the values of each eigenvalue sorted in ascending order for each graph.
The resulting plot can be seen in Figure \ref{figure:oneeigen}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\linewidth]{figures/eigen-one-component.eps}
    \caption{Box plot of the eigenvalues of 100 graphs with 20 nodes and one connected component, sorted in ascending order for each graph.}
    \label{figure:oneeigen}
\end{figure}

\section{Graphs with multiple connected components} \label{section:multiplecom}
We now want to identify which node belongs to which connected component
when the graph has the number of connected components $k \geq 2$
as the one in Figure \ref{figure:plotga}.

First, we rearrange the matrix $L$ grouping together nodes in the same connected component.
In this way, $L$ is a block diagonal matrix with subgraphs $L_1, \dots, L_k$.
Now, each subgraph has one and only one connected component
and therefore it has one eigenvalue that is equal to zero,
as proven in Section \ref{section:twocom}.

We also know from linear algebra that the set of eigenvalues of $L$
is composed of the eigenvalues of $L_1, \dots, L_k$.
Also, each corresponding eigenvector in $L_i$ is a valid eigenvector in $L$ with zero padding
in the other block positions.
It follows that the matrix $L$ has $k$ eigenvalues with value equals to zero and
one combination of eigenvalues valid in that eigenspace is composed of indicator vectors.
This propriety can be seen empirically in Figure \ref{figure:mcomp},
where each graph is again a Watts–Strogatz model.

\begin{figure}[hb]
    \hfill
    \subfigure[Graphs with two connected components.]{\includegraphics[width=0.3\linewidth]{figures/2-components.eps}}
    \hfill
    \subfigure[Graphs with four connected components.]{\includegraphics[width=0.3\linewidth]{figures/4-components.eps}}
    \hfill
    \subfigure[Graphs with eight connected components.]{\includegraphics[width=0.3\linewidth]{figures/8-components.eps}}
    \hfill
    \label{figure:mcomp}
    \caption{Box plot of the first 10 eigenvalues of 100 graphs with 64 nodes and different connected components, sorted in ascending order for each graph.}
\end{figure}

Since in this scenario the eigenspace of the eigenvalues with value zero
is a linear combination of indicator vectors, we use them to identify the connected components.
In particular, we build a matrix $V \in \mathit{R}^{n \times k}$ where each column corresponds to
an eigenvector of the previous eigenspace.
Finally, K-means is performed over the matrix $V$ to group data,
since groups in $V$ are linearly separable.

In order to see if the eigenvectors described before are good indicator vectors,
we build a graph with three connected components:
the first cluster corresponds to the first three nodes,
the second cluster is composed of the last two and the third has all the remaining ones.
Each connected component of the graph is a complete graph itself.
Figure \ref{figure:eivects} represents this configuration graphically.

\begin{figure}[hb]
    \hfill
    \subfigure{\includegraphics[width=0.3\linewidth]{figures/0-eigenvectors.eps}}%
    \hfill
    \subfigure{\includegraphics[width=0.3\linewidth]{figures/1-eigenvectors.eps}}%
    \hfill
    \subfigure{\includegraphics[width=0.3\linewidth]{figures/2-eigenvectors.eps}}%
    \hfill
    \caption{Example of eigenvectors used as indicator vectors.}%
    \label{figure:eivects}%
\end{figure}

\section{Considerations about noise} \label{section:noisecom}
Until now we considered the situation in which each detected community
corresponds to a connected component of the graph.

If we instead assume that each cluster is a tightly coupled group of nodes
with a low number of inter-community edges, we can treat these edges as noise added
to the situations described in the previous sections.
Given the noiseless adjacency matrix $A$,
we can view the edges presented before as a perturbation $H$ that produces the final
matrix $W = A + H$.

Thanks to the Davis–Kahan theorem \cite{davis1970rotation} we can state that small perturbations
in the Laplacian matrix lead to small perturbations in its eigenvectors and eigenvalues.
This means that if the noise (in our case, the addition of unwanted edges) is small,
it is still possible to perform spectral clustering.
In particular, we choose the number of communities looking at each eigengap:
small eigengaps are considered produced by noise,
while the first eigengap greater than the previous ones is assumed to be the dividing
indicator between eigenvectors used as indicator vectors and other eigenvectors.
Using the eigengap as a criterion to decide the number of communities in a graph
can be justified in other ways, well documented in \cite{DBLP:journals/corr/abs-0711-0189}.

To empirically verify the quality of using the eigengap to identify the number of clusters,
we start from a graph composed of three connected components which are complete graphs themselves and
we add iteratively more edges until all nodes are connected to all the others.
As we can see from Figure \ref{figure:addingnoise}:
\begin{itemize}
    \item The smallest eigenvalue is zero all the time since there is always at least one connected component in a graph.
    \item The first three eigenvalues in ascending order are initially zero when there is no noise and then they grow when edges are added.
    \item The eigengap between the third and the fourth eigenvalue is bigger than the others until there is too much noise.
\end{itemize}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\linewidth]{figures/adding-noise.eps}
    \caption{Example of the effect of noise in the eigenvalues intensities in a graph with 3 connected components.}
    \label{figure:addingnoise}
\end{figure}

\section{Spectral clustering when data is not a graph} \label{section:nograph}
We have previously seen how to use spectral clustering to identify communities in graphs.
In this section, we extend this concept to a set of data points $x_1, x_2, \dots, x_n$.

First, we introduce the family of similarity functions $s(x_i, x_j) \geq 0$ which can be used to build an adjacency matrix.
$s(x_i, x_j)$ can be defined in a lot of ways, for instance:
\begin{itemize}
    \item $s(x_i, x_j) = e^{ - \frac{||x_i - x_j||^2}{2\sigma^2}}$ (Gaussian similarity function): it produces a graph in which all nodes are connected with each other, but everyone with different weights.
    \item $s(x_i, x_j) = ||x_i - x_j||_2 < \epsilon$: it produces an unweighted graph.
    \item $s(x_i, x_j) = \frac{1}{\sqrt{2}} \sqrt{\sum_k (\sqrt{x_i(k)} - \sqrt{x_j(k)})^2}$ (Hellinger distance \cite{PPN243919689_0136}): useful when each data point is a discrete probability distribution.
\end{itemize}

Then, we perform spectral clustering using the adjacency matrix built before.
As previously mentioned, thanks to this technique it is possible to perform clustering in situations in which
clusters are not linearly separable.

To compare it with K-means, we generate some synthetic data.
During this experiment, spectral clustering uses $s(x_i, x_j) = ||x_i - x_j||_2 < \epsilon$ as a similarity function.
Figure \ref{figure:comparison} shows the dataset generated and the best results obtained with these two methods.
Clearly, spectral clustering outperforms K-means.

\begin{figure}[ht]
    \hfill
    \subfigure[Ground truth.]{\includegraphics[width=0.3\linewidth]{figures/ground-truth.eps}}%
    \hfill
    \subfigure[Clustering using K-means.]{\includegraphics[width=0.3\linewidth]{figures/kmeans.eps}}%
    \hfill
    \subfigure[Clustering using spectral clustering.]{\includegraphics[width=0.3\linewidth]{figures/spectral-on-comparison.eps}}%
    \hfill
    \caption{Comparison between K-means and spectral clustering.}%
    \label{figure:comparison}%
\end{figure}

So far we have manually tweaked the hyperparameter $\epsilon$ and the number of clusters to find the best solution for each dataset.
We now ask ourselves if there is a way to automatically choose the right value for them.
To do so, we introduce the modularity quality function (presented in \cite{fortunato2010community}):
\begin{equation*}
    Q = \frac{1}{2m} \sum_{i=1}^n \sum_{j=1}^n (A_{ij} - \frac{k_i k_j}{2m}) \mathbbm{1}_{[C_i == C_j]}
\end{equation*}
where $m$ is the number of edges of the unweighted graph for which the modularity is computed,
$A$ is the adjacency matrix,
$k_i$ is the degree of the node $i$
and $C_i$ the index of the connected component of the node $i$.
This function measures how different the considered unweighted graph is from a graph with the same degree sequence but with
edges connected randomly.  High values of modularity correspond to the presence of clusters;
vice versa low modularity suggests a graph without sharp partitions.

Since binarizing the graph produces multiple connected components if the threshold is chosen through modularity,
we hypothesize is that it might help spectral clustering.
To empirically test our idea, we use the "Mall customers" dataset.\footnote{\url{https://www.kaggle.com/kandij/mall-customers}}
We then compute the modularity with 100 different thresholds and we pick the largest threshold between the ones with high modularity.
Looking at the eigenvalues near zero, we identify 57 clusters.
We then perform spectral clustering considering true the number of clusters found.
The previous steps are then compared with spectral clustering using the Gaussian similarity function manually tweaking the number of clusters
and the $\sigma$ parameter. As we can see from Figure \ref{figure:realworld},
manually tweaking the hyperparameters outperforms our results.
Looking at the literature, \cite{DBLP:journals/corr/abs-0711-0189} founds no theoretical justification in different methods proposed to overcome this issue.
Since the cited review article is from 2007, we searched in the literature but without finding anything useful from a theoretical point of view.

\begin{figure}[hb]
    \hfill
    \subfigure[Raw data.]{\includegraphics[width=0.3\linewidth]{figures/mall-raw.png}}%
    \hfill
    \subfigure[Spectral clustering with modularity.]{\includegraphics[width=0.3\linewidth]{figures/mall-spectral.png}}%
    \hfill
    \subfigure[Spectral clustering.]{\includegraphics[width=0.3\linewidth]{figures/mall-manual.png}}%
    \hfill
    \caption{Comparison between spectral clustering using the Gaussian similarity function and spectral clustering using modularity first.}%
    \label{figure:realworld}%
\end{figure}

\pagebreak
\nocite{*}
\bibliography{references}
\bibliographystyle{plain}

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{multicol}
\usepackage[formats]{listings}
\lstloadaspects{formats}
\usepackage{verbatim}
\usepackage{color}
\usepackage{geometry}
\usepackage{float}
\usepackage{amsmath}
\usepackage{multirow}% http://ctan.org/pkg/multirow
\usepackage{hhline}% http://ctan.org/pkg/hhline
\usepackage{caption}
\usepackage{pdflscape}
\usepackage{caption}
\usepackage[font={color=black},figurename=Fig.,labelfont={it}]{caption}
\usepackage{hyperref}
\setlength{\belowcaptionskip}{-10pt}
\setlength{\abovecaptionskip}{-30pt}
\floatstyle{boxed}
\restylefloat{figure}
\usepackage{graphicx}
\usepackage{subcaption}
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
\lstdefinestyle{mystyle}{
	backgroundcolor=\color{backcolour},
	commentstyle=\color{codegreen},
	keywordstyle=\color{blue},
	numberstyle=\tiny\color{codegray},
	stringstyle=\color{codepurple},
	basicstyle=\footnotesize,
	breakatwhitespace=false,
	breaklines=true,
	captionpos=b,
	keepspaces=true,
	numbers=left,
	numbersep=5pt,
	showspaces=false,
	showstringspaces=false,
	showtabs=false,
	tabsize=2
}
\lstset{style=mystyle}
\title{Data Mining\\
		Home work 12\\Clustering, PCA, OLAP}
\author{Aqeel Labash\\ \textbf{Lecturer:} Jaak Vilo}
\geometry{
	a4paper,
	total={170mm,257mm},
	left=10mm,
	top=5mm,
}
\begin{document}
	\maketitle
\section*{First Question}
For this question I tried K-means. I used the following code :
\begin{lstlisting}[language=Python]
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
with open ('DM2016_org.txt') as f:
    d = {}
    headers = f.readline().split(' ')
    values = map(lambda x:x.split(),f.readlines())
    for i in range(len(values[0])):
        d[i]=[]
        for v in values:
            d[i].append(v[i])
data = pd.DataFrame(d)
npdata = data[data.columns[1:]].as_matrix()
npdata = np.array(npdata,dtype=int)
random_state = 170
nc = 339
y_pred = KMeans(n_clusters=nc, random_state=random_state,precompute_distances=True).fit_predict(npdata)
output=[]
for i in range(nc):
    print i
    map(lambda x:output.append(x),map(lambda x:data[data.index[0]][x].values,np.where(y_pred==i))[0])
f = open('Kmeans_399.txt','w')
for element in output:
    f.write('\n{}'.format(element))
f.close()
\end{lstlisting}
I tried 3 values as number of clusters 2,10,30,399. The results were as following:
\begin{figure}[H]
\begin{subfigure}{.5\textwidth}
  \centering
\includegraphics[scale=0.15]{kmeans_2.png}
  \caption{KMeans 2 clusters}
  \label{fig:sfig1}
\end{subfigure}
\begin{subfigure}{.5\textwidth}
  \centering
\includegraphics[scale=0.15]{kmeans_10.png}
  \caption{KMeans 10 clusters}
  \label{fig:sfig2}
\end{subfigure}

\begin{subfigure}{.5\textwidth}
\vspace{5mm}
  \centering
\includegraphics[scale=0.15]{kmeans_30.png}
  \caption{KMeans 30 clusters}
  \label{fig:sfig3}
  \vspace{5mm}
\end{subfigure}
\begin{subfigure}{.5\textwidth}
\vspace{5mm}
  \centering
\includegraphics[scale=0.15]{kmeans_399.png}
  \caption{KMeans 30 clusters}
  \label{fig:sfig4}
  \vspace{5mm}
\end{subfigure}
\caption{Shows Comparsion between different number of clusters using K-Means}
\end{figure}
Only getting a cluster values won't work because in that case we only know the cluster elements.Which won't let us know the order of the cluster. That information is crucial in our case.\\
After that I tried hierarchical clustering using Scipy library. The full size dendogram was as following :
\newgeometry{
 	a4paper,
 	total={170mm,257mm},
 	left=0mm,
 	top=10mm,
 	right=0mm,
 	bottom=0mm
 }
\begin{landscape}

\begin{figure}[H]
\includegraphics[scale=0.7,trim={4cm 1cm 5cm 2cm},clip]{dendogram_full.png}
\caption{Full dendogram}
\end{figure}

\begin{figure}[H]
\includegraphics[scale=0.69,trim={4cm 0cm 5cm 2cm},clip]{dendogram_truncated_clustersize.png}
\caption{Truncated Dendogram}
\end{figure}
\end{landscape}
\newgeometry{
	a4paper,
	total={170mm,257mm},
	left=10mm,
	top=5mm,
}
\section*{Second Question}
For this question I read the file in python and extract the data as matrix and then used sklearn for PCA and used matplotlib for 3d plotting. Here is the code :
\begin{lstlisting}[language=Python]
import numpy as np
from sklearn.decomposition import PCA
import pandas as pd
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
with open ('DM2016_org.csv') as f:
    d = {}
    headers = f.readline().split(' ')
    values = map(lambda x:x.split(),f.readlines())
    for i in range(len(values)):
        d[headers[i]]=[]
        for v in values:
            d[headers[i]].append(v[i])
data = pd.DataFrame(d)
npdata = data[data.columns[1:]].as_matrix()
pca = PCA(n_components=3)
pca.fit(npdata)
output = pca.transform(npdata)
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
n = 100
for x, y, z in output:
    ax.scatter(x, y, z)

ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')

plt.show()
\end{lstlisting}
And here is the output figure:
\begin{figure}[H]
\includegraphics[scale=0.5,trim={8cm 2cm 10cm 5cm},clip]{3D.png}
\caption{X: PCA1 , Y:PCA2 , Z:PCA3}
\end{figure}
\section*{Third Question}
For this question I used the following code to find pivot table on basis of Gender and education :
\begin{lstlisting}[language=Python]
import pandas as pd
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
data = pd.read_csv('extract_medium.csv',sep=';')
data.head(1)
table = pd.pivot_table(data,values='Earnings',index=['Sex', 'Education'],aggfunc=np.mean)
##OR###
temp1 = data.groupby(['Sex', 'Education']).Earnings.mean()
temp1
temp1 = temp1.values
test=temp1.reshape(2,17)
test.shape
plt.pcolor(test,cmap=plt.cm.Reds,vmin=np.min(test), vmax=np.max(test))
plt.yticks([0,1],range(3))
plt.xticks(range(17),range(17))
plt.title('Heat map of average Earnings Gender Vs Education')
plt.show()
plt.close()
\end{lstlisting}
And here is the pivot table :
\begin{table}[H]
  \centering
  \begin{tabular}{|c|c|c|}
  \hline
    Sex&Education&Average earning\\ \hline
    \multirow{16}{*}{Male}&Not in universe (Under 3 years)&0.000000\\ \hhline{~--}
	&No schooling completed&3504.784689\\ \hhline{~--}
	&Nursery school to 4th grade&690.859232\\ \hhline{~--}
	&5th or 6th grade&5083.671988\\ \hhline{~--}
	&7th or 8th grade&4073.961606\\ \hhline{~--}
	&9th grade&9596.498516\\ \hhline{~--}
	&10th grade&11185.848485\\ \hhline{~--}
	&11th grade&11167.638889\\ \hhline{~--}
	&12th grade,no diploma&19404.356436\\ \hhline{~--}
	&High school graduate&24012.275826\\ \hhline{~--}
	&college,less than 1 year&28201.210614\\ \hhline{~--}
	&college 1+ years, no degree&28488.347335\\ \hhline{~--}
	&Associate degree&35081.001821\\ \hhline{~--}
	&Bachelor,s degree&53294.264282\\ \hhline{~--}
	&Master.s degree&70755.173611\\ \hhline{~--}
	&Professional degree&94245.204918\\ \hhline{~--}
	&Doctorate degree&61467.676768 \\\hline
    \multirow{16}{*}{Female}&Not in universe (Under 3 years)&0.000000\\ \hhline{~--}
	 &No schooling completed&947.790323\\ \hhline{~--}
	 &Nursery school to 4th grade&387.342995\\ \hhline{~--}
	 &5th grade or 6th grade&2771.055195\\ \hhline{~--}
	 &7th or 8th grade&2256.137405\\ \hhline{~--}
	 &9th grade&3326.657609\\ \hhline{~--}
	 &10th grade&4281.338798\\ \hhline{~--}
	 &11th grade&5003.164557\\ \hhline{~--}
	 &12th grade,no diploma&9200.885781\\ \hhline{~--}
	 &High school graduate&11515.832571\\ \hhline{~--}
	 &college,less than 1 year&16305.045514\\ \hhline{~--}
	 &college 1+ years, no degree&16949.317489\\ \hhline{~--}
	 &Associate degree&21991.480263\\ \hhline{~--}
	 &Bachelor,s degree&33193.713496\\ \hhline{~--}
	 &Master.s degree&44412.231834\\ \hhline{~--}
	 &Professional degree&46821.961290\\ \hhline{~--}
	 &Doctorate degree&41476.065574 \\ \hline
  \end{tabular}
  \label{T:peak}
\end{table}
And here is the heatmap :
\begin{figure}[H]
\includegraphics[scale=0.42,trim={3cm 0cm 7cm 2cm},clip]{hmGEE.png}
\caption{Heatmap gender vs education in earnings}
\end{figure}
It's really cool. Have profissinoal degree or master is better than have a doctoral. I think that's because most doctoral degree work in education which is not the highest salaries.
\section*{Fourth Question}
For this question I will show the relation between marital status , gender and earnings seemed interesting :) and here is the code :
\begin{lstlisting}[language=python]
table = pd.pivot_table(data,values='Earnings',index=['Sex', 'Marriage'],aggfunc=np.mean)
table = table.values
test=table.reshape(2,5)
fig = plt.figure()
ax = fig.add_subplot(111)
ax.pcolor(test,cmap=plt.cm.Reds,vmin=np.min(test), vmax=np.max(test))
ax.set_yticks([1,2])
ax.set_yticklabels(Genders)
ax.set_xticks(range(6))
ax.set_xticklabels(MarriageState)
for tick in ax.get_xticklabels():
    tick.set_rotation(45)
plt.gcf().subplots_adjust(bottom=0.20)
ax.set_title('Heat map of average Earnings Gender Vs Mariage')
plt.show()
plt.close()
\end{lstlisting}
The resulted heat map :
\begin{figure}[H]
\includegraphics[scale=0.4,trim={4cm 2.5cm 0cm 2cm},clip]{hmGME.png}
\caption{Heatmap gender vs marriage status in earnings}
\end{figure}
depending on this data if you want to increase your earning just get married :D but don't kill your partner that will decrease your income dramatically.
\section*{Fifth Question}
I did read most of the article but the operation were quite more understandable and clear on the wikipedia web page.
For this task I'll aggregate (Sex,Marriage,Hours) as (X,Y,Z) please notice that Y we can convert it to days -> weeks depending on the value we have.
The operations are :
\begin{enumerate}
\item \textbf{Slicing}: by this we can take one males or females.
\item \textbf{Dicing}: by this we can take group of hours (or group of days if we generalize this value).
\item \textbf{Drill-up and drill-down}:let's say Hours is genralized to days and we can generalize it to weeks and currently we are using days.At this point drill down will go to Hours for specific values and Drill-up will generalize to weeks.
\item \textbf{Pivoting}: is about changing axes like changing axes between sexes and hours.
\end{enumerate}
To calculate the data cube I used the following code \\
\textbf{Note:}The following code depending on the previous code.
\begin{lstlisting}[language=Python]
table = pd.pivot_table(data,values='Earnings',index=['Sex', 'Marriage','Hours'],aggfunc=np.mean)
\end{lstlisting}
The previous code will calculate the required data pivot but I could not find a good way to show interactive plot.

I hope I understood the question.
\section*{Sixth Question}
For this question I used python as usual and used google TSP to increase the speed and performance. (even though it's not a straight forward solution). Here is the code :
\begin{lstlisting}[language=Python]
__author__ = 'Modified by :aqeel'
import pandas as pd
import numpy as np
import RandomMatrix
import ShortestPathFinder
# Copyright 2010-2014 Google
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Traveling Salesman Sample.
   This is a sample using the routing library python wrapper to solve a
   Traveling Salesman Problem.
   The description of the problem can be found here:
   http://en.wikipedia.org/wiki/Travelling_salesman_problem.
   The optimization engine uses local search to improve solutions, first
   solutions being generated using a cheapest addition heuristic.
   Optionally one can randomly forbid a set of random connections between nodes
   (forbidden arcs).
"""
import random
import gflags
from ortools.constraint_solver import pywrapcp
# function to Calculate the distance which can be replaced with any Type of Distance
# ex. Google API to Calculate the distance between two GPS Coordinations
def distance(p1, p2):
    dist = np.linalg.norm(p1-p2)
    return dist

class RandomMatrix(object):
    """Random matrix."""

    def __init__(self,points):
        self.lst = points
        """Initialize random matrix."""
        self.matrix = {}
        for from_node in xrange(len(self.lst)):
            self.matrix[from_node] = {}
            for to_node in xrange(len(self.lst)):
                if from_node == to_node:
                    self.matrix[from_node][to_node] = 0
                else:
                    self.matrix[from_node][to_node] = distance(self.lst[from_node], self.lst[to_node])
    def Distance(self, from_node, to_node):
        return self.matrix[from_node][to_node]
with open ('DM2016_org.csv') as f:
    d = {}
    headers = f.readline().split(' ')
    values = map(lambda x:x.split(),f.readlines())
    for i in range(len(values[0])):
        d[i]=[]
        for v in values:
            d[i].append(v[i])
data = pd.DataFrame(d)
npdata = data[data.columns[1:]].as_matrix()
npdata = np.array(npdata,dtype=int)
class ShortestPathFinder:
    def __init__(self, pointslist):
        reload(gflags)
        self.FLAGS = gflags.FLAGS
        self.points = pointslist

        gflags.DEFINE_integer('tsp_size', len(pointslist),
                              'Size of Traveling Salesman Problem instance.')
        gflags.DEFINE_boolean('tsp_use_random_matrix', True,
                              'Use random cost matrix.')
        gflags.DEFINE_integer('tsp_random_forbidden_connections', 0,
                              'Number of random forbidden connections.')
        gflags.DEFINE_integer('tsp_random_seed', 0, 'Random seed.')
        gflags.DEFINE_boolean('light_propagation', False, 'Use light propagation')

    def FindApproximateShortestWay(self):
        # Create routing model
        if self.FLAGS.tsp_size > 0:
            # Set a global parameter.
            param = pywrapcp.RoutingParameters()
            param.use_light_propagation = self.FLAGS.light_propagation
            pywrapcp.RoutingModel.SetGlobalParameters(param)

            # TSP of size FLAGS.tsp_size
            # Second argument = 1 to build a single tour (it's a TSP).
            # Nodes are indexed from 0 to FLAGS_tsp_size - 1, by default the start of
            # the route is node 0.
            routing = pywrapcp.RoutingModel(self.FLAGS.tsp_size, 1)

            parameters = pywrapcp.RoutingSearchParameters()
            # Setting first solution heuristic (cheapest addition).
            parameters.first_solution = 'PathCheapestArc'
            # Disabling Large Neighborhood Search, comment out to activate it.
            parameters.no_lns = True
            parameters.no_tsp = False

            # Setting the cost function.
            # Put a callback to the distance accessor here. The callback takes two
            # arguments (the from and to node inidices) and returns the distance between
            # these nodes.
            matrix = RandomMatrix(self.points)
            matrix_callback = matrix.Distance
            if self.FLAGS.tsp_use_random_matrix:
                routing.SetArcCostEvaluatorOfAllVehicles(matrix_callback)
            else:
                routing.SetArcCostEvaluatorOfAllVehicles(RandomMatrix.distance)
            # Forbid node connections (randomly).
            rand = random.Random()
            rand.seed(self.FLAGS.tsp_random_seed)
            forbidden_connections = 0
            while forbidden_connections < self.FLAGS.tsp_random_forbidden_connections:
                from_node = rand.randrange(self.FLAGS.tsp_size - 1)
                to_node = rand.randrange(self.FLAGS.tsp_size - 1) + 1
                if routing.NextVar(from_node).Contains(to_node):
                    print 'Forbidding connection ' + str(from_node) + ' -> ' + str(to_node)
                    routing.NextVar(from_node).RemoveValue(to_node)
                    forbidden_connections += 1

            # Solve, returns a solution if any.
            assignment = routing.SolveWithParameters(parameters, None)
            if assignment:
                # Solution cost.
                cost =  assignment.ObjectiveValue()
                # Inspect solution.
                # Only one route here; otherwise iterate from 0 to routing.vehicles() - 1
                route_number = 0
                node = routing.Start(route_number)
                result =[]
                while not routing.IsEnd(node):
                    result.append(self.points[int(node)])
                    node = assignment.Value(routing.NextVar(node))
                result.append(self.points[0])
                return cost,result
            else:
                print 'No solution found.'
        else:
            print 'Specify an instance greater than 0.'
resut = ShortestPathFinder(npdata)
totaldistance,plan = resut.FindApproximateShortestWay()
lst=[]
i=0
for p in plan:
    lst.append(np.argmax(np.sum(npdata==p,axis=1)))
    i+=1
    if i%100==0:
        print i
output = map(lambda x:data[data.index[0]][x],lst)
f = open('output.txt','w')
for element in output:
    f.write('\n{}'.format(element))
f.close()
\end{lstlisting}
The output Image was really close :
\begin{figure}[H]
\includegraphics[scale=0.3]{TSP_OUTPUT.png}
\caption{Shows the scrumbled image after applying TSP}
\end{figure}
The problem in Figure 4 is deciding the first row. To decide which is the best cut I just searched for maximum distance between two neighbours and take the first one as the start.Here is the code that should be added the the previous one to fix the problem:
\begin{lstlisting}[language=Python]
#Find the Best Cut
maxvalue=0
maxindex=-1
for i in range(len(plan)-1):
    dist = np.linalg.norm(plan[i]-plan[i+1])
    if (dist>maxvalue):
        maxvalue = dist
        maxindex=i

lst2=output[maxindex:] + output[:maxindex]
\end{lstlisting}
\begin{figure}[H]
\includegraphics[scale=0.3]{TSP_OUTPUT_Perfect.png}
\caption{Image after optimizing the start point}
\end{figure}
Just for fun I applied the same software in the second image and I got the following result:
\begin{figure}[H]
\includegraphics[scale=0.4]{TSP_OUTPUT2.png}
\caption{Image 2 after recovering using TSP with optimized start point}
\end{figure}
\section*{Seventh Question}

For this question I used mysql.And I used the following steps to achieve the pivot table using sql
\begin{enumerate}
\item Created table in the database using the following code:
\begin{lstlisting}[language=SQL]
CREATE TABLE `census` (`State` varchar(50) DEFAULT NULL,  `House_id` varchar(50) DEFAULT NULL,  `Weight` varchar(50) DEFAULT NULL,  `House_relation` varchar(50) DEFAULT NULL,  `Sex` varchar(50) DEFAULT NULL,  `Age` varchar(50) DEFAULT NULL,  `Race` varchar(50) DEFAULT NULL,  `Marriage` varchar(50) DEFAULT NULL,  `Education` varchar(50) DEFAULT NULL,  `Ancestry` varchar(50) DEFAULT NULL,  `Language` varchar(50) DEFAULT NULL,  `Employment_status` varchar(50) DEFAULT NULL,  `Traveltime` varchar(50) DEFAULT NULL,  `Industry` varchar(50) DEFAULT NULL,  `Occupation` varchar(50) DEFAULT NULL,  `Hours` varchar(50) DEFAULT NULL,  `Weeks` varchar(50) DEFAULT NULL,  `Salary` varchar(50) DEFAULT NULL,  `Income` varchar(50) DEFAULT NULL,  `Earnings` int(20) DEFAULT NULL)
\end{lstlisting}
\item After that I loaded the data into the table using the following code :
\begin{lstlisting}[language=SQL]
#Show the directory that mysql can accept files from
SHOW VARIABLES LIKE "secure_file_priv";
LOAD DATA INFILE '/var/lib/mysql-files/extract_large.csv'
INTO TABLE census
FIELDS TERMINATED BY ';'
\end{lstlisting}

\item At the end, I used the following code to generate the pivot table :
\begin{lstlisting}[language=SQL]
SELECT Sex,Education,avg(Earnings)
FROM census
group by Sex,Education
\end{lstlisting}
Which clearly find the average earning for each set of (Sex,Education)
\end{enumerate}
Here is the output table :\\
\begin{tabular}{|c|c|c|}
\hline
Sex&Education&Avg(Earnings)\\ \hline
1&00&0.0000\\ \hline
1&01&3436.9163\\ \hline
1&02&1018.5143\\ \hline
1&03&5719.2300\\ \hline
1&04&4585.9244\\ \hline
1&05&7892.1493\\ \hline
1&06&8702.8604\\ \hline
1&07&10341.6579\\ \hline
1&08&16704.6764\\ \hline
1&09&22399.7355\\ \hline
1&10&26831.3657\\ \hline
1&11&29900.6087\\ \hline
1&12&35249.8174\\ \hline
1&13&52814.0881\\ \hline
1&14&63902.4224\\ \hline
1&15&97160.4380\\ \hline
1&16&69790.5412\\ \hline
2&00&0.0000\\ \hline
2&01&1358.9230\\ \hline
2&02&386.2795\\ \hline
2&03&1996.0910\\ \hline
2&04&1656.8459\\ \hline
2&05&3048.7202\\ \hline
2&06&3878.9171\\ \hline
2&07&4711.3237\\ \hline
2&08&7794.4182\\ \hline
2&09&10865.3802\\ \hline
2&10&15102.8411\\ \hline
2&11&16310.1780\\ \hline
2&12&20708.8457\\ \hline
2&13&27804.7783\\ \hline
2&14&35058.9334\\ \hline
2&15&44529.5096\\ \hline
2&16&44325.0832\\ \hline
\end{tabular} \\ \\

\textbf{Note:}All .py,.ipython,.tex,.pdf etc.. exist on\href{https://github.com/aqeel13932/DM/tree/master/HW12}{github}
\begin{thebibliography}{9}
\bibitem{1}
\href{https://joernhees.de/blog/2015/08/26/scipy-hierarchical-clustering-and-dendrogram-tutorial/}{SciPy Hierarchical Clustering and Dendrogram Tutorial}
\end{thebibliography}
\begin{center}
\textbf{E.O.F}
\end{center}

\end{document}


% technical background material:
\section{Background}
\label{sec:background}

% define CHC's:
\subsection{Constrained Horn Clauses}
\label{sec:chcs}

\subsubsection{Structure}
% definition of CHC
A Constrained Horn Clause is a logical implication where the
antecedent is called the body and the consequent is called the head.
%
The body is a conjunction of a logical formula, called the constraint,
and a vector of uninterpreted predicates. The constraint is an arbitrary
formula in some background logic, such as linear integer arithmetic.
%
The uninterpreted predicates are applied to variables which may or may
not appear in the constraint.
%
A head can be either an uninterpreted predicate applied to variables or $False$.
%
A clause where the head is $False$ is called a query. A CHC can be
defined structurally:
\begin{align*}
\cc{chc} \Coloneqq&~\cc{head} \gets \cc{body} \\
\cc{head} \Coloneqq&~False \\
  \mid&~\cc{pred} \\
\cc{body} \Coloneqq&~\varphi \wedge \cc{preds} \\
\cc{preds} \Coloneqq&~True \\
  \mid&~\cc{pred} \wedge \cc{preds} \\
\cc{pred} \Coloneqq&~\textsf{\emph{an uninterpreted predicate applied to variables}} \\
\varphi \Coloneqq&~\textsf{\emph{a formula}} \\
\end{align*}
%
% define a body function, head function, and recursion-free
For a given CHC $C$, $\bodyof{C}$ denotes the vector of uninterpreted
predicates in the body and $\consof{C}$ denotes the constraint in the
body.
%
If $C$ is not a query, then $\headof{C}$ denotes the uninterpreted
predicate in the head.
%
%define a CHC system
A CHC system is a set of CHCs where exactly one clause is a query.
%
For a given CHC system $S$, $\predof{S}$ denotes the set of all
uninterpreted predicates and $\cc{query}$ denotes the body of the
query clause.

To explain the structure of a CDD system, we need terminology that relates
predicates in a CHC system including the terms \emph{predicate dependency},
\emph{transitive predicate dependency}, and \emph{sibling}.
% define dependency predicates
\begin{defn}
  Given a CHC system $\cc{S}$ and two uninterpreted predicates
  $\cc{P}$ and $\cc{Q} \in \predof{S}$, if $\exists \cc{C} \in \cc{S}$
  such that $\cc{P} = \headof{C}$ and $\cc{Q} \in \bodyof{C}$, then
  $\cc{Q}$ is a \emph{predicate dependency} of $\cc{P}$.
\end{defn}
%
\begin{ex}
  In $\mcchc$, because $\cc{L}_4$ is in the body of clause (4) and
  $\cc{L}_8$ is the head of clause (4), $\cc{L}_4$ is a predicate
  dependency of $\cc{L}_8$.
\end{ex}
%
Given a CHC system $\cc{S}$ and an uninterpreted predicate $\cc{P}$,
$\depsof{P}$ denotes the set of all predicate dependencies of $\cc{P}$
in $\cc{S}$.
%
%define transitive dependency predicates
\begin{defn}
  Given a CHC system $\cc{S}$ and three uninterpreted predicates
  $\cc{P}, \cc{Q}$, and $\cc{R} \in \predof{S}$, if $\cc{Q} \in
  \depsof{P}$ then \cc{Q} is a \emph{transitive predicate dependency} of
  \cc{P}.
  %
  If \cc{Q} is a transitive predicate dependency of \cc{P} and \cc{R}
  is a transitive predicate dependency of \cc{Q}, then $\cc{R}$ is a
  \emph{transitive predicate dependency} of $\cc{P}$.
\end{defn}
%
\begin{ex}
  In $\mcchc$, because $\cc{L}_4$ is a predicate dependency of
  $\cc{L}_8$, $\cc{L}_4$ is a transitive predicate dependency of
  $\cc{L}_8$.
  %
  And because $\cc{L}_8$ is an transitive predicate dependency of
  $\cc{L}_9$, $\cc{L}_4$ is a transitive predicate dependency of
  $\cc{L}_9$.
\end{ex}
%
Given a CHC system $\cc{S}$ and an uninterpreted predicate $\cc{P}$,
$\tdepsof{P}$ denotes the set of all transitive predicate dependencies
of $\cc{P}$ in $\cc{S}$.
%
%define siblings
\begin{defn}
  Given a CHC system $\cc{S}$ and two uninterpreted predicates
  $\cc{P}$ and $\cc{Q} \in \predof{S}$, if $\exists \cc{C} \in \cc{S}$
  such that $\cc{P} \in \bodyof{C}$ and $\cc{Q} \in \bodyof{C}$, then
  $\cc{Q}$ and $\cc{P}$ are siblings.
\end{defn}
%
\begin{ex}
  Because uninterpreted predicates $\cc{L}_9$ and $\cc{dbl}$ both
  appear in the body of clause (7), $\cc{L}_9$ and $\cc{dbl}$ are
  siblings.
  %
\end{ex}
%
%
For a given CHC system $\cc{S}$, if there is no uninterpreted
predicate $\cc{P} \in \predof{S}$ such that $\cc{P} \in \tdepsof{P}$,
then $\cc{S}$ is a \emph{recursion-free} CHCs system.
%

A solution to a CHC system $S$ is a map from each predicate $\cc{P}
\in \predof{S}$ to its corresponding interpretation which is a
formula.
%
For a solution to be valid, each clause in $S$ must be valid after
substituting each predicate by its interpretation.

% interpolants:
\subsection{Logical interpolation}
\label{sec:itps}
%
All logical objects in this paper are defined over a fixed space of first-order
variables, $X$.
%
For a theory $T$, the space of $T$
formulas over $X$ is denoted $\tformulas{T}$.
%
For each formula $\varphi \in \tformulas{T}$, the set of free
variables that occur in $\varphi$ (i.e., the \emph{vocabulary} of
$\varphi$) is denoted $\vocab(\varphi)$.
%
% define models, satisfaction, entailment
For formulas $\varphi_0, \ldots, \varphi_n, \varphi \in
\tformulas{T}$, the fact that $\varphi_0, \ldots, \varphi_n$
\emph{entail} $\varphi$ is denoted $\varphi_0, \ldots, \varphi_n
\entails \varphi$.

% introduce interpolation:
An interpolant of a pair of mutually inconsistent formulas
$\varphi_0$ and $\varphi_1$ in $\tformulas{T}$ is a formula $I$ in $\tformulas{T}$ over
their common vocabulary that explains their inconsistency.
%
\begin{defn}
  \label{defn:itps}
  % define:
  For $\varphi_0, \varphi_1, I \in \tformulas{T}$, if
  %
  \textbf{(1)} $\varphi_0 \entails I$, %
  \textbf{(2)} $I \land \varphi_1 \entails \false$, and %
  \textbf{(3)} $\vocab(I) \subseteq \vocab(\varphi_0) \intersection
  \vocab(\varphi_1)$,
  %
  then $I$ is an \emph{interpolant} of $\varphi_0$ and $\varphi_1$.
\end{defn}
%
For the remainder of this paper, all spaces of formulas will be
defined for a fixed, arbitrary theory $T$ that supports
interpolation, such as the theory of linear
arithmetic.
% introduce decision procedure:
Although determining the satisfiability of formulas in such theories
is NP-complete in general, decision procedures~\cite{moura08} and
interpolating theorem provers~\cite{mcmillan04} for such theories have
been proposed that operate on such formulas efficiently.
%

%
We define \sys in terms of an abstract interpolating theorem
prover for $T$ named $\solveitp$.
%
Given two formulas $\varphi_0$ and $\varphi_1$, if $\varphi_0$ and $\varphi_1$ are mutually
inconsistent, $\solveitp$ returns the interpolant of $\varphi_0$ and $\varphi_1$.
%
Otherwise, $\solveitp$ returns $\none$.


\documentclass[12pt]{cdblatex}
\usepackage{examples}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% ============================================================================================
\bgroup
\CdbSetup{action=hide}
\begin{cadabra}
   import cdblib
   checkpoint_file = 'tests/semantic/output/example-04.json'
   cdblib.create (checkpoint_file)
   checkpoint = []
\end{cadabra}
\egroup

\clearpage

% ============================================================================================
\section*{Example 4 Python functions}

\begin{cadabra}
   {a,b,c,d,e,f,h,i,j,k,l,m,n,o,p,q,r,s,t,u#}::Indices.

   def truncate (poly,n):

       # define the weight and give it a label
       x^{a}::Weight(label=\epsilon).

       # start with an empty espression
       ans = Ex("0")

       # loop over selected terms in the source
       for i in range (0,n+1):

          foo := @(poly).
          bah  = Ex("\epsilon = " + str(i))

          # extract a single term
          keep_weight (foo, bah)

          # update the running sum
          ans = ans + foo

       # all done, return final answer
       return ans

   Quartic := c^{a}
            + c^{a}_{b} x^b
            + c^{a}_{b c} x^b x^c
            + c^{a}_{b c d} x^b x^c x^d
            + c^{a}_{b c d e} x^b x^c x^d x^e.   # cdb (ex-04.100,Quartic)

   Cubic = truncate (Quartic,3)                  # cdb (ex-04.101,Cubic)

   checkpoint.append (Cubic)
\end{cadabra}

\begin{align*}
   p(x) &= \Cdb{ex-04.100}\\
   q(x) &= \Cdb{ex-04.101}
\end{align*}

\clearpage

% ============================================================================================
% export to json format

\bgroup
\CdbSetup{action=hide}
\begin{cadabra}
   for i in range( len(checkpoint) ):
      cdblib.put ('check{:03d}'.format(i),checkpoint[i],checkpoint_file)
\end{cadabra}
\egroup

\end{document}


\section{Lists}\label{Lists}

This library includes the following files:

\begin{itemize}

\item {\tt List.v} contains definitions of (polymorphic) lists,
  functions on lists such as head, tail, map, append and prove some
  properties of these functions. Implicit arguments are used in this
  library, so you should read the Reference Manual about implicit
  arguments before using it.

\item {\tt ListSet.v} contains definitions and properties of finite
  sets, implemented as lists.

\item {\tt TheoryList.v} contains complementary results on lists. Here
  a more theoretic point of view is assumed : one extracts functions
  from propositions, rather than defining functions and then prove them.

\item {\tt Streams.v} defines the type of infinite lists (streams). It is a
  coinductive type. Basic facts are stated and proved. The streams are
  also polymorphic.

\end{itemize}


\documentclass{article}
\usepackage{amsmath,amsthm,amsfonts,microtype,amssymb}
\usepackage{enumitem}
\usepackage{parskip}
\usepackage{mathpazo}
\usepackage{tikz-cd}
\usepackage{geometry}
    \geometry{top=1in}

\usepackage{hyperref}
    \hypersetup{%
        colorlinks=true,
        linkcolor=red,
        filecolor=red,
        urlcolor=red,
        bookmarks=true,
        pdfpagemode=FullScreen,
    }

\title{Homework 02 \\ Basepoints and CW structures}
\author{Algebraic Topology - Winter 2021}
\date{Due: \textbf{January 28, 2021, 11:59 pm}}

\begin{document}
\pagenumbering{gobble}
\maketitle

\begin{enumerate}
    \item Let $(X,x)$ and $(Y,y)$ be pointed topological spaces. Prove that
        \begin{align*}
            \pi_1(X \times Y, (x,y)) \cong \pi_1(X,x) \times \pi_1(Y,y).
        \end{align*}

    \item  For a topological space $X$, define the \emph{fundamental groupoid}, denoted $\Pi_1(X)$, to be the category whose objects are points in $X$ and whose morphisms are path-homotopy classes of paths i.e. for $x,y \in X$ define
    \begin{align*}
        \Pi_1(x,y) := \left\{ [\gamma] \mid \gamma: [0,1] \to X, \partial_0 \gamma = x, \partial_1 \gamma = y \right\}.
    \end{align*}
    \begin{enumerate}
        \item A groupoid is a category in which every morphism is invertible.
        Show that $\Pi_1(X)$ is a groupoid.
    \end{enumerate}
    Let $\mathbf{Cat}$ denote the category whose objects are categories and whose morphisms are functors.
    \begin{enumerate}[resume]
        \item Show that $\Pi_1$ naturally extends to a functor $\Pi_1: \mathbf{Top} \to \mathbf{Cat}$.
    \end{enumerate}
    Let $x$ be a point in $X$. We can think of the fundamental ``group'', $\pi_1(X,x)$, as the full subcategory of $\Pi_1(X)$ with exactly one object $x$.
    \begin{enumerate}[resume]
        \item Show that if $X$ is path-connected, then the categories $\Pi_1(X)$ and $\pi_1(X,x)$ are equivalent.
    \end{enumerate}
    Thus, the fundamental groupoid is a basepoint-independent variant of the fundamental group.
\end{enumerate}
For the following questions, you do not have to provide completely rigorous proofs. It's enough to draw sufficiently descriptive pictures.
\begin{enumerate}[resume]
    \item
    Let $n$ be a positive integer.
        \begin{enumerate}
            \item Construct a CW structure for $S^1$ with $n$ 0-cells and $n$ 1-cells.
            \item Construct a CW structure for $S^n$ with only two cells total.
            \item Construct a CW structure for $S^3$ two 0-cells, two 1-cells, two 2-cells, and two 3-cells. (Optional: Can you generalize this to $S^{n}$?)
            \item Construct a CW complex with one 0-cell, one 1-cell, one 2-cell, and one 3-cell that is homotopy equivalent to $S^3$. (Optional: Can you generalize this to $S^{2n-1}$?)
            \item Let $v, e, f$ be positive integers with $v - e + f = 2$. Construct a CW structure for $S^2$ having $v$ 0-cells, $e$ 1-cells, and $f$ 2-cells.
        \end{enumerate}

    \item
    Let $X$ be a topological space that can be given a CW structure with a finite number of cells. Let $n_i$ denote the number of cells in dimension $i$ in this CW structure. The \emph{Euler characteristic} of $X$ is defined as
    \begin{align*}
        \chi(X) := \sum_{i \in \mathbb{N}} (-1)^i n_i.
    \end{align*}
    Assume that the Euler characteristic is well-defined i.e. does not depend on the choice of the CW structure. Further assume that the Euler characteristic is a homotopy invariant.

    \begin{enumerate}
        \item Compute the Euler characteristic of spheres and genus $g$ surfaces.
        \item Does there exist a CW complex with one 0-cell, one 1-cell, one 2-cell, \dots, one $2n$-cell that is homotopy equivalent to $S^{2n}$?
        \item What can you say about homeomorphisms between the various spheres and genus $g$ surfaces, using just the Euler characteristic?
    \end{enumerate}

\end{enumerate}

\section*{Suggested exercises for practice from Hatcher}

\begin{description}
    \item[Pg. 18] 2
    \item[Pg. 19] 9, 15, 16
    \item[Pg. 38] 1, 2, 3
    \item[Pg. 39] 10, 11, 15, 17
\end{description}

\end{document}


\documentclass{article}

\usepackage[latin1]{inputenc}
\usepackage{color}
\usepackage{listings}
\usepackage[english]{babel}
\usepackage{graphicx}
\usepackage{indentfirst}
\usepackage{amsmath,amsfonts,graphicx}

%\usepackage{fontspec}
%\usepackage[utf8]{inputenc}
%\setmainfont{Futura}[ItalicFont={Futura Italic}]

\definecolor{colKeys}{rgb}{0,0,1}
\definecolor{colIdentifier}{rgb}{0,0,0}
\definecolor{colComments}{rgb}{0,0.5,1}
\definecolor{colString}{rgb}{0.6,0.1,0.1}

\parskip 5pt plus 2pt minus 2pt
\textwidth=17.5cm
\oddsidemargin=-0.5cm
\evensidemargin=-0.5cm
\topmargin=-0.25cm
\headheight=0cm
\headsep=1cm
\textheight=23cm
\parindent=0cm

\begin{document}
\thispagestyle{empty}
\begin{center}
\fbox{\large\bf Frequency Response Techniques in Feedback Control Systems}
\end{center}
\bigskip

Let's assume $u(t)$, $y(t)$, and $G(t)$ represents the input, output,
and transfer function representation of an input-output continuous time
system.

In order to characterize frequency response of a dynamical system,
the test signal is
%
\begin{align*}
 u(t) = e^{j \omega t}
\end{align*}
%
which is an artificial complex periodic signal with a
frequency of $\omega$. If we assume that the system is ``stable'' or system is a part of
closed loop system and closed loop behavior is stable then
at steady state we have
%
\begin{align*}
y_{ss}(t) &= G(j \omega) e^{j \omega t} \\
&= | G(j \omega) | e^{i \omega t + \angle [ G(j \omega) ] }
\\
&= M e^{i \omega t + \theta}
\end{align*}
%
In other words complex periodic signal is scaled and phase shifted
based on the following operators
%
\begin{align*}
M &= | G(j \omega) | \\
\theta &= \angle G(j \omega)
\end{align*}
%
It is very easy to show that for a general real time domain signal
$u(t) = \sin (\omega t + \phi)$, the output $y(t)$ at steady state
is computed via
%
\begin{align*}
  y_{ss}(t) = M \sin(\omega t + \phi + \theta)
\end{align*}

\section*{2. Plotting Frequency Response: Polar Plot}

We can consider the frequency response function $G(j \omega)$
as a mapping from positive $j \omega$ axis to a curve in
the complex plane. In polar plot, we simply draw the frequency response
function starting from $\omega = 0$ (or $\omega \to 0^+$) to $\omega
\to \infty$ on the complex plane.

\textbf{Ex:} Let's draw the polar plots of
%
\begin{align*}
 G_1(s) &= s
 \quad , \quad
 G_2(s) = \frac{1}{s}
 \quad , \quad
 G_3(s) = s + 2
\\
 G_4(s) &= 2 + \frac{1}{s}
\quad , \quad
 G_5(s) = 2 + s +  \frac{1}{s}
\end{align*}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.99\textwidth]{figs/polar}
    \end{center}
  \end{minipage}

\newpage

\textbf{Ex:} Draw the polar plots of
%
\begin{align*}
 G_1(s) &= \frac{1}{s+1}
 \quad , \quad
 G_2(s) = \frac{s}{s+1}
\end{align*}

Let's analyze $G_1(j \omega)$ for $\omega \in [0 , \infty)$
%
\begin{align*}
 G_1(j \omega) &= \frac{1}{j \omega +1} = \frac{1 - j \omega}{\omega^2 +1}
= \frac{1}{\omega^2 +1} - \frac{\omega}{\omega^2 +1} j
\\
| G_1(j \omega) | &= \frac{1}{ \sqrt{1 + \omega^2} }
\\
\angle [ G_1(j \omega) ] &= \arctan (-\omega)
\end{align*}

Now let's analyze $G_2(j \omega)$ for $\omega \in [0 , \infty)$
%
\begin{align*}
 G_2(j \omega) &= \frac{j \omega}{j \omega +1} = \frac{j \omega + \omega^2}{\omega^2 +1}
= \frac{\omega^2}{\omega^2 +1} + \frac{\omega}{\omega^2 +1} j
\\
| G_2(j \omega) | &= \sqrt{ \frac{ \omega^2 }{ 1 + \omega^2 } }
\\
\angle [ G_2(j \omega) ] &= \arctan (1 / \omega)
\end{align*}

Polar plots of $G_1(s)$ and $G_2(s)$ are illustrated below.

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.8\textwidth]{figs/polar2}
    \end{center}
  \end{minipage}

\vspace{6 pt}

\textbf{Ex:} Draw the polar plot of  $G(s) = \frac{1}{(s+1)^2}$
%
\begin{align*}
  G(j \omega ) &= \frac{1}{ (j \omega + 1)^2 } = \frac{ (-j \omega + 1)^2 }{( \omega^2 +1 )^2 }
\\
&= \left[ \left( 1 - \omega^2 \right) + j ( - 2 \omega) \right] \frac{1}{( \omega^2 +1 )^2 }
\end{align*}
%
Some important points and associated features on the polar plot can be computed as
\begin{align*}
  \omega &\to 0 \ \Rightarrow G(j \omega) = 1
\\
 \omega &\to 1 \ \Rightarrow G(j \omega) = -0.5 j
\\
 \omega &\to  \infty \ \Rightarrow | G(j \omega) | \to 0 \quad \& \quad \angle  [ G(j \omega) ] \to -\pi
\end{align*}

Resultant polar plot is illustrated below

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.45\textwidth]{figs/polar4}
    \end{center}
  \end{minipage}

\vspace{6 pt}

\textbf{Ex:} Draw the polar plot of  $G(s) = \frac{1}{(s+1)^3}$
%
\begin{align*}
  G(j \omega ) &= \frac{1}{ (j \omega + 1)^3 } = \frac{ (-j \omega + 1)^3 }{( \omega^2 +1 )^3 }
\\
&= \left[ \left( 1 - 3 \omega^2 \right) + j (\omega^3 - 3 \omega) \right] \frac{1}{( \omega^2 +1 )^3 }
\end{align*}
%
Some important points and associated features on the polar plot can be computed as
\begin{align*}
  \omega &\to 0 \ \Rightarrow G(j \omega) = 1
\\
 \omega &\to \sqrt{1/3} \ \Rightarrow G(j \omega) = -0.65 j
\\
 \omega &\to \sqrt{3} \ \Rightarrow G(j \omega) = -1/8
\\
 \omega &\to  \infty \ \Rightarrow | G(j \omega) | \to 0 \quad \& \quad \angle  [ G(j \omega) ] \to \pi/2
\end{align*}

Resultant polar plot is illustrated below

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.6\textwidth]{figs/polar5}
    \end{center}
  \end{minipage}

\vspace{6 pt}

\section*{2. Nyquist Contour \& Nyquist Plot}

Nyquist plot is another tool that we use to to investigate
the stability and robustness of a feedback system. The technique
utilizes the frequency response characteristics of a system.

\textbf{Definition:} A contour $\Gamma_s$ is a closed path with a direction
in a complex plane.

\textbf{Remark:} A continuous function $F(s)$ maps a contour
$\Gamma_s$ in $s-$plane to another contour $\Gamma_{F(s)}$
in $F(s)$ plane.

The figure below illustrates a clock-wise contour
$\Gamma_s$ and its map $\Gamma_{F(s)}$ which is also clock-wise
in this example.

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.75\textwidth]{figs/cmap}
    \end{center}
  \end{minipage}

\vspace{6 pt}

Let's consider an LTI transfer function $G(s)$ that
has \textit{no zeros or poles on the imaginary axis}. Nyquist contour/path,
$\Gamma_s$, is defined in a way that it covers the
whole open-right half plane (i.e. whole unstable region).

As illustrated in the Figure below,
Nyquist contour is technically a half-circle for which the radius, $R
\to \infty$. After that, one can draw the Nyquist plot, which is the
mapped contour $\Gamma_{G(s)}$. Figure below illustrates a Nyquist
contour and associated Nyquist plot.

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.65\textwidth]{figs/nyq}
    \end{center}
  \end{minipage}

\newpage

\textbf{Ex:} Let's draw the Nyquist plot of $G(s) = \frac{1}{s+1}$

\textbf{Solution:}  Based on the Nyquist contour we have three major
paths.
%
\begin{enumerate}
  \item This part corresponds to the polar plot that we already solved previously. We previously
  plotted $G(j \omega)$, where $\omega : 0 \to \infty$ on the complex plane.
%
  \item This is the mapping of the infinite radius circular path on
    Nyquist contour. In this case if we write $s$ in polar form, we get
   $s = R e^{j \theta}$ where $\theta : \pi/2 \to -\pi/2$.  Then
   we can derive that
   \begin{align*}
     & G \left( R e^{j \theta} \right)  \approx \frac{1}{R e^{j
       \theta}} = \frac{e^{j (-\theta)}}{R}
       \quad  \Rightarrow \quad | G \left( R e^{j \theta} \right) | \approx 0
   \end{align*}
   %
   \item Last path (mapping of negative imaginary axis) is simply
    the conjugate of polar plot with reverse direction.
\end{enumerate}

If we follow the procedure, we obtain the following Nyquist plot.

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.7\textwidth]{figs/ex2}
    \end{center}
  \end{minipage}

\vspace{6 pt}

\textbf{Ex:} Let's draw the Nyquist plot of
$G(s) = \frac{1}{(s+1)^2}$

\textbf{Solution:} Let's analyze the Nyquist paths
%
\begin{enumerate}
  \item Mapping of first path corresponds to the polar plot that we covered in the previously.
%
  \item Mapping of the infinite radius circular path. Let  $s = R e^{j \theta}$ where $\theta : \pi/2 \to -\pi/2$.  Then
   we can derive that
   \begin{align*}
     & G \left( R e^{j \theta} \right)  \approx \frac{1}{R^2 e^{j
       2 \theta}} = \frac{e^{j (-2 \theta)}}{R^2}
       \quad \quad
    \Rightarrow | G \left( R e^{j \theta} \right) | \approx 0
   \end{align*}
   %
   \item Last path (mapping of negative imaginaty axis) is again
   the conjugate of polar plot with reverse direction.
\end{enumerate}

If we follow the procedure, we obtain the following Nyquist plot.

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.65\textwidth]{figs/ex3}
    \end{center}
  \end{minipage}

\vspace{12 pt}

\textbf{Ex:} Draw the Nyquist plot of $G(s) =
\frac{1}{(s+1)^3}$

\textbf{Solution:} First let's analyze the Nyquist paths
%
\begin{enumerate}

  \item This is the polar plot that we have solved previously.
    %
  \item Mapping of the infinite radius circular path on
    Nyquist contour. Let $s = R e^{j \theta}$ and $\theta : \pi/2 \to -\pi/2$.  Then
   we can derive that
   \begin{align*}
     & G \left( R e^{j \theta} \right) \approx \frac{1}{R^3 e^{j
       3 \theta}} = \frac{e^{j (-3 \theta)}}{R^3}
	\quad
    \Rightarrow  \quad | G \left( R e^{j \theta} \right) | \approx 0
   \end{align*}
   %
   \item Last path (mapping of negative imaginary axis) is again
   the conjugate of polar plot with reverse direction.
\end{enumerate}

If we follow the procedure, we obtain the following Nyquist plot.

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.75\textwidth]{figs/ex4}
    \end{center}
  \end{minipage}

\section*{3. Nyquist Stability Criterion for Feedback Systems}

In control theory and its applications Nyqist plot and Nyquist stability test are dominantly used for
analyzing feedback topologies. The figure below illustrates the fundamental feedback system topology for a SISO system

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.5\textwidth]{figs/unityfeedback}
    \end{center}
  \end{minipage}

\vspace{6 pt}

We know that the closed-loop transfer function, $T(s)$, for this
system has the following form
%
\begin{align*}
  T(s) = \frac{G(s)}{1 + G(s) H(s)} = \frac{G(s)}{1 + G_{OL}(s)}
\end{align*}
%
where $G_{OL}(s)$ is the open-loop transfer function for the given
topology. We know that poles of $T(s)$ are the roots that satisfy
$1 + G_{OL}(s) = 0$.

In order to characterize and ``quantify'' the stability of the whole
closed system, we will utilize the Nyquist plot of the open-loop
transfer function, $G_{OL}(s)$.

In this part (frequency response analysis) of the course, we will
limit to a specific (but very important and general) class of systems
and assume that
%
\begin{itemize}
  \item $G_{OL}(s)$ is a \textit{minimum-phase} system, i.e.
    \begin{itemize}
      \item No poles/zeros in the Open Right Half Plane
      \item $\lim_{\omega \to \infty} [ \frac{G_{OL}(s)}{s} ]_{s = \j \omega} = 0 $
    \end{itemize}
  \item The feed-back system is Type $0-2$ (i.e. no integrator of order larger than 3 in the open-loop transfer function).
    \item Polar plot of $G_{OL}(j \omega)$ crosses the negative real-axis
      at most once.
\end{itemize}
%

Under these assumptions, Nyquist stability criterion is reduced to the
following definition

\vspace{3pt}

\textbf{Def:} A closed-loop system is BIBO stable, if the Nyquist plot
of the open-loop transfer function neither encircle nor intersect the point $(-1
+ 0 j )$ point on the complex plane.

\vspace{6pt}

\textbf{Def:} T(s) is BIBO stable, if the Nyquist plot of $G_{OL}(s)$ does not encircle $\mathbf{(-1 + 0 j )}$

\vspace{6pt}

\vspace{3pt}

Indeed, it very easy to understand why the system becomes BIBO
unstable, when the Nyquist plot intersect the $(-1 + 0 j )$ point. If
Nyquist plot intersects $(-1 + 0 j )$, it means that $\exists \;
\omega \in \mathbb{R}$ such that $G(j \omega) = -1$ and $1 + G(j \omega) = 0$,
which further implies that the closed-loop system has at least one
pole on the imaginary axis.

\newpage

\textbf{Ex:} Analyze the stability of the following feedback system
using Nyquist plot.

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.5\textwidth]{figs/ex1block}
    \end{center}
  \end{minipage}

\vspace{6 pt}

\textbf{Solution:} For this given system $G_{OL}(s) = \frac{1}{s+1}$.
In the previous lecture we already derived the Nyquist plot for $\frac{1}{s+1}$
which is illustrated below

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.7\textwidth]{figs/ex2}
    \end{center}
  \end{minipage}

\vspace{3 pt}

It is clear that Nyquist plot of $G_{OL}(s)$ neither encircles nor
intersect the $(-1 + 0 j)$ point, thus the closed-loop system is
BIBO stable.

\vspace{6 pt}

\textbf{Ex:} Find the range of positive $K$ values that makes the following
closed-loop system stable.

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.45\textwidth]{figs/ex2block}
    \end{center}
  \end{minipage}

\vspace{6 pt}

\textbf{Soln:} We would like to test the stability of the closed-loop system
for different values of $K$, moreover we would like to derive
the range of $K > 0$ values that makes the closed-loop system
stable. Indeed, we don't need to re-draw the Nyquist plot
for each $K$ that we want to test.

Instead, we will simply analyze the
effect of a positive gain $K$, on the Nyquist plot. Let
$\Gamma_1$ be the Nyquist plot of $G_{OL}(s)$ when $K = 1$. Since $K$
is a simple positive gain it only radially scales the Nyquist plot
without affecting the phase, i.e. Nyquist plot for an arbitrary gain
$K$ will be $\Gamma_K = K \Gamma_1$. Since we are interested in only
the encirclement  of $(-1 + j)$, it would be sufficient to concentrate of the
intersection of Nyquist plot with the negative real axis.

We already derived the Nyquist plot for $\frac{1}{(s+1)^2}$.
We can easily obtain the Nyquist plot of $\frac{K}{(s+1)^2}$
by scaling. Both Nyquist plots are illustrated below.

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.6\textwidth]{figs/exK}
    \end{center}
  \end{minipage}

\vspace{6 pt}

We can see from the derived Nyquist plots that the closed-system
would be stable for all values of $K$.

\vspace{6 pt}

\textbf{Ex:} Find the range of $K$ values that makes the
following closed-loop system stable.

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.6\textwidth]{figs/ex3block}
    \end{center}
  \end{minipage}

\vspace{6 pt}

\textbf{Solution:} In the previous lecture we already derived the
Nyquist plot for $\frac{1}{(s+1)^2}$. Now let's illustrate the Nyquist
plots of both $\frac{1}{(s+1)^2}$ and $\frac{K}{(s+1)^2}$

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.8\textwidth]{figs/exK2}
    \end{center}
  \end{minipage}

\vspace{6 pt}

As we can see from the Nyquist plot pf $\frac{K}{(s+1)^3}$,
intersects the negative real axis at the point of $(-K/8 + 0 j)$.
We can easily see that when $K = 8$, Nyquist plot intersects
the $(-1 + 0 j)$ point, and when $K > 8$, Nyquist plot encircles
the $(-1 + 0 j)$. As a results, the closed loop system is BIBO
stable for $K \in (0 , 8)$.

\section*{Nyquist Plot and Stability Test with  Open-Loop Poles/Zeros on the Imaginary Axis}

If you remember, when we introduced Nyquist contour and Nyquist stability test, we assumed that there is no pole/zero on the imaginary axis (in addition to other assumptions). However, it is especially common to have poles at the origin since it corresponds to the simple integrator. In this course, we will explicitly cover the case when there exists a pole or zero at the origin.

Let's assume that $G_{OL}(s)$ has a pole (or zero, or double pole)
at the origin. We simply modify the Nyquist contour by adding an
infinitesimal notch at the origin to the original Nyquist contour.

The figure below illustrates this modified Nyquist contour and
an illustrative Nyquist plot.

\vspace{12 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.8\textwidth]{figs/originnyq}
    \end{center}
  \end{minipage}

\vspace{6 pt}

\newpage

\textbf{Ex:} Find the range of positive $K$ values that makes the
following closed-loop system stable.

\vspace{12 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.65\textwidth]{figs/ex5block}
    \end{center}
  \end{minipage}

\vspace{12 pt}

\textbf{Solution:} For this given system, when $K = 1$ open-loop
transfer function takes the form $ G_{OL}(s) = \frac{1}{s (s+1) }$. Note, there exist a pole at the origin, thus
we need to utilize the modified Nyquist Contour.
In the modified Nyquist contour there exist 4 major paths (as opposed to three paths in classical Nyquist contour),
and we need to draw the Nyquist plot based on these 4 paths.

\begin{enumerate}
  \item This is the polar plot, where we need to plot $G(j \omega)$, where $\omega : 0 \to
    \infty$.
    %
    \begin{align*}
      G(j \omega) = \frac{1}{j \omega (j \omega +1)} = \frac{ -j  - \omega }{\omega
      (\omega^2 + 1) } = \frac{-1}{\omega^2 + 1} + \frac{-1/\omega}{
      \omega^2 + 1} j
    \end{align*}
%
   Some observations about polar plot
    \begin{align*}
       Re \lbrace G(j \omega) \rbrace < 0 \quad & \&  \quad  Im \lbrace G(j
                                                \omega) \rbrace < 0 \quad , \ \forall  \omega > 0
      \\
        \lim_{\omega \to 0} Re \lbrace G(j \omega) \rbrace = - 1
       \quad & \& \quad
       \lim_{\omega \to 0} Re \lbrace G(j \omega) \rbrace = - \infty
        \\
       \lim_{\omega \to \infty} | G(j \omega) | = 0
        \quad & \& \quad
      \lim_{\omega \to \infty} \angle [ G(j \omega) ] = -\pi
      \end{align*}
 %
  \item This part is the mapping of in the infinite radius half-circle part of the Nyquist contour.
    Let $s = R e^{j \theta}$ and $\theta : \pi/2 \to -\pi/2$.  Then
   we can derive that
   \begin{align*}
     & G \left( R e^{j \theta} \right) \approx \frac{1}{R^2 e^{j
       2 \theta}} = \frac{e^{j (-2 \theta)}}{R^2}
       \\
    &\Rightarrow | G \left( R e^{j \theta} \right) | \approx 0
   \end{align*}
   %
   Note that when $\theta : \pi/2 \to -\pi/$, the infinite-small
   contour around origin rotates in CCW direction.
   %
   \item This part is simply the conjugate of polar plot with reverse
     direction.
  \item This part is new. We need to deal with this path   since there is a pole at the origin.
  	Let's write $s$ in polar coordinates
     $s = \epsilon e^{j \phi}$, where $\epsilon \to 0$ and $\phi :
     -\pi/2 \to \pi/2$ (CCW direction).  Then  we can derive that
   \begin{align*}
     & G \left( \epsilon e^{j \phi} \right) \approx \frac{1}{\epsilon e^{j
        \phi}} = R e^{j (-\phi)}
       \\
    &\Rightarrow | G \left(\epsilon e^{j \phi} \right) | \approx
     R \ll 1
   \quad , \quad \angle [ G \left( \epsilon e^{j \phi} \right) ] \approx -\phi
   \end{align*}
\end{enumerate}

As a result, we obtain the following Nyquist plot for
$ \frac{1}{s (s+1)}$

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.75\textwidth]{figs/ex5}
    \end{center}
  \end{minipage}

\vspace{6 pt}

Since gain $K$ only scales the Nyquist plot, we can clearly see that
Nyquist plot never circulates $\mathbf{-1}$ point for all possible values of $K > 0$.
Thus, the system is stable $\forall \; K > 0$.

\vspace{12 pt}

\textbf{Ex:} Draw the Nyquist plot of the open-loop transfer function
for the following feedback topology and apply Nyquist stability test

\vspace{12 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.5\textwidth]{figs/block_doubleint}
    \end{center}
  \end{minipage}

\vspace{12 pt}

\textbf{Solution:} For this given system, open-loop
transfer function has the form $ G_{OL}(s) = \frac{s+1}{s^2}$. Note, there exist a repeated pole at the origin, thus
we need to utilize the modified Nyquist Contour.
In the modified Nyquist contour there exist 4 major paths,
and we need to draw the Nyquist plot based on these 4 paths.

\begin{enumerate}
  \item This is the polar plot, where we need to plot $G(j \omega)$, where $\omega : 0 \to
    \infty$.
    %
    \begin{align*}
      G(j \omega) = \frac{j \omega + 1}{ (j \omega)^2} =  -\frac{1}{ \omega^2} - \frac{1}{ \omega} j
    \end{align*}
%
   One can easily see that
   %
    \begin{align*}
       Re \lbrace G(j \omega) \rbrace < 0 \quad & \&  \quad  Im \lbrace G(j
                                                \omega) \rbrace < 0 \quad , \ \forall  \omega > 0
     \end{align*}
   %
   This implies that polar plot is only located in the third quadrant of the complex plane. Now let's analyze the behavior of the
   polar plot for infinitely small $\omega$, i.e. $\omega = \frac{1}{R}$, where $R \to \infty$
   %
    \begin{align*}
    	Re \lbrace G(j \omega) \rbrace = -R^2  \quad & \Rightarrow \quad  \lim_{R \to \infty} Re \lbrace G(j \omega) \rbrace = -\infty
	\\
	Im \lbrace G(j \omega) \rbrace = -R  \quad & \Rightarrow \quad  \lim_{R \to \infty} Im \lbrace G(j \omega) \rbrace = -\infty
      \end{align*}
      %
      As once can see both Real and Imaginary parts goes to $-\infty$ when $\omega \to 0$. However real part goes faster due to
      square term. Thus the phase of the polar when $\omega \to 0$ can be find as
      %
	\begin{align*}
      		\lim_{\omega \to 0} \frac{Im \lbrace G(j \omega) \rbrace}{Re \lbrace G(j \omega) \rbrace} = 0
		\quad & \Rightarrow \quad  \lim_{\omega \to 0}  \angle [ G(j \omega) ] = \arctan( 0 ) -\pi = -\pi
	\end{align*}
 %
   Now let's analyze the behavior of the polar plot for infinitely large $\omega$, i.e. $\omega = R$, where $R \to \infty$
   %
    \begin{align*}
    	Re \lbrace G(j \omega) \rbrace = -\frac{1}{R^2}  \quad & \Rightarrow \quad  \lim_{R \to \infty} Re \lbrace G(j \omega) \rbrace = -0
	\\
	Im \lbrace G(j \omega) \rbrace = -\frac{1}{R}   \quad & \Rightarrow \quad  \lim_{R \to \infty} Im \lbrace G(j \omega) \rbrace = -0
      \end{align*}
      %
         As once can see both the Real and Imaginary parts goes to $0$ when $\omega \to \infty$. However real part goes faster due to
      square term. Thus the phase of the polar when $\omega \to \infty$ can be find as
     %
	\begin{align*}
      		\lim_{\omega \to 0} \frac{Im \lbrace G(j \omega) \rbrace}{Re \lbrace G(j \omega) \rbrace} = \infty
		\quad & \Rightarrow \quad  \lim_{\omega \to 0}  \angle [ G(j \omega) ] = \arctan( \infty ) -\pi = -\frac{\pi}{2}
	\end{align*}
    %

  \item This part is the mapping of in the infinite radius half-circle part of the Nyquist contour.
    Let $s = R e^{j \theta}$ and $\theta : \pi/2 \to -\pi/2$.  Then
   we can derive that
   \begin{align*}
     & G \left( R e^{j \theta} \right) \approx \frac{R e^{j \theta}}{R^2 e^{j 2 \theta}} = \frac{e^{j (-2 \theta)}}{R^2}
    \quad
    \Rightarrow \quad | G \left( R e^{j \theta} \right) | \approx 0
   \end{align*}
   %
   Note that whole path is mapped to a single ``point'' at the origin.
   %
   \item This part is simply the conjugate of polar plot with reverse
     direction.
  \item Let's write $s$ in polar coordinates
     $s = \epsilon e^{j \phi}$, where $\epsilon \to 0$ and $\phi :
     -\pi/2 \to \pi/2$ (CCW direction).  Then  we can derive that
   \begin{align*}
     & G \left( \epsilon e^{j \phi} \right) \approx \frac{1}{\epsilon^2 e^{j 2 \phi}} = R^2 e^{j (-2 \phi)}
       \\
    &\Rightarrow | G \left(\epsilon e^{j \phi} \right) | \approx
     R \to \infty
   \quad , \quad \angle [ G \left( \epsilon e^{j \phi} \right) ] \approx -2 \phi
   \end{align*}
\end{enumerate}

As a result, the mapping of this contour is an infinitely large almost full circle in CW direction.

If we combine all four paths the Nyquist plot of $ \frac{s+1}{s^2}$ takes the following form

\vspace{6 pt}

  \begin{minipage}[h]{1\linewidth}
    \begin{center}
      \includegraphics[width=0.95\textwidth]{figs/doubleint}
    \end{center}
  \end{minipage}

\vspace{6 pt}

We can see that Nyquist plot never circulates $\mathbf{-1}$ point.
Thus, the closed-loop system is BIBO stable.

\end{document}


%!TEX root =  ../main.tex

\subsection{Pecking Order}

\objective{Build and decompose a composition of functions and their derivatives.}

In elementary school, we learned that a mathematics sentence has a
certain order of operations, that somethings must happen before others.
If we want to add before we multiply, we must notate parentheses,
because normally multiplication is repeated addition, something of a higher
order than simple addition.  These parentheses lead to a phenomenon
in mathematics that is difficult and unnatural, compared to all other
languages: inside to outside reading.

For example, in the function $f(x)=\frac{2(x^2+1)}{3}$, you know to follow a
sequence of operations on any given input:
\begin{itemize}
\item square it
\item add 1
\item multiply by two
\item divide by three
\end{itemize}
Natural language does something like this, but it is quite confusing to read
(try saying it aloud to yourself): I saw the child the man my girl hates fathered.
It might be better to make this entirely left-branching: I saw the child who
the man fathered who my girl hates!

Our example utilizes only elementary functions: squaring, adding, multiplying,
and dividing.  If we want to use arbitrary functions, we must write
$f(g(h(x)))$ or $(f\circ g\circ h)(x)$.

\subsection{Domain and Range}
Taking a function as a composition of two other (e.g. $f(x)=g(h(x))$), we observe
a ``chain'' of inputs to outputs.  This might be compared to the game of telephone,
where your ear receives a message from the person before you, garbles it in your
brain, and then outputs a transformed version from your mouth, to the next person's
ear.  In math, there is a domain for $h(x)$, a set of numbers it can take for input.
These numbers are mapped onto a range of outputs, which are then fed into
$g(x)$.  Now $g(x)$ had its own domain, and the output from $h$ is a subset of that
absolute domain.  Having received some input from a portion of its domain,
$h(x)$, now outputs a unique range.

\begin{figure}
\begin{center}
\begin{tikzpicture}[scale=0.8,
      >=stealth,
      bullet/.style={
        fill=black,
        circle,
        inner sep=1pt
      },
      projection/.style={
        ->,
        thick,
        shorten <=2pt,
        shorten >=2pt
      },
    ]

    \draw (0, 0) circle [x radius=2, y radius=3];
    \node [bullet, label=below:\(x\)] (x) at (-1, -0.5) {};
    \node[font=\large] (X) at (0, 4) {\(X\)};

    \begin{scope}[xshift=4cm]
      \draw (0, 0) circle [x radius=1, y radius=3.5]; \node [bullet,
      label=above:\(h(x)\)] (fx) at (0.3, 2) {};
      \node[font=\large] (Y) at (0, 4) {\(Y\)};
    \end{scope}
    \begin{scope}[xshift=8cm]
      \draw (0, 0) circle [x radius=2, y radius=1.5]; \node [bullet,
      label=below:\(g(h(x))\)] (gfx) at (-0.5, -0.1) {};
      \node[font=\large] (Z) at (0, 4) {\(Z\)};
    \end{scope}

    \draw [projection] (x) -- (fx);
    \draw [projection] (fx) -- (gfx);
    \draw [projection] (X) -- (Y)
          node [pos=0.5, above] {\(h\)};
    \draw [projection] (Y) -- (Z)
          node [pos=0.5, above] {\(g\)};
    \draw [out=45, in=180-45, projection, line width=1.5pt, red!80!black]
          (X) .. controls ++(1, 1) and ++(-1, 1) .. (Z)
          node [pos=0.5, above] {\(f = g \circ h\)};
\end{tikzpicture}
\caption{Composition of functions \cite{sxtikzcomp}.}
\end{center}
\end{figure}

\subsection{Decomposition}
Any function with more than one operation can be written as the composition of
other functions.  For example, if $f(x)=x^2+1$ then we could construct
functions $g(x)=x^2$ and $h(x)=x+1$ such that $f(x)=h(g(x))$ ($g(h(x))$ would be
$(x+1)^2$).

\subsection{The Chain Rule}\index{derivative!chain rule}
The algebra of functions continues, even into the realm of derivatives.  Two functions
added would have a simple derivative: the sum of their respective derivatives.  But what
of two functions composed?  What is the derivative of $g(h(x))$?

Here, Leibniz's notation is more helpful:

$$f(x)=g(h(x)), \quad \frac{df}{dx}=\frac{dg}{dh}\cdot\frac{dh}{dx}$$

\paragraph{Trigonometric Derivatives}\index{derivative!of sine}
While you are not responsible for proving and understanding trigonometric functions
until Part~\ref{pt:trig}, it is helpful to being memorizing facts about them now.  This will allow
you to practice manipulating the functions like any other.  However, to do so, you will
need to know the derivative of sine and cosine.\index{derivative!of cosine}

\begin{itemize}
\item $f(x) = \sin(x)$
\item $f'(x) = \cos(x)$
\item $f''(x) = -\sin(x)$
\item $f'''(x) = -\cos(x)$
\item $f^{(4)}(x) = \sin(x)$
\end{itemize}
As you can see, a straightforward, four-step cycle emerges.  The derivative of sine is
cosine.  The derivative of cosine is negative sine.


% Bismillahi-r-Rahmani-r-Rahim
\documentclass{article}

\title{Approximate Entailment}

\author{Daoud Clarke}

\date{\today}

\usepackage{color}
\usepackage{graphicx}
\usepackage{amssymb}

\begin{document}

\maketitle

\section{Introduction}

We have been treating the hyponymy relation given by WordNet as a
partial ordering on the vector space of contexts, however this is not
consistent if we take into account synonyms, if we treat them as
satisfying $u \le v$ and $v\le u$, where $u$ and $v$ are the context
vectors of two terms, which is only possible if $u = v$. This can be
satisfied either by constructing a quotient space to make the vectors
equal, or by relaxing the constraints on the relationship $\le$. Here
we consider the latter option.

\begin{figure}
\def\svgwidth{5cm}
\centering
\input{approx-cone.pdf_tex}
\caption{The set of points $u\gtrsim 0$ are those such that $u\ge-\delta$.}
\end{figure}

\begin{figure}
\def\svgwidth{5cm}
\centering
\input{synonym.pdf_tex}
\caption{Approximate synonym vectors $u$ and $v$ satisfy $|u - v| \le \delta$ (the
  darker shaded region).}
\end{figure}

Given a partially ordered vector space $V$ and a vector $\delta \in
V$, we define a new relation $\lesssim$ on $V$ by $u \lesssim v$ if
and only if $u \le v + \delta$. We can then define a relation $\simeq$
by $u \simeq v$ if and only if $u\lesssim v$ and $v \lesssim u$; its
interpretation is that the terms associated with $u$ and $v$ are
approximately synonymous. Note that this is only possible if $\delta
\ge 0$, since $u \le v + \delta \le u + 2\delta$.

\section{Learning Approximate Relations}

If we assume this more general situation, we now have the problem of
learning not just the ordering $\le$, but also the vector
$\delta$.

One option is to assume that $u \lesssim v$ means there exists a
$\delta_{uv}$ such that $u \le v + \delta_{uv}$ and
$\|\delta_{uv}\|_2\le \epsilon$ for some fixed $\epsilon$. This leads
to an overall value of $\delta =
(\epsilon,\epsilon,\ldots,\epsilon)$. In this construction, we have to
learn a single parameter $\epsilon$, which acts in a similar manner to
the bias in support vector machines.

Another option is to try and learn $\le$ and $\delta$ simultaneously
using support vector machines. Let $p_i$ be a set of hyperplanes and
$b_i$ a set of associated biases, which together define a set of
half-spaces $S_i$, by $u\in S_i$ if and only if $p_i\cdot u \ge
b_i$. If the $p_i$ are linearly independent, then the intersection of
these half-spaces forms a displaced cone: let $M$ be the matrix with
$p_i$ as rows, and $\delta$ the vector with $b_i$ as elements, then
$u$ is in the intersection of half spaces if $Mu \ge \delta$.

\end{document}