\section{\texorpdfstring{Hash functions, Merkle trees, MACs}{Hash functions, Merkle trees, MACs}}
\vspace{5mm}
\large

\begin{definition}
	Hash function $f$ is
	\[ f:\{0, 1\}^{\ast} \to \{ 0,1 \}^b \]

	Typically used for signatures.
\end{definition}

\begin{properties} Secure hash function:
	\begin{enumerate}
		\item collision-free: there are no $x \neq y: h(x) = h(y)$
		\item no second preimage: for given $x$, find $y$: $h(x) = h(y)$.
		\item no inversion: for given $z$ find $x: h(x) = z$.
	\end{enumerate}

\end{properties}

\subsection{Merkle-Damgard construction}
Given a \emph{compression function} $f:\{0, 1 \}^b \times \{0, 1 \}^b \to \{0, 1 \}^b$. Construct $h$:

	\includegraphics[scale=0.4]{md.eps}

\begin{theorem}[Collision resistant hash]
	If $f$ is collision resistant compression function $\Rightarrow h$ is collision resistant hash function.
\end{theorem}
\begin{proof}
	Indirect proof. Assume we found a collision of $h$, meaning $h(x_1,...,x_n) = h(x_1^{\prime},...,x_n^{\prime})$

	Let's consider 2 options:\\
	1) $n \neq n^{\prime}$. In the last step we compress $n$ and $n^{\prime}$. As outputs are identical $\Rightarrow$ the last block containing size is also identical. Therefore we have found collision of $f$.\\
	2) $n = n^{\prime}$. Using induction we examine compressed blocks. As outputs are identical: either the blocks were identical or we have found collision of $f$. Then, as messages are not identical, there exist two different blocks encoded to the same output. Which implies collision of $f$.
\end{proof}

\subsubsection{Length extension property of M-D hash} Suppose we have a message $x$ and prefix $p$.
For a Random function, knowing prefix $h(p)$ should help us producing full $h(p | s)$. Where $s = x - p$ suffix.

From the M-D internal structure, hash of the suffix depends on the hash of prefix.
Therefore, an attacker, knowing hash of the prefix, can add his data to the message and compute new signature (hash).

We should take this property into consideration using M-D hashes.

\subsubsection{Compression function for M-D}

Having a block cipher, we can use it to create compression function as the following:
\[ f(u,v) := E_u(v) \oplus v \]

XOR is required because otherwise finding collision is easy. Suppose $f(u,v) := E_u(v)$, then
\[ E_u(v) = y \]
Taking another key, e.g. $b$, we decrypt $y$ using $b$
\[ D_b(y) = a \Rightarrow f(u,v) = y = f(a,b) \]

Using DES as block cipher is also a bad idea, since $E_{\overline{k}}(\overline{x}) = \overline{E_k(x)}$:
\[ f(\overline{a}, \overline{c}) = E_{\overline{a}}(\overline{c}) = \overline{E_a(c)} \oplus \overline{c} = E_a(c) \oplus c = f(a,c) \]

For $v := D_u(0), f(u,v) = E_u(v) \oplus v = 0 \oplus v = v$. Which cannot happen for random hash function.

\begin{theorem}[Collision resistant hash]
	With an ideal block cipher (for every key a random permutation) $f$ is collision resistant. In particular, for an attack evaluating E/D $q$ times ($q \leq 2^{b/2}$)
	\[ Pr[collision] \leq q^2/2^b \]
\end{theorem}
\begin{proof}
	WLOG: attacker do not ask redundant questions (E of the text he already knows).

	From the properties of ideal block cipher, results of encryption is a bijection. Can be viewed as table of values.

	Asking E and D questions attacker gets following answer:
	\[ E_u(v) = f(u,v) = E_u(v) \oplus v \land D_u(v) = f(u, D_u(v)) = v \oplus D_u(v) \]

	We find first collision in step $i: 1 \geq i \leq q$. We found $f(l,k)$ matching some known value, having $(i - 1)$ of them.
	So it was either the answer to E or D question, which is exactly 1 value.

	For every pair, known and new answers:
	\[ Pr[collision] = \frac{1}{num\ of\ possible\ ans.} \leq \frac{1}{2^b - i - 1} \land i \leq 2^{i/2} \leq 2^{b/2} \Rightarrow Pr[collision] \leq \frac{1}{2^{b-1}} \]

	\[ Pr[collision] \leq \frac{1}{2^{b-1}} pairs \leq \frac{1}{2^{b-1}} \binom{q}{2} \leq \frac{q^2}{2*2^{b-1}} = \frac{q^2}{2^{b}} \]

\end{proof}

\subsubsection{Finding collision of hash function}

\begin{enumerate}
	\item Brute force: by the Birthday paradox expect a match in $\approx 2^{b/2}$ steps. Lots of memory.
	\item With constant memory. Assume $\forall x: |x| = |h(x)|$.
		Let's construct a graph where vertexes are all possible blocks and edges are $(x, h(x))$.
		Degree of every $x$ is 1. Iterating $h(h(... h(x))$ we create a path in Graph.
		At some point there will be a loop, as Grah is finite.

		A subgraph reachable from $x$ looks like lollipop. Path + cycle.
		In order to find collision in such subgraph we choose 2 travelers in the graph: tortoise (1 step at time) and rabbit (2 steps at time).
		Starting at $x$, they move in graph. Claim: they will meet in vertex which connects path with cycle.
	\item In many cases we need to find meaningful collision (to send evil message instead of innocent).
		And the party signing messages refuses to sign random noise.

		In order to find collision, we construct parametrized message. Let's say Dear Bob, Dear Sean etc.
		With $k$ places of parametrization we can produce $2^k$ innocent messages.

		Both previous attacks work with such messages. However, edges in the Graph for 2nd type of attach will be $(x, h(parametrize(x)))$.
	\item Meet-in-the-middle attack. We want $h(evil) = h(innocent)$.

		Let A,B two random subsets of all hashes. Than, with high probability they have non empty intersection. $|A| = |B| = \sqrt{n}$.
		A will be hashes of $2^{b/2}$ innocent messages, B - hashes of $2^{b/2}$ evil messages.
	\item For M-D hashes we can product lots of collision, as easy as 1.
		By Birthday paradox, we find first collision: $x_1 \neq x_1^{\prime}: f(IV, x_1) = f(IV, x_1^{\prime}) = y_1$.
		Then we repeat $k$ times using $y_{i-1}$ as IV. As we can use both $x_i$ and $x_i^{\prime}$ to get $y_{i-1}$.
		We get $2^k$ combinations which hash to the same result.

		As a result, concat of 2 hashes, both with length $b$ does not have security level $2b$ if either of functions is M-D.
		Because we can find $2^{b/2}$ colliding messages in time $b/2 * 2^{b/2}$. Than $h_2$ is likely to have 2 collisions.
		Therefore, we have found collision in $h = h_1 | h_2$ in time $b/2 * 2^{b/2}$.
\end{enumerate}

\subsection{Sponge construction}
Sponge construction has 2 phases:\\
1) Absorbing the input (updating internal state depending on input)\\
2) Squeezing the output

\begin{example}
	SHA-3 sponge construction. Suppose we have a permutation $\pi$ on blocks of size $w = r + c$.
	Where $w$ is width, $r$ - rate and $c$ - capacity.

	\includegraphics[scale=0.3]{sponge.eps}

	Security against BF attacks:\\
	1) By the Birthday paradox we can attack the output in $2^r$ steps.\\
	2) Internal collisions (chosen plain text attack):\\
	By the Birthday paradox in $2^c$ steps we can find
	\[ i < j: S_i = S_j \]
	In order to get collision, we feed the sponge $0^i$ as first part of input.
	Then we feed $(t_i \oplus t_j)$ to get $r = t_i$ back and feed zeros till the end $0^{j - 1}$.
	As a result, after $j$-th stage sponge squeezes the same output.

	However, we know that for a random $\pi$ security level $\geq \min(r/2, c/2)$.
\end{example}

\subsection{Other hash functions}

\textbf{Shake-128}, an XOR(extensible-output functions)
Designed for variable size output. Can be used as PRNG, that is initialized by some value.
Can be run for many \# of steps.

\begin{definition}
	Merkle trees. Divide data into blocks, then hash and concat some of then.
	Which forms a tree.

	\includegraphics[scale=0.3]{merkle.eps}

	Claim: if hash function $f$ is collision-free $\Rightarrow$ whole tree is collision-free.

	Can be run in parallel, also updating a single block of data result in path update only.
	Used in practice to sign a DB state.
\end{definition}

\begin{observation}
	Naive Merkle tree is not secure to \emph{extension attacks}.
	As we can take old tree, add another subtree(that includes our evil data) and a new root.

	Fix: add 1 bit to every node, 1 is for Root. (see Sakura coding as example).
\end{observation}

\subsection{MACs}
MACs are used to sign some information (either plain or cipher text).

\includegraphics[scale=0.3]{mac_0.eps}

Usually, sign is deterministic and verification is just sign + compare with received signature from the counterparty.
Typically sign is parametrized hash.

\paragraph{Security Model}
Attacker has access to \emph{signing oracle} and can ask it to sign $(x_1, ..., x_n)$ plain texts.
Cryptographic primitive is secure iff the attacker is able to produce new valid signature for $x \neq x_i$.

How to sign? One can include a key to the hash alongside the text. There are several ways to do so:

\begin{itemize}
	\item $h(k | x)$ is secure for random $h$, but not for Merkle-Damgard.
		M-D hashes are not protected against extension attacks. SHA-3 is believed to be secure for such usage. Such construction is included to the standard as \textbf{KMAC}.
	\item $h(x | k)$ will not work either, because the attacker can precompute all combinations independent from key, then try with specific key. Therefore, $h(x | k)$ should be also avoided.
	\item Much better option is \textbf{HMAC}$_h := h(k \oplus C_{out} | h((k \oplus C_{in}) | x))$.
	$C_{in}, C_{out}$ are derived from master key. Where $h((k \oplus C_{in}) | x)$ is collision-resistant. Consequently, we can even use broken SHA-1,2.
\end{itemize}

\paragraph{Combining auth with enc}
Typically, we want both security and authenticity, so that the attacker cannot neither decrypt nor change the message. To achieve that, MAC with encryption is used.
Again, there are several possibilities to combine 2 primitives.

\begin{itemize}
	\item Encrypt and MAC independently is a bad idea, since the signature of the same message encrypted with different cipher is leaking their identity.
	\item MAC then Encrypt has several advantages over the opposite option:\\
		1) Signature should reflect plain text as close as possible. Should be same for different ciphers.\\
		2) Authenticity is more important than secrecy (e.g. for nuclear missiles control system)\\
		3) MAC is protected by the cipher

		Unfortunately, described scheme is vulnerable to the \emph{oracle timing attack}.
		Specifically, reducing the padding by 1 block will result in 1 MAC iteration less.
		Which can be observed by the attacker. As a result, almost all protocols that includes MAC then Encrypt are broken because of this attack.

		However, MAC then Encrypt is used in TLS, To avoid the attack, chosen MAC is not the block type.
	\item Encrypt then MAC is the recommended scheme.
\end{itemize}

\paragraph{CBC-MAC}
We can construct MAC from symmetric cipher in CBC mode.

\includegraphics[scale=0.4]{mac_1.eps}

\begin{properties}
In order to be secure, CBC-MAC should have these properties

\begin{itemize}
	\item Fixed IV. If the IV is random (not fixed), it should be transfered over the network.
		As a result, Mallory can flip bits in both IV and the first block of the message.
		Which will alter the message, but the sign remains valid.
	\item Intermediate steps of the CBC should be kept secret.
	\item No message is a prefix of another message.
\end{itemize}
With these properties, assuming ideal cipher, CBC-MAC is proved to be secure.
\end{properties}

\begin{definition}
	An MAC is \emph{Shannon secure} iff given a known pair $(x, sign(x))$ all signatures of $x^{\prime} \neq x$ are equally possible for a random key.
	Where $sign(x)$ is a signing function which uses a single random key.

	Alternatively, a pair $(x, sign(x))$ gives no info about other messages.
\end{definition}

\begin{definition}
	A family of functions from set X to set Y is $\mathcal{H} := \{ h_k | k \in \mathcal{K} \}$.
	Technically a multiset.
\end{definition}

\subsubsection{2 independent MAC}

\begin{definition}
	$\mathcal{H}$ is 2-independent iff
	\[ \forall x, x^{\prime} \in X \forall y, y^{\prime} \in Y: Pr_h[h(x) = y \land h(x^{\prime}) = y^{\prime}] = \frac{1}{|Y|^2} \]
\end{definition}

Typically we use Galois field (GF) as X and Y.

\begin{example}
	$h_{a,b}(x) := ax + b$ is a 2-independent family of functions. To construct a MAC from this family we:\\
	1) select $h \in \mathcal{H}$ at random \\
	2) key is $(a,b)$ - parameters of $h$\\
	3) signature is $h(x)$.

	To prove 2-independence we calculate ($x_1, x_2, y_1, y_2$ are given constants):
	\[ Pr_f[ h_k(x_1) = y_1 \land h_k(x_2) = y_2] = Pr_{a,b}[ax_1 + b = y_1 \land ax_2 + b = y_2] \]
	As we have system of 2 linear equations with 2 unknowns: a and b, it has an unique solution.
	Therefore the probability is $\frac{1}{|Y|^2}$, all possible choices of parameters a, b.
\end{example}

\begin{theorem}[2-ind MAC security]
	2-ind MAC is Shannon secure.
\end{theorem}
\begin{proof}
	By the conditional probability:
	\[ Pr_f[ h_k(x_1) = y_1 | h_k(x_2) = y_2] = \frac{Pr_f[ h_k(x_1) = y_1 \land h_k(x_2) = y_2]}{Pr_f[h(x_1) = y_1]} \]
	By the 2-independence
	\[ Pr_f[ h_k(x_1) = y_1 \land h_k(x_2) = y_2] = \frac{1}{|Y|^2} \]
	%TODO do not understand why (lecture 11.25, from 1:14:00)
	$Pr_f[h(x_1) = y_1]$ can be calculated as a disjoint sum of events
	\[ \sum_{y_i} Pr_h[h(x) = y \land h(x_i) = y_i] = \frac{|Y|}{|Y|^2} \]
	Combining 2 results
	\[ Pr_f[ h_k(x_1) = y_1 | h_k(x_2) = y_2] = \frac{1}{|Y|^2} \div \frac{1}{|Y|} = \frac{1}{|Y|} \]
	Which satisfies Shannon security.
\end{proof}

Downsides of 2-ind MAC:
\begin{itemize}
	\item key is twice as large as a message.
	\item no key can be reused
	\item sequence number is required
	\item message size = signature size
\end{itemize}

\subsubsection{Polynomial based MAC}
Suppose, F is Galois field, $X = F^n, Y = F, K = F^2$. Then
\[ h_{a,b}(x_1,...,x_n) = x_1a^n + ... + x_na + b \]
Evaluation the polynomial at some point could be done in $\bigO(n)$.

\begin{theorem}[Polynomial MAC security]
	Polynomial MAC is not Shannon secure.
\end{theorem}
\begin{proof}
\[ Pr_{a,b} [h_{a,b}(x_1^{\prime},...,x_n^{\prime}) = y^{\prime} | h_{a,b}(x_1,...,x_n) = y] = Pr_{a,b}[L | R] = \frac{Pr[L \land R]}{Pr[R]} \]
Substracting 1st equation from second we get
\[ Pr[L \land R] = (x_1 - x_1^{\prime})a^n + ... + (x_n - x_n^{\prime})a = y - y^{\prime} \]
Which is a non zero polynomial, since messages are different $\Rightarrow$ at most $n$ values of a.
Also, initial equation was satisfied for exactly one $b \Rightarrow$ at most $n$ pair $(a, b)$. Yielding:
\[ Pr[L \land R] \leq \frac{n}{|F|^2} \]
Then
\[ Pr[R] = \frac{|F|}{|F|^2} = \frac{1}{|F|} \]
And
\[ Pr_{a,b}[L | R] \leq \frac{n}{|F|} \]
Which does not satisfy Shannon security, as $n$ is in numerator. However, we can choose reasonably big Field.

\end{proof}

And yet, polynomial MAC is not practical as we need a new random key for every message.
Which can be solved by using PRNG parametrized by key.
\begin{observation}
	If parameter $a$ is fixed and $b$ is generated by PRNG security level does not decrease.
\end{observation}

\subsubsection{GCM mode of block ciphers}

\begin{definition}
	GCM (Galois counter mode) is an auth encoding with additional data (e.g. sign, sequence number)

	\includegraphics[scale=0.4]{gcm.eps}

	We use additional data, encoded data, sizes and IV as a coef of a Polynomial. And $P(E_k(0))$ is a signature, where $E_k(0)$ is a constant than has to be secret!

	As elements are 128-bit strings, addition is XOR multiplication of polynomials over field $\Z_2$.
	Which is a \emph{carry-less} multiplication mod some irreducible polynom Q.
\end{definition}

\begin{example}
	Poly 1305 [Bernstein], is a MAC construction from block ciphers over $GF(2^{130} - 5)$.
	Size of the field is choosen close to power of 2 for speedup. Moreover, points $a$ in which polynomial is evaluated are also some specific subset of the field for the same reason.
	Security level does not decrease significantly.
\end{example}


\section{Isomorphisms}

\begin{outcome}
  \begin{enumerate}
  \item Apply the concepts of one to one and onto to transformations
    of vector spaces.
  \item Determine if a linear transformation of vector spaces is an
    isomorphism.
  \item Determine if two vector spaces are isomorphic.
  \end{enumerate}
\end{outcome}


\subsection{Budget constraints}

Economic agents can choose elements from a set. We can define this set both by what is included, and what is excluded.

An agent could choose inputs from an interval of real numbers. In this case the finding the value which maximises the utility function across all real numbers may not be a valid solution. For example an agent may optimally wish to consume \(20\) apples, but only have \(10\).

Similarly an agent may be limited in combinations of choices. For example an agent could choose how much to work and how much to enjoy leisure, but be constrained by the amount of time in the day.


\chapter{分治法}

\section{Pow(x,n)} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{sec:pow}

\subsubsection{描述}
Implement pow(x, n).

\subsubsection{分析}
二分法，$x^n = x^{n/2} \times x^{n/2} \times x^{n\%2}$

\subsubsection{代碼}
\begin{Code}
//LeetCode, Pow(x, n)
// 二分法，$x^n = x^{n/2} * x^{n/2} * x^{n\%2}$
// 時間複雜度O(logn)，空間複雜度O(1)
class Solution {
public:
    double myPow(double x, int n) {
        long long nn = n // 使用 64 bit 來處理 INT_MIN 由負變正 case
        if (nn < 0) return 1.0 / power(x, -nn);
        else return power(x, nn);
    }
private:
    double power(double x, long long nn) {
        if (nn == 0) return 1;
        double v = power(x, nn / 2);
        if (nn % 2 == 0) return v * v;
        else return v * v * x;
    }
};
\end{Code}

\subsubsection{相關題目}

\begindot
\item Sqrt(x)，見 \S \ref{sec:sqrt}
\myenddot

\section{Sqrt(x)} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{sec:sqrt}

\subsubsection{描述}
Implement int \fn{sqrt(int x)}.

Compute and return the square root of \fn{x}.

\subsubsection{分析}
二分查找

\subsubsection{代碼}
\begin{Code}
// LeetCode, Sqrt(x)
// 二分查找
// 時間複雜度O(logn)，空間複雜度O(1)
class Solution {
public:
    int mySqrt(int x) {
        int left = 1, right = x / 2;
        int last_mid;  // 記錄最近一次mid

        if (x < 2) return x;

        while(left <= right) {
            const int mid = left + (right - left) / 2;
            if(x / mid > mid) { // 不要用 x > mid * mid，會溢出
                left = mid + 1;
                last_mid = mid;
            } else if(x / mid < mid) {
                right = mid - 1;
            } else {
                return mid;
            }
        }
        return last_mid;
    }
};
\end{Code}

\subsubsection{相關題目}
\begindot
\item Pow(x)，見 \S \ref{sec:pow}
\myenddot

\section{The Skyline Problem}
\label{sec:the-skyline-problem}

\subsubsection{描述}
A city's skyline is the outer contour of the silhouette formed by all the buildings in that city when viewed from a distance. Now suppose you are given the locations and height of all the buildings as shown on a cityscape photo (Figure A), write a program to output the skyline formed by these buildings collectively (Figure B).

\begin{center}
\includegraphics[width=250pt]{the-skyline-problem.png}\\
\figcaption{Skyline Problem}\label{fig:the-skyline-problem}
\end{center}

The geometric information of each building is represented by a triplet of integers [Li, Ri, Hi], where Li and Ri are the x coordinates of the left and right edge of the ith building, respectively, and Hi is its height. It is guaranteed that 0 ≤ Li, Ri ≤ INT_MAX, 0 < Hi ≤ INT_MAX, and Ri - Li > 0. You may assume all buildings are perfect rectangles grounded on an absolutely flat surface at height 0.

For instance, the dimensions of all buildings in Figure A are recorded as: [ [2 9 10], [3 7 15], [5 12 12], [15 20 10], [19 24 8] ] .

The output is a list of "key points" (red dots in Figure B) in the format of [ [x1,y1], [x2, y2], [x3, y3], ... ] that uniquely defines a skyline. A key point is the left endpoint of a horizontal line segment. Note that the last key point, where the rightmost building ends, is merely used to mark the termination of the skyline, and always has zero height. Also, the ground in between any two adjacent buildings should be considered part of the skyline contour.

For instance, the skyline in Figure B should be represented as:[ [2 10], [3 15], [7 12], [12 0], [15 10], [20 8], [24, 0] ].

Notes:

\begin{enumerate}
\item The number of buildings in any input list is guaranteed to be in the range [0, 10000].
\item The input list is already sorted in ascending order by the left x position Li.
\item The output list must be sorted by the x position.
\item There must be no consecutive horizontal lines of equal height in the output skyline. For instance, [...[2 3], [4 5], [7 5], [11 5], [12 7]...] is not acceptable; the three lines of height 5 should be merged into one in the final output as such: [...[2 3], [4 5], [12 7], ...]
\end{enumerate}

\subsubsection{分析}
Nil

\subsubsection{分治法}
\begin{Code}
// 時間複雜度O(nlogn)，空間複雜度O(n)
class Solution {
public:
    vector<vector<int>> getSkyline(vector<vector<int>>& buildings) {
        return DFS(buildings, 0, buildings.size());
    }
private:
    vector<vector<int>> DFS(vector<vector<int>>& buildings, int start, int end)
    {
        if (start == end)
        {
            return vector<vector<int>>();
        }
        else if (end - start == 1)
        {
            vector<vector<int>> result;
            int& startX = buildings[start][0];
            int& startY = buildings[start][2];
            int& endX = buildings[start][1];
            result.push_back({startX, startY});
            result.push_back({endX, 0});

            return result;
        }

        int mid = start + (end - start) / 2;
        auto left = DFS(buildings, start, mid);
        auto right = DFS(buildings, mid, end);

        return mergeSkylines(left, right);
    }
    vector<vector<int>> mergeSkylines(const vector<vector<int>>& left
                                      , const vector<vector<int>>& right)
    {
        int lN = left.size();
        int rN = right.size();
        int lp, rp, curP;
        int ly, ry, curY;
        int x, maxY;
        vector<vector<int>> output; output.reserve(lN + rN);
        lp = rp = curP = 0;
        ly = ry = curY = 0;
        x = maxY = 0;

        while (lp < lN && rp < rN)
        {
            if (left[lp][0] < right[rp][0])
            {
                x = left[lp][0];
                ly = left[lp][1];
                lp++;
            }
            else
            {
                x = right[rp][0];
                ry = right[rp][1];
                rp++;
            }

            maxY = max(ly, ry);

            if (curY != maxY)
            {
                UpdateOutput(output, x, maxY);
                curY = maxY;
            }
        }

        if (lp < lN)
            AppendSkylines(output, left, lp, lN, curY);
        else
            AppendSkylines(output, right, rp, rN, curY);

        return output;
    }
    void UpdateOutput(vector<vector<int>>& output, int x, int y)
    {
        if (output.empty() || output.back()[0] != x)
            output.push_back({x, y});
        else
            output.back()[1] = y;
    }
    void AppendSkylines(vector<vector<int>>& output
                        , const vector<vector<int>>& skyline, int p, int N, int curY)
    {
        while (p < N)
        {
            const int& x = skyline[p][0];
            const int& y = skyline[p][1];
            p++;

            if (curY != y)
            {
                UpdateOutput(output, x, y);
                curY = y;
            }
        }
    }
};
\end{Code}


%\documentclass[a4paper,landscape]{article}
\documentclass[a4paper]{article}
\usepackage{amsmath, amssymb}
\usepackage{multirow}
\usepackage{mathrsfs}
\usepackage{graphicx}
\usepackage{amsthm}
\usepackage{multicol}

%\usepackage[font={small}, margin=1cm]{caption}
\usepackage[margin=.1in]{geometry}
\setlength{\parindent}{0pt}
\renewcommand{\arraystretch}{0.8}
\newcommand{\C}{\mathbb{C}}
\newcommand{\F}{\mathcal{F}}
\newcommand{\K}{\mathbb{K}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\ra}{\rightarrow}

\begin{document}

\fontsize{10.5}{6}
\selectfont
\begin{multicols}{2}
%\section*{CHAPTER 0:}
%\input{Def_0_5}
%\input{Def_0_6}
%\input{Thm_0_11}
%\input{Thm_0_21}
%\input{Def_0_29}

\section*{CHAPTER 1: FOURIER SERIES}
%\subsubsection*{Real Fourier Series}

\underline{\textsl{\bf REAL FOURIER SERIES}}
\input{Thm_1_1}
\input{Thm_1_2}
%\subsubsection*{Complex Fourier Series}

\underline{\textsl{\bf COMPLEX FOURIER SERIES}}
\input{Thm_1_20}
%\subsubsection*{Convergence Theorems}

\underline{\textsl{\bf CONVERGENCE THEOREMS}}
\input{Thm_1_21}
\input{Thm_1_22}
\input{Thm_1_28}
\input{Thm_1_30}
\input{Lem_1_33}
\input{Thm_1_35}
\input{Thm_1_39}
\input{Thm_1_40}

\section*{CHAPTER 2:FOURIER TRANSFORM}
\input{Thm_2_1}
%\newline
\input{Thm_2_6}
\input{Thm_2_9and10}
\input{Thm_2_12}
%\subsubsection*{Linear Filters}

\underline{\textsl{\bf LINEAR FILTERS}}
\input{Def_2_13}
\input{Lem_2_16}
\input{Thm_2_17}
\input{Thm_2_19}
\input{Thm_2_20}
%\subsubsection*{The Sampling Theorem}

\underline{\textsl{\bf THE SAMPLING THEOREM}}
\input{Def_2_22}
\input{Thm_2_23}

\section*{CHAPTER 3: DISCRETE FOURIER TRANSFORM}
\input{Intro_3_1_1}
\input{Thm_3_3}
%\newline
\input{Thm_3_4}

\section*{CHAPTER 4: HAAR WAVELET ANALYSIS}
{\bf Haar Scaling function:} The Haar scaling function is defined as $\phi(x) = 1$ if $x\in[0,1]$.
{\bf Definition:} Suppose $j$ is any nonnegative integer. The space of step functions at level $j$, denoted by $V_j$, is defined to be the space spanned by the set $\{\dots,\phi(2^j+1),\phi(2^j),\phi(2^j-1),\phi(2^j-2),\dots\}$.
{\bf Theorem 4.5:} A function $f(x)$ belongs to $V_0$//$V_j$ if and only if $f(2^jx)$//$f(2^{-j}x)$) belongs to $V_j$//$V_0$.
{\bf Theorem 4.6:} The set of functions $\{2^{j/2}\phi(2^jx-k);k\in\Z\}$ is an orthonormal basis of $V_j$.
{\bf Haar Wavelet:} The Haar wavelet is function $\psi(x) = \phi(2x)-\phi(2x-1)$.
{\bf Theorem 4.8:} Let $W_j$ be the space of functions of the form $\sum_{k\in\Z}a_k\psi(2^jx-k)$, $a_k\in\R$ (only a finite number of $a_k$ are nonzero). $W_j$ is the orthogonal complement of $V_j$ in $V_{j+1}$ and $V_{j+1}=V_j\bigoplus W_j$.
{\bf Theorem 4.9:} The space $L^2(\R)$ can be decomposed as an infinite orthogonal direct sum $L^2(\R)=V_0\bigoplus W_0\bigoplus W_1\dots$. In particular, each $f\in L^2(\R)$ can be written as $f=f_0+\sum_{j=0}^{\infty}w_j$, where $f_0\in V_0$ and $w_j\in W_j$.

\underline{\textsl{\bf SAMPLE}}
If the signal is continuous, $y=f(t)$, where $t$ represents time, choose the top level $j=J$ so that $2^j$ is larger than the Nyquist rate for the signal. Let $a_k^J=f(k/2^J)$. The top level $a_k^J$ is set equal to the $k$th term in the sampled signal, and $2^J$ is taken to be the sampling rate. In any case, we have the highest-level approximation to $f$ given by $f_J=\sum_{k\in\Z}a_k^j\phi(2^Jx-k)$.

\underline{\textsl{\bf DECOMPOSITION}}
{\bf Lemma 4.10:} The following relations hold for all $x\in\R$. $\phi(2^jx)=\left(\phi(2^{j-1}x)+\psi(2^{j-1}x)\right)/2$. $\phi(2^jx-1)=\left(\phi(2^{j-1}x)-\psi(2^{j-1}x)\right)/2$.
{\bf Theorem 4.12:} Suppose $f_j(x)=\sum_{k\in\Z}a_k^j\phi(2^jx-k)\in V_j$. Then $f_j$ can be decomposed as $f_j=w_{j-1}+f_{j-1}$, where $w_{j-1}=\sum_{k\in\Z}b_k^{j-1}\psi(2^{j-1}x-k)\in W_{j-1}$ and $f_{j-1}=\sum_{k\in\Z}a_k^{j-1}\phi(2^{j-1}x-k)\in V_{j-1}$, with $b_k^{j-1}=\frac{a_{2k}^j-a_{2k+1}^j}{2}$ and $a_k^{j-1}=\frac{a_{2k}^j+a_{2k+1}^j}{2}$.

\underline{\textsl{\bf RECONSTRUCTION}}
{\bf Theorem 4.12:} If $f=f_0+w_0+w_1+\dots +w_{j-1}$ with $f_{0}(x)=\sum_{k\in\Z}a_k^0\phi(x-k)\in V_0$ and $w_{j}=\sum_{k\in\Z}b_k^{j}\psi(2^jx-k)\in W_{j}$ for $0\leq j\leq J$, then $f(x)=f_J(x)=\sum_{k\in\Z}a_k^J\phi(2^Jx-k)\in V_J$. The $a_k^J$ are determined recursively by $a_k^j = a_l^{j-1}+b_l^{j-1}$ if $k=2l$ is even and $a_k^j = a_l^{j-1}-b_l^{j-1}$ if $k=2l+1$ is odd.

\section*{CHAPTER 5: MULTIRESOLUTION ANALYSIS}
{\bf Definition:} Let $V_j, j=\dots -1,0,1\dots,$ be a sequence of subspaces of cuntions in $L^2(\R)$. The collection of spaces $\{V_j,j\in\Z\}$ is called a \textit{multiresolution analysis with scaling funciton $\phi$} if the following conditions hold. {\bf 1.} (Nested) $V_j\subset V_{j+1}$. {\bf 2.} (Density) $\overline{\cup V_j}=L^2(\R)$. {\bf 3.} (Separation) $\cap V_j=\{0\}$ {\bf 4.} (Scaling) See Theorem 4.5(b) {\bf 5.} (Orthonormal basis) The function $\phi$ belongs to $V_0$ and the set $\{\phi(x-k);k\in\Z\}$ is an orthonormal basis for $V_0$.
{\bf Theorem 5.5:} Suppose $\{V_j; j\in\Z\}$ is a multiresolution analysis with scaling function $\phi$. Then for any $\in\Z$, the set of functions $\{\phi_{jk}(x)=2^{j/2}\phi(2^jx-k);k\in\Z\}$ is an orthonormal basis for $V_j$.

\underline{\textsl{\bf THE SCALING RELATION}}
{\bf Theorem 5.6:} Suppose $\{V_j; j\in\Z\}$ is a multiresolution analysis with scaling function $\phi$. Then the following scaling relation holds: $\phi(x) = \sum_{k\in\Z} p_k\phi(2x - k)$, where $p_k = 2\int_{-\infty}^{\infty}\phi(x)\overline{\phi(2x - k)}dx$. Moreover, we also have $\phi_{j-1,l}=2^{-1/2}\sum_{k\in\Z}p_{k-2l}\phi_{jk}$. \newline {\bf Remark:} When the support of $\phi$ is compact, only a finite number of $p_k$ are nonzero, because when $|k|$ is large enough, the support of $\phi(2x - k)$ will be outside of the support of $\phi(x)$.
{\bf Theorem 5.9:} Suppose $\{V_j; j\in\Z\}$ is a multiresolution analysis with scaling function $\phi$. Then, provided the scaling relation can be integrated termwise, the following equalities hold: {\bf 1.} $\sum_{k\in\Z}p_{k-2l}\overline{p_k}=2\delta_{l0}$. {\bf 2.} $\sum_{k\in\Z}|p_{k}|^2=2$. {\bf 3.} $\sum_{k\in\Z}p_{k}=2$. {\bf 4.} $\sum_{k\in\Z}p_{2k}=1$ and $\sum_{k\in\Z}p_{2k+1}=1$.
{\bf Theorem 5.10:} Suppose $\{V_j; j\in\Z\}$ is a multiresolution analysis with scaling function $\phi = \sum_{k}p_{k}\phi(2x-k)$.  Let $W_j$ be the span of $\{\psi(2^jx-k); k\in\Z\}$ where $\psi(x) = \sum_{k\in\Z}(-1)^{k}\overline{p_{1-k}}\phi(2x-k)$. Then, $W_j\subset V_{j+1}$ is the orthogonal complement of $V_j$ in $V_{j+1}$. Furthermore, $\{\psi_{jk}(x) := 2^{j/2}\psi(2^jx - k); k\in\Z\}$ is an orthonormal basis for the $W_j$.
{\bf Theorem 5.11:} Let $\{V_j, j\in\Z\}$ be a multiresolution analysis with scaling function $\phi$. Let $W_j$ be the orthogonal complement of $V_j$ in $V_{j+1}$. Then $L^2(\R)=\dots\bigoplus W_{-1}\bigoplus W_{0}\bigoplus W_{1}\bigoplus\dots$. In particular, each $f\in L^2(\R)$ can be uniquely expressed as a sum $\sum_{k\in\Z}w_k$ with $w_k\in W_k$ and where the $w_k$'s are mutually orthogonal. Equivalently, the set of all wavelets, $\{\psi_{jk}\}_{j,k\in\Z}$, is an orthonormal basis for $L^2(\R)$.

\underline{\textsl{\bf DECOMPOSITION}}
{\bf Orthogonal Form:} $a^{j-1}_l = 2^{-1}\sum_{k\in\Z}\overline{p_{k-2l}}a_k^j$ and $b^{j-1}_l = 2^{-1}\sum_{k\in\Z}(-1)^kp_{1-k+2l}a_k^j$
{\bf ON Form:} $\left\langle f,\phi_{j-1,l}\right\rangle = 2^{-1/2}\sum_{k\in\Z}\overline{p_{k-2l}}\left\langle f,\phi_{jk}\right\rangle$ and $\left\langle f,\psi_{j-1,l}\right\rangle = 2^{-1/2}\sum_{k\in\Z}(-1)^kp_{1-k+2l}\left\langle f,\phi_{jk}\right\rangle$

\underline{\textsl{\bf RECONSTRUCTION}}
{\bf Orthogonal Form:} $a^{j}_k = \sum_{l\in\Z}p_{k-2l}a_l^{j-1}+\sum_{l\in\Z}(-1)^k\overline{p_{1-k+2l}}b_l^{j-1}$
{\bf ON Form:} $\left\langle f,\phi_{jk}\right\rangle = 2^{-1/2}\sum_{l\in\Z}p_{k-2l}\left\langle f,\phi_{j-1,l}\right\rangle+2^{-1/2}\sum_{l\in\Z}(-1)^k\overline{p_{1-k+2l}}\left\langle f,\psi_{j-1,l}\right\rangle$
\section*{APPENDIX}
\input{Apendix}

\end{multicols}
\end{document}


\section{The declarative formulation}
\label{sec:declarative-formulation}

In this section we give the formulation of type theory in a declarative way
which minimizes the number of judgments, is better suited for a semantic account, but is
not susceptible to an algorithmic treatment.

\subsection{Syntax}
\label{sec:syntax}

Contexts:
%
\begin{equation*}
  \G
  \begin{aligned}[t]
    \bnf   {}& \ctxempty & & \text{empty context}\\
    \bnfor {}& \ctxextend{\G}{\x}{\T} & & \text{context $\G$ extended with $\x : \T$}
  \end{aligned}
\end{equation*}
%
Terms ($\e$) and types $(\T, \U)$:
%
\begin{equation*}
  \e, \T, \U
  \begin{aligned}[t]
    \bnf   {}& \Type & & \text{universe}\\
    \bnfor {}& \Prod{x}{\T} \U & & \text{product}\\
    \bnfor {}& \JuEqual{\T}{\e_1}{\e_2} & & \text{equality type} \\
    \bnfor {}&  \x   &&\text{variable} \\
    \bnfor {}&  \lam{\x}{\T_1}{\T_2} \e  &&\text{$\lambda$-abstraction} \\
    \bnfor {}&  \app{\e_1}{\x}{\T_1}{\T_2}{\e_2}  &&\text{application} \\
    \bnfor {}&  \juRefl{\T} \e  &&\text{reflexivity} \\
  \end{aligned}
\end{equation*}
%
Note that $\lambda$-abstraction and application are tagged with extra types not usually
seen in type theory. An abstraction $\lam{\x}{\T_1}{\T_2} \e$ speficies not only the type
$\T_1$ of $\x$ but also the type $\T_2$ of $e$, where $\x$ is bound in $\T_2$ and $\e$.
Similarly, an application $\app{\e_1}{\x}{\T_1}{\T_2}{\e_2}$ specifies that $\e_1$ and
$\e_2$ have types $\Prod{\x}{\T_1} \T_2$ and $\T_2$, respectively. This is necessary
because in the presence of exotic equalities (think ``$\mathsf{nat} \to \mathsf{bool}
\equiv \mathsf{nat} \to \mathsf{nat}$'') we must be \emph{very} careful about
$\beta$-reductions.

The annotations on an application matter also for determining when two
terms are equal. For example, if $X,Y : \Type$, $f : \mathsf{nat}\to X$ and $$e
: \JuEqual{\Type}{\mathsf{nat}{\to}X}{\mathsf{nat}{\to}Y},$$ then
$(\app{f}{\_}{\mathsf{nat}}{X} 0) : X$ and $(\app{f}{\_}{\mathsf{nat}}{Y} 0) : Y$,
so the two identical-but-for-annotations terms have different types and thus
cannot be equivalent.

\subsection{Judgments}
\label{sec:judgments}

\begin{align*}
& \isctx{\G} & & \text{$\G$ is a well formed context} \\
& \isterm{\G}{\e}{\T} & & \text{$\e$ is a well formed term of type $\T$ in context $\G$} \\
& \eqterm{\G}{\e_1}{\e_2}{\T} & & \text{$e_1$ and $e_2$ are equal terms of type $\T$ in context $\G$}
\end{align*}
%
The judgement ``$\T$ is a type in context $\G$'' is a special case of term formation, namely
$\istype{\G}{\T}$. Similarly, equality of types is just equality of terms at $\Type$.

\subsection{Contexts}
\label{sec:contexts}

\begin{mathpar}
  \infer[\rulename{ctx-empty}]
  { }
  {\isctx{\ctxempty}}

  \infer[\rulename{ctx-extend}]
  {\isctx{\G} \\
   \istype{\G}{\T}
  }
  {\isctx{\ctxextend{\G}{\x}{\T}}}
\end{mathpar}

% In rules that extend the context, we leave implicit the premise that the extended context be well formed.
% XXX Is this covered by the term-var rule?

\subsection{Terms and types}

\paragraph{General rules}
\begin{mathpar}
  \infer[\rulename{term-conv}]
  {\isterm{\G}{\e}{\T} \\
   \eqtype{\G}{\T}{\U}
  }
  {\isterm{\G}{\e}{\U}}

  \infer[\rulename{term-var}]
  {\isctx{\G} \\
   (\x{:}\T) \in \G
  }
  {\isterm{\G}{\x}{\T}}
\end{mathpar}

\paragraph{Universe}

\begin{mathpar}
  \infer[\rulename{ty-type}]
  {\isctx{\G}
  }
  {\istype{\G}{\Type}}
\end{mathpar}

\paragraph{Products}

\begin{mathpar}
  \infer[\rulename{ty-prod}]
  {\istype{\G}{\T} \\
   \istype{\ctxextend{\G}{\x}{\T}}{\U}
  }
  {\istype{\G}{\Prod{\x}{\T}{\U}}}

  \infer[\rulename{term-abs}]
  {\isterm{\ctxextend{\G}{\x}{\T}}{\e}{\U}}
  {\isterm{\G}{(\lam{\x}{\T}{\U}{\e})}{\Prod{\x}{\T}{\U}}}

  \infer[\rulename{term-app}]
  {\isterm{\G}{\e_1}{\Prod{x}{\T} \U} \\
   \isterm{\G}{\e_2}{\T}
  }
  {\isterm{\G}{\app{\e_1}{\x}{\T}{\U}{\e_2}}{\subst{\U}{\x}{\e_2}}}
\end{mathpar}

\paragraph{Equality types}
\label{sec:equality}

\begin{mathpar}
  \infer[\rulename{ty-eq}]
  {\istype{\G}{\T}\\
   \isterm{\G}{\e_1}{\T}\\
   \isterm{\G}{\e_2}{\T}
  }
  {\istype{\G}{\JuEqual{\T}{\e_1}{\e_2}}}

  \infer[\rulename{term-refl}]
  {\isterm{\G}{\e}{\T}}
  {\isterm{\G}{\juRefl{\T} \e}{\JuEqual{\T}{\e}{\e}}}
  \end{mathpar}

\subsection{Equality}

\paragraph{General rules}

\begin{mathpar}
  \infer[\rulename{eq-refl}]
  {\isterm{\G}{\e}{\T}}
  {\eqterm{\G}{\e}{\e}{\T}}

  \infer[\rulename{eq-sym}]
  {\eqterm{\G}{\e_2}{\e_1}{\T}}
  {\eqterm{\G}{\e_1}{\e_2}{\T}}

  \infer[\rulename{eq-trans}]
  {\eqterm{\G}{\e_1}{\e_2}{\T}\\
   \eqterm{\G}{\e_2}{\e_3}{\T}}
  {\eqterm{\G}{\e_1}{\e_3}{\T}}

  \infer[\rulename{eq-conv}]
  {\eqterm{\G}{\e_1}{\e_2}{\T}\\
    \eqtype{\G}{\T}{\U}}
  {\eqterm{\G}{\e_1}{\e_2}{\U}}
\end{mathpar}

\paragraph{Equality reflection}
%
\begin{mathpar}
  \infer[\rulename{eq-reflection}]
  {\isterm{\G}{\e}{\JuEqual{\T}{\e_1}{\e_2}}}
  {\eqterm{\G}{\e_1}{\e_2}{\T}}
\end{mathpar}

\paragraph{Computations}

\begin{mathpar}
\infer[\rulename{prod-beta}]
  {\eqtype{\G}{\T_1}{\U_1}\\
    \eqtype{\ctxextend{\G}{\x}{\T_1}}{\T_2}{\U_2}\\\\
    \isterm{\ctxextend{\G}{\x}{\T_1}}{\e_1}{\T_2}\\
    \isterm{\G}{\e_2}{\U_1}}
  {\eqterm{\G}{\bigl(\app{(\lam{\x}{\T_1}{\T_2}{\e_1})}{\x}{\U_1}{\U_2}{\e_2}\bigr)}
              {\subst{\e_1}{\x}{\e_2}}
              {\subst{\T_2}{\x}{\e_2}}}
\end{mathpar}

\paragraph{Extensionality}

%
\begin{mathpar}
  \infer[\rulename{eq-eta}]
  {\isterm{\G}{\e'_1}{\JuEqual{\T}{\e_1}{\e_2}} \\
    \isterm{\G}{\e'_2}{\JuEqual{\T}{\e_1}{\e_2}}
  }
  {\eqterm{\G}{\e'_1}{e'_2}{\JuEqual{\T}{\e_1}{\e_2}}}

  \infer[\rulename{prod-eta}]
  {\isterm{\G}{\e_1}{\Prod{\x}{\T}{\U}}\\
   \isterm{\G}{\e_2}{\Prod{\x}{\T}{\U}}\\\\
   \eqterm{\ctxextend{\G}{\x}{\T}}{(\app{\e_1}{\x}{\T}{\U}{\x})}
          {(\app{\e_2}{\x}{\T}{\U}{\x})}{\U}
  }
  {\eqterm{\G}{\e_1}{\e_2}{\Prod{\x}{\T}{\U}}}
\end{mathpar}

\subsubsection{Congruences}

\paragraph{Type formers}

\begin{mathpar}
  \infer[\rulename{cong-prod}]
  {\eqtype{\G}{\T_1}{\U_1}\\
   \eqtype{\ctxextend{\G}{\x}{\T_1}}{\T_2}{\U_2}}
  {\eqtype{\G}{\Prod{\x}{\T_1}{\T_2}}{\Prod{\x}{\U_1}{\U_2}}}

  \infer[\rulename{cong-eq}]
  {\eqtype{\G}{\T}{\U}\\
   \eqterm{\G}{\e_1}{\e'_1}{\T}\\
   \eqterm{\G}{\e_2}{\e'_2}{\T}
  }
  {\eqtype{\G}{\JuEqual{\T}{\e_1}{\e_2}}
              {\JuEqual{\U}{\e'_1}{\e'_2}}}
\end{mathpar}

\paragraph{Products}

\begin{mathpar}
  \infer[\rulename{cong-abs}]
  {\eqtype{\G}{\T_1}{\U_1}\\
    \eqtype{\ctxextend{\G}{\x}{\T_1}}{\T_2}{\U_2}\\
    \eqterm{\ctxextend{\G}{\x}{\T_1}}{\e_1}{\e_2}{\T_2}}
  {\eqterm{\G}{(\lam{\x}{\T_1}{\T_2}{\e_1})}
              {(\lam{\x}{\U_1}{\U_2}{\e_2})}
              {\Prod{\x}{\T_1}{\T_2}}}

  \infer[\rulename{cong-app}]
  {\eqtype{\G}{\T_1}{\U_1}\\
   \eqtype{\ctxextend{\G}{\x}{\T_1}}{\T_2}{\U_2}\\\\
   \eqterm{\G}{\e_1}{\e'_1}{\Prod{\x}{\T_1}{\T_2}}\\
   \eqterm{\G}{\e_2}{\e'_2}{\T_1}}
  {\eqterm{\G}{(\app{\e_1}{\x}{\T_1}{\T_2}{\e_2})}{(\app{\e'_1}{\x}{\U_1}{\U_2}{\e'_2})}{\subst{\T_2}{\x}{\e_2}}}
\end{mathpar}

\paragraph{Equality types}

%
\begin{mathpar}
\infer[\rulename{cong-refl}]
{\eqterm{\G}{\e_1}{\e_2}{\T}\\
 \eqtype{\G}{\T}{\U}}
{\eqterm{\G}{\juRefl{\T} \e_1}{\juRefl{\U} \e_2}{\JuEqual{\T}{\e_1}{\e_1}}}
\end{mathpar}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\subsection{Comparison of Fit vs Non-Fit}

The method proposed by P. Hudzovic is compared  to  itself  with  and  without a
least squared fit of the parameters $T$ and  $r$. Similarly, the method proposed
by  L.  Sani is also compared to itself with and without a least squared fit  of
$T$ and $r$.

The input curve in Sani's case  was  a 4th order Hudzovic transfer function with
$r=\frac{1}{6}$ and $T=1$.  The  input  curve in Hudzovic's case was a 4th order
Sani transfer function with $r=0.5$  and  $T=1$.

The reason for choosing an ``unsuitable'' input curve was to  make it impossible
for the method  under  test  to  calculate a perfectly correct transfer function
such that the error would be 0.

The test was performed twice for  each method, once with no input noise and once
with  random noise (20\% normalised amplitude) modulated onto the input  signal.

\begin{figure}[t]
    \includegraphics[width=\linewidth]{images/hudzovic_vs_fit}
    \caption{Comparison of a normal Hudzovic Tu/Tg lookup to a least square refinement. Order n=4, Noise=20\% and 0\%}
    \label{fig:hudzovic_vs_fit}
\end{figure}

\begin{figure}[t]
    \includegraphics[width=\linewidth]{images/sani_vs_fit}
    \caption{Comparison of a normal Sani Tu/Tg lookup to a least square refinement. Order n=4, Noise=20\% and 0\%}
    \label{fig:sani_vs_fit}
\end{figure}

The   results   can   be   seen   in   figures   \ref{fig:hudzovic_vs_fit}   and
\ref{fig:sani_vs_fit}. The blue circles show the raw  input  data  to the method
under test. The red  curve  shows  the  step  response of the resulting transfer
function \textit{without} fitting and the yellow curve  shows  the step response
\textit{with} a least squares fit.

Further, the two purple  curves  on  a  second  Y  axis show the \textit{squared
error}  of the output function to the original non-noisy input function of  each
data point. The dashed purple line is the error of the fitted result whereas the
continuous purple line is the error of the non-fitted result.

It is fairly clear that in all  four  cases  the  least  square  fit yields more
accurate results. The maximum squared error of the non-fitted methods (with 20\%
input  noise)  is  around $2x10^{-3}$, whereas the maximum squared error of  the
fitted  methods  is  around  $1x10^{-4}$. This leads to an improvement factor of
about 5.


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}

\title{MATH 542 Homework 5}
\author{Saket Choudhary\\skchoudh@usc.edu}

\begin{document}
\maketitle
\section*{Problem 1}

%
%    Miscellaneous Exercise 1: #2 on page 15
%    Exercise 2a: #1 and #3(b) on page 19
%    Exercise 2b: #2, #3, #6, #8 on page 23-24
\subsection*{Problem 1a.}
Using variance-covariance expansion:

\begin{align*}
Var(X_1-2X_2+X_3) &= Var(X_1) + Var(-2X_2) + Var(X_3) + 2Cov(X_1,-2X_2) + 2Cov(-2X_2,X_3) + 2Cov(X_3,X_1)\\
&= Var(X_1) + 4Var(X_2) + Var(X_3) -4 Cov(X_1,X_2) -4Cov(X_2,X_3) + 2Cov(X_3,X_1)\\
&= 5+4(3)+3-4(2) -4(0) + 2(3)\\
&= 18
\end{align*}

\begin{align*}
Y &= \begin{pmatrix} Y_1\\ Y_2 \end{pmatrix}\\
&= \begin{pmatrix}
X_1+X_2\\
X_1+X_2+X_3\\
\end{pmatrix}\\
&= \begin{pmatrix}
1 & 1 & 0\\
1 & 1 & 1
\end{pmatrix} \begin{pmatrix}
X_1\\
X_2\\
X_3
\end{pmatrix}
\end{align*}

Now using $Var(AX) = AVar(X)A'$

\begin{align*}
Var(Y) &= \begin{pmatrix}
1 & 1 & 0\\
1 & 1 & 1
\end{pmatrix} Var(X) \begin{pmatrix}
1 & 1 \\
1 & 1 \\
0 & 1\\
\end{pmatrix}\\
&= \begin{pmatrix}
12 & 15 \\
15 & 21
\end{pmatrix}
\end{align*}

\section*{Ex2a Problem 1}

$f(y_1,y_2) = k^{-1}\exp(-\frac{1}{2}(2y_1^2+y_2^2+2y_1y_2-22y_1-14y_2+65))$

\begin{align*}
2y_1^2+y_2^2+2y_1y_2-22y_1-14y_2+65 &= \begin{pmatrix}y_1-\mu_1 & y_2-\mu_2\end{pmatrix} \begin{pmatrix}
a & b\\
b & c
\end{pmatrix} \begin{pmatrix}
y_1-\mu_1\\
y_2-\mu_2
\end{pmatrix}\\
&= a(y_1-\mu_1)^2+2b(y_1-\mu_1)(y_2-\mu_2) + c(y_2-\mu_2)^2\\
&= ay_1^2+cy_2^2+2by_1y_2-y_1(2a\mu_1+2b\mu_2)-y_2(2b\mu_1+2c\mu_2) + (a\mu_1^2+2b\mu_1\mu_2+c\mu_2^2)
\end{align*}

Now comparing the coefficient of $y_1^2$ $\implies\ a=2$

Comparing coefficient of $y_2^2\ \implies\  c=1$

Comparing coefficient of $y_1y_2\ \implies \ b=1$

Comparing coefficient of $y_1 \implies \  4\mu_1+2\mu_2=22$

Comparing coefficient of $y_2 \implies \  2\mu_1+2\mu_2=14$

Thus, $\mu_1 = 4$ and $\mu_2=3$

Check: $4\mu_1^2 + 2\mu_1\mu_2+\mu_2^2 = 2(16)+24+9 = 65$

and hence $\Sigma^{-1} = \begin{pmatrix} 2 & 1\\ 1 & 1 \end{pmatrix}$

$det(\Sigma^{-1}) = 1$

$\Sigma = \begin{pmatrix}
1 & -1\\
-1 & 2
\end{pmatrix}$

Thus, $k^{-1} = \frac{1}{\sqrt{2\pi det(\Sigma)}}^{2/2} = \frac{1}{2\pi}$
Thus, $k = 2\pi$
\subsection*{2a Problem 1b}

\begin{align*}
E[Y] & = \begin{pmatrix}
\mu_1\\
\mu_2
\end{pmatrix}\\
&= \begin{pmatrix}
4\\
3
\end{pmatrix}
\end{align*}

\begin{align*}
Var[Y] & = \begin{pmatrix}
a & b\\
b & c
\end{pmatrix}\\
&= \begin{pmatrix}
1 & -1\\
-1 & 2
\end{pmatrix}
\end{align*}

\section*{Ex2a Problem 3(b)}
\begin{align*}
\Sigma &= \begin{pmatrix}
1 & \rho \\
\rho & 1
\end{pmatrix}
\end{align*}

Determining eigen values:
\begin{align*}
det(\Sigma-\lambda I) &=0 \\
(1-\lambda)^2 = \rho^2 \\
\lambda &= 1\pm \rho
\end{align*}

And the corresponding eigen values:
\begin{align*}
\begin{pmatrix}
1 & \rho \\
\rho & 1
\end{pmatrix}\begin{pmatrix} v_1 \\ v_2  \end{pmatrix} &= \lambda \begin{pmatrix}
v_1\\ v_2
\end{pmatrix}
\end{align*}

One set of eigen vectors are given by: for $\lambda_1=1+\rho$: $\frac{1}{\sqrt{2}}\begin{pmatrix}1\\1 \end{pmatrix}$
and for $\lambda_2=1-\rho$: $\frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -1 \end{pmatrix}$

Thus using eigen decomposition $\Sigma$ can be rewritten as:
\begin{align*}
\Sigma &= A\Lambda A'\\
&=\frac{1}{2}\begin{pmatrix}
1 & 1\\
1 & -1
\end{pmatrix}\begin{pmatrix}
1+\rho & 0\\
0 & 1-\rho
\end{pmatrix}\begin{pmatrix}
1 & 1\\
1 & -1
\end{pmatrix}
\end{align*}

And hence $\Sigma^{1/2} = A\Lambda^{1/2} A' = \frac{1}{2}\begin{pmatrix}
\sqrt{1+\rho} + \sqrt{1-\rho} & \sqrt{1+\rho} - \sqrt{1-\rho}\\
\sqrt{1+\rho} - \sqrt{1-\rho} & \sqrt{1+\rho} + \sqrt{1-\rho}
\end{pmatrix}$

\section*{Ex2b Problem 2}

$Y_i = \begin{pmatrix} 0 & 0 & \dots & 1_i & 0 \dots 0 \end{pmatrix}Y =a_i'Y$

Since $Y_i \sim N(\mu, \Sigma)$ Using Theorem 2.2
$Y_i \sim N(a_i'\mu, a_i'\Sigma a_i) = N(\mu_i, \sigma_{ii})$

\section*{Ex2b Problem 3}
Since $Y_1,Y_2,Y_3$ and $Y_1-Y_2$ are both normal, their joint distribution is normal too.
Consider: $A = \begin{pmatrix}
1 & 1 & 1\\
1 & -1 & 0
\end{pmatrix}$

Now, $Z = \begin{pmatrix}
Z_1\\
Z_2
\end{pmatrix} = AY$

and hence $Z \sim N(A\mu, A\Sigma A')$
$A\mu = \begin{pmatrix}
5\\
1
\end{pmatrix}$

$A\Sigma A' = \begin{pmatrix}
10 & 0\\
0 & 3
\end{pmatrix}$

Since $Z_1$ and $Z_2$ are normal, and they are independent $\sigma_{12}=0$ so the joint distribution is given by their product.

$\sigma_1^2 = 10$ ; $\sigma_2^2=3$
$\mu_1 = 5$, $\mu_2 =1$

\begin{align*}f(Z_1,Z_2) &= \frac{1}{2\pi \sigma_1 \sigma_2}\exp(-\frac{(Z_1-\mu_1)^2}{2\sigma_1^2}-\frac{(Z_2-\mu_2)^2}{2\sigma_2^2})\\
\end{align*}
\section*{Ex2b Problem 6}

Define $U_1 = Y_1+Y_2$
and $U_2 = Y_1-Y_2$ where $U_i \sim N(0,1)$

$Cov(U_1,U_2) = 0$

Rearranging gives:
\begin{align*}
Y_1 &= \frac{1}{2}(U_1+U_2)\\
Y_2 &= \frac{1}{2}(U_1-U_2)\\
\end{align*}

Thus, $Y_i \sim N(0, \frac{1}{4}$

Since any vector $a'Y$ has  a univariate normal distribution(mean=0) using Theorem 2.3, we see that $Y \sim N(\mu, \Sigma)$ where

\begin{align*}
Y &= \begin{pmatrix}
Y_1\\Y_2 \end{pmatrix}\\
\mu &= \begin{pmatrix}
\mu_1\\
\mu_2
\end{pmatrix}\\
&= \begin{pmatrix}
0\\
0
\end{pmatrix}\\
\end{align*}

To find $\Sigma$:

\begin{align*}
Cov(U_1,U_2) &= 0\\
Cov(Y_1+Y_2, Y_1-Y_2) &=0\\
Cov(Y_1,Y_1)+Cov(Y_1, Y_2)+Cov(Y_2,Y_1)+Cov(Y_2, -Y_2) &= 0\\
\sigma_{11}+2\sigma_{12}-\sigma_{22} &=0
\implies \sigma_{12} &=0 \text{ using } \sigma_{11} = \sigma_{22} =1
\end{align*}
Thus, $Y_1,Y_2$ have a bivariate normal ditribution. with $\mu= \begin{pmatrix}
0\\
0
\end{pmatrix}$ and $\Sigma = \begin{pmatrix}
1 & 0\\
0 & 1
\end{pmatrix}$

\section*{Ex2b Problem 8}

\begin{align*}
\begin{pmatrix} \bar{Y} & Y_1-\bar{Y}& Y_2-\bar{Y_3} \dots Y_n-\bar{Y}\end{pmatrix}' &= \begin{pmatrix}
1/n & 1/n & 1/n & \dots & 1/n\\
1-1/n & -1/n & -1/n & \dots & -1/n\\
1 & 1-1/n & -1/n & \dots & -1/n\\
\vdots \\
-1/n & -1/n & -1/n & \dots & 1-1/n\\
\end{pmatrix} \begin{pmatrix} Y_1 & Y_2 & Y_3 &\dots Y_n \end{pmatrix}'\\
%&= (I-\frac{1}{n}1_n1_n')\begin{pmatrix} Y_1 & Y_2 & Y_3 &\dots Y_n \end{pmatrix}'
Z &= AY
\end{align*}

Also $Z \sim N(A\mu, A\Sigma A')$

\begin{align*}
A\Sigma A' &=  AA' \text{ since } \Sigma = I\\
&= \begin{pmatrix}
\frac{n}{n^2} & 0 & 0 & \dots & 0 \\
0 & (1-\frac{1}{n})^2+\frac{n}{n^2}  &  -2/n(1-1/n) + \frac{n-2}{n} & \dots & -2/n(1-1/n) + \frac{n-2}{n}\\
\vdots \\
0 & -2/n(1-1/n) + \frac{n-2}{n} & -2/n(1-1/n) + \frac{n-2}{n} & \dots & (1-\frac{1}{n})^2+\frac{n}{n^2}\\
\end{pmatrix}\\
&= \begin{pmatrix}
\frac{1}{n} & 0 & 0 & 0 & \dots & 0\\
0 & \frac{1}{n}+(1-\frac{1}{n})^2 & -\frac{1}{n} & -\frac{1}{n} &\dots & -\frac{1}{n}\\
\vdots\\
0 & -\frac{1}{n} & -\frac{1}{n} & -\frac{1}{n} & \dots & \frac{1}{n}+(1-\frac{1}{n})^2
\end{pmatrix}\\
&=B
\end{align*}

%where $A=I-\frac{1}{n}1_n1_n'$

Thus M.g.f. of $Z=AY$ is (using Theorem 2.2 with $d=0$)

\begin{align*}
E[\exp(t'AY)] &= \exp(t'A\mu + \frac{1}{2}t'A\Sigma A't)\\
&= \exp(t'A\mu + \frac{1}{2}t'AA't)) \text{ using } \Sigma = I\\
&=\exp(t'A\mu + \frac{1}{2}t'Bt))
\end{align*}

And hence $Z=\begin{pmatrix} \bar{Y} & Y_1-\bar{Y}& Y_2-\bar{Y_3} \dots Y_n-\bar{Y}\end{pmatrix}'$ follows a multivariate distribution such that $Cov(\bar{Y}, Y_i-\bar{Y}) = 0$ $\implies$ $\bar{Y}$ and $Y_i-\bar{Y}$ are independent (for all $i$)

Let's call $X=\begin{pmatrix} Y_1-\bar{Y}& Y_2-\bar{Y_3} \dots Y_n-\bar{Y}\end{pmatrix}'$

Then, from above we have $\bar{Y}$ and $X$ are indepedent (also follows from rom theorem 2.4)

Then,

$$\sum_i(Y_i-\bar{Y})^2 = X'X$$

Since $\bar{Y},X$ are independent $\implies $ $\bar{Y},X'X$ are independent

\end{document}


\chapter{Models}

Typing sentences can be seen as a Markovian process. To correct sequences using HMMs, we can assume the
hidden states to be representing the intended words, while the observations are the typed words.
We will consider two correction tasks. The first one chooses the states that are individually most likely, maximising
the expected number of correct individual states. The model that implements this criterion is the Noisy Channel
Model~\cite{martin2009speech}. The second criterion estimates the most likely state sequence, or \textit{trellis path}.
The model used to implement this criterion is the Hidden Markov Model with the Viterbi
algorithm~\cite{tarniceriu2015hmm}.

\section{Noisy Channel Model}
\label{section:ncm}
The \textbf{error model} implemented in this project is called \textit{Noisy Channel Model}. In this model we treat the
original word (the misspelled word) as if a correctly spelled word had been “distorted” by passing through a noisy
communication channel. This channel introduces “noise” in the form of substitutions and other changes to the letters,
making it hard to recognise the “true” word.

We use a slightly different approach when the word typed by the user does not belong to our dictionary (non-word
errors) than when it does (real-word errors).

To correct \textbf{non-word errors} we first generate a list of \textbf{candidates}, real words with a similar letter
sequence to the typed text. We search for candidates up to a certain edit distance, given as a parameter to the model
(\texttt{edit\_distance}). We evaluated distances 1 and 2 in our experiments, since the run time of the algorithm grows
exponentially with respect to this parameter. Furthermore, most of the errors occurring in real-world usage are at
distance 1 from the intended word.

This noisy channel model is, therefore, a kind of Bayesian inference.
Having observed a misspelled word $x$, we want to find the intended word $w$ that generated this observation.
Out of all possible words in our language model $L$ we want to find the word $\hat{w}$ that maximise the probability
$P(\hat{w}|x)$:
\begin{equation}\label{eq:4.1}
\hat{w} = \arg\max_{w \in L} P(w|x) \mbox{.}
\end{equation}

Applying Bayes's rule, factoring out the denominator, since $P(x)$ doesn’t change for each word because we are always
asking about the most likely word for the same observed error $x$,  and limiting $L$ such that we're going to consider
only a set of candidates $C$ at a maximum edit distance for the word $x$, we get the formula:
\begin{equation}\label{eq:4.2}
\hat{w} = \arg\max_{w \in C} {P(x|w)P(w)} \mbox{.}
\end{equation}

The distance used to limit the set of candidates $C$ is the \textbf{Levenshtein} edit distance~\cite{martin2009speech}.

\begin{figure}[H]
	\centering
	\includegraphics[width=10cm]{NoisyChannel.png}
	\caption{Diagram of the Noisy Channel Model}
	\label{fig:noisychannel}
\end{figure}

Once obtained a set of candidates, the formula \ref{eq:4.2} requires us to compute the two components,
respectively, the prior probability of a hidden word $w$ and the channel model (or likelihood).
The prior probability $P(w)$ is given by the language model, that is obtained by counting the frequencies of
each word in a corpus of text.
The channel model $P(x|w)$ used in this project makes the assumption that $P(\mbox{balbo}|\mbox{bilbo}) =
P(\mbox{a}|\mbox{i})$. The channel error model is trained on a corpus of spelling errors coming from different
datasets. In particular, it is estimated just using the number of times that the a letter $i$ was substituted for the
letter $a$.

Considering each character of an alphabet $A$, generally, we'll have a confusion matrix $|A|\times|A|$ for each
kind of channel model we're going to use. The following are the channel models used in this project:
\begin{itemize}
	\item \textsc{character deletions}: $\mbox{del}[x, y] = \frac{\mbox{count (xy  typed as x)}}
	{\mbox{count (xy)}}$
	\item \textsc{character insertions}: $\mbox{ins}[x, y] = \frac{\mbox{count (x  typed as xy)}}
	{\mbox{count (x)}}$
	\item \textsc{substitution of characters}: $\mbox{sub}[x, y] = \frac{\mbox{count (x  typed as y)}}
	{\mbox{count (x)}}$
	\item \textsc{transposition of adjacent characters}: $\mbox{swap}[x, y] = \frac{\mbox{count (xy  typed as
	yx)}}
	{\mbox{count (xy)}}$
\end{itemize}

This model is appropriate for estimating the likelihood of \textbf{non-word spelling errors}, or errors were the
misspelled word isn't in the vocabulary (e.g. writing \textsl{giraffe} as \textsl{graffe}).
When no candidates are found, the model keeps the original word as the only candidate.

\textbf{Real-word errors}, or errors were the misspelled word is in the vocabulary (e.g. writing \textsl{work}  as
\textsl{worm}), need a slightly different approach.

We're still searching for the candidate that maximizes formula \ref{eq:4.2}, but the channel model is treated
differently.
We need to assume, since the word is in the vocabulary, that the input word is not necessarily an error. We will call
$P(w|w)$ as $\alpha$. We can make various assumptions about what the value of this measure should be,
according to situation the text was written in. For example, we could choose an alpha of $0.99$ for a carefully edited
text, while casually texting someone has an alpha of $0.80$.

So, given a typed word $x$, let the channel model $P(x|w)$ be alpha when $x = w$, and then distribute the remaining
$1-\alpha$ evenly over all other candidates $C(x)$.

\begin{equation}\label{eq:4.3}
	P(x|w) = \begin{cases}
	\alpha & \mbox{if } x = w \\
	\frac{1-\alpha}{|C(x)|} & \mbox{if }  x \in C(x) \\
	0 & \mbox{otherwise}
	\end{cases}
\end{equation}

We'll then replace the edit probability of the various confusion matrices for non-word spelling errors with an
equal distribution of $1-\alpha$, while keeping the logic of the model intact.

\section{Hidden Markov Model}
\label{section:hmm}
A \textbf{Hidden Markov Model} (HMM) allows us to talk about both observed events, like misspelled words that
we see in the input, and hidden events, like the intended words, that we think of as causal factors in our
probabilistic model.

Our HMM is specified by the following components:
\begin{itemize}
	\item $Q = q_1q_2 \dots q_N$: a set of $N$ \textbf{states}
	\item $A=a_{11}	\dots a_{ij} \dots a_{NN}$: a \textbf{transition probability matrix} $A$. \\ Each $a_{ij}$
	representing the probability of moving from state $i$ to state $j$, such that $\sum_{j=1}^N a_{ij}=1 \quad
	\forall i$. In our case, the state transitions are given by the probability of one word given its predecessor,
	obtained from a certain corpus of text.
	\item $B = b_i (o_t )$: a sequence of \textbf{observation likelihoods}, also called \textbf{emission
		probabilities}, each expressing the probability of an observation to being generated from a state $i$. These
	are calculated as $P(q_i|o)$, where $q_i$ represents the \textbf{intended} word for $o$, that is the \textbf{observed} misspelling.
	\item $\pi = \pi_1,\pi_2,\dots,\pi_N$: an \textbf{initial probability distribution} over states. $\pi_i$ is the
	probability that the Markov chain will start in state $i$.
\end{itemize}

\begin{figure}[H]
	\centering
	\includegraphics[width=10cm]{hmm.png}
	\caption{Illustration of Hidden Markov Model}
	\label{fig:hmm}
\end{figure}

We consider a \textit{first-order} Hidden Markov Model, since it allows us to make two simplifying assumptions. First,
as with a first-order Markov chain, the probability of a particular state depends only on the previous state. Second,
the probability of an output observation $o_i$ depends only on the state that produced the observation $q_i$
and not on any other states or any other observations.

For every observed word, we will consider a subset of all possible words as its correct candidates, given by
the error model described in Section~\ref{fig:noisychannel}.

Typing being a sequential process, as the HMM proceeds from state to state, we will also have to limit the
candidates generated by each observation with only those having an actual transition from state $Q_{i-1}$ to
$Q_i$.

In Figure~\ref{fig:trellis} we present an example of the evolution of our HMM.
\begin{figure}[H]
	\centering
	\includegraphics[width=15cm]{TrellisExample.png}
	\caption{Example of a trellis graph generated by our model}
	\label{fig:trellis}
\end{figure}
In this example, the most likely state sequence is “\textsl{someone else always has to carry on the \textbf{starry}}”.
The algorithm partially fails in this case, since the intended sentence was “\textsl{someone else always has to carry
on the story}”.

\section{Most Likely State Sequence}

The Viterbi algorithm calculates the most likely sequence of hidden states, the intended words. It is a probabilistic
extension of the minimum edit distance. Instead of computing the “minimum edit distance” between two strings, Viterbi
computes the “maximum probability alignment” of one string with another.

The initial probability of being in a state $i$, $\pi_i$, in our case the probability of intend a word $i$, and the
transition probabilities $A_{ij}$, or the transition from the word $i$ to the next word $j$, are given. Since we have
observed the output $y_1, y_2, \dots , y_t$, that is the sentence written with typos, it is possible to compute the
most likely state sequence $x_1, x_2, \dots , x_t$ starting from the following expression:

\begin{equation}
\begin{aligned}
V_{1,t+1} &= P(x_1, \dots, x_t, x_{t+1}, y_1, \dots, y_t,  y_{t+1}) = \\
&= \arg\max_{x_{1:t}} p(x_1, \dots, x_t | y_1, \dots, y_t) = \\
& =  \alpha \cdot p(y_{t+1}|x_{t+1})\cdot\max_{x_t} \Big( p(x_{t+1}|x_t) \max p(x_1, \dots, x_{t}|y_1,
\dots, y_t)\Big)
\end{aligned}
\end{equation}

The initial state probabilities $\pi$ are computed from the word frequencies in the reference text corpus, the state
transition probabilities are the probability of a word given its predecessor, and the emission probabilities are the
probabilities to type word $i$ when word $j$ was intended.

In our implementation, we construct the \textit{trellis} choosing the state-to-state connection with the local maximal
probability. We decided to implement the \textbf{Viterbi} algorithm instead of the Forward-Backward algorithm, given
the results obtained by experiments carried out in the literature.
The HMM-Based Error Correction Mechanism for Five-Key Chording Keyboards article~\cite{tarniceriu2015hmm} explains that
the Forward-Backward algorithm estimates the most likely state for each observation, but the resulting state sequence
may not be a valid succession of words in natural language (or a very unlikely word sequence) and produce inferior
results.