%
% set up Oct. 2013
%
\section{Calculating $(00|r_{12}|00)^{m}$ integrals}
%
%
The $(00|r_{12}|00)^{m}$ type of integrals is fundamental
for recursive relation calculation. This type of integrals
is needed to be calculated directly from incomplete Gamma
function (see \ref{nuclear_attraction_direct_int_eq:20} and
\ref{OS_ERI_complementary_result}):
\begin{equation}
 (0_{A}0_{B}|r_{12}|0_{C}0_{D})^{(m)} =
 2\left( \frac{\rho}{\pi}\right)^{\frac{1}{2}}(0_{A}|0_{B})
(0_{C}|0_{D})\int^{1}_{0} t^{2m} e^{-(\rho|PQ|^{2})t^{2}} dt
\label{fm_ssssm_eq:0}
\end{equation}
Where we have the arguments as:
\begin{align}
 \overrightarrow{P} &= \frac{\alpha \overrightarrow{A} + \beta
\overrightarrow{B}}{\alpha + \beta} \nonumber \\
\overrightarrow{Q} &= \frac{\gamma \overrightarrow{C} + \delta
\overrightarrow{D}}{\gamma + \delta} \nonumber \\
\rho &= \frac{(\alpha+\beta)(\gamma + \delta)}
{(\alpha+\beta)+(\gamma + \delta)}
\end{align}

The integral in \ref{fm_ssssm_eq:0} could be directly calculated through
incomplete Gamma function. However, such way is expensive because the
$(00|r_{12}|00)^{(m)}$ is inside the primitive Gaussian loop. Suggest
we have 1000 basis sets, and each basis set's contraction is 3; then
for producing the normal four center ERI integrals the number $(00|r_{12}|00)^{(m)}$
for each $m$ could be approximated as: $\dfrac{1}{8}*1000^{4}*3^{4} =
1.0125\times 10^{13}$! Therefore, even a small increasing of the calculation
cost to the $(00|r_{12}|00)^{(m)}$ will lead to a dramatical cost increase
to the integral calculation. Therefore, it's necessary to have a simpler
way to calculate the $(00|r_{12}|00)^{(m)}$ integrals.

There has been a lot of literatures to discuss the integral calculation here(see
\cite{harris1983sssm, gill1991two} etc. and the paper cited by them). The discussion
made in this section is mainly based on the results in the above reference.

\subsection{$f_{m}(t)$}
\label{fmt_function_assessment}
%
%
Firstly, let's discuss the integral inside the \ref{fm_ssssm_eq:0}:
\begin{equation}\label{fm_ssssm_fmt_eq:1}
 f_{m}(t) = \frac{2}{\sqrt{\pi}}\int^{1}_{0} u^{2m} e^{-tu^{2}} du
\end{equation}

When the $m=0$, this function is back to the error function:
\begin{equation}
\begin{split}
 f_{0}(t) &= \frac{2}{\sqrt{\pi}}\int^{1}_{0} e^{-tu^{2}} du  \\
          &= t^{-\frac{1}{2}}\frac{2}{\sqrt{\pi}}\int^{1}_{0}
          e^{-(\sqrt{t}u)^{2}} d (\sqrt{t}u) \\
          &= t^{-\frac{1}{2}}\frac{2}{\sqrt{\pi}}\int^{\sqrt{t}}_{0} e^{-x^{2}} dx \\
          &= t^{-\frac{1}{2}} erf(t^{\frac{1}{2}})
\end{split}
\label{fm_ssssm_fmt_eq:2}
\end{equation}
The error function is fast to converge(this function in standard C++ library is several
hundreds time faster than the incomplete gamma function defined in boost library), and
we note; this function can be found
in standard C++ library\footnote{We note that the standard efc function in C library has
the $\dfrac{2}{\sqrt{\pi}}$}.

For $m>0$, by integration by parts it's easy to find a recursive relation
for compute $f_{m}(t)$:
\begin{equation}
 \begin{split}
  f_{m}(t) &= \frac{2}{\sqrt{\pi}}\int^{1}_{0} u^{2m} e^{-tu^{2}} du \\
           &= -\frac{1}{t\sqrt{\pi}}\int^{1}_{0} u^{2m-1} e^{-tu^{2}} d(-tu^{2}) \\
           &= -\frac{1}{t\sqrt{\pi}}\int^{1}_{0} u^{2m-1} d\left( e^{-tu^{2}} \right) \\
           &= -\left.\frac{1}{t\sqrt{\pi}}u^{2m-1} e^{-tu^{2}}\right|^{1}_{0} +
           \frac{2m-1}{t\sqrt{\pi}}\int^{1}_{0} u^{2m-2} e^{-tu^{2}} du \\
           &= -\frac{1}{t\sqrt{\pi}}e^{-t} + \frac{2m-1}{2t} f_{m-1}(t) \Rightarrow \\
           &= \frac{1}{2t}\left( (2m-1)f_{m-1}(t) - \frac{2}{\sqrt{\pi}}e^{-t}\right)
 \end{split}
 \label{fm_ssssm_fmt_eq:3}
\end{equation}
This function provides the easiest way to compute the $f_{m}(t)$. However, if we use
such recursive relation to direct compute $f_{m}(t)$ from the $f_{0}(t)$, it's
easy to see the error is propagating very quickly (see the example in
\ref{error_propagation_numerical} for more details). Therefore, to directly use the
\ref{fm_ssssm_fmt_eq:3} from $f_{0}(t)$ for computing is not applicable.

The error propagation in this example is because we have
$f_{m}(t) > f_{m+1}(t)$\footnote{such expression could be easily derived from
Cauchy–Schwarz inequality}. It's easy to see that $f_{m}(t)$ is always larger than 0.
As the $m$ grows larger, the difference between $f_{m}(t)$ and $f_{m+1}(t)$ is getting
smaller. This is the reason why error begins propagating when we calculate $f_{m+1}(t)$
from $f_{m}(t)$.

However, such recursive relation may be applicable for some given $m$ and $t$ combinations,
as long as the error is within some small range. For testify this idea, we made an
investigation for all of $m$ and $t$ combinations within error limit of $1.0E^{-10}$.
The results indicate that for $m=1$ to $m=10$ and $t>1$, by using this ``UP''
recursive relation we could get the accurate results(within error of $1.0E^{-10}$)
by climbing from error function\footnote{for a detailed testing, we study the $t$
from $1.0$ to $30.0$(when $t>30.0$ we can use simpler form to express $f_{m}(t)$, see
the discussion below) with step length of $1.0E^{-6}$ for $m=1$ to $m=10$. All of results
are comparing with the one calculated from incomplete gamma function from boost library
in terms of error limit in $1.0E^{-10}$}.

On the other hand, such circumstance indicates that the reverse recursive calculation
is applicable for calculation, that is; to compute the $f_{m}(t)$ from $f_{m+1}(t)$.
It's easy to show, that through the ``downward recursive relation'' the computation
of $f_{m}(t)$ will never lose the accuracy\footnote{We tested the cases for $m=1$
to $m=10$ and $t$ ranges from $0.0$ to $30.0$ with same step length etc. shown above}.
Therefore, in real application we will use the following expression:
\begin{equation}
 \label{fm_ssssm_fmt_eq:4}
 f_{m}(t) = \frac{1}{2m+1}\left( 2tf_{m+1}(t) + \frac{2}{\sqrt{\pi}}e^{-t}\right)
\end{equation}
We have tested for $m=0$ to $m=30$ for t ranging from $0.001$ to $30.00$ (with
step wide of 0.001 too), by using
the downward recursive relation all of results could be accurate within the error
range of $1.0^{-10}$. Therefore, the problem here is that how we compute the
$f_{m_{max}}(t)$.

The most easiest way to calculate $f_{m}(t)$ is perhaps through it's series expansion. In
paper of Harris\cite{harris1983sssm}, equation 9 provides a series expansion:
\begin{equation}
 \label{fm_ssssm_fmt_eq:5}
 f_{m}(t) = \frac{2}{\sqrt{\pi}}e^{-t}\sum_{k=0}^{\infty}\frac{(2m-1)!!}{(2m+2k+1)!!}
 (2t)^{k}
\end{equation}
We can see that for small t (perhaps $t<=1$), such expression will converge in
a very quick way. For larger $t$, it's obvious that it needs more term for
converging\footnote{We have tested the $m=1$ to $m=10$ cases with respect to
$t<=1$ in same testing condition mentioned above. All of results are in good
accuracy within error range of $1.0^{-10}$ with 12 terms expansion}.

However, for the large $t$ situation, the integral of $f_{m}(t)$ should be faster
approaching to zero compared with these with small $t$. From the paper
\cite{harris1983sssm}, it has another series expansion to understand it:
\begin{equation}
\begin{split}
 f_{m}(t) &= \frac{(2m-1)!!}{(2t)^{m}t^{1/2}}-\frac{2}{\sqrt{\pi}}\frac{e^{-t}}{2t}
 \left( 1+\frac{2m-1}{2t} \right. \\
 &+ \left. \frac{(2m-1)(2m-3)}{(2t)^{2}} + \frac{(2m-1)(2m-3)(2m-5)}{(2t)^{3}}+ \cdots
 \right)
\end{split}
 \label{fm_ssssm_fmt_eq:6}
\end{equation}
It's easy to see, when $t>25$; then $\dfrac{2}{\sqrt{\pi}}\dfrac{e^{-t}}{2t}$
will be less than $10^{-12}$ therefore we can always omit the series inside
\ref{fm_ssssm_fmt_eq:6} so that when $t>=25$, the $f_{m}(t)$ could be expressed
as:
\begin{equation}
 \label{fm_ssssm_fmt_eq:7}
 f_{m}(t) = \frac{(2m-1)!!}{(2t)^{m}t^{1/2}}
\end{equation}

Now let's give a summary to calculate the $f_{m}(t)$:
\begin{enumerate}
 \item if $M_{max} = 0$, use erf function;
 \item if $M_{max} >= 1$ and $M_{max} <= 10$:
 \begin{enumerate}
  \item if $t<=1$, calculate $f_{Mmax}(t)$ by using power series in
  \ref{fm_ssssm_fmt_eq:5} with 12 terms, then use down recursive relation
  in \ref{fm_ssssm_fmt_eq:4} for the rest of $f_{m}(t)$;
  \item if $t>1$ and $t<25$, calculate $f_{0}(t)$ with error function and
  use up recursive relation in \ref{fm_ssssm_fmt_eq:3} to derive other $f_{m}(t)$;
  \item if $t>=25$, calculate $f_{Mmax}(t)$ with \ref{fm_ssssm_fmt_eq:7} and use down
  recursive relation in \ref{fm_ssssm_fmt_eq:4} to derive other $f_{m}(t)$
 \end{enumerate}
 \item if $M_{max} >= 11$:
 \begin{enumerate}
  \item if $t<25$, calculate $f_{Mmax}(t)$ from incomplete gamma function (for example,
  from boost library function), and use down
  recursive relation in \ref{fm_ssssm_fmt_eq:4} to derive other $f_{m}(t)$
  \item if $t>=25$, calculate $f_{Mmax}(t)$ with \ref{fm_ssssm_fmt_eq:7} and use down
  recursive relation in \ref{fm_ssssm_fmt_eq:4} to derive other $f_{m}(t)$
 \end{enumerate}
\end{enumerate}
We note, that with such arrangement, all of S, P and D shell integrals (up to second
derivatives order) could be derived without incomplete gamma function, which is
expensive. for $m>11$, the down recursive relation grantees that only one $f_{m}(t)$
is calculated, which also saves a lot of times.

\subsection{Compute Bottom Integrals}
%
%
Now let's use the above results and consider the full expression of
\ref{fm_ssssm_eq:0}:
\begin{equation}
 (00|r_{12}|00)^{(m)} = O\rho^{\frac{1}{2}}f_{m}(T)
\label{fm_ssssm_eq:1}
\end{equation}
Where the parameters are given as\footnote{``O'' here indicates
overlap integrals}:
\begin{equation}
 \begin{split}
  \sigma_{P} &= \frac{1}{\alpha + \beta} \\
  \bm{P}  &=(\alpha{A} + \beta\bm{B})\sigma_{P} \\
  O_{P} &= (\pi\sigma_{P})^{\frac{3}{2}}e^{-\alpha\beta\sigma_{P}
  |\bm{A}-\bm{B}|^{2}} \\
   \sigma_{Q} &= \frac{1}{\gamma + \delta} \\
  \bm{Q}  &= (\gamma\bm{C} + \delta\bm{D})\sigma_{Q} \\
  O_{Q} &= (\pi\sigma_{Q})^{\frac{3}{2}}e^{-\gamma\delta\sigma_{Q}
  |\bm{C}-\bm{D}|^{2}} \\
  O &= O_{P}O_{Q} \\
  \rho &= \frac{1}{\sigma_{P} + \sigma_{Q}} \\
  R &= |\bm{P}-\bm{Q}| \\
  T &= \rho R^{2}
 \end{split}
 \label{fm_ssssm_eq:2}
\end{equation}
Here all of the parameters except $\rho$, $O$, $R$ and $T$ could be
pre-computed outside the integral loop.

For the $m=0$, the $(00|r_{12}|00)^{(0)}$ is:
\begin{equation}
 \begin{split}
  (00|r_{12}|00)^{(0)} &= O\rho^{\frac{1}{2}}f_{0}(T) \\
  &= O\rho^{\frac{1}{2}} T^{-\frac{1}{2}} erf(T^{\frac{1}{2}}) \\
  &= \frac{O}{R}erf(T^{\frac{1}{2}})
 \end{split}
 \label{fm_ssssm_eq:3}
\end{equation}

For the $m>0$ case, the down recursive relation is:
\begin{equation}
\begin{split}
 f_{m}(T) &=\frac{1}{2m+1}\left( 2Tf_{m+1}(T) + \frac{2}{\sqrt{\pi}}e^{-T}\right)
 \Rightarrow \\
 (00|r_{12}|00)^{(m)} &= \frac{1}{2m+1}
 \left( 2t(00|r_{12}|00)^{(m+1)} + \frac{2}{\sqrt{\pi}}O\rho^{\frac{1}{2}}e^{-T} \right)
\end{split}
\label{fm_ssssm_eq:4}
\end{equation}
Here the factor of $\frac{1}{T\sqrt{\pi}}$ and term
$O\rho^{\frac{1}{2}}e^{-T}$ need to
be computed in the integral loop (but only once!).

The up recursive relation is:
\begin{equation}
 \begin{split}
  f_{m}(T) &= \frac{1}{2T}\left( (2m-1)f_{m-1}(T) - \frac{2}{\sqrt{\pi}}e^{-T}\right)
  \Rightarrow \\
  (00|r_{12}|00)^{(m)} &= \frac{1}{2T}\left( (2m-1)(00|r_{12}|00)^{(m-1)} -
  \frac{2}{\sqrt{\pi}}O\rho^{\frac{1}{2}}e^{-T}\right)
 \end{split}
 \label{fm_ssssm_eq:5}
\end{equation}

The power series becomes:
\begin{equation}
  (00|r_{12}|00)^{(m)} = \frac{2}{\sqrt{\pi}}O\rho^{\frac{1}{2}}e^{-T}
  \sum_{k=0}^{\infty}\frac{(2m-1)!!}{(2m+2k+1)!!}(2T)^{k}
 \label{fm_ssssm_eq:6}
\end{equation}
and the \ref{fm_ssssm_fmt_eq:7} becomes:
\begin{equation}
 \label{fm_ssssm_fmt_eq:7}
 (00|r_{12}|00)^{(m)} = O\frac{(2m-1)!!}{(2T)^{m}R}
\end{equation}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{tcolorbox}
\usepackage{float}
\usepackage{amsfonts}
\usepackage{svg}
\date{}

\title{\textbf{Quantum Computing: An Applied Approach}\\\vspace*{1cm}
Chapter 4 Problems: Complexity Theory
}

\begin{document}

\maketitle

\section{}

This problem statement is somewhat misleading. The discrete log problem and the factoring problem are not isomorphic; factoring an integer $N$ is not sufficient to solve the discrete log problem modulo $N$. Rather, computation of log modulo $N$ is sufficient to factor $N$ and calculation of the discrete log problem modulo $p_i$, where $p_i$ are the prime factors of $N$, is sufficient to factorize $N$.

The following proof that these two problems are equivalent is adapted from the original publication by Eric Bach (Bach 1984). I recommend reading the original proof, as it is fairly concise and there is not much added here.\newline
(https://www2.eecs.berkeley.edu/Pubs/TechRpts/1984/CSD-84-186.pdf)

\subsection{First Direction: Discrete Logarithms $\rightarrow$ Factoring}

 Let $\phi(n)$ be the Euler totient function. Let $n$ be an integer that is not a power of a prime, with factorization $n=p_1^{c_1}...p_k^{c_k}$, so that the corresponding group decomposition via the Chinese Remainder Theorem: $\mathbb{Z}_n^{*}\equiv\mathbb{Z}_{p_1^{c_1}}\times ...\times\mathbb{Z}_{p_k^{c_k}}^{*}$ consists of direct factors of order $\phi_i=(p_i-1)p_i^{c_i-1}$ dividing $\lambda=\text{lcm}(\phi_1,..,\phi_k)$.

Using a theorem proven previously (Miller 1976), the solution $a^x\cong 1(\text{ mod }n$ for $x\neq 0$ enables selection of some $a\in\mathbb{Z}_n^{*}$, such that with probability greater than $\frac{1}{2}$ $a$ can be used to find a factor $p_i$
 of $n$ in polynomial time.

 Specifically, letting $K=\{a\in\mathbb{Z}_n^{*}:a^{\frac{\lambda}{2}}\cong\pm 1(\text{ mod }n)\}$ and $\nu=\text{max}\{k:2^k|(p_1-1)\}\geq\text{max}\{k:2^k|(p_i-1)\}$ for all $i$, letting $a$ have order $\phi_1$ and order $\frac{\phi_1}{2}$ modulo the other prime powers yields an $a\not\in K$ by results from number theory.

 Let $\alpha$ be the order of this $a$ in $\mathbb{Z}_n^{*}$ and $\lambda=\alpha\beta_0$, $x=\alpha\beta_1\cdot 2^\nu$ for some odd integers $\beta_0,\beta_1$.

 Taking $k=\nu+1$, this yields a solution $\text{gc}(n,(a^\frac{x}{2^k}+1)\text{mod }n)$ as a proper factor of $n$, so that $x$ can be computed quickly if $x$ itself can be accessed in polynomial time.

 \subsection{Second Direction: Factoring $\rightarrow$ Discrete Logarithms}

Select a prime $p_i\not |\phi(n)$. Then $p$ satisfies $(a^p)^y\cong a(\text{mod }n)$. Substituting $x=py-1$, $a^x\cong 1$ for some prime factor $p_i$.

Since $\mathbb{Z}_{p^c}^*$ is cyclic, it admits an isomorphic decomposition $\mathbb{Z}_{p^c}^*\cong\mathbb{Z}_[^*\times\mathbb{Z}_{p^{c-1}}^+$. Assume the projection onto $\mathbb{Z}_{p^{c-1}}^+$ can be computed in polynomial time, and compute $x_2$ such that $x_2\codt\theta(a)\cong\theta(b)(\text{ mod }p^{c-1})$.

This yields a coupled set of congruities, $x\cong x_1(\text{mod }p-1)$ and $x\cong x_2(\text{mod }p^{c-1})$. that together satisfy $a^x\cong b(\text{mod }p^c)$.

To compute $\theta$, take the (polynomial-time) isomorphism $\lambda:U\rightarrow\mathbb{Z}_{p^{c-1}}^+$ and set $\theta=\lambda\circ\pi$, where $\pi$ is the prime-counting function.

Overall, this shows that given a solution $a^{x_1}\cong b(\text{mod }p_i)$, there is a polynomial-time approach to determine an $x$ such that $a^x\cong b(\text{mod }p_i^c)$.

Finally, taking a prime factorization $n=p_1^{c_1}...p_r^{c_r}$ of $n$, with solutions $x_i,x$ with $a^{x_i}\cong b(\text{mod }p_i)$ and $a^x\cong b(\text{mod }n)$, this shows that $a^{x_i}\cong b(\text{mod }p_i^{c_i})$.

The last step is to compute the order of $a$ modulo some prime power of $n$, with solution:
\begin{align}
    \Pi_{q|\phi(P),q\text{ prime}}q^{\nu_q(\text{order}(a))}
\end{align}

where $\nu_q(\text{order}(a))=\text{min}\{k:a^\frac{\phi(P)}{q^k}\cong 1(\text{ mod }P)\}$.
\newline\newline
This shows that, given a factorization of $n=\Pi p_i^{c_i}$, a polynomial-time algorithm can be used to solve $a^x\cong b(\text{ mod }n$ for integers $a$ and $b$, and vice-versa.

\section{}

This question is also misleading; the graph isomorphism problem is in fact in the complexity class NP. For the sake of this problem, I assume that Hidary is asking to discuss the consequences and existing indications regarding whether or not it is NP-complete.

Schöning demonstrated that the graph isomorphism problem lies in the low hierarchy of NP , so that the problem "is not NP-complete...unless the polynomial-time hierarchy collapses to some finite level" (Schöning 1987). On the other hand, many particular cases of the graph isomorphism problem can be solved in polynomial time, including that for trees, planar graphs, interval graphs, permutation graphs, circulant graphs, and bounded-parameter graphs. However, it is unclear whether these results necessarily generalize in the case of an arbitrary graph.

\section{}
Ewin Tang demonstrated a faster classical solution to the recommendation problem, inspired by an efficient quantum solution.

\textbf{Recommendation Problem}: The recommendation problem asks to approximately determine an appropriate feature for a given user, where $m$ users and $n$ features are given as rows and columns, respectively, in a matrix $A\in \mathbb{C}^{m\times n}$.

Effectively, the problem can be posed as sampling from a row $D_i$, where $D$ is an approximate version of $A$. It corresponds roughly to picking a feature for a given user based on a table of information, where $D$ corresponds to a lower-dimensional version of that information.

\subsection{(a)}

Tang's strategy is essentially to use a dequantized version of the phase estimation algorithm in quantum computing. Tang takes into account sample and query access to input as an additional step, making it possible to compare state preparation in the quantum case with a similar step in the classical case and draw relatively similar time complexities.

Specifically, Tang frames a quantum algorithm as a problem with some set of $C$ input states $|\phi_1\rangle,...,|\phi_C\rangle$ and output $|\psi\rangle$ (possibly equivalent to scalar $\lambda$ to some precision), and defines a dequantization operation that yields a corresponding classical algorithm. The `dequantized' classical algorithm takes as input versions of the input state with sample and query access, SQ($\phi_1$),...,SQ($\phi_C$) and yields output in the form SQ($\psi$) or $\lambda$.

To `dequantize' the quantum solution to the recommendation problem, Tang uses a dequantized version of low-rank approximation. This mirrors the use of phase estimation in the quantum algorithm, but offers a significant speedup over the classical solution.

\textbf{Low-rank approximation} takes some $A\in\mathbb{C}^{m\times n}$, SQ($A$), a threshold $k$, and an error parameter $\epsilon$ and outputs a low-rank approximation of $A$ with number of queries polynomial in $k$ and $\epsilon^{-1}$. This provides the crucical step needed to convert $A$ into $D$ in the statement of the recommendation problem.

Specifically, $D$ can be expressed in terms of $A$ as $D=AS^Y\hat{U}\hat{\Sigma}^{-1}(\hat{\Sigma}^{-1})^T\hat{U}^TS$.
Sampling from this matrix $D_i=A_iS^TMS$ can be performed by approximating $A_iS^T$ using another `dequantized' algorithm, this time inner product protocols.

\textbf{Inner product protocols} accept some $x,y\in\mathbb{C}^n$, SQ($x$) and Q($y$) and estimates $\langle x,y\rangle$ up to some error, with some probability of success.

$A_iS^TM$ can be computed with standard matrix-vector multiplication, and $A_iS^T\hat{U}\hat{U}^TS$ can be evaluated with a final `dequantized' algorithm, the thin matrix-vector product.

The \textbf{thin matrix-vector product} accepts an input $V\in\mathbb{C}^{n\times k}$ and $w\in\mathbb{C}^k$ and approximates SQ($VW$) with queries polynomial in $k$, taking SQ($V^\dagger$) and Q($w$) as input.

$\Sigma$ follows as an output of low-rank approximation, and as a diagonal matrix can be efficiently multiplied, making it possible to sample. Overall, the preceding steps make it possible to sample from an approximation of $D$ efficiently.

\subsection{(b)}

Certainly, it is often difficult to prove that there exists no more efficient algorithm, even when low time complexity solutions already exist.

For example, the most efficient known algorithm for classical integer factorization is the general number field sieve (GNFS), with asymptotic running time\newline $O(\text{exp}\left(\left(\sqrt[3]{\frac{64}{8}}+o(1)\right)(\text{ln }n)^\frac{1}{3}(\text{ln ln }n)^\frac{2}{3}\right)$ for integer $n$. However, Shor's Algorithm for a general quantum computer performs factorization in $O(b^3)$ time and $O(b)$ space; there may be an algorithm with greater time complexity but lower time complexity than that of GNFS.

In contrast, the Deutsch-Jozsa algorithm demonstrates a proven speedup over the classical case; the fastest deterministic classical solution has exponentially greater time complexity. There may, however, be some non-deterministic algorithm with a lower time complexity.

\newpage
\section*{References}
Babai, László, and Eugene M. Luks. "Canonical labeling of graphs." Proceedings of the fifteenth annual ACM symposium on Theory of computing. 1983.\newline
Bach, Eric. "Discrete logarithms and factoring." (1984).\newline
Miller, Gary L. "Riemann's hypothesis and tests for primality." Journal of computer and system sciences 13.3 (1976): 300-317.\newline
Schöning, Uwe. "Graph isomorphism is in the low hierarchy." Journal of Computer and System Sciences 37.3 (1988): 312-323.\newline

\end{document}


\documentclass[twoside]{MATH77}
\usepackage[\graphtype]{mfpic}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\opengraphsfile{pl02-09}
\begmath 2.9 Incomplete Elliptic Integrals

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

An integral of the form
\begin{equation}
\label{O1}\int R\left( t,P(t)^{1/2}\right) dt
\end{equation}
in which P($t)$ is a polynomial of the third or fourth degree that has no
multiple roots, and R is a rational function of $t$ and P($t)^{1/2}$, is
either elementary, or is an {\em elliptic integral}. It is always possible to
express integrals of the form of Eq.~(1) linearly in terms of elementary
functions and three {\em elliptic integrals of canonical form}. These functions
are described more completely in \cite{ams55:Ellip-Int} and \cite{Byrd:1971:HEI}.
Several canonical forms have been proposed, but the most widely used are
due to Jacobi, Legendre and Carlson. In each of Eqs.~(2)--(4) we present
first Jacobi's and then Legendre's form of the canonical elliptic integrals:
\begin{gather}
\label{O2}
\begin{split}
F(\varphi ,k)&=\int_0^y\left( 1-t^2\right) ^{-1/2}\left(
1-k^2t^2\right) ^{-1/2}dt\\
&=\int_0^\varphi \left( 1-k^2\sin ^2\theta \right) ^{-1/2}d\theta
\end{split}\\
\label{O3}
\begin{split}
E(\varphi ,k)&=\int_0^y\left( 1-t^2\right) ^{-1/2}\left(
1-k^2t^2\right) ^{1/2}dt\\
&=\int_0^\varphi \left( 1-k^2\sin ^2\theta \right) ^{1/2}d\theta
\end{split}
\end{gather}
\begin{multline}
\label{O4}
\Pi (\varphi ,\alpha ,k)\\
=\int_0^y\left( 1-\alpha ^2t^2\right) ^{-1}\left( 1-t^2\right)
^{-1/2}\left( 1-k^2t^2\right) ^{-1/2}dt\\
=\int_0^\varphi \left( 1-\alpha ^2\sin ^2\theta \right)
^{-1}\left( 1-k^2\sin ^2\theta \right) ^{-1/2}d\theta
\end{multline}
in which $y=\sin \varphi $. If $\varphi $ is equal to $\pi /2$, the
integrals are said to be {\em complete}, otherwise they are {\em
incomplete}. Carlson's forms of the canonical elliptic integrals are
\begin{equation}\label{O5}
\hspace{-15pt}R_D(a,b,c)=\frac 32\int_0^\infty \left( t+a\right)
^{-1/2}\left( t+b\right) ^{-1/2}\left( t+c\right) ^{-3/2}dt
\end{equation}
in which $a$ and $b$ are nonnegative such that $a+b>0$ and $c$ is positive;
if either $a$ or $b$ is zero, the integral is {\em complete}, otherwise it is
{\em incomplete},
\begin{equation}\label{O6}
\hspace{-15pt}R_F(a,b,c)=\frac 12\int_0^\infty \left( t+a\right)
^{-1/2}\left( t+b\right) ^{-1/2}\left( t+c\right) ^{-1/2}dt
\end{equation}
in which $a$, $b$ and $c$ are nonnegative and at most one of them is zero;
if one of $a$, $b$ or $c$ is zero, the integral is {\em complete},
otherwise it is {\em incomplete}, and
\begin{multline}\label{O7}
\hspace{-10pt}R_J(a,b,c,r)\\
\hspace{-12pt}=\frac 32 \int_0^\infty \!\!\left( t+r\right)
^{-1}\left( t+a\right)^{-1/2}\left( t+b\right) ^{-1/2}\left( t+c\right)
^{-1/2}dt\hspace{-10pt}
\end{multline}
in which $a$, $b$ and $c$ are nonnegative, and at most one of them is zero,
and $r$ is nonzero; if one of $a$, $b$ or $c$ is zero, the integral is
$complete$, otherwise it is $incomplete$. Notice that $R_D(a$, $b$, $c)=R_J(a$, $%
b$, $c$, $c)$. But the necessity to compute $R_D(a$, $b$, $c)$ arises
frequently in practice, and a procedure especially tailored to compute $R_D(a
$, $b$, $c)$ is more efficient than computing $R_J(a$, $b$, $c$, $c)$. The
function $R_C(a$, $b)=R_F(a$, $b$, $b)$ is elementary, but also appears
frequently. A procedure is provided to compute $R_C(a$, $b).$

Identify a, $b$ and $c$ such that $a\leq b\leq c$, and assume $a<c$. Then
\begin{align}\label{O8}
c^{3/2}R_D(a,b,c)&=\frac 3{k^2\sin ^3\varphi }\left[ F(\varphi
,k)-E(\varphi ,k)\right]\hspace{-1in}\\
\label{O9}c^{1/2}R_F(a,b,c)&=\frac{F(\varphi ,k)}{\sin \varphi }\\
\label{O10}c^{3/2}R_J(a,b,c,r)&=\frac 3{\alpha ^2\sin ^3\varphi }\left[ \Pi
(\varphi ,\alpha ,k)-F(\varphi ,k)\right]
\end{align}
where $\cos ^2\varphi =a/c$, $k^2=(c-b)/(c-a)$ and $\alpha ^2=(c-r)/(c-a).$

The subprograms described in this chapter evaluate the canonical forms of
incomplete elliptic integrals, using either the Legendre or the Carlson
parameterization.

\subsection{Usage}

\subsubsection{Program Prototype, Single Precision, Legendre's Form, E and F}

\begin{description}
\item[REAL]  \ {\bf PHI, K, F, E}

\item[INTEGER]  \ {\bf IERR}
\end{description}

Assign values to PHI and K.
$$
\fbox{{\bf CALL SELEFI (PHI, K, F, E, IERR)}}
$$

\paragraph{Argument Definitions}

\begin{description}
\item[PHI]  \ [in] Argument, $\varphi $, of the elliptic integral. Require $|%
\text{PHI}|\leq \pi /2.$

\item[K]  \ [in] Modulus, $k$. Require $|\text{K}|\leq 1.0.$

\item[F]  \ [out] F($\varphi $, $k)$ with $\varphi $ given by PHI and $k$
given by K.

\item[E]  \ [out] E($\varphi $, $k)$ with $\varphi $ given by PHI and $k$
given by K.

\item[IERR]  \ [out] status indicator:

\begin{itemize}
\item[0 =]  no errors

\item[1 =]  Magnitude of argument too large, $|$PHI$| >\pi /2.$

\item[2 =]  Magnitude of Modulus too large, $|$K$|>1.0.$

\item[3 =]  $|$PHI$| =\pi /2$ and $|$K$|=1$, F is infinite.
\end{itemize}
\end{description}

\subsubsection{Program Prototype, Single Precision, Legendre's Form, $\Pi $}

\begin{description}
\item[REAL]  \ {\bf PHI, K2, ALPHA2, PI}

\item[INTEGER]  \ {\bf IERR}
\end{description}

Assign values to PHI, K2 and ALPHA2.
$$
\fbox{{\bf CALL SELPII (PHI, K2, ALPHA2, PI, IERR)}}
$$

\paragraph{Argument Definitions}

\begin{description}
\item[PHI]  \ [in] Argument, $\varphi $, of the elliptic integral. Require $|%
\text{PHI}|\leq \pi /2.$

\item[K2]  \ [in] Square of the modulus, $k^2$. Require $k^2\sin
^2\varphi \leq 1.0$. See Section E.

\item[ALPHA2]  \ [in] Characteristic, $\alpha ^2$. Require $\alpha
^2\sin ^2\varphi \leq 1.0$. See Section E.

\item[PI]  \ [out] $\Pi (\varphi ,\alpha ,k)$, with $\varphi $ given by PHI,
$\alpha ^2$ given by ALPHA2 and $k^2$ given by K2.

\item[IERR]  \ [out] status indicator.  If IERR = 0, there were no
errors.  Other values are produced by procedures SRFVAL and SRJVAL (see
Sections B.5 and B.6) which are
used in computing $\Pi (\phi, \alpha, k).$
\end{description}

\subsubsection{Program Prototype, Single Precision, Carlson's Form, $R_C$}

\begin{description}
\item[REAL]  \ {\bf X, Y, RC}

\item[INTEGER]  \ {\bf IERR}
\end{description}

Assign values to X and Y.
$$
\fbox{\bf CALL SRCVAL (X, Y, RC, IERR)}
$$
\paragraph{Argument Definitions}

\begin{description}
\item[X, Y]  \ [in] Arguments of the elliptic integral. Require X $\geq 0$,
Y $\neq 0$. See Section E.

\item[RC]  \ [out] The computed value of $R_C($X, Y).

\item[IERR]  \ [out] Status indicator:

\begin{itemize}
\item[0 =]  no errors

\item[1 =]  X $<0.0$ or Y = 0.0.

\item[2 =]  X + $|\text{Y}|$ too small (See Section E).

\item[3 =]  X or $|\text{Y}|\text{ or X} +|\text{Y}|$ too large (See Section E).

\item[4 =]  Y $<0$ and $|\text{Y}|$ too large and X too small (See Section
E).
\end{itemize}
\end{description}

\subsubsection{Program Prototype, Single Precision, Carlson's Form, $R_D$}

\begin{description}
\item[REAL]  \ {\bf X, Y, Z, RD}

\item[INTEGER]  \ {\bf IERR}
\end{description}

Assign values to X, Y and Z.
$$
\fbox{\bf CALL SRDVAL (X, Y, Z, RD, IERR)}
$$
\paragraph{Argument Definitions}

\begin{description}
\item[X, Y, Z]  \ [in] Arguments of the elliptic integral. Require X $\geq 0$%
, Y $\geq 0$, X + Y $>0$, Z $>0$. See Section E.

\item[RD]  \ [out] The computed value of $R_D($X, Y, Z).

\item[IERR]  \ [out] Status indicator:

\begin{itemize}
\item[0 =]  no errors

\item[1 =]  X $<0.0$ or Y $<0.0$ or Z $<0.0.$

\item[2 =]  X + Y too small or Z too small (See Section E).

\item[3 =]  X or Y or Z too large (See Section E).
\end{itemize}
\end{description}

\subsubsection{Program Prototype, Single Precision, Carlson's Form, $R_F$}

\begin{description}
\item[REAL]  \ {\bf X, Y, Z, RF}

\item[INTEGER]  \ {\bf IERR}
\end{description}

Assign values to X, Y and Z.
$$
\fbox{\bf CALL SRFVAL (X, Y, Z, RF, IERR)}
$$
\paragraph{Argument Definitions}

\begin{description}
\item[X, Y, Z]  \ [in] Arguments of the elliptic integral. Require X $\geq 0$%
, Y $\geq 0$, Z $\geq 0$, at most one of X, Y or Z equal zero. See Section E.

\item[RF]  \ [out] The computed value of $R_F($X, Y, Z).

\item[IERR]  \ [out] Status indicator:

\begin{itemize}
\item[0 =]  no errors

\item[1 =]  X $<0.0$ or Y $<0.0$ or Z $<0.0.$

\item[2 =]  X + Y or X + Z or Y + Z too small (See Section E).

\item[3 =]  X or Y or Z too large (See Section E).
\end{itemize}
\end{description}

\subsubsection{Program Prototype, Single Precision, Carlson's Form, $R_J$}

\begin{description}
\item[REAL]  \ {\bf X, Y, Z, R, RJ}

\item[INTEGER]  \ {\bf IERR}
\end{description}

Assign values to X, Y, Z and R.
$$
\fbox{\bf CALL SRJVAL (X, Y, Z, R, RJ, IERR)}
$$
\paragraph{Argument Definitions}

\begin{description}
\item[X, Y, Z, R]  \ [in] Arguments of the elliptic integral. Require X $%
\geq 0$, Y $\geq 0$, Z $\geq 0$, at most one of X, Y or Z equal zero, R $\neq
0$. See Section E.

\item[RJ]  \ [out] The computed value of $R_J($X, Y, Z, R).

\item[IERR]  \ [out] Status indicator:

\begin{itemize}
\item[0 =]  no errors

\item[1 =]  X $<0.0$ or Y $<0.0$ or Z $<0.0$ or R $=0.0.$

\item[2 =]  X + Y or X + Z or Y + Z or $|$R$|$ too small (See Section E).

\item[3 =]  X or Y or Z or $|$R$|$ too large (See Section E).
\end{itemize}
\end{description}

\subsubsection{Modifications for Double Precision}

For double precision usage, change the REAL type statements to DOUBLE
PRECISION and change the subprogram names SELEFI, SELPII, SRCVAL, SRDVAL,
SRFVAL and SRJVAL to DELEFI, DELPII, DRCVAL, DRDVAL, DRFVAL and DRJVAL,
respectively.

\subsection{Examples and Remarks}

\subsubsection{Related Functions}

Logarithms, inverse circular functions and inverse hyperbolic functions can
be expressed in terms of $R_C$, see [9,~pp.~163,~186]:
$$
\begin{array}{ll}
(\ln x)/(x-1) = R_C((\frac{1}{2}+\frac{1}{2}x)^2,x), & x>0;\\
(\sin ^{-1} x)/x = R_C(1-x^2,1), & -1\leq x\leq 1;\\
(\sinh^{-1} x)/x = R_C(1+x^2,1),& -\infty <x<\infty;\\
(\cos ^{-1} x)/(1-x^2)^{\frac{1}{2}} = R_C(x^2,1), & 0\leq x\leq 1;\\
(\cosh ^{-1} x)/(x^2-1)^{\frac{1}{2}} = R_C(x^2,1), & x\geq 1;\\
(\tan ^{-1} x)/x = R_C(1,1+x^2), & -\infty <x<\infty ;\\
(\tanh ^{-1} x)/x = R_C(1,1-x^2), & -1<x<1;\\
\cot ^{-1} x = R_C(x^2,x^2+1), & 0\leq x<\infty;\\
\coth ^{-1} x = R_C(x^2,x^2-1), & x>1.
\end{array}
$$
The first seven of these allow computing nearly indeterminate forms
with more accuracy than would be possible using the na\"\i ve
formulation.

Heuman's lambda function \cite{Heuman:1941:TCE} is a variant of Legendre's third integral:
\begin{multline}\label{O11}
\frac{\left( 1-\cos ^2\alpha \sin ^2\beta \right) ^{1/2}}{\cos
^2\alpha \sin \beta \cos \beta }\Lambda (\alpha ,\beta ,\varphi )\\
=\sin \varphi \,R_F(\cos ^2\varphi ,1-\sin ^2\alpha
\sin ^2\varphi ,1)\vspace{2pt} \\ \displaystyle\qquad +\frac{\sin ^2\alpha
\sin ^3\varphi }{3\left( 1-\cos ^2\alpha \sin ^2\beta \right) }\times
R_J \Bigl(\cos ^2\varphi ,\\
1-\sin ^2\alpha \sin^2\varphi ,1,1-\frac{\sin ^2\alpha \sin ^2\varphi }
{1-\cos ^2\alpha \sin^2\beta }\Bigr)
\end{multline}
\begin{multline}\label{O12}
\frac \pi 2\Lambda _0(\alpha ,\beta )=\Lambda (\alpha ,\beta
,\pi /2)\vspace{2pt}\\
=\sin \beta \left[ R_F(0,\cos^2\alpha ,1)-\frac 13\sin ^2\alpha
R_D(0,\cos ^2\alpha ,1)\right]\\
\times R_F\left( \cos ^2\beta ,1-\cos^2\alpha \sin ^2\beta ,1\right)
-\frac 13\cos ^2\alpha \sin ^3\beta\\
\times R_F(0,\cos ^2\alpha ,1)\times R_D(\cos ^2\beta ,1-\cos ^2\alpha
\sin ^2\beta ,1)
\end{multline}
The variants of Legendre's integrals used by Bulirsch in
\cite{Bulirsch:1965:NCE} and \cite{Bulirsch:1969:NCE} are
\begin{equation}
\label{O13}el1(x,k_c)=xR_F\left( 1,1+k_c^2x^2,1+x^2\right) ,
\end{equation}
\begin{multline}\label{O14}
el2(x,k_c,a,b)=axR_F\left( 1,1+k_c^2x^2,1+x^2\right)\\
+\frac 13(b-a)x^3R_D\left( 1,1+k_c^2x^2,1+x^2\right)
\end{multline}
\begin{multline}\label{O15}
ele3(x,k_c,p)=xR_F\left( 1,1+k_c^2x^2,1+x^2\right)\\
+\frac 13(1-p)x^3R_J\left( 1,1+k_c^2x^2,1+x^2,1+p x^2\right)
\end{multline}
\begin{equation}\label{O16}
\hspace{-15pt}cel(k_c,p,a,b)=aR_F(0,k_c^2,1)=\frac 13(b-pa)R_J\left(
0,k_c^2,1,p\right)
\end{equation}
\subsubsection{Which Procedure Should Be Used?}

Several factors influence the choice of procedure. If one
needs to write a simple program and use it once, one should probably choose
the procedure that evaluates the functions in the form most similar to the
way the problem is posed. If one needs to write a program that will have
substantial use, one should usually prefer SELEFI to SRDVAL and SRFVAL, as
the former is up to~30 times faster than the latter two. An exception to
this rule occurs if one needs to compute $R_D(a,b,c)$ with $c < \max (a,b)$,
in which case the parameters for SELEFI will be out of range. If accuracy is
an issue but speed is not, one may prefer SRDVAL and SRFVAL to SELEFI, at
least for computing F($\varphi ,k).$ (See testing in Section D below).

SRCVAL is somewhat slower, on an IBM PC/AT with a numeric data processor,
than using the equivalent Fortran intrinsic functions. This is no surprise,
as most of the intrinsic functions are implemented by hardware. But the
inverse hyperbolic functions are not. SRCVAL is roughly the same speed as
the procedures in Chapter~2.1. As mentioned above, it may be
advantageous to use SRCVAL to compute nearly indeterminate forms.

SELPII is implemented by using SRJVAL and SRFVAL (see Eqs.~(9) and
(10) above).
Thus, there is no special advantage in speed or accuracy to one or the
other. The sole criterion is how closely the forms of the functions
evaluated directly by the procedures match the forms of the functions the
user needs to evaluate.

\subsection{Functional Description}

\subsubsection{Properties of the Functions}

The first form given in Eqs.~(2)--(4) is the Jacobi or algebraic form. When
expressed in this form Eq.~(2) is finite for all real and complex $y$, including $%
\infty $, has a simple pole of order~1 for $y = \infty $, and is
logarithmically infinite for $y = 1/\alpha ^2.$

\subsubsection{Method of Computation}

The procedure SELEFI is based upon a procedure ELLPI developed by Allan V.
Hershey and modified by Alfred H.  Morris, described in
\cite{ahm:lib:ELLPI}.  The procedure uses series expansions due to
DiDonato and Hershey, described in \cite{DiDonato:1959:NFC}.  The
procedure SELPII is based upon a procedure EPI developed by Alfred H.
Morris, described in [5].  It computes $\Pi (\varphi ,k^2,\alpha ^2)$
using Eqs.~(9) and (10), as computed by SRFVAL and SRJVAL.  The procedures
SRCVAL, SRDVAL, SRFVAL and SRJVAL are based on procedures developed by B.
C.  Carlson and Elaine M.  Notis, described in \cite{Carlson:1979:CEI} and
\cite{Carlson:1981:AAI}.  All of the referenced procedures were revised to
be consistent with low level modules and naming conventions of MATH77.
\vspace{10pt}

%
% \input prepic.tex
% \input pictex.tex
% \input postpic.tex
% \beginpicture \ninepoint
% \setcoordinatesystem units <  1.797in,  2.800in>
% \setplotarea x from    .000 to   1.600, y from    .000 to   1.000
% \axis bottom label {}
%  ticks in withvalues 0 .2 .4 .6 .8 1.0 1.2 1.4 1.6 /
%  at 0 .2 .4 .6 .8 1.0 1.2 1.4 1.6 / /
% \axis left label {$k$}
%  ticks in numbered from   .0     to  1.0     by   .2     /
% \axis top ticks in quantity  9 /
% \axis right ticks in quantity  6 /
% \put {.2} at    .200   .400
% \put {.4} at    .400   .600
% \put {.6} at    .600   .400
% \put {.8} at    .830   .600
% \put {1} at   1.025   .400
% \put {1.2} at   1.220   .300
% \put {1.4} at   1.405   .200
% \setquadratic \plot
%   .2000   .00000   .2000   .00000   .2000   .00500   .2000   .01500
%   .2000   .02500   .2000   .03500   .2000   .04500   .2000   .05500
%   .2000   .0700   .2000   .0850   .2000   .1000   .2000   .1150
%   .2000   .1350   .2000   .1500   .2000   .1700   .2000   .1900
%   .2001   .2100   .2001   .2300   .2001   .2550   .2001   .2800
%   .2001   .3100   .2002   .3400   .2002   .3750
% /
% \setquadratic \plot
%   .2002   .4250   .2003   .4550   .2003   .4850   .2004   .5200
%   .2004   .5600   .2005   .6000   .2005   .6400   .2006   .6800
%   .2007   .7250   .2008   .7700   .2009   .8150   .2010   .8750
%   .2012   .9350   .2013   .9650   .2014  1.0000
% /
% \setquadratic \plot
%   .4115  1.0000   .4097   .9250   .4080   .8473   .4066   .7750
%   .4052   .6950   .4047   .6600   .4042   .6250
% /
% \setquadratic \plot
%   .4035   .5750   .4029   .5200   .4023   .4650   .4018   .4150
%   .4014   .3600   .4011   .3200   .4008   .2800   .4006   .2450
%   .4004   .2050   .4003   .1800   .4002   .1500   .4002   .1300
%   .4001   .1050   .4001   .0900   .4001   .0700   .4000   .0550
%   .4000   .04000   .4000   .03000   .4000   .02000   .4000   .01000
%   .4000   .00000
% /
% \setquadratic \plot
%   .6000   .00000   .6000   .00429   .6000   .01000   .6000   .02000
%   .6000   .03000   .6001   .04000   .6001   .05000   .6001   .06000
%   .6002   .0750   .6003   .0900   .6004   .1100   .6006   .1300
%   .6008   .1550   .6012   .1850   .6016   .2150   .6022   .2550
%   .6030   .2950   .6038   .3350   .6049   .3750
% /
% \setquadratic \plot
%   .6063   .4250   .6073   .4550   .6083   .4850   .6106   .5450
%   .6133   .6050   .6153   .6450   .6172   .6800   .6212   .7450
%   .6253   .8050   .6305   .8700   .6365   .9350   .6400   .9687
%   .6435  1.0000
% /
% \setquadratic \plot
%   .9273  1.0000   .9120   .9662   .8983   .9300   .8856   .8900
%   .8734   .8450   .8640   .8044   .8550   .7600   .8480   .7203
%   .8416   .6800   .8381   .6550   .8341   .6250
% /
% \setquadratic \plot
%   .8281   .5750   .8244   .5400   .8210   .5050   .8180   .4700
%   .8156   .4400   .8114   .3800   .8080   .3207   .8058   .2750
%   .8037   .2200   .8026   .1850   .8017   .1500   .8012   .1250
%   .8007   .09500   .8005   .08000   .8003   .06000   .8002   .04500
%   .8001   .03000   .8000   .02000   .8000   .01000   .8000   .00500
%   .8000   .00000
% /
% \setquadratic \plot
%   1.0000   .00000   1.0000   .00228   1.0000   .01000   1.0001   .02000
%   1.0001   .03000   1.0002   .04000   1.0003   .05000   1.0006   .06500
%   1.0009   .0800   1.0014   .1000   1.0020   .1200   1.0029   .1450
%   1.0042   .1750   1.0061   .2100   1.0081   .2400   1.0115   .2850
%   1.0157   .3300   1.0177   .3500   1.0206   .3750
% /
% \setquadratic \plot
%   1.0270   .4250   1.0314   .4550   1.0362   .4850   1.0416   .5150
%   1.0480   .5477   1.0551   .5800   1.0636   .6150   1.0720   .6456
%   1.0842   .6850   1.0960   .7183   1.1110   .7550   1.1252   .7850
%   1.1416   .8150   1.1574   .8400   1.1760   .8652   1.2000   .8924
%   1.2246   .9150   1.2594   .9400   1.2972   .9600   1.3360   .9746
%   1.3760   .9851   1.4320   .9940   1.4880   .9983   1.5280   .9996
%   1.5679  1.0000
% /
% \setquadratic \plot
%   1.5680   .8748   1.5360   .8598   1.5040   .8422   1.4771   .8250
%   1.4480   .8032   1.4240   .7822   1.4000   .7579   1.3763   .7300
%   1.3578   .7050   1.3415   .6800   1.3243   .6500   1.3115   .6250
%   1.2960   .5907   1.2857   .5650   1.2748   .5350   1.2651   .5050
%   1.2563   .4750   1.2497   .4500   1.2425   .4200   1.2371   .3950
%   1.2320   .3694   1.2285   .3500   1.2243   .3250
% /
% \setquadratic \plot
%   1.2171   .2750   1.2152   .2600   1.2123   .2350   1.2098   .2100
%   1.2075   .1850   1.2053   .1550   1.2031   .1200   1.2020   .0950
%   1.2011   .07000   1.2007   .05500   1.2003   .04000   1.2002   .03000
%   1.2001   .02000   1.2000   .01000   1.2000   .00000
% /
% \setquadratic \plot
%   1.4000   .00000   1.4000   .00154   1.4000   .01000   1.4001   .02000
%   1.4003   .03000   1.4005   .04000   1.4008   .05000   1.4013   .06500
%   1.4020   .0800   1.4031   .1000   1.4045   .1200   1.4070   .1500
%   1.4096   .1750
% /
% \setquadratic \plot
%   1.4161   .2250   1.4201   .2500   1.4245   .2750   1.4295   .3000
%   1.4339   .3200   1.4398   .3450   1.4451   .3650   1.4507   .3850
%   1.4584   .4100   1.4669   .4350   1.4761   .4600   1.4863   .4850
%   1.4960   .5067   1.5073   .5300   1.5206   .5550   1.5353   .5800
%   1.5516   .6050   1.5586   .6150   1.5680   .6278
% /
% \endpicture\vspace{-5pt}
%\centerline{$\varphi $}\vspace{5pt}
%\centerline{E($k,\varphi)$}\vspace{20pt}
%
% \beginpicture \ninepoint
% \setcoordinatesystem units <  1.797in,  2.800in>
% \setplotarea x from    .000 to   1.600, y from    .000 to   1.000
% \axis bottom label {}
%  ticks in withvalues 0 .2 .4 .6 .8 1.0 1.2 1.4 1.6 /
%  at 0 .2 .4 .6 .8 1.0 1.2 1.4 1.6 / /
% \axis left label {$k$}
%  ticks in numbered from   .0     to  1.0     by   .2     /
% \axis top ticks in quantity  9 /
% \axis right ticks in quantity  6 /
% \put {.2} at    .200   .300
% \put {.4} at    .400   .500
% \put {.6} at    .600   .300
% \put {.8} at    .780   .500
% \put {1} at    .990   .300
% \put {1.2} at   1.150   .500
% \put {1.4} at   1.290   .600
% \put {1.6} at   1.400   .700
% \put {1.8} at   1.450   .800
% \put {2.0} at   1.450   .900
% \setquadratic \plot
%   .2000   .00000   .2000   .00500   .2000   .01500   .2000   .02500
%   .2000   .03500   .2000   .04500   .2000   .06000   .2000   .07000
%   .2000   .0850   .2000   .1000   .2000   .1150   .2000   .1350
%   .2000   .1550   .2000   .1700   .2000   .1900   .1999   .2100
%   .1999   .2350   .1999   .2550   .1999   .2750
% /
% \setquadratic \plot
%   .1999   .3250   .1998   .3550   .1998   .3850   .1998   .4150
%   .1997   .4500   .1997   .4850   .1996   .5200   .1996   .5600
%   .1995   .6000   .1995   .6450   .1994   .6900   .1993   .7300
%   .1992   .7700   .1991   .8100   .1990   .8550   .1989   .9050
%   .1988   .9550   .1987   .9750   .1987  1.0000
% /
% \setquadratic \plot
%   .3897  1.0000   .3910   .9350   .3922   .8700   .3935   .7950
%   .3947   .7150   .3956   .6550   .3964   .5950   .3968   .5600
%   .3972   .5250
% /
% \setquadratic \plot
%   .3977   .4750   .3981   .4300   .3985   .3800   .3988   .3350
%   .3992   .2850   .3994   .2450   .3996   .2050   .3997   .1750
%   .3998   .1400   .3999   .1200   .3999   .0950   .3999   .0800
%   .4000   .06000   .4000   .05000   .4000   .03500   .4000   .02500
%   .4000   .01500   .4000   .01000   .4000   .00000
% /
% \setquadratic \plot
%   .6000   .00000   .6000   .00500   .6000   .01500   .6000   .02500
%   .6000   .03500   .5999   .04500   .5999   .06000   .5998   .07500
%   .5997   .0950   .5996   .1150   .5993   .1400   .5991   .1650
%   .5987   .1950   .5983   .2250   .5977   .2600   .5976   .2650
%   .5975   .2750
% /
% \setquadratic \plot
%   .5965   .3250   .5953   .3750   .5940   .4250   .5926   .4700
%   .5911   .5150   .5884   .5900   .5850   .6700   .5832   .7100
%   .5810   .7550   .5785   .8050   .5757   .8550   .5713   .9300
%   .5669  1.0000
% /
% \setquadratic \plot
%   .7262  1.0000   .7319   .9600   .7374   .9200   .7440   .8692
%   .7513   .8100   .7559   .7700   .7609   .7250   .7651   .6850
%   .7690   .6450   .7731   .6000   .7766   .5600   .7778   .5450
%   .7794   .5250
% /
% \setquadratic \plot
%   .7831   .4750   .7849   .4500   .7871   .4150   .7897   .3700
%   .7921   .3250   .7943   .2750   .7962   .2250   .7974   .1850
%   .7984   .1450   .7989   .1200   .7993   .0950   .7996   .0750
%   .7998   .05500   .7998   .04500   .7999   .03000   .8000   .01500
%   .8000   .00000
% /
% \setquadratic \plot
%   1.0000   .00000   1.0000   .00500   1.0000   .01500    .9999   .02500
%    .9998   .03500    .9997   .04500    .9996   .05500    .9993   .07000
%   .9990   .0850   .9985   .1050   .9979   .1250   .9969   .1500
%   .9956   .1800   .9937   .2150   .9918   .2450   .9908   .2600
%   .9897   .2750
% /
% \setquadratic \plot
%   .9856   .3250   .9838   .3450   .9814   .3700   .9782   .4000
%   .9754   .4250   .9712   .4600   .9674   .4900   .9633   .5200
%   .9589   .5500   .9543   .5800   .9495   .6100   .9440   .6427
%   .9383   .6750   .9327   .7050   .9269   .7350   .9200   .7693
%   .9125   .8050   .9048   .8400   .8960   .8784   .8880   .9121
%   .8786   .9500   .8723   .9750   .8658  1.0000
% /
% \setquadratic \plot
%    .9857  1.0000    .9982   .9700   1.0084   .9450   1.0202   .9150
%   1.0318   .8850   1.0411   .8600   1.0537   .8250   1.0640   .7952
%   1.0724   .7700   1.0821   .7400   1.0930   .7050   1.1019   .6750
%   1.1104   .6450   1.1185   .6150   1.1263   .5850   1.1336   .5550
%   1.1406   .5250
% /
% \setquadratic \plot
%   1.1514   .4750   1.1564   .4500   1.1611   .4250   1.1664   .3950
%   1.1705   .3700   1.1743   .3450   1.1772   .3250   1.1812   .2950
%   1.1849   .2650   1.1886   .2300   1.1920   .1926   1.1945   .1600
%   1.1966   .1250   1.1976   .1050   1.1986   .0800   1.1991   .0650
%   1.1996   .04500   1.1997   .03500   1.1999   .02500   1.1999   .01500
%   1.2000   .00000
% /
% \setquadratic \plot
%   1.4000   .00000   1.4000   .00500   1.3999   .01500   1.3998   .02500
%   1.3996   .03500   1.3994   .04500   1.3991   .05500   1.3985   .07000
%   1.3978   .0850   1.3966   .1050   1.3948   .1300   1.3931   .1500
%   1.3916   .1650   1.3877   .2000   1.3837   .2300   1.3791   .2600
%   1.3749   .2850   1.3703   .3100   1.3663   .3300   1.3610   .3550
%   1.3565   .3750   1.3517   .3950   1.3441   .4250   1.3387   .4450
%   1.3316   .4700   1.3240   .4950   1.3161   .5200   1.3078   .5450
%   1.2973   .5750
% /
% \setquadratic \plot
%   1.2786   .6250   1.2686   .6500   1.2561   .6800   1.2474   .7000
%   1.2340   .7300   1.2240   .7514   1.2102   .7800   1.2000   .8006
%   1.1849   .8300   1.1717   .8550   1.1581   .8800   1.1440   .9050
%   1.1296   .9300   1.1178   .9500   1.1040   .9729   1.0960   .9859
%   1.0872  1.0000
% /
% \setquadratic \plot
%   1.1723  1.0000   1.1897   .9800   1.2067   .9600   1.2234   .9400
%   1.2400   .9196   1.2560   .8995   1.2750   .8750   1.2901   .8550
%   1.3084   .8300   1.3227   .8100   1.3400   .7850   1.3534   .7650
%   1.3697   .7400   1.3760   .7300   1.3791   .7250
% /
% \setquadratic \plot
%   1.4092   .6750   1.4234   .6500   1.4343   .6300   1.4475   .6050
%   1.4600   .5800   1.4697   .5600   1.4800   .5378   1.4901   .5150
%   1.5007   .4900   1.5087   .4700   1.5183   .4450   1.5255   .4250
%   1.5341   .4000   1.5406   .3800   1.5468   .3600   1.5525   .3400
%   1.5593   .3150   1.5631   .3000   1.5680   .2795
% /
% \setquadratic \plot
%   1.5680   .6607   1.5536   .6800   1.5382   .7000   1.5222   .7200
%   1.5057   .7400   1.4887   .7600   1.4720   .7791   1.4712   .7800
%   1.4640   .7880
% /
% \setquadratic \plot
%   1.4298   .8250   1.4154   .8400   1.3957   .8600   1.3907   .8650
%   1.3806   .8750
% /
% \setquadratic \plot
%   1.3760   .8795   1.3600   .8950   1.3440   .9102   1.3280   .9251
%   1.3062   .9450   1.2894   .9600   1.2720   .9752   1.2560   .9890
%   1.2431  1.0000
% /
% \setquadratic \plot
%   1.3017  1.0000   1.3200   .9881   1.3399   .9750   1.3600   .9614
%   1.3768   .9500   1.3920   .9395   1.4126   .9250
% /
% \setquadratic \plot
%   1.4640   .8875   1.4806   .8750   1.5001   .8600   1.5191   .8450
%   1.5377   .8300   1.5520   .8182   1.5680   .8048
% /
% \endpicture\vspace{-5pt}
%\centerline{$\varphi $}\vspace{5pt}
%\centerline{F($k,\varphi)$}
%
\subsubsection{Testing}

The single precision programs for E($\varphi ,k)$, F($\varphi ,k)$, $%
R_D(a,b,c)$ and $R_F(a,b,c)$ were tested on an IBM PC/AT (using IEEE
arithmetic) by comparison to double precision results, as described below.
The relative precision of IEEE single precision arithmetic is $\rho =
2^{-23} \approx 0.119\times 10^{-6}.$

The accuracy of procedure SELEFI was assessed by comparing its results to
double precision results obtained by applying  Eqs.~(8) and (9), with $%
R_D(a,b,c)$ and $R_F(a,b,c)$ evaluated by DRDVAL and DRFVAL, respectively.
The accuracy of procedures SRDVAL and SRFVAL was assessed by comparing their
results to double precision results obtained by applying Eqs.~(8) and (9),
with E($\varphi ,k)$ and F($\varphi ,k)$ evaluated by DELEFI.

To test SELEFI, the rectangular region $0 \leq \varphi \leq \pi /2 \times 0
\leq k \leq 1$ of the $\varphi \times k$ plane was divided into 2000
regions, and a point was randomly selected in each region. To test SRDVAL
and SRFVAL, the argument $c$ was set to~1.0, the rectangular region $0 \leq
a < 1 \times 0 \leq b < 1$ of the $a\times b$ plane was divided into~2000
regions, and a point was randomly selected in each region. The maximum
relative and absolute errors are summarized in the following table.

\begin{center}
\begin{tabular}{lrr}
& \multicolumn{1}{c}{\bf Max. Rel.} & \multicolumn{1}{c}{\bf Max. Abs.}\\
{\bf Function} & \multicolumn{1}{c}{\bf Error} & \multicolumn{1}{c}{\bf Error}\\
E($\varphi ,k)$ & $.82\rho $\rule{.2in}{0pt} & $.98\rho $\rule{.2in}{0pt}\\
F($\varphi ,k)$ & $5.24\rho $\rule{.2in}{0pt} & $15.91\rho $\rule{.2in}{0pt}\\
$R_D(a,b,1)$ & $1.20\rho $\rule{.2in}{0pt} & $3.01\rho $\rule{.2in}{0pt}\\
$R_F(a,b,1)$ & $1.35\rho $\rule{.2in}{0pt} & $2.55\rho $\rule{.2in}{0pt}
\end{tabular}
\end{center}

Errors in F($\varphi ,k)$ increase as the arguments approach the infinite
singularity at $\varphi =\pi /2$ and $k=1.$

\nocite{Carlson:1977:SFA}
\bibliography{math77}
\bibliographystyle{math77}

\subsection{Error Procedures and Restrictions}

The procedure SELEFI requires $|\varphi | \leq \pi /2$, and $|k| \leq 1$.

Procedure SELPII computes $\Pi (\varphi ,k^2,\alpha ^2)$ from $R_J(a,b,c,r)$
and $R_F(a,b,c)$ using Eqs.~(9) and (10). The initial values for the arguments
are $a = \cos ^2\varphi $, $b = 1-k^2\sin ^2\varphi $, $r = 1-\alpha ^2\sin
^2\varphi $, and $c = \max (a,b,r)$. Then a, $b$ and $r$ are replaced by $%
a\times c$, $b\times c$ and $r\times c$, respectively. SELPII requires $%
|\varphi | \leq \pi /2$. Restrictions on $k^2$ and $\alpha ^2$ are enforced
indirectly by restrictions on a, $b$ and $c$ imposed by SRFVAL and SRJVAL,
described below.

The ranges for $\varphi $ and $k^2$ can be extended using formulae 113.01,
113.02, 114.01, 115.01, 115.02, 160.02, 161.02 and~162.02 from [2],
or formulae~17.4.1 through~17.4.18 from [1].

Denote the largest representable magnitude by $\Omega $, and the smallest
nonzero representable magnitude by $\omega $. General restrictions on the
arguments to procedures SRCVAL, SRDVAL, SRFVAL and SRJVAL were described
above in Section B.

SRCVAL requires $\text{X}+|\text{Y}| \geq 5\omega ,\ \text{X} \leq
\Omega /5$, $|\text{Y}| \leq \Omega /5$, and, if $\text{Y} < -2.236/\sqrt
\omega $ it requires $\text{X} \geq (\omega \Omega )^2/25$.

Denote the machine round-off level by $\rho $, that is, $\rho $ is the
smallest positive number such that
the representation of $1+\rho $\ is different from~1. Let $\varepsilon $ be
the solution of the equation $\rho = 3\varepsilon ^6(1-\varepsilon )^{-3/2}$,
$\Omega _D = 2\Omega ^{-2/3}$ and $\omega _D = \varepsilon \omega ^{-2/3}/10$%
. SRDVAL requires $\text{X}+\text{Y} \geq \omega _D$ and $\text{Z}
\geq \omega _D,\ \text{X} \leq  \Omega _D,\ \text{Y} \leq \Omega _D$
and $\text{Z} \leq \Omega _D$.

SRFVAL requires $\text{X} + \text{Y} \geq 5\omega ,\ \text{X} +
\text{Z} \geq 5\omega $, $\text{Y} + \text{Z} \geq 5\omega ,\ \text{X} \leq
\Omega /5,\ \text{Y} \leq \Omega /5$ and
$\text{Z} \leq \Omega /5$.

Let $\Omega _J = (\Omega /5)^{1/3}/5$ and $\omega _J = (5\omega )^{1/3}$.
SRJVAL requires $\text{X}+\text{Y} \geq \omega _J,\ \text{Y} +\text{Z}\geq
\omega _J,\ \text{X} +\text{Z} \geq \omega _J$, $|\text{R}| \geq
\omega _J,\ \text{X} \leq \Omega _J,\ \text{Y} \leq \Omega _J,\ \text{Z} \leq
\Omega _J$ and $|\text{R}| \leq \Omega _J$.

The accessible ranges of the
arguments may be extended beyond the ranges admissible in the procedures by
using the homogeneity of the functions:
\begin{align*}
R_F(ka, kb, kc) &= k^{-1/2} R_F(a, b, c),\quad \text{and}\\
R_J(ka,kb, kc, kr) &= k^{-3/2} R_J(a,\ b,\ c,\ r).
\end{align*}
If any of the restrictions above is violated, all procedures return an error
indicator in the argument named IERR, and invoke the error message
processor (see Chapter~19.2) with LEVEL = 0. The procedure ERMSET (see
Chapter~19.2) may be used to affect the default error processing action.

\subsection{Supporting Information}

The source language for these subroutines is ANSI Fortran 77.

The procedures SELEFI and SELPII were written by W. V. Snyder in December
1990, based on earlier procedures described by Alfred H. Morris, Naval
Surface Warfare Center, Dahlgren, VA in [5]. The procedures SRCVAL,
SRDVAL, SRFVAL and SRJVAL were written by W. V. Snyder in December 1990,
based on earlier procedures described by Carlson and Notis in \cite{Carlson:1981:AAI}.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt} \\
DELEFI & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DELEFI, DERM1, DERV1, DLNREL, ERFIN, ERMSG\rule[-5pt]{0pt}{8pt}}\\DELPII & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DELPII, DERM1, DERV1, DRCVAL, DRFVAL, DRJVAL, ERFIN, ERMSG\rule[-5pt]{0pt}{8pt}}\\DRCVAL & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DERM1, DERV1, DRCVAL, ERFIN, ERMSG\rule[-5pt]{0pt}{8pt}}\\DRDVAL & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DERM1, DERV1, DRDVAL, ERFIN, ERMSG\rule[-5pt]{0pt}{8pt}}\\DRFVAL & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DERM1, DERV1, DRFVAL, ERFIN, ERMSG\rule[-5pt]{0pt}{8pt}}\\DRJVAL & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DERM1, DERV1, DRCVAL, DRFVAL, DRJVAL, ERFIN, ERMSG\rule[-5pt]{0pt}{8pt}}\\SELEFI & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, SELEFI, SERM1, SERV1, SLNREL\rule[-5pt]{0pt}{8pt}}\\SELPII & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, SELPII, SERM1, SERV1, SRCVAL, SRFVAL, SRJVAL\rule[-5pt]{0pt}{8pt}}\\SRCVAL & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, SERM1, SERV1, SRCVAL\rule[-5pt]{0pt}{8pt}}\\SRDVAL & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, SERM1, SERV1, SRDVAL\rule[-5pt]{0pt}{8pt}}\\SRFVAL & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, SERM1, SERV1, SRFVAL\rule[-5pt]{0pt}{8pt}}\\SRJVAL & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, SERM1, SERV1, SRCVAL, SRFVAL, SRJVAL\rule[-5pt]{0pt}{8pt}}\\\end{tabular}

\begcodenp
\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRSELI}\vspace{10pt}
\lstinputlisting{\codeloc{seli}}

\vspace{30pt}\centerline{\bf \large ODSELI}\vspace{10pt}
\lstset{language={}}
\lstinputlisting{\outputloc{seli}}

\closegraphsfile
\end{document}


% Created 2019-11-14 jue 22:00
\documentclass[letterpaper]{scrartcl}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{fixltx2e}
\usepackage{graphicx}
\usepackage{longtable}
\usepackage{float}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{textcomp}
\usepackage{marvosym}
\usepackage{wasysym}
\usepackage{amssymb}
\usepackage{hyperref}
\tolerance=1000
\usepackage{khpreamble}
\usepackage{geometry}
\geometry{top=20mm, bottom=20mm, left=22mm, right=18mm}
\author{Kjartan Halvorsen}
\date{\today}
\title{State feedback with observer exercise}
\hypersetup{
  pdfkeywords={},
  pdfsubject={},
  pdfcreator={Emacs 25.3.50.2 (Org mode 8.2.10)}}
\begin{document}

\maketitle

\subsection*{Complete the block diagram}
\label{sec-0-1}
\begin{center}
\includegraphics[width=\linewidth]{../../figures/state-feedback-with-observer-incomplete}
\end{center}

The control law is \(u(k) = l_0u_c(k) - L\hat{x}(k)\). The feedback to the observer is \(K\tilde{y}(k) = K\big(y_m(k) - \hat{y}(k)\big) = K\big(y(k) + n(k) - C\hat{x}(k)\big) \)

\subsection*{Complete the augmented state-space model}
\label{sec-0-2}

\begin{align*}
\bbm x(k+1)\\\hat{x}(k+1) \ebm &= \bbm & & & & & & & &&& & & &\\ &&& & & & & & & & & & &\ebm \bbm x(k)\\\hat{x}(k)\ebm + \bbm &&&\\&&&\\&&&\\&&& \ebm u_c(k) + \bbm &&&\\&&&\\&&&\\&&& \ebm n(k)\\
y(k) &= \bbm &&&&&& \ebm \bbm x(k)\\\hat{x}(k)\ebm
\end{align*}

\newpage

\subsection*{Control of a tank model}
\label{sec-0-3}
Consider the system of two tanks in the figure below. The input signal is the flow of water into the upper tank, and the output signal is the level of the second tank.

\begin{minipage}{0.6\linewidth}
In continuous-time the system is described by the state space system
\begin{align*}
\frac{dx}{dt} &= \bbm -0.0197 & 0\\0.0178 & -0.0129 \ebm x + \bbm 0.0263\\0 \ebm u\\
y &= \bbm 0 & 1 \ebm x.
\end{align*}
With sampling period \(h=12\) we obtain the discrete-time system
\begin{align*}
x(kh+h) &= \bbm 0.790 & 0\\ 0.176 & 0.857 \ebm x(kh) + \bbm 0.281\\ 0.0296 \ebm u(kh)\\
y(kh) &= \bbm 0 & 1 \ebm x(kh).
\end{align*}
\end{minipage}
\begin{minipage}{0.4\linewidth}
 \includegraphics[width=\linewidth]{../../figures/fig2-12-two-tank-system.png}
\end{minipage}

\begin{enumerate}
\item Determine an observer with poles that are twice as fast as the fastest mode of the plant.
\item Determine a state feedback gain \(L\) such that the closed-loop system has poles in \( 0.8 \pm 0.1\).
\end{enumerate}

\begin{center}
\includegraphics[width=0.45\linewidth]{../../figures/zgrid-crop}
\end{center}

\newpage

\subsection*{Implementing the controller}
\label{sec-0-4}
Complete the pseudo-code (matlab) below for the implementation of the controller
\begin{verbatim}
% The plant model
n = 2;
h = 12;
Phi = [0.79 0; 0.176 0.857];
Gamma = [0.281; 0.0296];
C = [0 1];

% The controller and observer gains

L =

K =

% Initialize variables
u = 0; xhat = [0; 0];
% Run the controller
while run_controller() % The function run_controller will return 0 if the controller should stop
   y = get_output();  % Returns the sampled output signal from the plant
   uc = get_setpoint();  % Returns the desired water leve in the second tank

   % Update the observer

   xhat =

   % Compute the control signal

   u =

   % Send the control signal to the plant
   write_control_signal(u);
   % Wait until next sampling instant
   sleep(h); % Assuming that the computational time is negligable
end
\end{verbatim}

\section*{Solution}
\label{sec-1}

\begin{enumerate}
\item The estimation error of the observer satisfies the difference equation
\[ \tilde{x}(kh+h) = \left( \Phi - KC \right) \tilde{x}(kh). \]
We can choose the poles of the matrix \( \Phi - KC\) freely if the system is \textbf{observable}. Here we have
\[ W_o = \bbm C\\ C \Phi \ebm = \bbm 0 & 1\\ 0.176 & 0.857 \ebm, \]
so \(\det W_o = -0.176 \neq 0\), i.e. the system is \textbf{observable}.

The poles of the continuous-time plant are \(-0.0197\)  and \(-0.0129\). The fastest poles is thus \(-0.0197\). The discrete-time observer poles should be twice as fast. This is obtained by placing the observer poles in
\[ \mexp{2(-0.0197)h} \approx 0.62. \]
Verify by studying the z-plane grid. This gives the desired characteristic polynomial
\[ (z-0.62)^2 = z^2 - 1.24z + 0.3844 \] for the observer.

We have
\begin{equation*}
\begin{split}
\Phi - KC &= \bbm 0.790 & 0\\ 0.176 & 0.857 \ebm - \bbm k_1\\k_2 \ebm \bbm 0 & 1 \ebm\\
&= \bbm 0.790 & 0\\ 0.176 & 0.857 \ebm - \bbm 0 & k_1\\ 0 & k_2 \ebm\\
&= \bbm 0.790 & -k_1\\ 0.176 & 0.857-k_2 \ebm.
\end{split}
\end{equation*}
Which gives the characteristic polynomial
\begin{equation*}
\begin{split}
\det \left( zI - (\Phi - KC) \right) &= \det \bbm z - 0.790 & k_1\\ -0.176 & z-0.857+k_2 \ebm \\
&= (z-0.790)(z-0.857+k_2) + 0.176k_1 = z^2 + (-0.790-0.857+k_2)z -0.790(-0.857+k_2) 0.176k_1\\
&= z^2 + (k_2 -1.647)z + 0.176k_1 - 0.790 k_2 + 0.677
\end{split}
\end{equation*}

Setting the coefficients of the two characteristic polynomials equal gives the system of equations
\begin{align*}
k_2 - 1.647 &= -1.24 \quad \Rightarrow \quad k_2 = 0.407\\
0.176k_1 - 0.790 k_2 + 0.677 &= 0.3844 \quad \Rightarrow \quad k_1 \approx 0.164.
\end{align*}

\item Now design the feedback gain \(L\). The system  is \textbf{reachable} since
\[W_c = \bbm \Gamma & \Phi\Gamma \ebm = \bbm 0.281   &  0.222 \\ 0.0296 & 0.0748  \ebm \]
\[ \det W_c = 0.0145 \neq 0\].

The desired characteristic polynomial is
\[ (z - 0.8 -0.1i)(z-0.8+0.1i) = z^2 -1.6z + 0.65. \]

The closed-loop system characteristic polynomial is obtained by
\[ \det \left(zI - (\Phi - \Gamma L) \right). \]
with
\[ \Gamma L = \bbm 0.281\\ 0.0296 \ebm \bbm l_1 & l_2 \ebm = \bbm 0.281 l_1 & 0.281 l_2\\ 0.0296 l_1 & 0.0296 l_2 \ebm \]
we get
\begin{equation*}
\begin{split}
\det \left(zI - (\Phi - \Gamma L) \right) &= \det \bbm z - 0.790 + 0.281l_1 & 0.281 l_2\\ -0.176 + 0.0296 l_1 & z - 0.857 + 0.0296 l_2 \ebm\\
&= (z-0.790 + 0.281 l_1)(z - 0.857 + 0.0296 l_2) - (-0.176 + 0.0296 l_1)(0.281 l_2)\\
&= z^2 + (-0.790 + 0.281 l_1 - 0.857 + 0.0296 l_2)z + (-0.790 + 0.281 l_1)(-0.857 + 0.0296 l_2) + 0.049 l_2 - 0.0296 \cdot 0.281 l_1 l_2 \\
&= z^2 + (0.281 l_1 + 0.0296 l_2 - 1.647)z + 0.677 - 0.023l_2 - 0.241 l_1 + 0.0296 \cdot 0.281 l_1 l_2 + 0.049 l_2 - 0.0296 \cdot 0.281 l_1 l_2 \\
&= z^2 + (0.281 l_1 + 0.0296 l_2 - 1.647)z + 0.677 - 0.241 l_1 + 0.026 l_2.
\end{split}
\end{equation*}
Setting the coefficients of the two characteristic polynomials equal gives the system of equations
\begin{align*}
0.281 l_1 + 0.0296 l_2 &= -1.6 + 1.647\\
-0.241 l_1 + 0.026 l_2 &= 0.65 - 0.677
\end{align*}
or
\[ \bbm 0.281 & 0.0296\\-0.241 & 0.026 \ebm \bbm l_1\\l_2 \ebm = \bbm 0.047\\-0.027 \ebm \]
with solution
\[ \bbm l_1\\l_2 \ebm = \frac{1}{0.281\cdot 0.026 + 0.241\cdot 0.0296} \bbm 0.026 & -0.0296\\0.241 & 0.281 \ebm \bbm 0.047\\ -0.027 \ebm = \bbm 0.14\\0.26 \ebm. \]
\end{enumerate}
% Emacs 25.3.50.2 (Org mode 8.2.10)
\end{document}


% % % % % % % % % % % % % % % % % % % % % % % % %
\chapter{4-Manifolds}
% % % % % % % % % % % % % % % % % % % % % % % % %

% ----------------------
\section{Bordism}
% ----------------------

\begin{definition}
	The $n$-dimensional oriented \textit{cobordism group}
	\index{cobordism group}
	over the space $X$ is
	\begin{equation*}
		\Omega_{n}[X] =
			\frac{ \{ f \colon M^{n} \rightarrow X \mid M^n \textrm{ oriented, closed, }
			n\textrm{-dim. manifold} \}}{\textrm{bordism}}
	\end{equation*}
	\marginnote{If $\alpha \colon M^{n} \rightarrow X$
		represents a bordism class, $M^n$ is allowed to
		have more than one component.}
\end{definition}

\begin{proposition}[{\citep[13.15, p. 319]{kauffman1987knots}}]
	\begin{itemize}
		\item Pushing forward a fundamental class
		\begin{align*}
			\Omega_n [X] & \rightarrow H_{n}(X ; \Z) \\
			[f \colon M^{n} \rightarrow X] & \mapsto f_{*}([M])
		\end{align*}
		is an isomorphism for $n \le 3$.

		\item The sequence
		\begin{equation*}
			\Omega_{4}[\ast] \rightarrow \Omega_{4}[X] \rightarrow H_{4}(X ; \Z)
		\end{equation*}
		is exact.
	\end{itemize}
\end{proposition}

%TODO

% ------------------------------------
\section{Mazur manifolds}
% ------------------------------------

\textbf{References}:
\begin{itemize}
	\item Homology spheres: \citep{saveliev2013invariants}
	\item \citep{akbulut1979mazur}
\end{itemize}

%TODO
TODO

% ------------------------------------
\section{Andrews-Curtis Conjecture}
% ------------------------------------

\citep{SCP4}

\begin{definition}
	A \textit{balanced presentation} \index{presentation!balanced} is a presentation
	\[
		\langle g_1, \ldots, g_n \mid r_1, \ldots, r_n \rangle
	\]
	with the same number of generators and relations.

	The Andrew-Curtis moves on a balanced presentation are
	\begin{enumerate}[label=(\roman*)]
		\item \textbf{1-handle slides:} Replace a pair of generators
		$ \{ x, y \} $ by $ \{ x, xy \} $

		\item \textbf{2-handle slides:} Replace a pair of relations
		$ \{ r, s \} $ by $ \{ grg^{-1}s, s \} $, where $ g $ is any word in the generators

		\item \textbf{1-2-handle cancellations:} Add a generator together with
		a new relation killing it
	\end{enumerate}
	In particular, you are \underline{not} allowed to make a copy of a relation to
	keep for later use (which would correspond to
	\textbf{2-3-handle creation/cancellation}).
\end{definition}

\begin{conjecture*}[Andrews-Curtis conjecture \textcolor{red}{(Probably false?)}]
	Any balanced presentation of the trivial group can be transformed
	by Andrews-Curtis moves to the trivial presentation.
\end{conjecture*}

\begin{example}
	%TODO
	TODO
	\[
		\langle x, y \mid xyx = yxy, x^{5} = y^{4} \rangle
	\]
	is a balanced presentation of the trivial group,
	but until now nobody was able to find a sequence of
	Andrews-Curtis moves to transform it into the trivial presentation
	\[
		\langle x, y \mid x, y \rangle
	\]
\end{example}

%TODO
TODO
Explain how to construct homotopy $4$-spheres from balanced
presentations of the trivial group
\citep{akbulut1985potential}

\marginnote{
	In \citep[I.3]{kirby2006topology} Kirby suspected that the
	\textit{Dolgachev surface}
	\index{Dolgachev surface} $E(1)_{2,3}$
	(an exotic copy of the rational elliptic surface
	$ \CP{2} \# 9 \overline{\CP{2}} $)
	might require 1- and/or 3-handles.
	But in \citep{akbulut2008dolgachev} Akbulut found a
	handlebody presentation without those.
}
\begin{openquestion*}[]
	Does a simply connected, closed, smooth $4$-manifold need
	$1$-handles and/or $3$-handles?
\end{openquestion*}

\begin{observation}
	If exotic $\sphere{4}$ exist, their handle decomposition must
	contain $1$- or $3$-handles.

	For this suppose you have a handle decomposition of a manifold
	with the homology of $\sphere{4}$ and no $1$- and $3$-handles,
	then there could not be any $2$-handles either because these would
	give nontrivial second homology.
\end{observation}

\begin{definition}
	Use the following notation to denote the abelian group
	\begin{equation*}
		\Gamma_n =
		\frac{\textrm{orientation preserving diffeomorphisms of } \sphere{n-1}}
			 {\textrm{those that extend to a diffeomorphism of } \disk{n}}
	\end{equation*}
\end{definition}
\marginnote{
	\begin{itemize}
		\item $\Gamma_{1} = \Gamma_{2} = 0$
		\item Munkres, Smale: $\Gamma_{3} = 0$
		\item Cerf: $\Gamma_{4} = 0$
		\item Kervaire, Milnor: $\Gamma_{5} = \Gamma_{6} = 0$,
		$\Gamma_{7} = \Z / 28$
	\end{itemize}
	}

\begin{proposition}
	For $n \ge 5$ we can identify $\Gamma_{n}$
	with the set of oriented smooth structures
	of the topological $n$-sphere.
	I.e. in dimension $\ge 5$ all exotic spheres
	can be obtained by using a diffeomorphism of
	$\sphere{n-1}$ to glue two $n$-disks along their boundary.
\end{proposition}

\begin{theorem}[Cerf, \citep{geiges2010eliashberg}]
	Any diffeomorphism of the $3$-sphere $\sphere{3}$
	extends over the $4$-ball $\disk{4}$,
	in other words
	\[
		\Gamma_4 = 0.
	\]
\end{theorem}

\begin{observation}
	Cerf's theorem implies that there are no exotic structures
	on $\sphere{4}$ that can be obtained by gluing
	two $4$-disks along their
	boundary.
\end{observation}

\newpage
% ------------------------------------
\section{Trisections}
% ------------------------------------

\begin{center}
	\textsc{``Trisections are to $4$-manifolds as Heegaard splittings are to
		$3$-manifolds''}
\end{center}

\subsection{References}

\begin{itemize}
	\item Original paper: \citep{gay2016trisecting}
	\item Lecture notes: \citep{gay2019heegaard}
\end{itemize}

\subsection{Definitions}

\marginnote{
	\begin{itemize}
		\item $\#^n A$ is the connected sum of $n$ copies of $A$,
		with $\#^{0} A = \sphere{m}$

		\item TODO
		%TODO
	\end{itemize}}
\begin{definition}
	\begin{itemize}
		\item The standard genus $g$ surface is
		\[
			\Sigma_g = {\#}^{g} (\sphere{1} \times \sphere{1})
		\]
		\item The standard genus $g$ solid handlebody is
		\[
			H_g = \natural^g (\sphere{1} \times \disk{2})
		\]
		with $\partial H_{g} = \Sigma_{g}$
		\marginnote{$\partial (A \natural B) = (\partial A) \# (\partial B)$}

		\item The standard $4$-dimensional $1$-handlebody
		(of ``genus $k$'') is
		\[
			Z_{k} =\natural^{k} (\sphere{1} \times \disk{3})
		\]
		i.e.\ a $4$-ball to which we attach $k$-many
		$4$-dimensional $1$-handles.
	\end{itemize}
\end{definition}
\begin{marginfigure}
	\begin{center}
		\includegraphics{./pictures/standard_manifolds.png}
	\end{center}
	\caption{Standard manifolds of dimension 2, 3}
	\label{fig:standard_manifolds}
\end{marginfigure}


\subsection{Iterative deepening depth-limited search}

Does a depth-limited search to a layer \(L\), increases the layer and starts again. The repeats are a waste, but earlier layers are much cheaper.

Informed: No

Time:

Space:

Complete:

Optimal:


\problemname{Flight Plans}
As you are probably aware, flight pricing can sometimes be surprisingly complex.
For example, it can often be cheaper to take a much longer flight with several legs instead of flying directly between two airports.
One of the reasons pricing seems to be complex is that airlines often try to obfuscate exactly how pricing works, in order to ensure that their customers choose more expensive flights.

One particular airline has deciced to take this obfuscation to the next level; they do not even offer an automated search of their flights.
Instead, they describe their flights in a very peculiar format.
For every one of their $N$ airports (which are numbered between $0$ and $N - 1$), they list either:
\begin{itemize}
    \item what airports they travel to from this airport, or
    \item what airports they \emph{do not} travel to from this airport.
\end{itemize}.

To compensate for this complexity, the airline sets the price of every direct flight between two airports to the same amount.

Can you write a program that, given the descriptions of all the flights the airline provides, determine the minimum number of flights required to travel from airport $s$ to airport $t$?

\section*{Input}
The first line of input contains an integer $1 \le N \le 10^5$, the number of airports, and the two integers $s$ and $t$ ($0 \le s, t < N$, $s \neq t$).

The next $N$ lines each describe the outgoing flights of an airport, starting with airport $0$.
The line starts with a letter.
If this letter is \texttt{N}, you will get a list of all destination airports from this airport.
If this letter is \texttt{C}, you will get a list of all airports that are \emph{not} destinations from this airport.

Following this letter is an integer $m$, the number of airports in the list.
Finally, there will $m$ unique numbers $a_i$ ($0 \le a_i < N$) on the line, the airports in the list.

The sum of $m$ over all airports is at most $2 \cdot 10^5$.

\section*{Output}
Output a single integer, the minimum number of flights required to travel from airport $s$ to airport $t$.

If there is no path, output ``\texttt{impossible}''.

\section*{Explanation of Sample Input 1}
The only flight from airport $0$ is to airport $2$.
From airport $2$, there is also only a single flight going to airport $3$.
From airport $3$, you can fly to any airport except airport $1$.

Since no airport has a direct flight to airport $1$, there cannot be any possible flight plan from airport $0$ to $1$, so the answer is \texttt{impossible}

\section*{Explanation of Sample Input 2}
The only flight from airport $0$ is to airport $2$.
From airport $2$, there is also only a single flight going to airport $3$.
From airport $3$, you can fly to any airport except airport $0$.

Thus, there is a flight plan from $0$ to $1$ going from $0$ to $2$, to $3$, to $1$, which results in 3 flights.
This is also the shortest flight plan.


\chapter{Model Quality and the Bias-Variance Tradeoff \label{chapter:biasvariance}}

We have now seen several different algorithms (and meta-algorithms) for supervised learning. We've examined linear models for classification and regression (Chapters~\ref{chapter:linreg} and \ref{chapter:logreg}), KNN (Chapters~\ref{chapter:classification} and \ref{chapter:regression}), decision trees (Chapters~\ref{chapter:decisiontrees}), and ensemble methods -- usually based on decision trees -- including random forests (Chapter~\ref{chapter:randomforests}) and boosting (Chapter~\ref{chapter:boosting}). In this short chapter, we'll take a step back and talk about some key themes that are relevant to all of these methods.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Measuring Error: Loss and Objective Functions}

Most of us think of error as a fairly intuitive concept, and so far in our discussions of model building and model quality, we've avoided any formal definitions. However, as we begin to extend the concepts we've learned from classification and regression to more challenging problem classes (e.g., survival analysis), a more formal conception of what error means and how to minimize it algorithmically becomes increasingly crucial. Here are some examples:

\begin{itemize}
\item In classification, error typically means either the total number of misclassified points,
$$ \text{error} = \sum_{i=1}^n \mathcal{I}\left(y^{(i)} \neq \hat{y}^{(i)}\right), $$
or a weighted average of the number of misclassified points per class. Another way of quantifying classification error is \textbf{AUC}, the area under the receiver operating characteristic curve, which is the probability that a randomly chosen positive example (where $y=1$) will receive a higher score from the model (higher $\hat{y}$) than a randomly chosen negative example (where $y=0$).
\item In regression, error typically means the \textbf{mean-squared error (MSE)},
$$ \text{error} = \frac{1}{n} \sum_{i=1}^n \left( y^{(i)} - \hat{y}^{(i)} \right)^2, $$
although there are also other ways of quantifying error. For example, you could use the \textbf{mean absolute error}
$$ \text{error} = \frac{1}{n} \sum_{i=1}^n \vert y^{(i)} - \hat{y}^{(i)} \vert. $$
Neither is ``wrong'' or ``right'', but they have different properties. For example, the MSE gives a higher weight to large errors (outliers). It's also differentiable, which is why it's used so much more often in, e.g., neural networks than the mean absolute error.
\item In survival analysis, error is typically quantified using something called \textbf{Harrell's concordance index}, which takes into account censored observations\footnote{For a detailed explanation and some experimental results, see Kattan MW, Hess KR and Beck JR (1998). ``Experiments to determine whether recursive partitioning (CART) or an artificial neural network overcomes theoretical limitations of Cox proportional hazards regression.'' \emph{Computers and Biomedical Research}, 31(5), 363-373.}.
\end{itemize}

\vspace{2mm}

\begin{question}{question:harrell}
Harrell's concordance ($C$) index is probably unfamiliar to you. It is calculated like this:
\begin{enumerate}
\item Create a list of all possible pairs of patients. There will be $n(n-1)/2$ pairs.
\item Eliminate all pairs for which the patient with the shorter follow-up time does not experience the event of interest (i.e., is censored). The remaining patient pairs are considered ``usable'' since the patient with the shorter time-to-event is identifiable.
\item Count the number of usable patient pairs for which the patient with the shorter follow-up time had the higher predicted hazard for the event. That is, you want the number of pairs for which the model's predictions about relative times to event are consistent with the observed data.
\item The $C$ statistic is the number of consistent pairs divided by the number of usable pairs. The error is defined as $1-C$.
\end{enumerate}
Here are four patients and the predicted mean time to event from two different survival models (low value means lower time to event). All times are in years. Calculate the $C$ statistic for both models.

    \begin{center}
    \begin{tabular}{ccccc}
    \toprule
    Patient & Follow-up Time & Observed? & Model 1 Score & Model 2 Score \\
    \midrule
    1 & 8.3 & 1 & 4.6 & 5.2 \\
    2 & 6.5 & 0 & 2.3 & 7.1 \\
    3 & 2.7 & 1 & 0.6 & 6.7 \\
    4 & 7.4 & 1 & 4.7 & 6.6 \\
    \bottomrule
    \end{tabular}
    \end{center}

{\small
    \begin{center}
    \begin{tabular}{ccccc}
    \toprule
    First Patient & Second Patient & Usable & Model 1 & Model 2 \\
    & & & Consistent & Consistent \\
    \midrule
    1 & 2 & \\
    1 & 3 & \\
    1 & 4 & \\
    2 & 3 & \\
    2 & 4 & \\
    3 & 4 & \\
    \bottomrule
    \end{tabular}
    \end{center}
}
You should find that the $C$ statistic for Model 1 is 0.75, while the $C$ statistic for Model 2 is only 0.25. Model 1 is clearly the better model.
\end{question}

The definition of error used to optimize a particular model is called a \textbf{loss function}. Loss functions are a general concept from optimization and decision theory; they represent the cost associated with a decision. If we think of a supervised learning model as an engine for making decisions, the loss is how ``bad'', on average, those decisions will be. Loss functions are, in turn, part of a broader class of functions called \textbf{objective functions}. Most learning algorithms work by either minimizing some measure of ``badness'' (a loss function) or maximizing some measure of ``goodness'' (negative of the loss, alternatively called a \textbf{reward function}).
\vspace{2mm}
\begin{question}{}
Imagine what would happen if, in Question~\ref{question:harrell}, we simply set a follow-up time of $7$ years and treated the problem as a classification problem, calculating the AUC for the two models based on that time horizon and ignoring censoring.
    \begin{center}
    \begin{tabular}{ccccc}
    \toprule
    Positive Patient & Negative Patient & Model 1 & Model 2 \\
    & & Ranks Pos Higher? & Ranks Pos Higher? \\
    \midrule
    3 & 1 & 1 & 0 \\
    3 & 2 & 1 & 1 \\
    3 & 4 & 1 & 1 \\
    \bottomrule
    \end{tabular}
    \end{center}
We would calculate an AUC of 1.0 for Model 1 and 0.67 for Model 2. Do you think this is a good approach to quantifying error for this problem? Why or why not?
\end{question}

The choice of how to define error, and which loss function to use for quantifying that error, ultimately rests with the model builder. The process of model training is about adjusting the available parameters of the model to minimize the loss.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Goodness of Fit vs. Generalizability}

Once we have an appropriate objective function, we can set about building a model to optimize it. This requires us to think about what constitutes a ``good model''. It's a bit more complicated than simply minimizing the loss on our training data, because ideally we want a model that is both accurate and \textbf{parsimonious}, meaning that it is as simple as possible without sacrificing performance. Another way of thinking about this is that we create models to tell stories about the data we see. If they are good stories, they will do two things:
\begin{enumerate}
\item Explain the structure of the data that are used to train them (high \textbf{goodness of fit})
\item Make accurate predictions on new data (good \textbf{generalizability})
\end{enumerate}
For a supervised learning model, we quantify (1) using the \textbf{training error} (error on the training set) and (2) using the \textbf{test error} (error on an independent test set).

\vspace{2mm}

\begin{question}{}
Here we see three decision boundaries for KNN with different values of $K$ (the number of neighbors considered in making a prediction). The data are for the two-class classification problem first discussed in Chapter~\ref{chapter:classification}. From left to right, $K=50, 15,$ and $3$. What are the tradeoffs in moving from left to right in terms of (a) training error/goodness of fit and (b) test error/generalizability?
\begin{center}
\includegraphics[width=0.3\textwidth]{img/esl-knn-50.png}
\includegraphics[width=0.3\textwidth]{img/esl-knn-15.png}
\includegraphics[width=0.3\textwidth]{img/esl-knn-3.png}
\end{center}
\end{question}

In supervised learning, \textbf{model complexity} (loosely defined as the effective number of parameters the model must fit) is, therefore, a very important consideration. A model that is not complex enough will fail to capture all of the structure in the training data, and both its training and test error will suffer as a result. We call this situation \textbf{underfitting}. Conversely, a model that is too complex may fit the training data very well -- maybe perfectly -- but will fail to generalize well to new data. We call this situation \textbf{overfitting}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Bias vs. Variance}

It turns out that there is a general principle governing model complexity in supervised learning called the \textbf{bias-variance tradeoff}. It is perhaps best illustrated by this figure, which comes from the excellent (and free) book \emph{Elements of Statistical Learning}, by Tibshirani, Hastie, and Friedman (Figure 2.11):

\begin{center}
\includegraphics[width=0.8\textwidth]{img/l03-bias-variance-tradeoff.png}
\end{center}

The figure shows us that test error, our measurement of generalization error, comes from two different sources:
$$ \text{test error} = \text{bias} + \text{variance}. $$
The term \textbf{bias} refers to error that results from underfitting, while \textbf{variance} results from overfitting.

Error due to bias can often be reduced by introducing more/different features or by using a different learning algorithm. Error due to variance can often be reduced by increasing the size or diversity of the training data. That's why it's important to know which situation you're in; gathering more training data when your model is too simple is unlikely to help you, and deploying an extremely fancy (and complicated) deep learning model when logistic regression has already overfit your training data is also unlikely to help.

Another consideration is that even the ``sweet spot'' (minimum test error) on the bias-variance tradeoff curve can still be high. This occurs when the data you have available simply aren't sufficient to answer the question -- maybe you have the wrong features or there is no relationship between the features and the outcome. Or maybe your features aren't measured correctly. In my experience, these types of issues are not considered often enough in the clinical domain.

\vspace{2mm}

\begin{question}{}
A recent review article had the provocative title ``A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models.'' (Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. \emph{Journal of Clinical Epidemiology}, 2019, 110:12-22). Assuming this finding is true, what is it telling us about the nature of the error in most clinical risk prediction models? What does it suggest about what we should be doing to improve the performance of these models?
\end{question}

Another way of thinking about bias and variance is in terms of what happens when you make slight changes to the training data (for example, by collecting new data from the same patient population, or drawing repeated bootstrap samples from the same training set). The figure below is from the article \emph{Understanding the Bias-Variance Tradeoff}, by Scott Fortmann-Roe:

\begin{center}
\includegraphics[width=0.7\textwidth]{img/targets-bias-variance.png}
\end{center}

Think of each dot as representing a single test example evaluated under the same model trained on slightly different datasets. The center of the target is the prediction the model should make for that test example. In the case of high bias and low variance, all of the models are off, but they are ``wrong in the same way''. Even if you average their predictions, the answer is still way off the mark. In the case of high variance, the models all make incorrect predictions, but their predictions are off in different directions. As a result, if you average their outputs, you'll get closer to the right answer.

\vspace{2mm}

\begin{question}{}
We saw random forests in Chapter~\ref{chapter:randomforests} and were introduced to boosting in Chapter~\ref{chapter:boosting}. It is interesting to compare the two methods because they reduce test error in different ways: one primarily tackles variance, while the other primarily tackles bias. Which one is which, and how does each algorithm succeed in reducing its respective form of error? Hint: The image below may help. One column represents three trees from a random forest; the other represents three trees from a boosting model.

\begin{center}
\includegraphics[width=0.7\textwidth]{img/bias-variance-reduction-targets.png}
\end{center}
\end{question}