\documentclass{article}
\newsavebox{\oldepsilon}
\savebox{\oldepsilon}{\ensuremath{\epsilon}}
\usepackage[minionint,mathlf,textlf]{MinionPro} % To gussy up a bit
\renewcommand*{\epsilon}{\usebox{\oldepsilon}}
\usepackage[margin=1in]{geometry}
\usepackage{graphicx} % For .eps inclusion
%\usepackage{indentfirst} % Controls indentation
\usepackage[compact]{titlesec} % For regulating spacing before section titles
\usepackage{adjustbox} % For vertically-aligned side-by-side minipages
\usepackage{array, amsmath, mathrsfs, mhchem}
\usepackage{hyper ref}
\usepackage{courier, subcaption}
\usepackage{multirow, color}
\usepackage[autolinebreaks,framed,numbered]{mcode}

\usepackage{float}
\restylefloat{table}
\newcommand{\Lapl}{\mathcal{L}}

\pagenumbering{gobble}
\setlength\parindent{0 cm}
\renewcommand{\arraystretch}{1.2}
\begin{document}
\large

\section*{Recap of the two-step negative autorepression system}

\begin{itemize}
\item Last time, we began our study of negative autoregulation using control theory.
\item Intuition suggested that this system could be used to maintain the concentration of a repressor at a desired value, dampening any fluctuations that might occur e.g. due to varying levels of transcriptional activation.
\item I warned you, however, that negative autoregulation might not always work as planned. In some cases it leads to chronic oscillations and/or amplifies small deviations from the desired value.
\item We focused our attention on a transcriptional repressor that regulates its own production with negative feedback that is proportional to the error (that is, to the deviation of the repressor's concentration from its desired value, which we assumed for simplicity to be 0).
\item This system can be described by the block diagram
\begin{center}\includegraphics[width=0.5 \textwidth]{autorepression_twostep.pdf}\end{center}

\item By considering the topology of our control loop, we determined the closed loop transfer function $\tilde{p}/\tilde{x}$ can be written as:

\[ \tilde{p}(s) = G(s) \tilde{a}(s) \implies \frac{\tilde{p}(s)}{\tilde{x}(s)} = \frac{G(s)}{1 + kG(s)} \]

\item Our motivation for the proportionality assumption was that if the repressor binds its own promoter hyperbolically, and the error term is small, then the slope of the repressor's binding curve is approximately linear with a slope we called $k$ (the repressor's ``gain").

\item To find a form for the open loop transfer function $G(s)$, we had to specify some particulars about the \textit{plant} -- that is, the LTI system relating the net repressor mRNA expression ($a(t)$) to the concentration of the repressor protein. We did this using a pair of differential equations:
\begin{eqnarray*}
\frac{dm}{dt} & = & c_m a(t) - \gamma_m m(t)\\
\frac{dp}{dt} & = & c_p m(t) - \gamma_p p(t)
\end{eqnarray*}

\item By taking the Laplace transform of these, and assuming that the initial concentrations of protein and mRNA were zero for simplicity, we found the open loop transfer function:

\[ G(s) = \frac{\tilde{p}(s)}{\tilde{a}(s)} = \frac{c_p c_m}{\left( s + \gamma_p \right)\left( s + \gamma_m \right)} \]

\item Na\"{i}vely, it appeared that choosing a large value of $k$ would ensure that the error in the system's output, i.e. $\tilde{p}(s)$-0, stayed small. However, I warned that increasing $k$ might make the system unstable.

\item We discussed how $\tilde{p}(s)$ (for an appropriate choice of $\tilde{x}(s)$) could be rewritten using partial fraction decomposition into a sum containing terms of the form $\frac{z_i}{s-\alpha_i}$, where the $\alpha_i$ were the \textit{poles} of the closed loop transfer function (multiplied by $\tilde{x}(s)$).

\item This implied that $p(t)$ would be equal to a sum of terms of the form $e^{-\alpha_i t}$. In other words, if a pole $\alpha_i$ of the closed loop transfer function had a positive real part, then that term would grow indefinitely with time, and so $p(t)$ would diverge.

\item We therefore wanted to know whether increasing $k$ could make one or more of the $\alpha_i$ have a positive real part. If so, choosing a large gain would cause our system to become unstable.

\end{itemize}

\section*{Continuation of the discussion of the Nyquist stability criterion}

\begin{itemize}

\item Where are the poles of our closed loop transfer function? The numerator, $G(s)$, contributes two poles: $s=-\gamma_m$ and $s=-\gamma_p$. Neither of these poles have positive real part. That means that if the closed loop transfer function has any poles with positive real part, they must be the roots of the denominator, $1+kG(s)$.

\item We needed a method to determine whether there were any roots of $1+kG(s)$ in the right half of the complex plane. We began to introduce the Nyquist stability criterion for this purpose.

\item The Nyquist stability criterion relates the number or roots and poles of a function inside the Nyquist contour to the shape of its Nyquist plot. The Nyquist plot of $1+kG(s)$ is drawn by plugging every point $s$ on the Nyquist contour into $1+kG(s)$.

\item The Nyquist contour runs up the imaginary axis, then back to $-i\infty$ through the right half plane along a semicircular arc of radius infinity.

\item The number of times the Nyquist plot of $1+kG(s)$ encircles the origin going clockwise ($N$) is equal to the number of zeros of $1+kG(s)$ inside the Nyquist contour ($Z$) minus the number of poles of $1+kG(s)$ inside the Nyquist contour ($P$).

\item The parameter $k$ is not fixed in our model, so we can't easily draw a Nyquist plot of $1+kG(s)$. However, a handy feature of the way the plot shifts and scales is that $N$ is also equal to the number of times the Nyquist plot of $G(s)$ encircles the point $-1/k$ going clockwise. This allows us to draw one Nyquist plot (of $G(s)$) and easily consider different values of $k$.

\item In order to find $Z$, we need both $N$ and $P$. $N$ we determine from the Nyquist plot; $Z$ we can get by examination of $1+kG(s)$. The poles of $1+kG(s)$ must be the same as the poles of $G(s)$. (If not convinced, try combining these terms under a common denominator.)

\item We already know that the poles of $G(s)$ are $s=-\gamma_m$ and $s=-\gamma_p$, neither of which have positive real part. Therefore, $Z=0$.

\item To find $N$, we can draw the Nyquist plot if the $c_i$ and $\gamma_i$ are fixed.  This can be done in MATLAB using \mcode{tf()} to define the transfer function $G(s)$ and \mcode{nyquist()} to draw the plot. [Draw what the result looks like.]

\item To convince ourselves that this is not a property of our choice of parameters, we can make a graphical argument. In order for $-1/k$ to be enclosed, the Nyquist plot would have to cross the real axis at least once at a negative real value. (This is not sufficient, but it is necessary). Now we will attempt to determine whether it does so by looking at $G(s)$ for points $s$ on the Nyquist contour.

\item On the semicircular arc, $|s|$ is infinite. Clearly $G(s)=0$ for all points on the semicircular arc.

\item To find where $G(s)$ maps the points on the imaginary axis, i.e., what is $G(i\omega)$ for $\omega \in (-\infty, \infty)$, we plug in $s=i\omega$ and rearrange to get a real denominator:

\begin{eqnarray*}
G(i \omega) & = &  \frac{c_p c_m}{\left( i\omega + \gamma_p \right)\left( i\omega + \gamma_m \right)} \cdot \frac{\left(- i\omega + \gamma_p \right)\left( - i\omega + \gamma_m \right)}{\left(- i\omega + \gamma_p \right)\left(- i\omega + \gamma_m \right)}\\
& = & \frac{c_p c_m \left[  - \omega^2 + \gamma_p \gamma_m -i \omega \left(\gamma_p + \gamma_m \right) \right]}{\left( \omega^2 + \gamma_p^2 \right)\left( \omega^2 + \gamma_m^2 \right)}
\end{eqnarray*}

\item This implies that the contour crosses the imaginary axis at a positive real value of $w$ (as well as at $w=0+0i$). In other words, no points on the negative real axis are ever enclosed, and it does not matter how large we choose $k$.
\end{itemize}

\section*{Comparison to a three-step pathway}

\begin{center}\includegraphics[width=0.5 \textwidth]{autorepression_threestep.pdf}\end{center}

\begin{itemize}

\item As we alluded to last time, something surprising happens when we make our ``plant" only slightly more complicated by adding a third step to the negative autoregulation.

\item Suppose the protein $p$ produces a chemical ($f$) that represses expression of the gene:

\begin{eqnarray*}
\frac{dm}{dt} & = & c_m a(t) - \gamma_m m(t)\\
\frac{dp}{dt} & = & c_p m(t) - \gamma_p p(t)\\
\frac{df}{dt} & = & c_f p(t) - \gamma_f f(t)
\end{eqnarray*}

\item We now consider the system's output to be $f(t)$, not $p(t)$. Similarly, $a(t) = x(t) - k f(t)$.

\item Performing the Laplace transforms and algebraic rearrangements as above, we find that when $m(0)=p(0)=f(0)$:

\[ G(s) = \frac{c_p c_m c_f}{\left( s + \gamma_p \right)\left( s + \gamma_m \right)\left( s + \gamma_f \right)} \]

\item Analysis of this system proceeds exactly as before. The closed loop transfer function $G(s)/(1+kG(s))$ has poles with positive real part only when the denominator, $1+kG(s)$, has roots with positive real part. The denominator has no poles with positive real part. Therefore, by the Nyquist stability criterion, $Z>0$ if and only if the Nyquist plot of $G(s)$ encloses the point $-1/k$.

\item As before, $G(s)=0$ on the semi-circular arc of the Nyquist contour, so we can draw the Nyquist plot by calculating $G(i \omega)$ for $\omega \in (-\infty, \infty)$.

\item Although this looks quite similar to the above, we will find that the system's behavior has changed qualitatively:
\begin{eqnarray*}
G(i \omega) & = &  \frac{c_p c_m c_f}{\left( i\omega + \gamma_p \right)\left( i\omega + \gamma_m \right)\left( i\omega + \gamma_f \right)} \cdot \frac{\left(- i\omega + \gamma_p \right)\left( - i\omega + \gamma_m \right)\left( -i\omega + \gamma_f \right)}{\left(- i\omega + \gamma_p \right)\left(- i\omega + \gamma_m \right)\left( -i\omega + \gamma_f \right)}\\
& = & \frac{c_p c_m c_f \left[ \gamma_p \gamma_m \gamma_f - \omega^2 \left(\gamma_p + \gamma_m + \gamma_f \right) + i \omega \left( \omega^2 - \left[ \gamma_p \gamma_m + \gamma_p \gamma_f + \gamma_m \gamma_f \right] \right) \right]}{\left( \omega^2 + \gamma_p^2 \right)\left( \omega^2 + \gamma_m^2 \right)\left( \omega^2 + \gamma_f^2 \right)}
\end{eqnarray*}

\item Now the plot crosses the real axis in two more places ($\omega \propto \pm \sqrt{ \gamma_p \gamma_m + \gamma_p \gamma_f + \gamma_m \gamma_f}$), and the real part of $G(i\omega)$ at those crossings is the same:

\[ c_p c_m c_f \left[ \gamma_p \gamma_m \gamma_f - \left( \gamma_p \gamma_m + \gamma_p \gamma_f + \gamma_m \gamma_f \right) \left(\gamma_p + \gamma_m + \gamma_f \right) \right] < 0  \]

\item The full Nyquist plot confirms our suspicion that some points on the negative real axis have been enclosed by the contour. If $-1/k$ falls within that range (i.e. if $k$ is sufficiently large),  the system becomes unstable. The gain will need to be kept below a threshold determined by the parameters above in order to prevent instabilty.

\item The more steps are included in a process, the easier it is for that process to become unstable; in a system with five components, the threshold gain $k$ would be even lower.

\item In these toy models, instability would imply that the system diverged. However, in biological systems, limited resources and the requirement that concentrations be non-negative ensures that a system will not really diverge. Instead, the system will likely either saturate or oscillate depending on the circumstances.

\item Long pathways are not the only sources of instability. The second problem on your fifth problem set will walk you through applying the Nyquist stability criterion in the case where we have a two-step system (i.e. we model just the repressor's mRNA and protein), but with a time delay between mRNA production and translation. To keep things simple, instead of solving the general case, you'll ultimately create a Nyquist plot for a particular set of parameter values. Every major step of the derivation is covered.

\item For now, however, we turn to another source of instability in negative autoregulation systems: non-linearity.
\end{itemize}

\section*{Goodwin oscillator}

\begin{itemize}
\item In the three-step pathway we just studied, when $k$ was too large, any small deviation of the input $x(t)$ from zero would cause the system to diverge. (If $x(t)=0$ for all time, then the mRNA, protein, and compound $f$ are never produced and $p(t)=0$ forever.) This manifests as oscillations that grow indefinitely in magnitude: not a very realistic outcome for a biological system (where concentrations do not fall below zero or rise indefinitel).

\item Through much of the course we have emphasized that biological systems are, for better or worse, slave to non-linear effects like hyperbolic binding, Michaelis-Menten kinetics, cooperativity, zero-order ultrasensitivity, and the like.

\item Since negative autoregulation is pervasive in biological systems, we might ask whether we have ``lost anything" by ignoring non-linearity. Might it be the case that these nonlinear terms foster stability in the system? Impede it?

\item We will see that a modified version of the three-step model studied earlier, assuming cooperativity in repressor binding, causes one of the behaviors commonly observed in natural negative autoregulation systems: stable oscillations.

\item We model an autoregulatory system where an mRNA $m$ begets a protein $p$ which begets a chemical compound $c$. The compound $f$ then acts cooperatively to inhibit expression of the mRNA (perhaps, by binding to a multimeric repressor to activate it).

\begin{eqnarray*}
\frac{dm}{dt}& = &\frac{a_1 K}{K + f^n} - b_1 m\\
\frac{dp}{dt} &= &a_2 m - b_2p\\
\frac{df}{dt} &= &a_3 p - b_3f
\end{eqnarray*}

\item We have a model with eight parameters. I will claim that four are superfluous, and that we can rewrite these equations in the form:

\begin{eqnarray*}
\frac{dz_1}{dt} &= &\frac{1}{1 + z_3^n} - c_1 z_1\\
\frac{dz_2}{dt} &= &z_1 - c_2 z_2\\
\frac{dz_3}{dt} &= &z_2 -c_3 z_3
\end{eqnarray*}

\item To achieve this, we will rescale all variables -- time and each concentration -- by an unknown constant. After substituting the scaled variables into our original equation, we can determine what the scaling factors should be.

\item We define scaling factors $w$ and $h_i$ such that $\tau = wt$, $z_1 = h_1 m$, $z_2 = h_2 p$, and $z_3 = h_3 f$. The first equation simplifies to:
\begin{eqnarray*}
\frac{dm}{dt} &=&\frac{w}{h_1} \frac{dz_1}{d\tau} = \left( a_1 \right) \frac{1}{1 + (\frac{z_3}{h_3 \sqrt[n]{K}})^n} - \left( \frac{b_1}{h_1} \right) z_1\\
\frac{dz_1}{d\tau}& =& \left( \frac{a_1h_1}{w} \right) \frac{1}{1 + (\frac{z_3}{h_3 \sqrt[n]{K}})^n} - \left( \frac{b_1}{w} \right) z_1
\end{eqnarray*}
The second equation to:
\begin{eqnarray*}
\frac{dp}{dt}& = &\frac{w}{h_2} \frac{dz_2}{d\tau} = \left( \frac{a_2}{h_1} \right) z_1 -  \left( \frac{b_2}{h_2} \right) z_2\\
\frac{dz_2}{d\tau}& = &\left( \frac{a_2 h_2}{wh_1} \right) z_1 -  \left( \frac{b_2}{w} \right) z_2
\end{eqnarray*}
And finally, by analogy, the third equation:
\begin{eqnarray*}
\frac{dz_3}{d\tau} = \left( \frac{a_3 h_3}{wh_2} \right) z_2 -  \left( \frac{b_3}{w} \right) z_3
\end{eqnarray*}

\item In order for these three equations to match the simplified version I proposed, we find that we need:

\[ h_3 = \frac{1}{\sqrt[n]{K}} \hspace{2 cm} \frac{a_1h_1}{w} = 1 \hspace{2 cm}  \frac{a_2h_2}{wh_1} = 1 \hspace{ 2 cm} \frac{a_3h_3}{wh_2} = 1 \]

\item The first condition can clearly be satisfied. To find $w$, note that if we multiple together equations two through four:

\[ \frac{a_1 a_2 a_3 h_3}{w^3} = 1 \implies w = \sqrt[3]{a_1 a_2 a_3 h_3} \]

Knowing $h_3$ and $w$ allows us to solve for $h_2$, and finally, for $h_1$.

\item The exact values of $w$ and the $h_i$ are not important: clearly we would be able to choose them appropriately to arrive at our simplified equations. The more valuable lesson is that while the binding constants and production/degradation rates are directly observable, it is more expedient and equally valid to consider three parameters in place of those seven. Ultimately we need only to track the Hill coefficient and three parameters that correspond to the effective degradation rates of each molecular species in the system:

\begin{eqnarray*}
\frac{dz_1}{dt} = \frac{1}{1 + z_3^n} - c_1 z_1\\
\frac{dz_2}{dt} = z_1 - c_2 z_2\\
\frac{dz_3}{dt} = z_2 -c_3 z_3
\end{eqnarray*}

\end{itemize}

\section*{Proof that the system is bounded}

\begin{itemize}
\item In an unstable system, one or more of the variables may diverge; the system may approach a limit cycle; or (in 3+ dimensions) the system may evolve chaotically.
\item If we assume that the initial concentrations in the Goodwin oscillator system are all non-negative, it turns out we can eliminate one of these possibilities right off the bat by showing that after a sufficiently long time, the concentrations in this system have upper bounds (and lower bounds at zero).

\item How can we show that this is so? When $z_i = 0$, its derivative $dz_i/d\tau$ is non-negative. Each $z_i$ will therefore never be decrease below zero: this establishes lower bounds for their values at zero.

\item To find the long-time upper bound for $z_1$, consider that since $z_3 >= 0$:
\[ \frac{dz_1}{d\tau} = \frac{1}{1 + z_3^n} -  c_1 z_1 \leq 1 - c_1 z_1\]
If $z_1$ begins at a value greater than $1/c_1$, over time it will decrease until it reaches at least $1/c_1$. Then, using the fact that $z < 1/c_1$ at long times,
\[ \frac{dz_2}{d\tau} = z_1 -  c_2 z_2 \leq \frac{1}{c_1}  - c_2 z_2\]
and now we can see that at sufficiently long times, if $z_2 > 1/c_1c_2$, it will decrease until it reaches at least $1/c_1c_2$. Similarly, $z_3 \leq 1/c_1c_2c_3$ at long times. So we have shown that all of the parameter values are bounded at long times -- our system cannot diverge.\\

\item This leaves three possibilities: our system could be stable, it could have a limit cycle, or it could be chaotic. A proof that the unstable solution for this system, if one exists, is a limit cycle would be beyond the scope of this class. However, if you are interested, you can consult a reasonably accessible proof for this (three-dimensional) case by Tyson\footnote{Tyson (1975). On the existence of oscillatory solution in negative feedback cellular control processes. \textit{J. Math. Biol.} \textbf{1}, 311-315.}, or a more general proof by Hastings et al.\footnote{Hastings, Tyson, and Webster (1977). Existence of periodic solutions for negative feedback cellular control systems. \textit{J. Diff. Eqn.}, \textbf{25}, 39-64.}. Henceforth we will take as granted that any non-stable solution is a limit cycle.
\end{itemize}

\section*{A single fixed point exists}
\begin{itemize}
\item Before determining when an unstable solution exists, we will find the steady-state solution $(z_1^*, z_2^*, z_3^*)$ by setting all derivatives equal to zero:

\[ 0 = \frac{1}{1 + \left( z_3^* \right)^n} - c_1 z_1^* \hspace{2 cm} 0 = z_1^* - c_2z_2^* \hspace{2 cm} 0 = z_2^* - c_3 z_3^* \]

\item Plugging the second and third equations into the first to eliminate two variables:

\[ c_1 c_2 c_3 z_3^* = \frac{1}{1 + \left( z_3^* \right)^n}  \]

\item The equation on the left is monotonically increasing with $z_3^*$ from zero to infinity; the one on the right is monotonically decreasing from one to zero. Therefore they must intersect exactly once at some positive value of $z_3^*$.
\item The intersection must occur before the left-hand side of the equation has a value greater than one (since the right-hand side is never greater than one), so:

\[ z_3^* < \frac{1}{c_1c_2c_3} \hspace{1 cm} z_2^*= c_3 z_3^* < \frac{1}{c_1 c_2} \hspace{1 cm} z_1^* = c_2 z_2^* < \frac{1}{c_1} \]

\item In other words, our steady-state solution is within the bounded region as expected.

\end{itemize}
\section*{Stability analysis}

\begin{itemize}

\item We would like to know whether the steady-state solution is stable. If not, trajectories that begin near the fixed point will approach the limit cycle instead of the fixed point.

\item Suppose that we begin at a position $\Delta z_i$ from the steady-state values $z_i^*$.  If this system is stable, these deviations will decrease to zero with time as the trajectory approaches the fixed point.

\item Our approach is to find three linear approximations for the deviation from the fixed point. Then, we could use either the Nyquist stability criterion or the fixed point analysis studied earlier in the course to determine when the system will be stable or unstable.

\item To simplify some algebra during the linearization, we define
\[ f(z_3) = \frac{1}{1 + \left( z_3 \right)^n} \]

\item The linear approximation around $z_3^*$ is:
\[ f(z_3) = f(z_3^* + \Delta x_3) \approx f(z_3^0) + \Delta z_3 f' (z_3^0) \]

\item This allows us to rewrite the first governing equation as:
\begin{eqnarray*}
\frac{d z_1}{d \tau} & = & f(z_3) - c_1 z_1\\
 \frac{d}{d \tau} \left( z_1^* + \Delta z_1\right) = 0 + \frac{d \left(\Delta z_1 \right)}{d \tau}  & \approx & f(z_3^*) + \Delta z_3 f' (z_3^*) - c_1 \left( z_1^* + \Delta z_1 \right)
\end{eqnarray*}

\item And since one of the steady-state conditions was:
\[ 0 = f(z_3^*) - c_1 z_1^* \]

The above can be further simplified to:
\[ \frac{d \left(\Delta z_1 \right)}{d \tau} \approx   \Delta z_3 f' (z_3^*) - c_1 \Delta z_1
\]

\item We need a more explicit expression for $f'(z_3^*)$. By taking the derivative of $f$ and using the steady-state condition $c_1c_2c_3 z_3^* = f(z_3^*)$:
\[ f'(z_3^*) = \frac{- n \left( z_3^* \right)^{n-1}}{\left[ 1 + \left(z_3^* \right)^n \right]^2} = -n c_1^2 c_2^2 c_3^2 \left( z_3^* \right)^{n+1} = -n c_1 c_2 c_3 \left( 1 - c_1c_2c_3 z_3^* \right) \]

This allows us to rewrite

\[ \frac{d \left(\Delta z_1 \right)}{d \tau} \approx   - \Delta z_3 n c_1 c_2 c_3 \left( 1 - c_1c_2c_3 z_3^* \right) -  c_1 \Delta z_1 \]

\item The second equation is  already linear:
\begin{eqnarray*}
\frac{d z_2}{d \tau} & = & z_1 - c_2 z_2\\
 \frac{d}{d \tau} \left( z_2^* + \Delta z_2\right) = \frac{d \left(\Delta z_2 \right)}{d \tau}  & \approx & z_1^* + \Delta z_1 - c_2 \left( z_2^* + \Delta z_2 \right)
 \end{eqnarray*}

\item From the second steady-state criterion:
\[ 0 =  z_1^* - c_2 z_2^* \]

which allows us to simplify the expression for $\Delta z_2/d \tau$:
\[  \frac{d \left(\Delta z_2 \right)}{d \tau}  \approx \Delta z_1 - c_2  \Delta z_2 \]

\item And analogously:
\[ \frac{d \left(\Delta z_3 \right)}{d \tau}  \approx \Delta z_2 - c_3  \Delta z_3 \]

\item We now have a linear system, which is stable if $\Delta z_i \to 0$ as $\tau \to \infty$.

\begin{eqnarray}
\frac{d}{d\tau} \begin{pmatrix} \Delta z_1 \\ \Delta z_2 \\ \Delta z_3 \end{pmatrix} & = & \begin{pmatrix} -c_1 & 0 & - n c_1 c_2 c_3 \left( 1 - c_1c_2c_3 z_3^* \right) \\ 1 & -c_2 & 0\\ 0 & 1 & -c_3 \end{pmatrix} \begin{pmatrix} \Delta z_1 \\ \Delta z_2 \\ \Delta z_3 \end{pmatrix} \label{eqn:goodwinlinearized}
\end{eqnarray}

\item Notice that the similarity to the system

\begin{eqnarray}
\frac{d}{dt} \begin{pmatrix} m \\ p \\ f \end{pmatrix} & = & \begin{pmatrix} -\gamma_m & 0 & - k c_m \\ c_p & -\gamma_p& 0\\ 0 & c_f & -\gamma_f \end{pmatrix} \begin{pmatrix} m \\ p\\ f \end{pmatrix} \label{eqn:threesteplinearized}
\end{eqnarray}

which is what the three-step pathway would have been if $x(t)=0$.

\item We know that the three-step pathway \ref{eqn:threesteplinearized} is unstable for large enough $k$. It therefore stands to reason that the fixed point Goodwin oscillator system \ref{eqn:goodwinlinearized} will be unstable, too, since it is just a particular choice of constants.

\item There is an interesting take-home point, however. Consider the effective ``repressor gain" for the Goodwin oscillator: it depends linearly on $n$. The implication is that if we can destabilize the system by choosing $n$ large enough -- that is, make the repressor binding \textit{more} cooperative! This result may be unintuitive in light of how cooperativity was required for stability in the mutual repressor and positive autoregulation bistable systems we examined earlier in the course.

\item Also note that instability of the fixed point in the Goodwin system, combined with the boundedness of solutions we showed earlier and the fact that the solutions are not chaotic (not proven here), suggest that there must be as table limit cycle.

\end{itemize}

\section*{Summary}

When we began our discussion of negative autorepression, we surmised that it would be useful for keeping the concentration of some protein (say, a transcriptional repressor) at or near a fixed value despite variation in the gene's expression due to inconsistent activation, $x(t)$. There was nothing sophisticated about that argument: if the protein's concentration got too high, it would decrease its own expression rate; if it got too low, its expression rate would increase.\\

Our closer exploration of negative autoregulation has revealed how dangerous such verisimilar arguments can be. We now know that for an unfortunate choice of parameters or in the case of a time delay, negative autoregulation can magnify small errors and drive oscillations. To make matters worse, any intuition developed earlier in the course that ``adding cooperativity will make it right" is quite far from the truth: increasing cooperativity can push a system across the threshold into instability.\\

Although this investigation has taken us down a rabbit hole of

\section*{Supplemental: full treatment of the Goodwin oscillator using the Nyquist stability criterion (paralleling Segel)}

Taking the Laplace transforms:

\begin{eqnarray*}
s \Delta \hat{z}_1 & = & f'(z_3^*) \Delta \hat{z}_3 - c_1 \Delta \hat{z}_1\\
s \Delta \hat{z}_2 & = & \Delta \hat{z}_1 - c_2 \Delta \hat{z}_2\\
s \Delta \hat{z}_3 & = & \Delta \hat{z}_2 - c_3 \Delta \hat{z}_3
\end{eqnarray*}

We can eliminate $\Delta \hat{z}_1 $ and $\Delta \hat{z}_2$ by substitution:
\begin{eqnarray*}
(s + c_1)(s + c_2)(s + c_3) \Delta \hat{z}_3 & = & f'(z_3^*) \Delta \hat{z}_3\\
\Delta \hat{z}_3 & = & \left[ \frac{f'(z_3^*)}{(s + c_1)(s + c_2)(s + c_3)} \right] \Delta \hat{z}_3 = f'(z_3^0) G(s) \Delta \hat{z}_3
\end{eqnarray*}
The gain of the system is $-f'(z_3^*)$ (a positive number, since $f$ is monotonically decreasing).

\subsection*{Nyquist stability criterion}

We begin by re-expressing $G(i\omega)$:
\begin{eqnarray*}
G(i\omega) & = & \frac{1}{(c_1 + i \omega)(c_2 + i \omega)(c_3 + i \omega)} \frac{(c_1 - i \omega)(c_2 - i \omega)(c_3 - i \omega)}{(c_1 - i \omega)(c_2 - i \omega)(c_3 - i \omega)}\\
& = & \frac{(c_1 - i \omega)(c_2 - i \omega)(c_3 - i \omega)}{(c_1^2 + \omega^2)(c_2^2 + \omega^2)(c_3^2 + \omega^2)}\\
& = & \frac{c_1c_2c_3 - \omega^2 (c_1 + c_2 + c_3) + i \omega \left[ \omega^2 - c_1c_2 - c_2c_3 - c_1 c_3 \right]}{(c_1^2 + \omega^2)(c_2^2 + \omega^2)(c_3^2 + \omega^2)}
\end{eqnarray*}
This function intercepts the real axis when $\omega = \omega_0 = 0$ and when
\[ \omega^2 = \omega_1^2 = c_1c_2 + c_2c_3 + c_1 c_3 \]
The system is unstable if the point $1/f'(z_3^*)$ (negative one over the gain constant) is encircled by $G(i\omega)$, i.e. $G(i \omega_1) < 1/f'(z_3^*) < G(0)$. The latter inequality is true since $G(0)$ is positive. So the system is unstable if:
\begin{eqnarray*}
G(i \omega_1) & < & \frac{1}{f'(z_3^*)}\\
\left| G(i \omega_1) \right| & > & \frac{1}{\left| f'(z_3^*) \right|}\\
\frac{\left| f'(z_3^*  \right|}{c_1c_2c_3}  \left| \frac{G(i \omega_1)}{G(0)} \right| & > & 1
\end{eqnarray*}

where on the last line we have used that $G(0) = 1/c_1c_2c_3$. Therefore the system is unstable if:

\[ n \left( 1 - c_1c_2c_3 z_3^* \right) \left| \frac{G(i \omega_1)}{G(0)} \right| > 1 \]

We would like to know the maximum value of the magnitude $| G(i \omega)/G(0) |$ so we can tell if and when this inequality will be met. This step requires employing trigonometric identities and optimizing subject to constraints (see Rapp 1975\footnote{Rapp (1975). A theoretical investigation of a large class of oscillators. \textit{Math. Biosci.} \textbf{25}, 165-188.})), with the result that
\[ \textrm{max } \left| \frac{G(i \omega_1)}{G(0)} \right| = \left[ \cos (\pi/3) \right]^3 = 1/8\]

Choosing $c_i=1$ for a comparative analysis\footnote{This choice is made arbitrarily to demonstrate how $n$ scales with the number of steps in the system.}, this system can be unstable if $n/8 > 1$, i.e., if $n=9$ or greater. A Hill coefficient of $n \geq 9$ may seem unlikely in a biological system, but e.g the two operators of bacteriophage $\lambda$ each bind six repressor molecules. If our pathway had more steps, a smaller Hill coefficient would suffice. With five steps (instead of three), we would only require $n > 3$; with eight steps, $n> 2$. The introduction of time delays also makes oscillations much more likely. Such oscillations can readily be observed in biological pathways with negative autoregulation.

\end{document}

lectures/Lecture 14 - When Negative Feedback Fails - The Goodwin Oscillator/lecture 14 notes.tex


\documentclass[10pt]{article}
\usepackage[a4paper, margin=1in]{geometry}
\usepackage{hyperref}
\usepackage{setspace}
\usepackage[style=iso]{datetime2}
\usepackage{amsmath}
\usepackage{eufrak}
\usepackage{amssymb}
\usepackage{mathtools}

\newcommand{\defeq}{\vcentcolon=}

\title{Automatic Differentiation Variational Inference - Mean Field Simplification}
\author{Harshad Deo \\
  \href{mailto:harshad@moreficent.com}{harshad@moreficent.com} \\
  \href{mailto:harshad@simianquant.com}{harshad@simianquant.com}
}
\date{}

\setlength{\parindent}{0cm}

\hypersetup{
  colorlinks,
  citecolor=blue,
  filecolor=blue,
  linkcolor=blue,
  urlcolor=blue
}

\setlength{\parskip}{\baselineskip}%

\begin{document}

\maketitle

This monograph is a follow up to the article introducing Auto Differentiation Variational Inference and develops simplifying
assumptions that allow the technique to be used to train large, complex models over large datasets, for example Bayesian
Deep Learning.

\section*{Decoupling The Latent Space}

The full rank approximation, developed in the previous monograph, allows all the latent variables ($\theta$) to be coupled
with each other through the correlation matrix of the variational approximation. As a result, the number of parameters
required to model the latent space grows quadratically with the size of the latent space. This makes it intractable for
use in all but the simplest problems, with a few thousand latent variables.

The mean field approximation assumes that all the latent variables are independent. Therefore, the number of parameters
required to model the latent space grows linearly with the size of the latent space, making it practical to scale the size
of the model. Formally,

\begin{align*}
  q(\zeta, \phi) \defeq& \mathcal{N}\big(\zeta; \mu, \text{diag}(\exp(\omega) ^2)\big) \\
  =& \prod_{k=1}^K \mathcal{N}(\zeta_k; \mu_k, \exp(\omega_k)^2)
\end{align*}

With this parameterization, the space of the variational parameters is given by:

\begin{equation*}
  \Phi = \{\mu_1, \ldots, \mu_k, \omega_1, \ldots, \omega_k\} = \mathbb{R}^{2K}
\end{equation*}

And the elliptical standardization is given by:

\begin{equation*}
  \eta_k = S_{\phi}(\zeta) = \frac{\zeta_k - \mu_k}{\exp \omega_k}
\end{equation*}

The correlation, if required, now needs to be modelled in explicitly as a function of the latent variables. For example,
the Lewandowski-Kurowicka-Joe transform \footnote{Lewandowski, D., Kurowicka, D., \& Joe, H. (2009). Generating random correlation matrices based on vines and extended onion method. Journal of multivariate analysis, 100(9), 1989-2001.} can be used to explicitly model a correlation matrix for a subset of the latent
variables.

\section*{Entropy Simplification}

The contribution of the entropy of the variational approximation to the ELBO can be simplified as:

\begin{align*}
  \frac{1}{2}\ln| \det \Sigma ^ 2| &= \frac{1}{2}\ln |\det \big(\text{diag}(\exp(\omega) ^ 2)\big)|  \\
  &= \frac{1}{2}\ln \prod_{k=1}^{K}\big(\exp(\omega_k)^2\big) \\
  &= \ln \prod_{k=1}^{K}\exp \omega_k \\
  &= \sum_{k=1}^{K}\omega_k
\end{align*}

Therefore the optimization objective can be simplified to:

\begin{equation*}
  \phi^* = \underset{\phi}{\text{argmax}} \, \mathbb{E}_{\mathcal{N}(\eta; 0, 1)}\big[\log p(x, T^{-1}(S_\phi^{-1}(\eta))) + \log |\det J_{T^{-1}}(S_\phi^{-1}(\eta))|\big] + \sum_{k=1}^{K}\omega_k
\end{equation*}

\section*{Joint Probability Simplification}

The joint probability of the data and the parameters, $p\big(x, T^{-1}(S_\phi^{-1}(\eta))\big)$ can be factorized as the product
of the probability of the data, given the parameters and the prior probability of the parameters. The objective can thus be
reexpressed as:

\begin{equation*}
  \phi^* = \underset{\phi}{\text{argmax}} \, \mathbb{E}_{\mathcal{N}(\eta; 0, 1)}\big[\log p(x | T^{-1}(S_\phi^{-1}(\eta))) + \log p(T^{-1}(S_\phi^{-1}(\eta))) + \log |\det J_{T^{-1}}(S_\phi^{-1}(\eta))|\big] + \sum_{k=1}^{K}\omega_k
\end{equation*}

Therefore, it is observed that the prior probability acts as a regularizing term on the distribution of the latent space. This gives
a clear and simple interpretation to standard regularization techniques like $L_1$ and $L_2$ regularization as assigning Laplace
and Gaussian priors respectively.

\section*{Graph Simplification}

To simplify the expression further, the family of constraint relaxation functions, $T(\theta)$ has to be restricted to those
for whom the Jacobian of the inverse transform is a triangular matrix. This is a benign assumption in practice because it is
true for virtually all of the commonly used transformations. Since the Jacobian of a triangular matrix is the product of
its diagonal, this assumption allows the computational graph to be factorized into mutually exclusive subgraphs $U_1 \ldots U_M, M \leq K$
such that $U_m, 1 \le m \leq M$ is only dependent on $U_1 \ldots U_{m - 1}$. Let

\begin{align*}
  z_m \defeq&\quad \text{The output of the m-th subgraph} \\
  \rho_m \defeq&\quad \text{The log prior probability of the m-th subgraph} \\
  \tau_m \defeq&\quad \text{The contribution of the jacobian of the inverse constraint transform} \\
  K_m \defeq&\quad \text{Indices for the m-th subgraph}
\end{align*}

The optimization objective can therefore be reexpressed as:

\begin{equation*}
  \phi^* = \underset{\phi}{\text{argmax}} \, \mathbb{E}_{\mathcal{N}(\eta; 0, 1)}\big[\log p(x | z_M) \big] + \sum_{m=1}^{M}\Big(\mathbb{E}_{\mathcal{N}(\eta_m; 0, 1)(\rho_m(z_1, \ldots, z_m ))}\Big)
\end{equation*}

\end{document}


\documentclass[]{report}

\usepackage{amsmath}

\newcommand{\bra}[1]{\left<#1\right|}
\newcommand{\ket}[1]{\left|#1\right>}
\newcommand{\bket}[2]{\left<#1|#2\right>}
\newcommand{\braket}[3]{\bra{#1}#2\ket{#3}}

% Title Page
\title{\texttt{mudirac} - a Dirac equation solver for muonic atoms
	\newline
	\large Development guide}
\author{Simone Sturniolo}

\begin{document}
\maketitle

\chapter*{Introduction}

This document contains information and references on the algorithms and methods employed for the development of \texttt{mudirac}. It is meant as a technical document, to keep track of technique used and allow for modification or bug fixing in the future.

\chapter{Theory}

\section{The Dirac equation}

The basic solver makes use of the radial Dirac equation written in its coupled form, here written in atomic units:

\begin{align}\label{dirac_sys}
	Q'(r) &= \frac{k}{r}Q + \left(mc-\frac{E-V(r)}{c}\right)P \\
	P'(r) &= -\frac{k}{r}P + \left(mc+\frac{E-V(r)}{c}\right)Q
\end{align}

as found for example in \cite{weinb2008, silbar2010, gross1999} and others. In this form, the final wavefunction can be reconstructed by combining it with the appropriate spherical harmonics:

\begin{align}\label{dirac_wavef}
\left<\mathbf{r}|\psi_{k\mu}\right> = \begin{pmatrix}
\frac{P_k}{r}\bket{\mathbf{\hat{r}}}{k\mu} \\
i\frac{Q_k}{r}\bket{\mathbf{\hat{r}}}{-k\mu}
\end{pmatrix}
\end{align}

where it must be kept in mind that the quantum number $k$ depends on the total angular momentum of the electron, namely, whether its angular and spin momenta are aligned or not. It is $k=-(l+1)$ when $j = l+1/2$ and $k=l$ when $j=l-1/2$, and the spin spherical harmonics are:

\begin{equation}\label{dirac_kmu}
\ket{k\mu} = \sum_{s=\pm 1/2} c(l,\frac{1}{2};\mu-s, s)\left|l,\mu-s\right>\Phi(s)
\end{equation}

with

\begin{align}\label{dirac_spharm}
\left<\mathbf{\hat{r}}|l,\mu-s\right> = Y_{l,\mu-s}(\mathbf{\hat{r}}) \\
\Phi\left(\frac{1}{2}\right) = \begin{pmatrix}
1 \\
0
\end{pmatrix}
\qquad
\Phi\left(-\frac{1}{2}\right) = \begin{pmatrix}
0 \\
1
\end{pmatrix}
\end{align}

with $Y_{lm}$ being the spherical harmonics, $c(l,\frac{1}{2};\mu-s, s)$ the Clebsch-Gordan coefficients \cite{weinb2008}, and $m=\mu-s$. The coefficients can also be conveniently expressed as

\begin{equation}\label{dirac_cgsym}
c(l,\frac{1}{2};\mu-s, s) = \begin{cases}
-\mathrm{sgn}(k)\sqrt{\frac{k+\frac{1}{2}-\mu}{2k+1}} \qquad s = \frac{1}{2} \\
\sqrt{\frac{k+\frac{1}{2}+\mu}{2k+1}} \qquad s = -\frac{1}{2}
\end{cases}
\end{equation}

as seen in \cite{gross1999, biedernharn1981}.\newline
Since this equation is one dimensional we don't need to worry about Q and P being complex-valued, as they will be always real. In general, P will be the `large' component, and in the limit of $c \rightarrow \infty$ we would have that $P^2(r) = r^2\Psi^2(r)$ where $\Psi$ is just the normal radial component of the Schr\"{o}dinger equation solution. A higher Q corresponds to a more `relativistic' behaviour.

\section{Energy corrections}

While the main contribution to the potential for a muonic atom is the Coulombic force generated by the nucleus, $-Z/r$, there are additional contributions that we need to consider due to the short radius and high energy of muonic orbitals \cite{borie1982}.

\subsection{Nuclear finite size}

Since muonic orbitals are squeezed in a much smaller space than electronic ones, the nucleus can not be considered as a point charge when computing their energy. The effective charge distribution of atomic nuclei is not obvious. In first approximation, one can treat the nucleus as a uniformly charged sphere of finite radius. Appropriate radii have been measured and listed in literature, and where available, these values are used \cite{angeli2013}. It should be noted that the values tabulated in the paper are values for the root mean square radius, which means they need to be multiplied by $\sqrt{5/3}$ to get the equivalent radius of a uniformly charged sphere. For any isotope for which the radius is not known, we fall back on the empirical formula \cite{borie1982}:

\begin{equation}\label{nuc_radius}
R = 1.2 A^{1/3} \mathrm{fm} = 2.267\cdot10^{-5}A^3
\end{equation}

, where the second version is in atomic units.\newline
While sufficient to achieve satisfying accuracy for most cases, this finite sphere approximation still produces significant errors in some extreme circumstances, in particular transitions between the first two shells in heavy elements such as lead. In this case, a further increase in accuracy can be achieved by smoothing the charge distribution from a sphere with a sharp cutoff to a Fermi 2-term distribution \cite{visscher1997}:

\begin{equation}\label{s:theory:fermi2}
\rho(r) = \rho_0\frac{1}{1+e^{4\ln (3) \frac{r-R_F}{\delta_F}}}
\end{equation}

, where $R_F$ and $\delta_F$ are two parameters corresponding roughly to the size of the nucleus and the thickness of its `skin', the shell across which the density falls to zero. In particular, $R_F$ is connected to the corresponding uniform sphere radius $R$  for the same nucleus by

\begin{equation}\label{s:theory:fermi2radius_heavy}
R_F = \sqrt{R^2-\frac{7}{3}\left(\frac{\pi \delta_F}{4\ln(3)}\right)^2}
\end{equation}

for atoms where in atomic mass units $A \ge 5$, while for lighter nuclei, where the above formula would fail, we set

\begin{equation}\label{s:theory:fermi2radius_light}
R_F = (2.2291\cdot10^{-5}A^{1/3}-0.90676\cdot10^{-5})
\end{equation}

in atomic units. In following with \cite{visscher1997}, we also take $\delta_F = 2.3 fm = 4.3464\cdot10^{-5}$ for all nuclei. The charge distribution in eq. \ref{s:theory:fermi2} does not produce a potential that can be described analytically.

\subsection{Vacuum polarizability}\label{vacpot}

The next term to include is the effect of the polarizability of the vacuum. The regular Coulomb law is based on the principle of vacuum having zero polarizability, but that is not an accurate description once one takes into account quantum field theory. Vacuum can be polarized by the appearance of charged particle pairs, which constitute Feynman diagrams with added loops in the photon propagator \cite{borie1982}. The biggest term is the diagram in which one electron-positron pair is created, of order $\alpha$. Diagrams with more loops will be of order $\alpha^2$ or even smaller, and diagrams involving pairs of more massive charged particles such as muons and antimuons also contribute much smaller contributions due to the higher energy required to form such particles, which makes them vanish exponentially. Accounting for this term gives rise to a correction to the Coulomb potential known as the Uehling potential \cite{johnson2001}:

\begin{equation}\label{uehl_point}
\delta V(r) = -\frac{2\alpha Z}{3\pi r}\int_0^1 du \sqrt{1-u^2}\left(\frac{1}{u}+\frac{1}{2}u\right)e^{-2cr/u}
\end{equation}

. The form seen in Eq. \ref{uehl_point} however is only valid for a point-like charge, which makes it not very useful for most muonic atoms. In presence of a charge distribution the Uehling potential becomes

\begin{equation}\label{uehl_distr}
\delta V_D(r) = -\frac{2\alpha^2}{3 r}\int_0^\infty dx x \rho(x)\int_0^1 du \sqrt{1-u^2}\left(1+\frac{1}{2}u^2\right)\left(e^{-2c|r-x|/u}+e^{-2c(r+x)/u}\right)
\end{equation}

. For the special case of a spherical distribution of charge of radius $R$ we have

\begin{equation}
\rho(x) = \begin{cases}
\frac{3}{4\pi}\frac{Z}{R^3} \qquad &x \leq R \\
0 \qquad &x > R \\
\end{cases}
\end{equation}

and thus the integral becomes

\begin{equation}\label{uehl_sphere}
\delta V_S(r) = -\frac{\alpha^2Z}{2\pi rR^3}\int_0^R dx x \int_0^1 du \sqrt{1-u^2}\left(1+\frac{1}{2}u^2\right)\left(e^{-2c|r-x|/u}+e^{-2c(r+x)/u}\right)
\end{equation}

The integral in $x$ can be carried out if we consider the two distinct cases of $x < r$ and $x \geq r$. For the first we have

\begin{multline}\label{intx_less}
X_{<}(r, u | R) = \int_0^R x \left(e^{-2c|r-x|/u}-e^{-2c(r+x)/u}\right)  dx = \\
e^{-2cr/u}\int_0^R x \left(e^{2cx/u}-e^{-2cx/u}\right)  dx = \\ e^{-2cr/u}\left[e^{2cR/u}\left(\frac{Ru}{2c}-\frac{u^2}{4c^2}\right)+e^{-2cR/u}\left(\frac{Ru}{2c}+\frac{u^2}{4c^2}\right)\right]
\end{multline}

while for the second

\begin{multline}\label{intx_more}
X_{>}(r, u | R) =\int_0^R x \left(e^{-2c|r-x|/u}-e^{-2c(r+x)/u}\right)  dx = \\
\int_0^R x \left(e^{-2c(x-r)/u}-e^{-2c(x+r)/u}\right)  dx = \\ \left(e^{2cr/u}-e^{-2cr/u}\right)\left[-\frac{Ru}{2c}e^{-2Rc/u}-\frac{u^2}{4c^2}(e^{-2Rc/u}-1)\right]
\end{multline}

. It then follows that the full term can be expressed as

\begin{multline}
\delta V_S(r) =  -\frac{\alpha^2Z}{2\pi rR^3}\int_0^1 du \sqrt{1-u^2}\left(1+\frac{1}{2}u^2\right) \times \\
\left[X_{<}(r, u | \min(r, R)) + X_{>}(r, u | R) -X_{>}(r, u | \min(r, R))\right]
\end{multline}

which is the expression used for the Uehling potential in the implementation. The integration over $u$ can be carried out numerically.

\subsection{Electronic shielding}

The effect of other electrons orbiting the nucleus in shielding its positive charge from the muon, thus shifting the energy, is usually quite small. It becomes bigger for heavier nuclei and higher muon transitions, but in practice it never seems to exceed $\sim100 \,\mathrm{eV}$ in most cases that matter \cite{borie1982}. Accounting for it in full would require one to solve the Schr\"{o}dinger or Dirac equation simultaneously for the muon and for all orbiting electrons, using methods such as Density Functional Theory to account for the many-body interactions. This method would be overly complex and time consuming for a minimal gain in accuracy. Here we elect instead to make use of the method developed by Tauscher et al. \cite{tauscher1978}. This approximation uses the following algorithm:

\begin{itemize}
	\item treat the muon as close enough to the nucleus to shield completely its charge, thus reducing it to Z-1;
	\item compute the density of the electrons in the n=1 shell as if they occupied unperturbed orbitals for a hydrogen-like atom;
	\item subtract the total charge of the electrons added in this way from the effective nuclear charge that determines the orbitals of the electrons in the shell above; repeat until all electrons are placed.
\end{itemize}

So, for example, for a neutral sodium atom (Z=11), one would construct a total density consisting of:

\begin{itemize}
	\item 2 electrons in a 1s orbital calculated with Z = 10;
	\item 8 electrons distributed between a 2s and a 2p orbital, both calculated with Z = 8;
	\item 1 electron in a 3s orbital calculated with Z = 1.
\end{itemize}

In this way we account crudely for the shielding effect that inner electrons have on outer ones. The potential from this charge background is then calculated with the integration technique described in section \ref{subsec_bkg_pot}, and the result is added to the other terms (Coulomb and Uehling). The Uehling correction to the muon-electron interactions is considered trivial and ignored. This method has been shown to give the correct electronic shielding energy contribution for muons to a 5\% accuracy.

\section{Dipole matrix elements}

To compute the transition probabilities between two states in presence of an electromagnetic field we need to compute the dipole matrix element between them. For spontaneous emission between two states $a$ and $b$, in atomic units, we have \cite{payne1955}

\begin{equation}\label{trans_prob}
W_{ab} = \frac{4}{3}\frac{E_{ab}}{c}|\braket{a}{|\vec{\alpha}|j_0(kr)}{b}|^2
\end{equation}

with $E_{ab}$ transition energy, the wave vector has magnitude $k=E_{ab}/c$, and $\vec{\alpha}$ is the vector of Dirac matrices $\alpha_x, \alpha_y, \alpha_z$.
If we consider the wavefunction written as seen in \ref{dirac_wavef},
then it follows that each individual matrix element can be written as

\begin{align}\label{matel_alphaj0}
\braket{a}{\alpha_ij_0(kr)}{b} = & i\braket{k_a\mu_a}{\sigma_i}{-k_b\mu_b}\int_{0}^{\infty}j_0(kr)P_aQ_b dr - \\
& i\braket{-k_a\mu_a}{\sigma_i}{k_b\mu_b}\int_{0}^{\infty}j_0(kr)P_bQ_a dr \nonumber
\end{align}

with the $\sigma_i$ being the Pauli matrices, keeping in mind that

\begin{equation}
\alpha_i = \begin{bmatrix}
0 & \sigma_i \\
-\sigma_i & 0
\end{bmatrix}
\end{equation}

. It should be recalled that in Eq. \ref{matel_alphaj0} the minus sign appears because the part in the bra is the adjoint, and not merely the complex conjugate, of the wavefunction. Now, keeping in mind Eq. \ref{dirac_kmu} and \ref{dirac_spharm}, and shortening the Clebsch-Gordan symbols seen in Eq. \ref{dirac_cgsym} as $c_{k\mu}^s = c(l,\frac{1}{2};\mu-s, s)$, we have

\begin{equation}
\ket{k\mu} = \begin{bmatrix}
c_{k\mu}^{1/2} Y_{l,\mu-1/2} \\
c_{k\mu}^{-1/2} Y_{l,\mu+1/2} \\
\end{bmatrix}
\end{equation}

and therefore,

\begin{equation}\label{matel_sz}
\braket{k'\mu'}{\sigma_z}{k\mu} = \left(c_{k'\mu'}^{1/2}c_{k\mu}^{1/2}-c_{k'\mu'}^{-1/2}c_{k\mu}^{-1/2}\right)\delta_{ll'}\delta_{\mu\mu'}
\end{equation}

\begin{equation}\label{matel_sp}
\braket{k'\mu'}{\sigma_+}{k\mu} = c_{k'\mu'}^{1/2} c_{k\mu}^{-1/2}\delta_{ll'}\delta_{\mu+1\mu'}
\end{equation}

\begin{equation}\label{matel_sm}
\braket{k'\mu'}{\sigma_-}{k\mu} = c_{k'\mu'}^{-1/2} c_{k\mu}^{1/2}\delta_{ll'}\delta_{\mu\mu'+1}
\end{equation}

. If we also set for brevity

\begin{equation}\label{pq_ints}
J_{ab} = \int_{0}^{\infty}j_0(kr)P_aQ_b dr
\end{equation}

, and we consider that if to a certain $k$ corresponds an $l$, then to $-k$ corresponds $l-\mathrm{sgn}(k)$, then Eq. \ref{matel_alphaj0} reduces to

\begin{align}\label{matel_alphaxj0}
\braket{a}{\alpha_xj_0(kr)}{b} = & i\left[\left(c_{k_a\mu_a}^{1/2} c_{-k_b\mu_b}^{-1/2}\delta_{\mu_a\mu_b+1}
+ c_{k_a\mu_a}^{-1/2} c_{-k_b\mu_b}^{1/2}\delta_{\mu_a+1\mu_b}\right)
\delta_{l_al_b-\mathrm{sgn}(k_b)} J_{ab}  - \right. \\
&\left.
\left(c_{-k_a\mu_a}^{1/2} c_{k_b\mu_b}^{-1/2}\delta_{\mu_a\mu_b+1}
+ c_{-k_a\mu_a}^{-1/2} c_{k_b\mu_b}^{1/2}\delta_{\mu_a+1\mu_b}\right)
\delta_{l_a-\mathrm{sgn}(k_a)l_b} J_{ba}
\right] \nonumber
\end{align}

\begin{align}\label{matel_alphayj0}
\braket{a}{\alpha_yj_0(kr)}{b} = & -\left[\left(c_{k_a\mu_a}^{1/2} c_{-k_b\mu_b}^{-1/2}\delta_{\mu_a\mu_b+1}
- c_{k_a\mu_a}^{-1/2} c_{-k_b\mu_b}^{1/2}\delta_{\mu_a+1\mu_b}\right)
\delta_{l_al_b-\mathrm{sgn}(k_b)} J_{ab}  - \right. \\
&\left.
\left(c_{-k_a\mu_a}^{1/2} c_{k_b\mu_b}^{-1/2}\delta_{\mu_a\mu_b+1}
- c_{-k_a\mu_a}^{-1/2} c_{k_b\mu_b}^{1/2}\delta_{\mu_a+1\mu_b}\right)
\delta_{l_a-\mathrm{sgn}(k_a)l_b} J_{ba}
\right] \nonumber
\end{align}

\begin{align}\label{matel_alphazj0}
\braket{a}{\alpha_zj_0(kr)}{b} = & i\left[\left(c_{k_a\mu_a}^{1/2} c_{-k_b\mu_b}^{1/2}-
 c_{k_a\mu_a}^{-1/2} c_{-k_b\mu_b}^{-1/2}\right)
\delta_{l_al_b-\mathrm{sgn}(k_b)} J_{ab}  - \right. \\
&\left.
\left(c_{-k_a\mu_a}^{1/2} c_{k_b\mu_b}^{1/2}-
c_{-k_a\mu_a}^{-1/2} c_{k_b\mu_b}^{-1/2}\right)\delta_{l_a-\mathrm{sgn}(k_a)l_b} J_{ba}
\right]\delta_{\mu_a\mu_b} \nonumber
\end{align}

. From these it becomes then easy to compute the transition probability as seen in Eq. \ref{trans_prob}. The transition rules are easily seen: only transitions with $\Delta l = \pm 1$ and $\Delta \mu = 0, \pm1$ are allowed. The integrals $J_{ab}$ can be computed numerically.

\chapter{Numerical implementation}

\section{Integration of the Dirac equation system}

\subsection{Grid}

The radial Dirac equation as written in \ref{dirac_sys} constitutes a system of two coupled first order differential equations. We use the shooting method to solve this problem. This approach requires one to choose the expected eigenvalue $E$ as a parameter, choose suitable guesses for the values at the
, then integrate numerically the equation from both sides of the domain up to a meeting point. The parameter is then adjusted based on how well the solutions match until stability is reached.\newline
There are various practical issues involved in solving the equation this way. The first thing is the grid to use. Here we choose to use a logarithmic radial grid, defined as:

\begin{equation}
r = r_0e^x \qquad x = \log\left(\frac{r}{r_0}\right)
\end{equation}

In this format the equations become

\begin{align}\label{dirac_sys_log}
\frac{\partial Q}{dx} &= kQ + r(x)\left(mc-\frac{E-V(x)}{c}\right)P \\
\frac{\partial P}{dx} &= -kP + r(x)\left(mc+\frac{E-V(x)}{c}\right)Q
\end{align}

The advantage is that an evenly spaced grid in $x$ is now much denser for small $r$, namely, for a Coulomb-like potential, close to the nucleus, where the wavefunction will vary more quickly.

\subsection{Boundary conditions}
Another issue is that of the boundary conditions. For $r \rightarrow \infty$, one can see how $V \rightarrow 0$ and the equations reduce to \cite{silbar2010}

\begin{align}\label{dirac_sys_inf}
Q' &\approx \left(mc-\frac{E}{c}\right)P \\
P' &\approx \left(mc+\frac{E}{c}\right)Q
\end{align}

leading to the conditions (defining $K = \sqrt{m^2c^2-E^2/c^2}$):

\begin{align}
P(r) &\approx e^{-Kr} \\
Q(r) &\approx -\frac{K}{mc+E/c}P(r)
\end{align}

whereas for $r \rightarrow 0$ we find different results depending on whether we use a `true' Coulomb potential (point charge) or one where the charge is assumed to have a finite spherical size. By expanding $P$, $Q$ and $V$ in power series for a point charge we find \cite{grant2009}

\begin{align}
P(r) &\approx r^{\gamma} \\
Q(r) &\approx -\frac{Z}{c(\gamma-k)}P(r)
\end{align}

with $\gamma = \sqrt{k^2-Z^2\alpha^2}$, while for a finite charge for $k < 0$ we have

\begin{align}
P(r) &\approx r^{-k} \\
Q(r) &\approx \frac{3Z}{2cR(-2k+1)}r^{-k+1}
\end{align}

whereas for $k > 0$

\begin{align}
P(r) &\approx r^{k+2} \\
Q(r) &\approx \frac{2cR(2k+1)}{3Z}r^{k+1}
\end{align}

making use of the fact that $V(0) = -3Z/2R$ for a uniformly charged spherical nucleus.

\subsection{Potential}\label{num_pot}

The electrostatic potential for the equation can be built out of multiple parts, representing the various contributions. Which ones are used depend on the desired level of accuracy.

\subsubsection{Coulomb potential}

The fundamental potential to apply is always the basic Coulomb potential. For a point-like nuclear charge, this is simply:

\begin{equation}
	V_N(r) = -\frac{Z}{r}
\end{equation}

It should be noted that since we're dealing with negative muons, all potential terms are implicit multiplied by one negative unit of charge, which is the cause of the minus sign. In case one considers a finite sized spherical nucleus of uniform charge and radius $R$, one finds:

\begin{equation}
V_N(r) = \begin{cases}
-\frac{Z}{R}\left(\frac{3}{2}-\frac{1}{2}\frac{r^2}{R^2}\right), & \text{$r\leq R$}.\\
-\frac{Z}{r}, & \text{$r > R$}.
\end{cases}
\end{equation}

\subsubsection{Potential from a radial charge distribution}\label{subsec_bkg_pot}
In presence of an arbitrary radial charge distribution, one builds the potential by integrating the equation:

\begin{equation}
\nabla^2 V = \rho
\end{equation}

which in radial coordinates becomes

\begin{equation}
	\frac{\partial}{\partial r}\left(r^2	\frac{\partial V}{\partial r}\right) = 4\pi r^2 \rho(r)
\end{equation}

where $4\pi r^2 \rho(r)$ represents the charge density of a spherical shell. Since we're using a logarithmic grid, when expressing the derivatives with respect to the $\log(r)$ we have

\begin{equation}
\frac{\partial^2 V}{\partial x^2} + \frac{\partial V}{\partial x}= 4\pi r^2 \rho(r)
\end{equation}

Due to the presence of a first order derivative this equation can not be integrated using Numerov's method. Instead, a finite order derivative scheme was used making use of the following approximations:

\begin{align}
\left.\frac{\partial V}{\partial x}\right|_i &\approx \frac{1}{h}\left(\frac{11}{6}V_i -3V_{i-1} +\frac{3}{2}V_{i-2}-\frac{1}{3}V_{i-3} \right) \\
\left.\frac{\partial^2 V}{\partial x^2}\right|_i &\approx \frac{1}{h^2}\left(2V_i -5V_{i-1} +4V_{i-2}-V_{i-3} \right) \\
\end{align}

setting the first three points of V with an approximation of constant charge for $r < r_0$ and shooting outwards for all the following ones.\newline
When the charge is defined on a limited radial grid, one also has the problem of what to use outside of its bounds. The convention adopted in the software is to always make it so that $V(\infty) = 0$ and to assume the charge to be constant inside the innermost radius and zero outside the outermost one. So if the grid gives us a charge distribution $\rho(r)$ defined between $R_{in}$ and $R_{out}$, and the resulting integrated potential is $V_{grid}(r)$, we will then define the total potential as

\begin{equation}
V_{bkg}(r) = \begin{cases}
-\frac{Q}{r}\qquad & r > R_{out} \\
V_{grid}(r) -V_{grid}(R_{out}) - \frac{Q}{R_{out}} \qquad & R_{out} \geq r > R_{in} \\
\frac{2}{3}\pi r^2\rho(R_{in})-V_{grid}(R_{out}) - \frac{Q}{R_{out}} \qquad & r \leq R_{in}
\end{cases}
\end{equation}

where

\begin{equation}
Q = \int_{R_{in}}^{R_{out}} 4\pi r^2\rho(r) dr + \frac{4}{3}\pi R_{in}^3\rho(R_{in})
\end{equation}

is the total charge of the background (including a term counting the assumed constant charge within the innermost radius of the grid).

\subsubsection{Uehling potential}

Both the nucleus and the charge distribution give rise to a further term due to vacuum polarizability. The full form for this has been presented in \ref{vacpot}.

\subsection{Integration scheme}

In order to integrate numerically the equations we make use of a fourth order Runge-Kutta method (RK4) \cite{press1992} adapted to work with coupled differential equations. Specifically, to integrate a system of the form:

\begin{align}\label{rk_qp}
Q'_i &= AA_i Q_i + AB_iP_i \\
P'_i &= BA_i Q_i + BB_iP_i
\end{align}

we apply the following rule:

\begin{align}
Q_i &= Q_{i-1} + \frac{1}{6}\left(k_{Q1}+2k_{Q2}+2k_{Q3}+k_{Q4}\right) \\
P_i &= P_{i-1} + \frac{1}{6}\left(k_{P1}+2k_{P2}+2k_{P3}+k_{P4}\right)
\end{align}

where we define

\begin{align}
k_{Q1} &= h\left(AA_{i-1}Q_{i-1}+AB_{i-1}P_{i-1}\right) \\
k_{Q2} &= h\left[AA_{i-1/2}\left(Q_{i-1}+\frac{k_{Q1}}{2}\right)+
AB_{i-1/2}\left(P_{i-1}+\frac{k_{P1}}{2}\right)\right] \\
k_{Q3} &= h\left[AA_{i-1/2}\left(Q_{i-1}+\frac{k_{Q2}}{2}\right)+
AB_{i-1/2}\left(P_{i-1}+\frac{k_{P2}}{2}\right)\right] \\
k_{Q4} &= h\left[AA_{i}(Q_{i-1}+k_{Q3})+AB_{i}(P_{i-1}+k_{P3})\right] \\
\end{align}

with similar expressions for the $k_P$, and with half-integers indices meaning $AA_{i-1/2} = (AA_i+AA_{i-1})/2$.

\subsection{Finding the eigenvalue}

The equations are integrated forward from $r\approx0$ and backwards from $r\approx\infty$ up to a meeting point. This is chosen to be the `turning point', namely the first point on the grid (going outwards) for which

\begin{equation}
E-mc^2 = B < V(r)
\end{equation}

where we call $B$ the `binding energy' of the particle, corresponding to the traditional energy eigenvalue in the Schr\"{o}dinger equation. This is the effective energy the particle has at its disposal to escape the potential, besides its rest mass energy term $mc^2$.\newline
Since the wavefunctions are not normalized yet, their values can all be scaled by a constant term. This means that comparing them directly is meaningless. Rather, in order to find whether the solutions `match', one can impose the condition:

\begin{equation}
\left. \frac{Q}{P}\right|_f = \left. \frac{Q}{P}\right|_b
\end{equation}

with f and b indicating forward or backwards integration. Since this is not usually the case, one will attempt a first value of $E$, then try to calculate

\begin{equation}
\frac{\partial}{\partial E}\left(\left. \frac{Q}{P}\right|_f - \left. \frac{Q}{P}\right|_b\right) =
\frac{\partial}{\partial E}\left. \frac{Q}{P}\right|_f - \frac{\partial}{\partial E}\left. \frac{Q}{P}\right|_b
\end{equation}

to then use a steepest descent algorithm to find the correct value of $E$. This tends to be a very well behaved function, so optimisation is straightforward. One approach to doing this is numerical - one simply integrates the equation both at $E$ and $E+\delta E$, then computes the derivative from the difference of the values found. However it's more efficient to look directly at the behaviour of $y = Q/P$ as its own function. By dividing $Q'$ by $P$ and defining for convenience:

\begin{equation}
g_\pm = \left(mc\pm\frac{E-V}{c}\right)
\end{equation}

we have

\begin{equation}
\frac{Q'}{P} = \frac{k}{r}\frac{Q}{P}+g_-
\end{equation}

and since (making use of the other equation for $P'$)

\begin{equation}
y' = \frac{Q'}{P} - \frac{Q}{P^2}P' = \frac{Q'}{P} + \frac{k}{r} y -g_+y^2
\end{equation}

we have

\begin{equation}\label{y_err_ode}
y' = 2\frac{k}{r}y-g_+y^2+g_-
\end{equation}

However what we are interested in is $\zeta = \partial y / \partial E$. We can derive and find:

\begin{equation}\label{zeta_err_ode}
\zeta' = 2\left(\frac{k}{r}-g_+y\right)\zeta-\frac{1}{c}(y^2+1)
\end{equation}

It should be noted that integrating this expression becomes problematic around nodes of the function $P$, where $y$ has a singularity. However, this problem can easily be fixed as a very similar expression can be written looking at the inverse ratio, $P/Q$, and substituting it in Eq. \ref{y_err_ode}:

\begin{equation}\label{eta_err_ode}
\eta' =\frac{\partial^2}{\partial E\partial x}\left(\frac{1}{y}\right) = -\frac{1}{y^2}\zeta' = -2\left(\frac{k}{r}+g_-\frac{1}{y}\right)\eta+\frac{1}{c}(\frac{1}{y^2}+1)
\end{equation}

Since we already know the values of Q and P, and thus of y, it's possible to integrate numerically Eq. \ref{zeta_err_ode}, switching then to Eq. \ref{eta_err_ode} when $y$ becomes too big (in the code we use $|y|>\alpha$ as our threshold). Then we can find

\begin{equation}
\frac{\partial}{\partial E}\left. \frac{Q}{P}\right|_f - \frac{\partial}{\partial E}\left. \frac{Q}{P}\right|_b = \zeta_f-\zeta_b
\end{equation}

and this can be used in combination with a steepest descent algorithm to minimize $y_f-y_b$ with respect to $E$ and thus find the correct eigenvalue.

\bibliographystyle{unsrt}
\bibliography{DevelopmentGuide.bib}

\end{document}


\begin{quote}\textit{
	``There was a mistranslation in Grothendieck's quote, `the rising sea:' he was actually talking about raising
	an \(X\)-wing fighter out of a swamp using the Force.''
	% the force-alg-geo analogy works surprisingly well
	% it's a little bit more like a religion
	% Kylo Ren is Lurie, ...
}\end{quote}
There are a lot of things that go under the scheme of algebraic geometry, but in this class we're going to use the
slogan ``algebra = geometry;'' we'll try to understand algebraic objects in terms of geometry and vice versa.

There are two main bridges between algebra and geometry: to a geometric object we can associate algebra via
functions, and the reverse construction might be less familiar, the notion of a spectrum. This is very similar to
the notion of the spectrum of an operator.

We will follow the textbook of Ravi Vakil,
\href{http://math.stanford.edu/~vakil/216blog/FOAGdec2915public.pdf}{\textit{The Rising Sea}}. There's also a
course website.\footnote{\url{https://www.ma.utexas.edu/users/benzvi/teaching/alggeom/syllabus.html}.} The
prerequisites will include some commutative algebra, but not too much category theory; some people in the class
might be bored. Though we're not going to assume much about algebraic sets, basic algebraic geometry, etc., it will
be helpful to have seen it.

Let's start. Suppose \(X\) is a space; then, there's generally a notion of \(\C\)-valued functions on it, and this
space might be \(F(X)\). For example, if \(X\) is a smooth manifold, we have \(C^\infty(X)\), and if \(X\) is a
complex manifold, we have the holomorphic functions \(\operatorname{Hol}(X)\).\footnote{The best examples here are
Riemann surfaces; when the professor imagines a ``typical'' or example algebraic variety, he sees a Riemann
surface.} Another category of good examples is \term{algebraic sets}, \(X\subset\C^n\) that is given by the common
zero set of a bunch of polynomials: \(X = \{f_1(x) =\dotsb= f_k(x) = 0\}\) for some
\(f_1,\dots,f_k\in\C[x_1,\dots,x_n]\). These have a natural notion of function, \term{polynomial functions}, which
are polynomials in \(\C[x_1,\dots,x_n]\) restricted to \(X\), If \(I(X)\) is the functions vanishing on \(X\), then
these functions are given by \(\C[x_1,\dots,x_n]/I\).

The point is, on all of our spaces, the functions have a natural ring structure.\footnote{In this class, all rings
will be commutative and have a \(1\). Ring homomorphisms will send \(1\) to \(1\).} In fact, there's more: the
constant functions are a map \(\C\to F(X)\), and since \(\C\) is a field, this map is injective. This means
\(F(X)\) is a \(\C\)-algebra, i.e.\ it is a \(\C\)-vector space with a commutative, \(\C\)-linear multiplication.

One of the things Grothendieck emphasized is that one should never look at a space (or an anything) on its own, but
consider it along with maps between spaces. For example, given a map \(\pi:X\to Y\) of spaces, we always have a
\term{pullback} homomorphism \(\pi^*: F(Y)\to F(X)\): if \(f:Y\to\C\), then its pullback is \(\pi^*y(x) =
y(\pi(x))\). This tells us that we have a \term{functor} from spaces to commutative rings.
\subsection*{Categories and Functors.}
This is all done in Vakil's book, but in case you haven't encountered any categories in the streets, let's revisit
them.
\begin{defn}
A \term{category} \(\fC\) consists of a set\footnote{This is wrong. But if you already know that, you know that
worrying about set-theoretic difficulties is a major distraction here, and not necessary for what we're doing, so
we're not going to worry about it.} of \term{objects} \(\operatorname{Ob}\fC\); if \(X\in\operatorname{Ob}\fC\), we
just say \(X\in\fC\). We also have for every \(X,Y\in\fC\) the set \(\Hom_\fC(X,Y)\) of \term{morphisms}. For every
\(X,Y,Z\in\fC\), there's a \term{composition map} \(\Hom_\fC(X,Y)\times\Hom_\fC(Y,Z)\to\Hom_\fC(Y,Z)\) and a unit
\(1_X\in\Hom_\fC(X,X) = \operatorname{End}_\fC(X)\) satisfying a bunch of axioms that make this behave like
associative function composition.
\end{defn}
To be precise, we want categories to behave like monoids, for which the product is associative and unital. In fact,
a category with one object is a monoid. Thus, we want morphisms of categories to act like morphisms of monoids:
they should send composition to composition.
\begin{defn}
A \term{functor} \(F:\fC\to\fD\) is a function \(F:\operatorname{Ob}\fC\to\operatorname{Ob}\fD\) with an
induced map on the morphisms:
\begin{itemize}
	\item If the map acts as \(\Hom_\fC(X,Y)\to\Hom_{\fD}(F(X),F(Y))\), \(F\) is called a \term{covariant}
	functor.
	\item If it sends \(\Hom_\fC(X,Y)\to\Hom_{\fD}(F(Y),F(X))\), then \(F\) is \term{contravariant}.
\end{itemize}
\end{defn}
When we say ``functor,'' we always mean a covariant functor, and here's the reason. Recall that for any monoid
\(A\) there's the \term{opposite monoid} \(A^{\text{op}}\) which has the same set, but reversed multiplication:
\(f\cdot_{\text{op}} g = g\cdot f\). Similarly, given a category \(\fC\), there's an \term{opposite category}
\(\fC\op\) with the same objects, but \(\Hom_{\fC\op}(X,Y) = \Hom_{\fC}(Y,X)\). Then, a contravariant functor
\(\fC\to\fD\) is really a covariant functor \(\fC\op\to\fD\). Hence, in this class, we'll just refer to
functors, with opposite categories where needed.
\begin{ex}
Show that a functor \(\fC\op\to\fD\) induces a functor \(\fC\to\fD\op\).
\end{ex}

When presented a category, you should always ask what the morphisms are; on the other hand, if someone tells you
``the category of smooth manifolds,'' they probably mean that the morphisms are smooth functions.

Now, we see that pullback is a functor \(F:\mathsf{Spaces}\to\mathsf{Ring}\op\). One of the major goals of this
class is to define a category of spaces on which this functor is an equivalence. This might not make sense,
\emph{yet}. This is the seed of ``algebra = geometry.''
\begin{defn} % TODO: pretty functor diagram!
Let \(F,G:\fC\rightrightarrows\fD\) be functors. A \term{natural transformation} \(\eta:F\Rightarrow G\) is a
collection of maps: for every \(X\in\fC\), there's a map \(\eta_X:F(X)\to G(X)\) satisfying a consistency
condition: for every \(f:X\to Y\) in \(\fC\), there's a commutative diagram
\[\xymatrix{
	F(X)\ar[r]^{\eta_X}\ar[d]_{F(f)} & G(X)\ar[d]^{G(f)}\\
	F(Y)\ar[r]^{\eta_Y} & G(Y)
}\]
\end{defn}
That is, a natural transformation relates the objects and the morphisms, and reflects the structure of the
category.
\begin{defn}
A natural transformation \(\eta\) is a \term{natural isomorphism} if for every \(X\in\fC\), the induced
\(\eta_X\in\Hom_{\fD}(F(X),G(X))\) is an isomorphism.
\end{defn}
This is equivalent to having a natural inverse to \(\eta\).

So one might ask, what is the notion for which two categories are ``the same?'' One might naïvely suggest two
functors whose composition is the identity functor, but this is bad. The set of objects isn't very useful: it
doesn't capture the structure of the category. In general, asking for equality of objects is worse than asking for
isomorphism of objects. Here's the right notion of sameness.
\begin{defn}
Let \(\fC\) and \(\fD\) be categories. Then, a functor \(F:\fC\to\fD\) is an \term{equivalence of
categories} if there's a functor \(G:\fD\to\fC\) such that there are natural isomorphisms \(FG\to\id_{\mathsf
D}\) and \(GF\to\id_{\fC}\).
\end{defn}
This is a very useful notion, and as such it will be useful to see an equivalence that is not an isomorphism.
\begin{ex}
Let \(k\) be a field, and let \(\fD = \mathsf{fdVect}_k\), the category of finite-dimensional vector spaces
and linear maps, and let \(\fC\) be the category whose objects are \(\Z_{\ge 0}\), the natural numbers, with an
object denoted \(\ang n\), and with \(\Hom(\ang n,\ang m) = \Mat_{m\times n}\). This is a category with composition
given by matrix multiplication.

Let \(F:\fC\to\fD\) send \(\ang n\mapsto k^n\), and with the standard realization of matrices as linear maps.
Show that \(F\) is an equivalence of categories.
\end{ex}
This category \(\fC\) has only some vector spaces, but for those spaces, it has all of the morphisms.
\begin{defn}
Let \(F:\fC\to\fD\) be a functor.
\begin{itemize}
	\item \(F\) is \term{faithful} if all of the maps \(\Hom_\fC(X,Y)\inj \Hom_\fD(F(X),F(Y))\) are injective.
	\item \(F\) is \term{fully faithful} if all of these maps are isomorphsism.
	\item \(F\) is \term{essentially surjective} if every \(X\in\fD\) is isomorphic to \(F(Z)\) for some
	\(Z\in\fC\).
\end{itemize}
\end{defn}
The following theorem will also be a useful tool.
\begin{thm}
A functor \(F:\fC\to\fD\) is an equivalence iff it is fully faithful and essentially surjective.
\end{thm}
So, to restate, we want a category of spaces that is the opposite category to the category of rings; this is what
Grothendieck had in mind. In fact, let's peek a few weeks ahead and make a curious definition:
\begin{defn}
The \term{category of affine schemes} is \(\mathsf{Rings}\op\).
\end{defn}
Of course, we'll make these into actual geometric objects, but categorically, this is all that we need.

Recall that if \(f:M\to N\) is a set-theoretic map of manifolds, then \(f\) is smooth iff its pullback sends
\(C^\infty\) functions on \(N\) to \(C^\infty\) functions on \(M\). The first step in this direction is the
following theorem, sometimes called \term{Gelfand duality}.
\begin{thm}[Gelfand-Naimark]
The functor \(X\mapsto C^0(X)\) (the ring of continuous functions) defines an equivalence between the category of
compact Hausdorff spaces and the (opposite) category of commutative \(C^*\)-algebras.
\end{thm}
This is an algebro-geometric result: it identifies a category of spaces with the opposite category of a category of
algebraic objects.

However, we need to think harder than Gelfand duality in terms of compact, complex manifolds or in terms of
algebraic spaces: for example, for \(X = \C\P^1\), \(\operatorname{Hol}(X) = \C\): the only holomorphic functions
are constant. The issue is that there are no partitions of unity in the holomorphic or algebraic world. This means
we'll need to keep track of local data too, which will lead into the next few lectures' discussions on \term{sheaf
theory}.

Returning to the example of algebraic sets, suppose \(X\) and \(Y\) are algebraic sets. What is the set of their
morphisms? We decided the ring of functions was the polynomial functions \(Y\to\C\), so we want maps \(X\to Y\) to
be those whose pullbacks send polynomial functions to polynomial functions. To be precise, the \term{ideal of
\(X\)} is \(I(X) = \{f\in\C[x_1,\dots,x_n]\mid f|_X = 0\}\), defining a map \(I\) from algebraic subsets of
\(\C^n)\) to ideals in \(\C[x_1,\dots,x_n]\). There's also a reverse map \(V\),\footnote{\(V\) stands for
``vanishing,'' ``variety,'' or maybe ``vendetta.''} sending an ideal \(I\) to \(V(I) = \{x\in\C^n\mid f(x) =
0\text{ for all } f\in I\}\). From classical commutative algebra, it's a fact that this is finitely generated, so
it's the vanishing locus of a finite number of polynomials, and therefore in fact an algebraic set.

The dictionary between algebraic sets and ideals of \(\C[x_1,\dots,x_n]\) is one of many versions of the
Nullstellensatz (more or less German for the ``zero locus theorem''): if \(J\) is an ideal, \(I(V(J)) = \sqrt J\),
its radical.
\begin{defn}
Let \(R\) be a ring and \(J\subset R\) be an ideal. Then, the \term{radical} of \(J\) is \(\sqrt J = \{r\in R\mid
r^n\in J\text{ for some } n > 0\}\). One says that \(J\) is \term{radical} if \(J = \sqrt J\).
\end{defn}
What this says is that \(J\) is radical iff \(R/J\) has no nonzero nilpotents.\footnote{Recall that if \(R\) is a
ring, an \(r\in R\) is \term{nilpotent} if \(r^n = 0\) for some \(n\).} Why are these kinds of ideals relevant? If
\(X\subset\C^n\) and \(f\) vanishes on \(X\), then so does \(f^n\) for all \(n\). That is, radicals encode the
geometric property of vanishing, which is why \(I(X)\) is a radical ideal.

This is an outline of what classical algebraic geometry studies: it starts by defining algebraic subsets, and
establishing a bijection between algebraic subsets of \(\C^n\) and radical ideals of \(\C[x_1,\dots,x_n]\). This
isn't yet an equivalence of categories. Radical ideals correspond to finitely generated \(\C\)-algebras with no
(nonzero) nilpotents: an ideal \(I\) corresponds to the \(\C\)-algebra \(\C[x_1,\dots,x_n]/I\).

This is all what the course is \emph{not} about; we're going to replace the category of finitely generated,
nilpotent-free \(\C\)-algebras with the category of \emph{all} rings, but we want to keep some of the same
intuition. This involves generalizing in a few directions at once, but we'll try to write down a dictionary; the
defining principle is to identify spaces \(X\) with rings \(R = F(X)\), their ring of functions.

A point \(x\in X\) is a map \(i_x:x\to X\), so we get a pullback \(i_x^*:F(X)\to\C\) given by evaluation at \(x\).
Let \(\m_x = \ker(i_x^*)\); since \(\C\) is a field, this is a maximal ideal.\footnote{Recall that an ideal
\(I\subset R\) is \term{maximal} iff \(R/I\) is a field. This is about the level of commutative algebra that we'll
be assuming.} If \(k\) is a field and \(R\) is a \(k\)-algebra, then \(R/I\) is also a \(k\)-algebra, so in
particular if \(I\) is maximal, then \(k\inj R/I\) is a map of fields, and therefore a field extension.  Thus, if
\(k\) is algebraically closed (e.g.\ we're studying \(\C\)) and \(R\) is a finitely generated \(k\)-algebra, then
maximal ideals of \(R\) are in bijection with homomorphisms \(R\to k\).

Thus, given a ring \(R\), we'll associate a set \(\MSpec(R)\), the set of maximal ideals of \(R\), such that \(R\)
should be its ring of functions. To do this, we'll say that an \(r\in R\) is a ``function'' on \(\MSpec(R)\) by
acting on an \(\m_x\subset R\) as \(r\bmod\m_x\). This is a ``number,'' since it's in a field, but the notion may
be different at every point in \(\MSpec(R)\)! For example, if \(R = \Z\), then \(\MSpec(\Z)\) is the set of primes,
and \(n\in\Z\) is a function which at \(2\) is \(n\bmod 2\), at \(3\) is \(n\bmod 3\), and so on.

A perhaps nicer example is when \(R = \R[x]\), which has maximal ideals \((x-t)\) for all \(t\in\R\). Here,
evaluation sends \(f(x)\mapsto f(x)\bmod (x-t) = f(t)\). That is, this is really evaluation, and here the quotient
field is \(\R\). So these look like good old real-valued functions, but these aren't all the maximal ideals:
\((x^2+1)\) is also a maximal ideal, and \(\R[x]/(x^2+1) \cong \C\). Then, we do get a kind of evaluation again,
but we have to identify points and their complex conjugates.

So we'll have to find a good notion of geometry which generalizes from \(\C\)-algebras to \(k\)-algebras for any
field \(k\), to any commutative rings. We'll also have to think about nilpotents: we threw them away by thinking
about zero sets, but they play a huge role in ring theory.


\section{Modules}
Here is a list of all modules\+:\begin{DoxyCompactList}
\item \contentsline{section}{Differentiation}{\pageref{group__numpp__differentiation}}{}
\begin{DoxyCompactList}
\item \contentsline{section}{Automatic Differentiation}{\pageref{group__numpp__differentiation__automatic}}{}
\begin{DoxyCompactList}
\item \contentsline{section}{Backward Automatic Differentiation}{\pageref{group__numpp__differentiation__backward__automatic}}{}
\item \contentsline{section}{Forward Automatic Differentiation}{\pageref{group__numpp__differentiation__forward__automatic}}{}
\end{DoxyCompactList}
\item \contentsline{section}{Finite Differentiation}{\pageref{group__numpp__differentiation__finite}}{}
\begin{DoxyCompactList}
\item \contentsline{section}{Backward Finite Differentiation}{\pageref{group__numpp__differentiation__finite__backward}}{}
\item \contentsline{section}{Central Finite Differentiation}{\pageref{group__numpp__differentiation__finite__central}}{}
\item \contentsline{section}{Forward Finite Differentiation}{\pageref{group__numpp__differentiation__finite__forward}}{}
\end{DoxyCompactList}
\item \contentsline{section}{Symbolic Differentiation}{\pageref{group__numpp__differentiation__symbolic}}{}
\end{DoxyCompactList}
\item \contentsline{section}{Krylov Subspace methods}{\pageref{group__numpp__krylov}}{}
\item \contentsline{section}{Root Finding}{\pageref{group__numpp__roots}}{}
\item \contentsline{section}{Structures}{\pageref{group__numpp__structures}}{}
\begin{DoxyCompactList}
\item \contentsline{section}{Matrices}{\pageref{group__numpp__structures__matrices}}{}
\begin{DoxyCompactList}
\item \contentsline{section}{Dense}{\pageref{group__numpp__structures__matrices__dense}}{}
\item \contentsline{section}{Sparse}{\pageref{group__numpp__structures__matrices__sparse}}{}
\end{DoxyCompactList}
\item \contentsline{section}{Vector}{\pageref{group__numpp__structures__vector}}{}
\end{DoxyCompactList}
\end{DoxyCompactList}


\documentclass[12pt]{article}
% We can write notes using the percent symbol!
% The first line above is to announce we are beginning a document, an article in this case, and we want the default font size to be 12pt
\usepackage[utf8]{inputenc}
% This is a package to accept utf8 input.  I normally do not use it in my documents, but it was here by default in Overleaf.
\usepackage{pgfplots}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
% These three packages are from the American Mathematical Society and includes all of the important symbols and operations
\usepackage{fullpage}
% By default, an article has some vary large margins to fit the smaller page format.  This allows us to use more standard margins.

\setlength{\parskip}{1em}
% This gives us a full line break when we write a new paragraph

\newcommand\w{\vec{w}}
\newcommand\wt{\vec{w}^T}
\newcommand\msb{S_B}
\newcommand\msw{S_W}

\begin{document}
	% Once we have all of our packages and setting announced, we need to begin our document.  You will notice that at the end of the writing there is an end document statements.  Many options use this begin and end syntax.

	\title{Notes on Fisher's Linear Discriminant}
	\author{Zhihan Yang}
	\maketitle

	\section{Binary classification problem}

	There are two classes, $C_1$ and $C_2$, each containing a non-zero number of data vectors, $\boldsymbol{x}^n$. For each class, $C_k$, we define some properties that will be used later on in this document.

	The number of vectors in class $k$ is given by $N_k$.

	The mean vector of class $k$ is given by:
	$$\boldsymbol{m}_k=\frac{1}{N_k} \sum_{n \in C_1}\boldsymbol{x}^n$$

	The projected mean of class $k$ is given by:
	$$m_k = \boldsymbol{w}^T \boldsymbol{m}_k$$

	The within-class variance of class $k$ is given by:
	$$s_k = \sum_{n \in C_k}(y^n - m_k)^2$$

	\section{Fisher's criterion}

	A linear combination between input variables, $x_i$, and weights $w_i$, can be written in vector form as:
	$$y=\boldsymbol{w}^T \boldsymbol{x}$$
	where $y$ can be interpreted as the projection of $\boldsymbol{x}$ onto $\boldsymbol{w}$.

	Given two classes, $C_1$ and $C_2$, we would like to select a $\boldsymbol{w}$ such that the projection of $\boldsymbol{x}$ maximizes class separation. Intuitively, this means maximizing between-class separation while minimizing within-class separation. Mathematically, we define between-class separation as $(m_2 - m_1)^2$, and define within-class separation as $s_1^2+s_2^2$. We can complete the two optimization tasks simultaneously by maximizing the following ratio, known as the Fisher criterion:

	$$J(\boldsymbol{w}) = \frac{(m_2 - m_1)^2}{s_1^2+s_2^2}$$

	Now, the question remains: how do we find the optimal $\boldsymbol{w}$ to maximize $J(\boldsymbol{w})$?

	\section{The Fisher criterion in matrix form}

	It turns out that the Fisher criterion is more conveniently optimized in matrix form. In this section, we convert the algebraic form derived in the last section into its matrix form.

	Expand the numerator:

	\begin{align}
	\text{numerator}(J(\boldsymbol{w}))
	&= (m_2 - m_1)^2 \\
	&= (m_2 - m_1)(m_2 - m_1) \\
	&= (\vec{w}^T \vec{x} - \vec{w}^T\vec{m_1}) (\vec{w}^T \vec{x} - \vec{w}^T\vec{m_2})
	\end{align}

	where both factors are scalars, so the order of multiplication does not matter within each factor,

	\begin{align}
	&= \vec{w}^T (\vec{x} - \vec{m}_1) (\vec{x} - \vec{m}_2)^T \vec{w}
	\end{align}

	Since matrix multiplication is associative, i.e., $A(BC) = (AB) C$, we can rewrite the numerator as:

	$$\text{numerator}(J(\boldsymbol{w})) = \vec{w}^T S_B \vec{w}$$

	where $S_B = (\vec{x} - \vec{m}_1) (\vec{x} - \vec{m}_2)^T $ and is called the between-class covariance matrix.

	Expand the denominator (somewhat tedious):

	\begin{align}
		\text{denominator}(J(\vec{w}))
		&= s_1 ^ 2 + s_2 ^2  \\
		&= \sum_{n \in C_1}(y^n - m_1)^2 + \sum_{n \in C_2}(y^n - m_2)^2 \\
		&=
		 \sum_{n \in C_1} (\vec{w}^T \vec{x} - \vec{w}^T \vec{m}_1) ^ 2 +
		 \sum_{n \in C_2} (\vec{w}^T \vec{x} - \vec{x}^T \vec{m}_2) ^ 2 \\
		 &=
		 \sum_{n \in C_1} \vec{w}^T (\vec{x} - \vec{m}_1) (\vec{x} - \vec{m}_1)^T \vec{w} +
		 \sum_{n \in C_2} \vec{w}^T (\vec{x} - \vec{m}_2) (\vec{x} - \vec{m}_2)^T \vec{w} \\
		&=
		\vec{w}^T \{ \sum_{n \in C_1}  (\vec{x} - \vec{m}_1) (\vec{x} - \vec{m}_1)^T \} \vec{w} +
	    \vec{w}^T \{ \sum_{n \in C_2}  (\vec{x} - \vec{m}_2) (\vec{x} - \vec{m}_2)^T \} \vec{w} \\
	    &=
	    \vec{w}^T
	    \{
	    \sum_{n \in C_1}  (\vec{x} - \vec{m}_1) (\vec{x} - \vec{m}_1)^T+
	    \sum_{n \in C_2}  (\vec{x} - \vec{m}_2) (\vec{x} - \vec{m}_2)^T
	    \}
	   	\vec{w} \\
		&= \vec{w}^T S_W \vec{w}
	\end{align}

	where $S_W= \sum_{n \in C_1}  (\vec{x} - \vec{m}_1) (\vec{x} - \vec{m}_1)^T+
	\sum_{n \in C_2}  (\vec{x} - \vec{m}_2) (\vec{x} - \vec{m}_2)^T$  and is called the within-class covariance matrix.

	Putting the numerator and the denominator together, obtain:

	$$J(\vec{w}) = \frac{\vec{w}^T S_B \vec{w}}{\vec{w}^T S_W \vec{w}}$$

	\section{Fisher's linear discriminant}

	Since $J(\vec{w})$ involves $\vec{w}$ in a quadratic fashion,
	$\frac
	{\partial J(\vec{w})}
	{\partial \vec{w}}
	$
	involves $\vec{w}$ in a linear fashion and hence
	$\frac
	{\partial J(\vec{w})}
	{\partial \vec{w}} = 0
	$
	has a unique solution for $\vec{w}$. First, let's find
	$\frac
	{\partial J(\vec{w})}
	{\partial \vec{w}}
	$.

	\begin{align}
		J(\w) = \frac{\wt \msb \w}{\wt \msw \w}
	\end{align}

	In order to obtain
	$\frac
	{\partial J(\vec{w})}
	{\partial \vec{w}}
	$,
	we need the quotient rule for matrix calculus, which requires us to define the numerator, $u$, the denominator, $v$, the derivative of numerator, $u'$, and the derivative of denominator, $v'$. Let's define them below:

	$$u = \wt \msb \w$$
	$$v = \wt \msw \w$$

	According to the Matrix Cookbook,

	$$u' = \frac{\partial u}{\partial \vec{w}} = (\msb + \msb^T) \w$$

	Derivation of $(\msb + \msb^T) \w = 2\msb\w$:

	$$
	\begin{aligned}
	S_{B} &=\left(\vec{m}_{2}-\vec{m}_{1}\right)\left(\vec{m}_{2}-\vec{m}_{1}\right)^{T} \\
	S_{B}^{\top} &= \left( \left(\vec{m}_2-\vec{m}_{1}\right)\left(\vec{m}_{2}-\vec{m}_{1}\right)^{T} \right)^T \\
	&=\left(\vec{m}_{2}-\vec{m}_{1}\right)^{T T}\left(\vec{m}_2-\vec{m}_{1}\right)^{T} \\ &=\left(\vec{m}_{2}-\vec{m}_{1}\right)\left(\vec{m}_{2}-\vec{m}_{1}\right)^{T} \\
    \end{aligned}
	$$

	Therefore, $S_B^T = S_B$ and $u'=2S_B \vec{w}$.

	Again, according to the Matrix Cookbook,

	$$v' = \frac{\partial u}{\partial \vec{w}} = (\msw + \msw^T) \w$$

	Derivation of $ (\msw + \msw^T) \w = 2\msw\w$:

	$$S_{W} = \sum_{n \in C_{1}}\left(\vec{x}^{n}-\vec{m}_{1}\right)\left(\vec{x}^{n}-\vec{m}_{1}\right)^{T}+
	\sum_{n \in C_{2}}\left(\vec{x}^{n}-\vec{m}_{2}\right)\left(\vec{x}^{n}-\vec{m}_{2}\right)^{T} $$

	Since a transpose of sums is a sum of transposes,

	$$
	\begin{aligned}
	S_{W}^{T}
	&=\sum_{n \in C_{1}}\left[\left(\vec{x}^{n}-\vec{m}_{1}\right)\left(\vec{x}^{n}-\vec{m}_{1}\right)^{T}\right]^{T}+
	\sum_{n \in C_{2}}\left[\left(\vec{x}^{n}-m_{2}\right)\left(\vec{x}^{n}-\vec{m}_{2}\right)^{T}\right]^{T} \\
	&= \sum_{n \in C_{1}}\left(\vec{x}^{n}-\vec{m}_{1}\right)\left(\vec{x}^{n}-\vec{m}_{1}\right)^{T}+
	\sum_{n \in C_{2}}\left(\vec{x}^{n}-\vec{m}_{2}\right)\left(\vec{x}^{n}-\vec{m}_{2}\right)^{T} \\
	\end{aligned}
	$$

	Therefore, $S_{W}^T = S_{W}$ and $u' = 2 S_W \vec{w}$.

	Using $u$, $u'$, $v$ and $v'$ and the quotient rule to find $\frac{\partial J(\vec{w})}{\partial \vec{w}}$:

	\begin{align}
	\frac{\partial J(\vec{w})}{\partial \vec{w}}
	&=\frac{u^{\prime} v-u v^{\prime}}{v^{2}} \\
	&=\frac{\left(2 S_{B} \vec{w}\right)\left(\vec{w}^{\top} S_W \vec{w}\right)-\left(\vec{w}^{\top} S_{B} \vec{w}\right)\left(2 S_{w} \vec{w}\right)}{\left(\vec{w}^{\top} S_{W} \vec{w}\right)\left(\vec{w}^{\top} S_{W} \vec{w}\right)} \\
	&=\frac{2 S_{B} \vec{w}}{\vec{w}^{\top} S_{W} \vec{w}}-\frac{\left(\vec{w}^{\top} S_{B} \vec{w}\right)\left(2 S_{W} \vec{w}\right)}{\left(\vec{w}^{\top} S_{W} \vec{w}\right)\left(\vec{w}^{\top} S_{W} \vec{w}\right)}
	\end{align}

	Setting $\frac{\partial J(\vec{w})}{\partial \vec{w}}=0$ and solving for this equation:

	\begin{align}
	\frac{\partial J(\vec{w})}{\partial \vec{w}}
	&=0 \\
	\frac{2 S_{B} \vec{w}}{\vec{w}^{T} S_{w} \vec{w}}
	&=\frac{\left(\vec{w}^{T} S_{B} \vec{w}\right)\left(2 S_{w} \vec{w}\right)}{\left(\vec{w}^{T} S_{w} \vec{w}\right)\left(\vec{w}^{T} S_{w} \vec{w}\right)} \\
	\underbrace{2\left(\vec{w}^{\top} S_{w} \vec{w}\right)}_{\text {scalar }} \left( S_{B} \vec{w}\right)
	&=\underbrace{2 \left(\vec{w}^{\top} S_{B} \vec{w}\right)}_{\text {scalar }} \left(S_{w} \vec{w}\right)
	\end{align}

	Since we do not care about the magnitude of $\vec{w}$, only its direction, we remove the scalars:

	$$S_{B} \vec{w} \propto S_{W} \vec{w}.$$

	Since $S_B = (\vec{m}_2 - \vec{m}_1)(\vec{m}_2 - \vec{m}_1)^T$ and $(\vec{m}_2 - \vec{m}_1)^T \vec{w}$ is a scalar, $S_B \vec{w}$ is always in the direction of $\vec{m}_2-\vec{m}_1$:

	$$(\vec{m}_2 - \vec{m}_1)\propto S_{W} \vec{w}$$

	Multiplying both sides by $S_W^{-1}$:

	$$\vec{w} \propto S_{W}^{-1}(\vec{m}_2 - \vec{m}_1)$$

	"This choice of $\vec{w}$ is known as Fisher’s linear discriminant, although it is strictly not a discriminant but rather a specific choice of direction for projection of data down to one dimension."

\end{document}


\documentclass[a4paper]{article}
\usepackage[a4paper, total={155mm, 250mm}]{geometry}
\usepackage{amssymb}
\usepackage[nodayofweek]{datetime}

\usepackage{wrapfig}
\usepackage{tikz}

\tikzset{
	r/.style={red!30},
	g/.style={green!30},
	b/.style={blue!30},
	ye/.style={yellow!40},
}

\newcommand{\half}{\frac{1}{2}}
\newcommand{\hexpr}[1]{\half(#1^2 + #1 + 2)}

\title{Splitting the Plane}
\author{Dyson}
\date{\today}

\begin{document}

\maketitle

\setlength{\parindent}{0em}
\setlength{\parskip}{1em}

Challenge question 3 of Review Exercise 2 in the Edexcel Further Maths Core Pure Book 1 says:\begin{center}Prove that if $n$ non-parallel lines divide the plane into $r$ regions, then $2n \leq r \leq \hexpr{n}$.\end{center}

I've also seen this problem presented as cutting pizza, and using circles looks nice, so I'll be doing that throughout.

\begin{figure}[h]
\centering
\begin{tikzpicture}[scale=3]
	\clip (0,0) circle[radius=1];

	\coordinate (A) at (0.125,0.375);

	\fill [r] (0,1) -- (0,0.5) -- (A) -- (0.75,1);
	\fill [g] (1,1) -- (0.75,1) -- (A) -- (0.5,0) -- (1,0);
	\fill [b] (A) -- (0,0.25) -- (0,0) -- (0.5,0);
	\fill [ye] (0,0.25) -- (0,0.5) -- (A);
	\fill [b] (0,1) -- (0,0.5) -- (-0.5,1);
	\fill [g] (-1,1) -- (-0.5,1) -- (0,0.5) -- (0,0.25) -- (-0.25,0) -- (-1,0);
	\fill [r] (0,0) -- (0,0.25) -- (-0.25,0);
	\fill [ye] (-0.25,0) -- (-1,-0.75) -- (-1,0);
	\fill [b] (0,0) -- (-0.25,0) -- (-1,-0.75) -- (-1,-1) -- (0,-1);
	\fill [r] (0,0) -- (0,-1) -- (1,-1) -- (1,-0.5) -- (0.5,0);
	\fill [ye] (0.5,0) -- (1,0) -- (1,-0.5);

	\draw (0,0) circle[radius=1];

	\draw (0,1) -- (0,-1);
	\draw (-1,0) -- (1,0);
	\draw (1,-0.5) -- (-0.5,1);
	\draw (-1,-0.75) -- (0.75,1);
\end{tikzpicture}
\hspace{10ex}
\begin{tikzpicture}[scale=3]
	\clip (0,0) circle[radius=1];

	\fill [r] (0,1) -- (0,0.5) -- (0.25,0.25) -- (1,1);
	\fill [g] (1,1) -- (0.25,0.25) -- (0.5,0) -- (1,0);
	\fill [b] (0.25,0.25) -- (0,0) -- (0.5,0);
	\fill [ye] (0,0) -- (0,0.5) -- (0.25,0.25);
	\fill [b] (0,1) -- (0,0.5) -- (-0.5,1);
	\fill [g] (-1,1) -- (-0.5,1) -- (0,0.5) -- (0,0) -- (-1,0);
	\fill [ye] (0,0) -- (-1,-1) -- (-1,0);
	\fill [b] (0,0) -- (-1,-1) -- (-1,-1) -- (0,-1);
	\fill [r] (0,0) -- (0,-1) -- (1,-1) -- (1,-0.5) -- (0.5,0);
	\fill [ye] (0.5,0) -- (1,0) -- (1,-0.5);

	\draw (0,0) circle[radius=1];

	\draw (0,1) -- (0,-1);
	\draw (-1,0) -- (1,0);
	\draw (1,-0.5) -- (-0.5,1);
	\draw (-1,-1) -- (1,1);
\end{tikzpicture}
\caption{An example of $n = 4$ with 11 regions (left) and 10 regions (right)}
\end{figure}

\section{Basis}

\begin{wrapfigure}[5]{l}{0.3\linewidth}
\vspace{-1em}
\begin{tikzpicture}[scale=2]
	\clip (0,0) circle[radius=1];

	\fill [r] (-1,1) rectangle (0,-1);
	\fill [g] (0,1) rectangle (1,-1);

	\draw (0,0) circle[radius=1];

	\draw (0,1) -- (0,-1);
\end{tikzpicture}
\end{wrapfigure}

For $n = 1$, we have one line, and we can only split the plane into 2 regions. This basis satisfies the inequality that we're looking for. For the minimum, $2n = 2 \times 1 = 2$ and for the maximum, $\hexpr{n} = \hexpr{1} = \half \times 4 = 2$. And of course, $2 \leq 2 \leq 2$, so the statement holds for $n = 1$.

\vspace{4em}

\section{Assumption}

We will now assume that if $k$ non-parallel lines cut the plane into $r$ regions, then $2k \leq r \leq \hexpr{k}$. We have shown this to be true for $k = 1$, so we can now use mathematical induction to show it to be true for any $k + 1$, and then we will have proved the original statement for all $n \in \mathbb{Z}^+$.

\newpage

\section{Induction}

We will start by showing the minimum number of regions for $k + 1$ lines. We are going to start with $k$ lines and add one more, so we start with $2k$ regions, as shown in the left circle here with $k = 3$.

Note that all the lines pass through a central point. This intersection point is shared by all the lines and is the only intersection point here. When we add our $(k + 1)$th line, we also want it to pass through this point to minimise how many new regions we create. When it does, it must cut through exactly two regions, turning two regions into four and giving us two new regions. Therefore, the minimum number of regions for $k + 1$ lines is $2k + 2 = 2(k + 1)$, as expected.

\begin{figure}[h]
\centering
\begin{tikzpicture}[scale=2]
	\clip (0,0) circle[radius=1];

	\fill [r] (0,1) -- (0,0) -- (1,1);
	\fill [g] (1,1) -- (0,0) -- (1,-1);
	\fill [b] (1,-1) -- (0,0) -- (0,-1);
	\fill [r] (0,-1) -- (0,0) -- (-1,-1);
	\fill [g] (-1,-1) -- (0,0) -- (-1,1);
	\fill [b] (-1,1) -- (0,0) -- (0,1);

	\draw (0,0) circle[radius=1];

	\draw (0,1) -- (0,-1);
	\draw (-1,-1) -- (1,1);
	\draw (-1,1) -- (1,-1);
\end{tikzpicture}
\hspace{15ex}
\begin{tikzpicture}[scale=2]
	\clip (0,0) circle[radius=1];

	\fill [r] (0,1) -- (0,0) -- (1,1);
	\fill [g] (1,1) -- (0,0) -- (1,0);
	\fill [ye] (1,0) -- (0,0) -- (1,-1);
	\fill [b] (1,-1) -- (0,0) -- (0,-1);
	\fill [r] (0,-1) -- (0,0) -- (-1,-1);
	\fill [g] (-1,-1) -- (0,0) -- (-1,0);
	\fill [ye] (-1,0) -- (0,0) -- (-1,1);
	\fill [b] (-1,1) -- (0,0) -- (0,1);

	\draw (0,0) circle[radius=1];

	\draw (0,1) -- (0,-1);
	\draw (-1,-1) -- (1,1);
	\draw (-1,1) -- (1,-1);
	\draw (-1,0) -- (1,0);
\end{tikzpicture}
\end{figure}

The maximum number of lines is a bit more complicated. All these lines are non-parallel by definition, which means that a $(k + 1)$th line will intersect all $k$ previous lines. If they all intersect at a single point, then we get the minimum number of regions, but if every intersection point is between only two lines, then we get the maximum number of regions.

When a new line intersects an existing one, it adds two new regions, as seen previously. When we add a $(k + 1)$th line, this is true for each of the $k$ previous lines, so we'd expect $2k$ new regions. However, $k - 1$ of these regions are double-counted, so we disregard them and get $k + 1$ new regions.

To understand this, consider the case of $k = 2$. We can start with two orthogonal lines and consider each separately. When we cut through the horizontal line shown here, we add 2 regions, one above the line and one below, shown in yellow and blue.

\begin{figure}[h]
\centering
\begin{tikzpicture}[scale=2]
	\clip (0,0) circle[radius=1];

	\fill [r] (-1,0) rectangle (1,-1);
	\fill [g] (-1,0) rectangle (1,1);

	\draw (0,0) circle[radius=1];

	\draw (1,0) -- (-1,0);
\end{tikzpicture}
\hspace{15ex}
\begin{tikzpicture}[scale=2]
	\clip (0,0) circle[radius=1];

	\fill [r] (-1,-1) -- (-1,0) -- (0.5,0) -- (1,-0.5) -- (1,-1);
	\fill [g] (-1,1) -- (-1,0) -- (0.5,0) -- (-0.5,1);
	\fill [b] (1,0) -- (0.5,0) -- (1,-0.5);
	\fill [ye] (1,1) -- (1,0) -- (0.5,0) -- (-0.5,1);

	\draw (0,0) circle[radius=1];

	\draw (-1,0) -- (1,0);
	\draw (1,-0.5) -- (-0.5,1);
\end{tikzpicture}
\end{figure}

When we cut through the vertical line, we similarly get 2 new regions.

\begin{figure}[h]
\centering
\begin{tikzpicture}[scale=2]
	\clip (0,0) circle[radius=1];

	\fill [r] (0,1) rectangle (-1,-1);
	\fill [g] (0,1) rectangle (1,-1);

	\draw (0,0) circle[radius=1];

	\draw (0,1) -- (0,-1);
\end{tikzpicture}
\hspace{15ex}
\begin{tikzpicture}[scale=2]
	\clip (0,0) circle[radius=1];

	\fill [r] (-1,1) -- (-1,-1) -- (0,-1) -- (0,0.5) -- (-0.5,1) -- (-1,1);
	\fill [g] (1,-1) -- (0,-1) -- (0,0.5) -- (1,-0.5);
	\fill [b] (0,1) -- (0,0.5) -- (-0.5,1);
	\fill [ye] (1,1) -- (0,1) -- (0,0.5) -- (1,-0.5);

	\draw (0,0) circle[radius=1];

	\draw (0,1) -- (0,-1);
	\draw (1,-0.5) -- (-0.5,1);
\end{tikzpicture}
\end{figure}

\newpage

So when we combine these two to create a diagram with 3 total lines, we could reasonably expect 8 total regions, but this isn't the case. We actually get 7, as shown. This is because we double-count one region when we combine the diagrams. In this case, we end up double-counting the resultant yellow region.

\begin{center}
\begin{tikzpicture}[scale=2]
	\clip (0,0) circle[radius=1];

	\fill [r] (0,0) rectangle (-1,-1);
	\fill [ye] (1,1) -- (1,0) -- (0.5,0) -- (0,0.5) -- (0,1);
	\fill [r] (0,0) -- (0,0.5) -- (0.5,0);
	\fill [g] (0,-1) -- (0,0) -- (0.5,0) -- (1,-0.5) -- (1,-1);
	\fill [g] (-1,1) -- (-1,0) -- (0,0) -- (0,0.5) -- (-0.5,1);
	\fill [b] (1,0) -- (0.5,0) -- (1,-0.5);
	\fill [b] (0,1) -- (0,0.5) -- (-0.5,1);

	\draw (0,0) circle[radius=1];

	\draw (0,1) -- (0,-1);
	\draw (-1,0) -- (1,0);
	\draw (1,-0.5) -- (-0.5,1);
\end{tikzpicture}
\end{center}

This line of reasoning continues to hold as we increase the number of lines, so the number of new regions given by the $(k + 1)$th line is $k + 1$. Since the maximum number of regions for $k$ lines is $\half(k^2 + k + 2)$, the maximum number for $k + 1$ lines is $\hexpr{k} + (k + 1)$.

This can be simplified down like so: \[\half(k^2 + k + 2 + 2(k + 1)) = \half(k^2 + 3k + 4) = \half((k + 1)^2 + k + 3) = \hexpr{(k + 1)}\] And now we have exactly what we wanted, with $k + 1$ in place of $k$ in the expressions for the minimum and the maximum.

\section{Conclusion}

We assumed that for $k$ lines, $2k \leq r \leq \hexpr{k}$ and then proved that if that's true, then for $k + 1$ lines, $2(k + 1) \leq r \leq \hexpr{(k + 1)}$. We proved that this statement holds for $k = 1$, so we have now proved that if $n$ non-parallel lines divide the plane into $r$ regions, then $2n \leq r \leq \hexpr{n}\ \forall\ n \in \mathbb{Z}^+$.

\hspace{\fill}$\square$

\end{document}