\section{Linear elasticity}

In this section, we study the simplest mathematical model for elastic
deformation of solids based on Hooke's law. For comparison,
consider~\cite{Braess97,Braess13}. For the full nonlinear model in all
mathematical detail refer to~\cite{Ciarlet88}.

\begin{Notation}{vector-diff-operators}
  Differential operators for vector fields $u:\R^d\to\R^d$
  are defined as follows:
  \begin{xalignat}2
    \nabla u &=
    \begin{pmatrix}
      \d_1 u_1 & \cdots & \d_d u_1\\
      \vdots && \vdots \\
      \d_1 u_d & \cdots & \d_d u_d
    \end{pmatrix}
    &&\text{(gradient)}
    \\
    \div u &= \sum_{i=1}^d \d_i u_i
    &&\text{(divergence)}
  \end{xalignat}

  For a tensor field $\sigma: \R^d\to \R^{d\times d}$, the divergence
  is a vector defined column-wise as
  \begin{gather}
    \div\sigma = \left(\sum_{i=1}^d \d_i \sigma_{ij}\right)_{j=1,\dots,d}
  \end{gather}
\end{Notation}

\begin{intro}
  The deformation of a solid body is described as a mapping $\Phi$
  from the \define{reference configuration} $\domain\subset \R^d$ to a
  deformed configuration $\deformed\domain \subset \R^d$, such that
  each undeformed point $x\in\domain$ is mapped to the point
  $\deformed x$ after deformation. The domain $d$ is 3 for physically
  relevant models, but we investigate two-dimensional problems in
  order to study mathematical properties and numerical methods more
  easily.

  Actually, we are not quite interested in this mapping $\Phi$, since
  it depends on the position of the points $x$. On the other hand, a
  basic principle of physical laws is frame invariance, namely, if we
  change from one Cartesian coordinate system to another, the physical
  law may only change by the same coordinate transformation, not in
  its physical implications. Therefore, only the differences
  $\deformed x-x$ should matter. Thus, we introduce the
  \define{displacement} $u$, such that
  \begin{gather*}
    \Phi = \id + u.
  \end{gather*}
  The symbol $\id$ will refer to all occurrences of identical mappings
  and their matrices.

  So far, by the introduction of $u$, we divide translations of the
  reference configuration out of our model. But, in addition, we have
  to eliminate the influence of rigid body rotations. These are
  operations, which leave all distances and angles unchanged. Thus, we
  investigate the change of the distance between $x$ and $x+z$ under
  the mapping $\Phi$. By definition of the derivative, we have
  \begin{align*}
    \abs{\Phi(x+z) - \Phi(x)}^2 &= \norm{\nabla\Phi z}^2 + o(\abs{z}^4)
    \\
                              &= z^T\nabla\Phi^T\nabla\Phi z + o(\abs{z}^4)
    \\
    &= z^T(I + \nabla u^T + \nabla u + \nabla u^T \nabla u) z + o(\abs{z}^4)
    \\
    &= \abs{z}^2 + 2 z^T\strain u z + o(\abs{z}^4),
  \end{align*}
  where
  \begin{gather}
    \tilde\epsilon(u) = \tfrac12
    \bigl(\nabla u^T + \nabla u + \nabla u^T \nabla u\bigr)
  \end{gather}
  is the \textbf{strain tensor}. From linear algebra, we know that a
  linear mapping which preserves all distances is orthogonal and thus
  also preserves angles. Thus, every deformation with $\strain u=0$
  is a rigid body transformation, namely a combination of translation
  and rotation.

  In this class we are concerned only with linear problems, which can
  be justified by the notion of infinitely small deformations
  $u$. Then, we only study first order effects in $u$, which implies
  that we are going to neglect the quadratic term in
  $\strain u$. This is justified by the fact that we obtain a model,
  which is sufficiently accurate for small deformations.
\end{intro}

\begin{Definition}{strain-tensor}
  The linearized \define{strain tensor} or \define{symmetric gradient}
  of $u$ is
  \begin{gather}
    \strain u = \frac{\nabla u + \nabla u^T}2.
  \end{gather}
\end{Definition}

\begin{intro}
  Next, we have to consider the interplay of forces and
  deformations. The basic principle is Newton's axiom of force
  balance. If a body force $f$ acts on a small volume $V$, there have
  to be surface forces acting against $f$ in order to keep $V$ at
  rest. Similarly, if a torque is applied inside this volume, there
  must be tangential forces on the surface balancing this torque. Due
  to Euler, we model these forces as a mapping $t$, which at each
  point $x$ maps a direction vector $n$ to a force vector
  $t(x,n)$. Thus, the balance of forces is written as
  \begin{alignat*}2
    \int_V f \dx &+ \int_{\d V} t(x,n) \ds &=&0\\
    \int_V x\times f \dx &+ \int_{\d V} x\times t(x,n) \ds &=&0.
  \end{alignat*}
  Due to Euler and Cauchy, this mapping $t(x,n)$ can be expressed as
  $\sigma(x)n$ by the \define{stress tensor} $\sigma$. Without proof,
  we note that the balance of torque implies that $\sigma$ is
  symmetric, while the force balance equation after integration by
  parts becomes
  \begin{gather}
    \label{eq:mixedintro:3}
    f + \div \sigma = 0.
  \end{gather}
  What is missing now is a relation between the displacement $u$ and
  the stress $\sigma$, which is not the result of fundamental
  principles, but of material properties.
\end{intro}

\begin{remark}
  At this point, we play again the card of small deformations, such
  that we do not have to distinguish whether equations are formulated
  on the reference domain $\domain$ or on the deformed domain
  $\deformed\domain$. Such a discussion becomes confusing easily and
  thus remains a subject for a more specialized class.
\end{remark}

\begin{Definition}{hooke}
  \define{Hooke's law} states that the stress depends linearly on the
  strain. Together with frame invariance, this implies the relation
  \begin{gather}
    \label{eq:mixedintro:4}
    \sigma = 2\mu \strain u + \lambda \operatorname{tr} \strain u \id,
  \end{gather}
  where $\lambda\ge 0$ and $\mu> 0$ are material properties called
  \define{Lamé-Navier parameters}.
\end{Definition}

\begin{remark}
  The trace of the strain operator is equal to the trace of the
  gradient. Thus, we have
  \begin{gather}
    \label{eq:mixedintro:5}
    \operatorname{tr} \strain u = \div u \,\id.
  \end{gather}
\end{remark}

\begin{intro}
  Equations~\eqref{eq:mixedintro:3} and~\eqref{eq:mixedintro:4}
  together define a second order partial differential equation, for
  which we have to impose boundary conditions. A natural choice, which
  keeps the mathematical analysis simple is the \define{Dirichlet
    boundary condition} $u=0$, corresponding to an elastic body whose
  boundary is fixed. The alternative is the traction free boundary
  condition $\sigma n=0$ with vanishing normal traces. Combinations
  are possible, for instance $u\cdot n=0$ for a boundary that allows
  sliding but no penetration. Constraining ourselves to Dirichlet
  condition on $\Gamma_D\subset \d\domain$ and traction free on
  $\gamma_N = \d\domain\setminus\Gamma_D$, we obtain the
  boundary value problem
  \begin{gather}
    \label{eq:mixedintro:lame-navier-bvp}
    \begin{aligned}
      -\div \sigma(x) &= f(x) & x&\in\domain,\\
      u(x) &= 0 & x&\in\Gamma_D, \\
      \sigma(x)n &=0& x&\in\Gamma_N,
    \end{aligned}
  \end{gather}
  together with the material law~\eqref{eq:mixedintro:4}.  Once we
  test and integrate by parts to obtain our weak formulation, we
  obtain
  \begin{gather*}
    \int_{\domain} -(\div \sigma) \cdot v\dx
    = \int_{\domain} \sigma : \nabla v\dx
    - \int_{\Gamma_N} \sigma n\cdot v\ds,
  \end{gather*}
  such that traction free is actually the natural boundary condition
  comparable to the Neumann condition for the Laplacian. Note that $:$
  is the double contraction or Frobenius product (see
  Problem~\ref{Problem:frobenius} below) of the two tensors.
\end{intro}

\begin{intro}
  We now walk the missing steps to obtain a weak formulation. first,
  we enter Hooke's law for $\sigma$ to obtain:
  \begin{gather*}
    \int_\domain \bigl[2\mu \strain u : \nabla v
    + \lambda (\div u \id) : \nabla v
    \bigr]\dx = \int_\domain f\cdot v\dx.
  \end{gather*}
  Then, we choose the space
  \begin{gather}
    V = H^1_{\Gamma_D}(\domain; \R^d) = \bigl\{v\in H^1(\domain;\R^d) \big\vert
    v_{|\Gamma_D} = 0 \bigr\}.
  \end{gather}
  We notice for the second term that
  \begin{gather*}
    \id : \nabla v = \sum_{i=1}^d \d_i v_i = \div v.
  \end{gather*}
  Furthermore, we use the result of Problem~\ref{Problem:frobenius} to
  obtain
  \begin{gather*}
    \strain u:\nabla v = \strain u : \strain v.
  \end{gather*}
\end{intro}

\begin{Definition}{weak-lame-navier}
  The weak formulation of the Lamé-Navier boundary value
  problem
  \begin{gather*}
    \begin{aligned}
      -\div \sigma(x) &= f(x) & x&\in\domain,\\
      u(x) &= 0 & x&\in\Gamma_D, \\
      \sigma(x)n &=0& x&\in\Gamma_N,
    \end{aligned}
  \end{gather*}
  is: find $u\in V = H^1_{\Gamma_D}(\domain;\R^d)$ such that
  \begin{gather}
    a(u,v) \equiv 2\mu\form(\strain u, \strain v)
    + \lambda \form(\div u, \div v)
    = \form(f,v)
    \quad\forall v\in V.
  \end{gather}
\end{Definition}

\begin{Problem}{frobenius}
  Given the vector space of square matrices $X = \R^{d\times d}$ with the
  Frobenius inner product
  \begin{gather}
    \label{eq:mixedintro:frobenius}
    \scal(A,B) = A:B = \sum_{ij} a_{ij}b_{ij}.
  \end{gather}
  Show that the subspaces of symmetric and skew-symmetric matrices,
  respectively, are orthogonal to each other and $X$ is the direct sum
  of those.
\begin{solution}
\begin{enumerate}
\item
  First, we note that for every matrix $A$, there holds
  \begin{gather*}
    A = A_S + A_A, \quad A_S = \frac{A+A^T}2, \quad A_A = \frac{A-A^T}2.
  \end{gather*}
  Furthermore, we see that $A_S$ is symmetric and $A_A$ is
  skew-symmetric. The sum of two symmetric matrices is symmetric,
  the same for skew-symmetric. Thus, the sets $X_S$ and $X_A$ of
  symmetric and skew-symmetric matrices, respectively, are vector
  spaces and $X=X_S+X_A$.

\item The only matrix which is symmetric and skew-symmetric is
  zero. Thus, $X=X_S\oplus X_A$.

\item It remains to show orthogonality. To this end, we first note
  that for any skew-symmetric matrix $A$, there holds $a_{ii}=0$
  for all diagonal elements. Thus, the inner product of $A$ with a
  symmetric matrix $S$ is
  \begin{align*}
    \scal(A,S) &= \sum_{i\neq j} a_{ij}s_{ij}\\
               &= \sum_{i<j} (a_{ij}s_{ij} + a_{ji}s_{ji})\\
               &= \sum_{i<j} (a_{ij}-a_{ji})s_{ij}\\
               &= 0.
  \end{align*}
\end{enumerate}
\end{solution}
\end{Problem}

\begin{intro}
  The form $a(\cdot,\cdot)$ is symmetric and thus semi-definite on $V$. It can
  also be bounded easily by the $H^1$-norm. But, for well-posedness of
  the weak formulation, we also require ellipticity. This question is
  indeed not trivial and rests on the fact that for a function
  $u\in V$, such that $\nabla u$ is skew-symmetric everywhere, there
  holds $\strain u\equiv 0$. Thus, such functions must be excluded by
  the boundary conditions. Note, that in particular for rigid body
  translations and rotations $\strain u = 0$. Therefore, the Dirichlet
  boundary conditions must exclude such solutions.

  The condition needed for well-posedness is called Korn inequality,
  and it will be posed as an assumption. We will give a proof for a
  simple case and refer the readers to a plethora of articles on more
  complicated cases.
\end{intro}

\begin{Assumption}{korn-inequality}
  We assume that the boundary conditions defining the space $V$ in the
  weak formulation of the Lamé-Navier equations are such that
  a \define{Korn inequality}
  \begin{gather}
    \label{eq:mixedintro:korn}
    c_K^2 \norm{u}_{H^1(\domain;\R^d)}^2
    \le \norm{u}_{L^2(\Omega;\R^d)}^2
    + \norm{\strain u}_{L^2(\Omega;\R^d)}^2,
  \end{gather}
  holds for all $u\in V$ with a uniform constant $c_K>0$.
\end{Assumption}

\begin{remark}
  We use the indefinite article `a' for this inequality, because it
  comes in many different forms, for instance replacing the $L^2$-norm
  by a boundary term. It's mathematics is not trivial. There is
  nevertheless a very simple interpretation of this inequality: the
  kernel of the gradient consists of constant functions, in the
  context of elastic deformations of translations by a constant
  vector. The kernel of the strain tensor contains translation
  \emph{and} rotations. Thus, a Korn inequality can only hold, if the
  boundary conditions rule out rotations.
\end{remark}

\begin{Lemma}{korn}
  Let $V = H^1_0(\domain;\R^d)$ for a Lipschitz domain
  $\domain$. Then, Korn's inequality holds on $V$ with a constant
  $c_K>0$.
\end{Lemma}

\begin{proof}
  While the inequality is of high importance in the mathematical
  and numerical analysis of problems in continuum mechanics, it a
  peripheral topic to this class. Therefore, we omit the proof and
  refer to Theorem 3.3 in~\cite[Section III.3.3]{DuvautLions76}.
\end{proof}

\begin{Problem}{elasticity-standard}
  Let the space $V=H^1_0(\domain;\R^d)$ be equipped with the inner
  product $\scal(u,v) = a(u,v)$ with the bilinear form of the
  Lamé-Navier equations and the corresponding norm $\norm{.}_V$.

  Show using techniques from the standard theory of elliptic partial
  differential equations:
  \begin{enumerate}
  \item The weak formulation has a unique solution for which there holds
    \begin{gather*}
      \norm{u}_V \le \norm{f}_{V^*}.
    \end{gather*}
  \item The ``energy estimate'' for conforming finite element
    approximation with a space $V_h\subset V$
    \begin{gather*}
      \norm{u-u_h}_V = \inf_{v_h\in V_h} \norm{u-v_h}_V.
    \end{gather*}
  \item The $H^1$-error estimate
    \begin{gather}
      \norm{u-u_h}_{H^1}
      \le \frac{2\mu+d\lambda}{2c_K\mu}
      \inf_{v_h\in V_h} \norm{u-v_h}_{H^1}.
    \end{gather}
    Use the fact that the space $V$ can be composed into the space
    $V^0$ of divergence-free functions ($\div v=0$) and its
    complement.
  \item For $\lambda \gg \mu$, the previous estimate is useless. Can
    it be improved easily? In view of the ``energy estimate'', can you
    think of conditions?
  \end{enumerate}
\begin{solution}
We consider the problem:\\
  Find $u\in V = H^1_{\Gamma_D}(\domain;\R^d)$ such that
  \begin{gather*}
    a(u,v) \equiv 2\mu\form(\strain u, \strain v)
    + \lambda \form(\div u, \div v)
    = \form(f,v)
    \quad\forall v\in V.
  \end{gather*}
  where the norm on $V$ is given through $a$ as
  \begin{align*}
    \norm{u}_V^2=a(u,u)=2\mu\norm{\strain u}_0^2+ \lambda \norm{\div u}_0^2
    \end{align*}
  \begin{enumerate}
   \item Testing symmetrically ($v=u$) gives
   \begin{align*}
   \norm{u}_V=\frac{a(u,u)}{\norm{u}_V}= \frac{\form(f,u)}{\norm{u}_V} \leq \norm{f}_{V^*}.
   \end{align*}
   \item Applying Céa's lemma gives
   \begin{align*}
   \norm{u-u_h}_V&=\frac{a(u-u_h, u-u_h)}{\norm{u-u_h}_V}\\
               &=\frac{a(u-u_h, u-v_h)}{\norm{u-u_h}_V}\leq \norm{u-v_h}_V\quad \forall v_h \in V_h
   \end{align*} and hence $\norm{u-u_h}_V = \inf_{v_h\in V_h} \norm{u-v_h}_V$.
   \item For the $H^1$-error estimate we make use of Korn's inequality to note
   \begin{align*}
    \norm{u}_V^2=2\mu\norm{\strain u}_0^2+ \lambda \norm{\div u}_0^2\geq
    2\mu(c_K^2\norm{u}_1^2-\norm{u}_0^2)
   \end{align*}
   and on the other hand
   \begin{align*}
    \norm{u}_V^2=2\mu\norm{\strain u}_0^2+ \lambda \norm{\div u}_0^2\leq
    (2\mu+\lambda d^2)\norm{u}_1^2
   \end{align*}
   where we used
   \begin{align*}
    \norm{\nabla \cdot u}_0^2 = \norm{\sum_i \partial_i u_i}_0^2 &\leq (\sum_{i,j} \norm{\partial_j u_i}_0\delta_{i,j})^2\\
    & \leq \sum_{i,j} \left(\norm{\partial_j u_i}_0^2\right) d^2 \leq \norm{u}_1^2 d^2.
   \end{align*}
   Using Poincaré's inequality
   \begin{align*}
    \norm{u}_0\leq C_P |u|_1
   \end{align*}
   we further notice
   \begin{align*}
    \norm{u}_V^2+2\mu\norm{u}_0^2\leq (1+C_P^2) \norm{u}_V^2.
   \end{align*}
   Combining these estimates gives
   \begin{align*}
   2\mu c_K^2\norm{u}_1^2\leq
    \norm{u}_V^2+2\mu \norm{u}_0^2\leq
    (1+C_P^2)\norm{u}_V^2\leq
    (1+C_P^2)(2\mu+\lambda d^2)\norm{u}_1^2.
   \end{align*}
   and therefore
   \begin{align*}
    \norm{u-u_h}_1^2\leq \inf_{v\in V_h} \frac{(1+C_P^2)(2\mu+\lambda d^2)}{2\mu c_K^2}\norm{u-v_h}_1^2.
   \end{align*}
   If the solution is even $H_0^1(\Omega)$-regular, we can improve the estimate to
   \begin{align*}
    \norm{u}_1^2\leq \inf_{v\in V_h}\frac{(\mu+(\lambda+\mu) d^2)}{\mu}\norm{u-v_h}_1^2.
   \end{align*}
   by using the identity
   \begin{align*}
    (\strain u, \strain v) = (\nabla u, \nabla v) + (\nabla \cdot u, \nabla \cdot v).
   \end{align*}

   \item A more refined estimate would be
   \begin{align*}
    \norm{u-u_h}_1^2\leq \inf_{v\in V_h}\frac{(1+C_P^2)}{2\mu c_K^2}\left(2\mu\norm{u-v_h}_1^2+\lambda \norm{\nabla\cdot (u-v_h)}_0^2\right).
   \end{align*}
   Choosing a divergence-preserving interpolation operator, the second term vanishes.
  \end{enumerate}
\end{solution}
\end{Problem}

\begin{example}
  We study the finite element approximation of the following problem:
  a square sheet of elastic material is hanging from the top, subject
  to gravity acting as body force pointing downward. We choose $\mu=1$
  and vary $\lambda$ from 1 to
  $10^5$. Figure~\ref{fig:elasticity-compressibility} shows
  approximations with standard bilinear finite elements (red) and a
  ``good'' approximation (blue). In fact, the red solution for
  $\lambda = 10^5$ is almost identical with the undeformed
  configuration, although the material is not as hard. This phenomenon
  was discovered early in finite element history and is called
  ``locking''.
\end{example}
\begin{figure}[tp]
  \centering
  \subfigure[{$\lambda = 1$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-0}}
  \subfigure[{$\lambda = 10$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-1}}
  \subfigure[{$\lambda = 100$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-2}}
  \subfigure[{$\lambda = 1000$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-3}}
  \subfigure[{$\lambda = 10000$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-4}}
  \subfigure[{$\lambda = 100000$}]{\includegraphics[width=.45\textwidth]{./graph/elasticity/stalactite-5}}
  \caption{Approximation with standard finite elements (red) and
    ``good'' elements (blue) for $\mu=1$ and different values of
    $\lambda$.}
  \label{fig:elasticity-compressibility}
\end{figure}

\begin{intro}
  As we could see in the preceding problem and example, approximation of the
  solution to the Lamé-Navier equations becomes difficult, if
  $\lambda \gg \mu$. In this case, the material is called almost
  incompressible, since the divergence measures compression or
  dilation and the dominating divergence term forces the divergence of
  the solution to be small. These cases are important in engineering
  and they initiated a lot of the research that resulted in the topics
  of this class.
\end{intro}

\begin{intro}
  A way to approach this problem is the introduction of an auxiliary variable
  \begin{gather*}
    p = -\lambda \div u.
  \end{gather*}
  Entering this definition into the Lamé-Navier equations, we obtain
  the following weak formulation.
\end{intro}

\begin{Definition}{displacement-pressure}
  The \define{displacement-pressure formulation} of the Lamé-Navier
  equations reads: find a pair $(u,p) \in V\times Q$ such that
  \begin{gather}
    \label{eq:lame-navier-mixed}
    \begin{aligned}
      2\mu\form(\strain u, \strain v) &- \form(p,\div v) &=&\form(f,v)
      &\forall v&\in V\\
      -\form(q,\div u) &-\tfrac1\lambda \form(p,q) &=&0
      &\forall q&\in Q.\\
    \end{aligned}
  \end{gather}
  Equivalently, we write this in a single equation as
  \begin{multline}
    2\mu\form(\strain u, \strain v) - \form(p,\div v)
    - \form(q,\div u) -\tfrac1\lambda \form(p,q)
    = \form(f,v)
    \\
    \qquad\forall v\in V, q\in Q,
  \end{multline}
  or in the nonsymmetric, (semi-)definite version
  \begin{multline}
    2\mu\form(\strain u, \strain v) + \form(p,\div v)
    - \form(q,\div u) +\tfrac1\lambda \form(p,q)
    = \form(f,v)
    \\
    \qquad\forall v\in V, q\in Q,
  \end{multline}
  Here, $V$ is as before and $Q=L^2(\domain)$.
\end{Definition}

\begin{remark}
  The three forms have different purposes and will be used
  accordingly. The first one highlights the fact that we now have a
  system of equations, each equation tested with its own test
  function. The second and third stress the fact that we now have a
  bilinear form on the product space $X=V\times Q$.

  The second form is symmetric, but we will see later that is
  indefinite. Thus, non of our tools from functional analysis
  apply. In contrast, the third form is nonsymmetric, but we have
  \begin{multline*}
    2\mu\form(\strain u, \strain u) + \form(p,\div u)
    - \form(p,\div u) +\tfrac1\lambda \form(p,p)
    \\
    = 2\mu\form(\strain u, \strain u) + \tfrac1\lambda \form(p,p)
    \ge \tfrac{2\mu}{c_K} \norm{u}_{H^1}^2 + \tfrac1\lambda \form(p,p).
  \end{multline*}
  Thus, we have ellipticity with respect to the norm
  \begin{gather}
    \norm{(u,p)}_X^2 = \norm{u}_{H^1}^2 + \norm{p}_{L^2}^2.
  \end{gather}
  Nevertheless, the ellipticity constant depends on $\lambda$, and for
  large $\lambda$, we loose sharpness of estimates again.
\end{remark}

\begin{Definition}{lame-navier-strong}
  Integrating the first equation by parts, we
  obtain the \textbf{strong form} of the Lamé-Navier equations
  \begin{gather}
    \arraycolsep.2ex
    \begin{matrix}
% Check signs
      - 2\mu \div \strain u &+& \nabla p &=& f \\
      \div u &+& \tfrac1\lambda p &=& 0
    \end{matrix}
  \end{gather}
\end{Definition}

\section{Abstract saddle-point systems}
%\subsection{Notation and basic properties}

\begin{intro}
  In order to study the mathematics not only of the Lamé-Navier
  equations but of more general systems of the form of
  equation~\eqref{eq:lame-navier-mixed}, we introduce abstract
  bilinear forms
  \begin{gather}
    \label{eq:mixedintro:1}
    \begin{aligned}
      a(u,v) &= 2\mu\form(\strain u, \strain u), &u,v&\in V \\
      b(u,p) &=  -\form(p,\div u), & u&\in V, p\in Q\\
      c(p,q) &= \tfrac1\lambda \form(p,q) & p,q&\in Q.
    \end{aligned}
  \end{gather}
\end{intro}

\begin{Definition}{saddle-point-operators}
  For each of the bilinear forms, we define associated operators
  \begin{gather}
    \begin{aligned}
      A&\colon V \to V^* & \scal(Au,v) &= a(u,v), \\
      B&\colon V \to Q^* & \scal(Bu,q) &= b(u,q), \\
      B^T&\colon Q \to V^* & \scal(B^T p,v) &= b(v,p), \\
      C&\colon Q\to Q^* & \scal(C p,q) &= c(p,q).
    \end{aligned}
  \end{gather}
  Here, $\scal(\cdot,\cdot)$ is the canonical pairing between an element of
  the dual space and the space itself.
\end{Definition}

\begin{Definition}{saddle-point-abstract}
  The abstract saddle-point problem in weak form reads: find a pair
  $(u,p)\in V\times Q$ such that
  \begin{gather}
    \label{eq:saddle-point-weak}
    \arraycolsep.1em
    \begin{matrix}
      a(u,v) &+& b(v,p) &=& f(v) &\quad&\forall v\in V, \\
      b(u,q) &-& c(p,q) &=& g(q) &&\forall q\in Q.
    \end{matrix}
  \end{gather}
  In operator notation, it reads
  \begin{gather}
    \label{eq:saddle-point-operator}
    \arraycolsep.1em
    \begin{matrix}
      Au &+& B^T p &=& f &\quad&\text{in } V^*, \\
      Bu &-& C p &=& g &&\text{in } Q^*.
    \end{matrix}
  \end{gather}
\end{Definition}

\begin{Notation}{saddle-point-form}
  In order to consider the whole bilinear form of the saddle-point
  system on the space $X = V\times Q$, we introduce
  \begin{gather}
    \mathcal A\mixedform(u,p,v,q)
      = a(u,v) + b(v,p) + b(u,q) - c(p,q)
  \end{gather}
\end{Notation}

\begin{Definition}{schur-complement}
  Let the operator $A:V\to V^*$ in the saddle-point system be
  invertible. Then, we define the \define{Schur complement} operator
  $S\colon Q\to Q^*$ of the system as
  \begin{gather}
      S= -B A^{-1} B^T - C.
  \end{gather}
\end{Definition}

\begin{Lemma}{schur-complement1}
  Formally, the saddle-point system~\eqref{eq:saddle-point-operator}
  can be solved in two steps by solving
  \begin{align}
    S p &= g - B A^{-1} f,\\
    A u &= f-B^T p.
  \end{align}
\end{Lemma}

\begin{proof}
  Formally solve the first equation
  of~\eqref{eq:saddle-point-operator} for $u$ and enter into the
  second.
\end{proof}

\begin{Lemma}{schur-definiteness}
  Let $a(\cdot,\cdot)$ be elliptic and $c(\cdot,\cdot)$ be positive semi-definite, and
  let $\ker B^T\cap\ker C \neq \{0\}$. Then, the bilinear form
  $\mathcal A(\cdot,\cdot)$ is indefinite.
\end{Lemma}

\begin{proof}
  First, we note that because of ellipticity of $a(\cdot,\cdot)$
  \begin{gather*}
    \mathcal A\mixedform(u,0,u,0) = a(u,u) \ge \gamma \norm{u}_V^2,
  \end{gather*}
  for some positive constant $\gamma$. Furthermore, $A$ is invertible
  and its inverse is positive definite. Furthermore,
  $A^{-1}B^T\colon Q\to V$. Then, choosing $v=-A^{-1}B^T p$ yields
  \begin{align*}
    \mathcal A\mixedform(v,p,v,p)
    &= a(A^{-1}B^T p,A^{-1}B^T p) - 2b(A^{-1}B^T p,p) - c(p,p) \\
    &= \scal(B^T p,A^{-1}B^T p) - 2\scal(A^{-1}B^T p,B^T p) - c(p,p)\\
    &= -\scal(B^T p,A^{-1}B^T p) - c(p,p).
  \end{align*}
  The first term is a quadratic term with the bilinear form associated
  with $A^{-T}$, which is positive definite. Since $b(\cdot,\cdot)$ is not the
  zero form, there is some $p$ such that $v\neq 0$. Therefore, with
  the minus sign and the semi-definiteness of $c(\cdot,\cdot)$ we have found a
  vector such that
  \begin{gather*}
    \mathcal A\mixedform(v,p,v,p) < 0.
  \end{gather*}
\end{proof}

\begin{remark}
  The previous result holds in particular for $c(\cdot,\cdot) \equiv 0$.
\end{remark}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Stokes equations}

\begin{intro}
  When we write the Lamé-Navier equations in mixed form according to
  Definition~\ref{Definition:displacement-pressure}, there is no
  parameter $\lambda$ tending to infinity when the material becomes
  more and more compressible. Instead, there is the parameter
  $1/\lambda$ tending to zero. Thus, we can simply consider the case
  of incompressible material by setting $1/\lambda=0$ or, in our
  abstract framework~\eqref{eq:mixedintro:1} setting $c(\cdot,\cdot) = 0$. The
  resulting system is not only important for incompressible
  elasticity, but also models the slow flow of a very viscous liquid,
  so called \putindex{creeping flow}.
\end{intro}

\begin{Definition}{stokes-eq1}
  The \define{Stokes equations} in strong form are
    \begin{gather}
      \label{eq:mixedintro:stokes-strong1}
      \arraycolsep.2ex
      \begin{matrix}
        -2\mu \div \strain u &+& \nabla p &=& f \\
        \div u && &=& 0.
      \end{matrix}
    \end{gather}
    In weak form, they are: find $u\in V \subset H^1(\domain;\R^d)$
    and $p\in Q \subset L^2(\domain)$ such that
  \begin{gather}
    \label{eq:mixedintro:stokes-weak1}
    \begin{aligned}
      2\mu\form(\strain u, \strain v) &- \form(\div v,p) &=&\form(f,v)
      + \text{bdry}
      &\forall v&\in V\\
      -\form(\div u,q) & &=&0+ \text{bdry}
      &\forall q&\in Q.\\
    \end{aligned}
  \end{gather}
  The subspaces $V$ and $Q$ are determined by boundary conditions.
\end{Definition}

\begin{Definition}{solenoidal}
  A vector-valued function $u$ is called \define{divergence-free} or
  \define{solenoidal}, if there holds
  \begin{gather*}
    \div u = 0.
  \end{gather*}
  Flow described by a solenoidal function is called
  \define{incompressible}.
\end{Definition}

\begin{Lemma}{stokes-equivalence}
  Let $V=H^1_0(\domain;\R^d)$. Then, for any solenoidal function $u\in
  V$ there holds
  \begin{gather}
    \label{eq:mixedintro:6}
    2\mu\form(\strain u, \strain v) = \mu\form(\nabla u, \nabla v)
    \qquad\forall v\in V.
  \end{gather}
\end{Lemma}

\begin{proof}
  We have
  \begin{align*}
    \strain u: \strain v
    &= \frac14\sum_{i,j=1}^d\left[ (\d_i u_j+\d_j u_i) (\d_i v_j+\d_j v_i)
      \right]
    \\
    &= \frac12\sum_{i,j=1}^d\left[ \d_i u_j\d_iv_j + \d_i u_j\d_j v_i\right].
  \end{align*}
  The first term is the desired result, thus we have to eliminate the
  other one. We integrate by parts to obtain
  \begin{align*}
    \int_\domain \d_i u_j\d_j v_i \dx
    &= - \int_\domain \d_{ij}u_j v_i \dx
%     \\ &
    = \int_\domain \d_j u_j \d_i v_i \dx
  \end{align*}
  Entering in the
  previous equation and summing over $i$ and $j$, we obtain
  \begin{gather*}
    2(\strain u,\strain v) = (\nabla u,\nabla v)+(\nabla\cdot u,\nabla\cdot v)
                           = (\nabla u,\nabla v).
  \end{gather*}
\end{proof}

\begin{intro}
  In order to simplify subsequent discussion, it is customary to use
  the previous lemma to simplify the Stokes equations and to replace
  the strain tensor by the gradient. As a result, we can avoid the use
  of a Korn inequality and operate directly with the inner product of
  $H^1$. We note though that this formulation, while mathematically
  simpler, is physically wrong if $u\neq H^1_0(\domain;\R^d)$.
\end{intro}

\begin{Definition}{stokes-eq2}
  The simplified \define{Stokes equations} in strong form are
    \begin{gather}
      \label{eq:mixedintro:stokes-strong2}
      \arraycolsep.2ex
      \begin{matrix}
        - \nu \Delta u &+& \nabla p &=& f \\
        \div u && &=& 0.
      \end{matrix}
    \end{gather}
    In weak form, they are: find $u\in V \subset H^1(\domain;\R^d)$
    and $p\in Q \subset L^2(\domain)$ such that
  \begin{gather}
    \label{eq:mixedintro:stokes-weak2}
    \begin{aligned}
      \nu\form(\nabla u, \nabla v) &- \form(\div v,p) &=&\form(f,v)
      + \text{bdry}
      &\forall v&\in V\\
      -\form(\div u,q) & &=&0+ \text{bdry}
      &\forall q&\in Q.\\
    \end{aligned}
  \end{gather}
  The subspaces $V$ and $Q$ are determined by boundary conditions.
\end{Definition}

\begin{Definition}{stokes-boundary2}
  Typical boundary conditions for the Stokes problem are
  \begin{xalignat}3
    \text{no-slip:}&& u &= 0,\\
    \text{free:}&& \d_n u + p n &= 0 ,\\
    \text{slip:}&& u_n &= 0 & \d_n u_\tau = 0,\\
    \text{friction:}&& u_n &= 0 & \d_n u_\tau = \alpha u_\tau.
  \end{xalignat}
  Here, $u_n$ and $u_\tau$ are the normal and tangential components of
  $u$ at the boundary.
\end{Definition}

\begin{remark}
  Very much the same way as for elliptic equations, boundary
  conditions on the function $u$ itself are essential boundary
  conditions which have to be incorporated into the space $V$, while
  those on the normal derivatives are the result of integration by
  parts and thus of the type of natural boundary conditions.

  All of the conditions above can also be imposed with nonzero data,
  where the physical meaning of such a condition might be debatable in
  some cases. Mathematically, inhomogeneous essential boundary
  conditions are achieved by lifting an arbitrary function with this
  boundary condition and modifying the right hand side of the
  equation, while inhomogeneous conditions on the normal derivative
  are implemented by boundary integrals on the right hand side.
\end{remark}

\begin{remark}
  Physically, the condition $u_n=0$ models an impermeable wall. It
  says in particular, that no mass is lost through this boundary and
  is thus related to the first principle of mass conservation.

  The conditions on the tangential velocity model the fact that
  molecules very close to the wall stick to the wall. While this claim
  is not supported by a first principle, it has been verified by
  measurements to very high accuracy. Nevertheless, in the study of
  turbulent flow, a friction condition comes up quite naturally.
\end{remark}

\begin{Lemma}{divergence-compatibility}
  For any solenoidal $u$ function there holds
  \begin{gather}
    \int_{\d\domain} u\cdot n\ds = 0.
  \end{gather}
  Furthermore, if the boundary condition $u_n=0$ holds on the whole
  boundary $\d\domain$, then the pressure $p$ is determined by the
  Stokes equations only up to a constant.
\end{Lemma}

\begin{proof}
  The first statement is simple application of the Gauss theorem
  \begin{gather*}
    \int_{\domain} \div u\dx = \int_{\d\domain} u\cdot n\ds.
  \end{gather*}
  For the second statement, we note that the only term in the
  equations which determines the pressure is $\form(\div v,
  p)$. Integrating by parts and using the boundary condition, we
  obtain
  \begin{gather*}
    \int_\domain \div vp\dx = -\int_\domain v\nabla p\dx +
    \int_{\d\domain} v\cdot n p\ds = -\int_\domain v\nabla p\dx.
  \end{gather*}
  Since the gradient of a constant is zero, we can add any constant
  function to a given solution $p$ without changing the term
  $\form(\div v,p)$, thus leaving $p$ determined only up to a
  constant.
\end{proof}

\begin{Notation}{pressure-constant}
  If in Definition~\ref{Definition:stokes-eq2} the space $V$ is chosen
  such that for all $v\in V$ there holds $v\cdot n =0$ on the whole
  boundary $\d\domain$, then the pressure solution $p$ cannot be
  determined uniquely in $L^2(\domain)$. In such cases, we choose the
  pressure space
  \begin{gather}
    L^2_0(\domain) = L^2(\domain)/\R
    = \biggl\{q\in L^2(\domain)\bigg| \int_\domain q\dx = 0\bigg\}.
  \end{gather}
\end{Notation}

\begin{remark}
  As we could see from the preceding lemma, solvability of
  equation~\eqref{eq:mixedintro:stokes-weak2} depends on some
  compatibility of the spaces $V$ and $Q$ we had not seen in the
  elliptic case. Indeed, we need a whole new tool from functional
  analysis to replace the Lax-Milgram lemma. This tool will be studied
  in Chapter~\ref{sec:mixed-wellposedness}.
\end{remark}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\appendix

\chapter{\label{app:prim-exists}Using logic of generalizations}

\section{Logic of generalization}

In mathematics it is often encountered that a smaller set $S$ naturally
bijectively corresponds to a subset $R$ of a larger set $B$. (In
other words, there is specified an injection from $S$ to $B$.) It
is a widespread practice to equate $S$ with $R$.
\begin{rem}
I denote the first set $S$ from the first letter of the word ``small''
and the second set $B$ from the first letter of the word ``big'',
because $S$ is intuitively considered as smaller than $B$. (However
we do not require $\card S<\card B$.)
\end{rem}
The set $B$ is considered as a generalization of the set $S$, for
example: whole numbers generalizing natural numbers, rational numbers
generalizing whole numbers, real numbers generalizing rational numbers,
complex numbers generalizing real numbers, etc.

But strictly speaking this equating may contradict to the axioms of
ZF/ZFC because we are not insured against $S\cap B\neq\emptyset$
incidents. Not wonderful, as it is often labeled as ``without proof''.

To work around of this (and formulate things exactly what could benefit
computer proof assistants) we will replace the set $B$ with a new
set $B'$ having a bijection $M:B\rightarrow B'$ such that $S\subseteq B'$.
(I call this bijection $M$ from the first letter of the word ``move''
which signifies the move from the old set $B$ to a new set $B'$).

The following is a formal but rather silly formalization of this situation
in ZF. (A more natural formalization may be done in a smarter formalistic,
such as dependent type theory.)

\subsection{The formalistic}

Let $S$ and $B$ be sets. Let $E$ be an injection from $S$ to $B$.
Let $R=\im E$.

Let $t=\subsets\bigcup\bigcup S$.

Let $M(x)=\left\{ \begin{array}{ll}
E^{-1}x & \text{if }x\in R;\\
(t,x) & \text{if }x\notin R.
\end{array}\right.$

Recall that in standard ZF $(t,x)=\{t,\{t,x\}\}$ by definition.
\begin{thm}
$(t,x)\notin S$.\end{thm}
\begin{proof}
Suppose $(t,x)\in S$. Then $\{t,\{t,x\}\}\in S$. Consequently $\{t\}\in\bigcup S$;
$\{t\}\subseteq\bigcup\bigcup S$; $\{t\}\in\subsets\bigcup\bigcup S$;
$\{t\}\in t$ what contradicts to the axiom of foundation (aka axiom
of regularity).\end{proof}
\begin{defn}
Let $B'=\im M$.\end{defn}
\begin{thm}
$S\subseteq B'$.\end{thm}
\begin{proof}
Let $x\in S$. Then $Ex\in R$; $M(Ex)=E^{-1}Ex=x$; $x\in\im M=B'$.\end{proof}
\begin{obvious}
$E$ is a bijection from $S$ to $R$.\end{obvious}
\begin{thm}
$M$ is a bijection from $B$ to $B'$.\end{thm}
\begin{proof}
Surjectivity of $M$ is obvious. Let's prove injectivity. Let $a,b\in B$
and $M(a)=M(b)$. Consider all cases: \end{proof}
\begin{description}
\item [{$a,b\in R$}] $M(a)=E^{-1}a$; $M(b)=E^{-1}b$; $E^{-1}a=E^{-1}b$;
thus $a=b$ because $E^{-1}$ is a bijection.
\item [{$a\in R$,~$b\notin R$}] $M(a)=E^{-1}a$; $M(b)=(t,b)$; $M(a)\in S$;
$M(b)\notin S$. Thus $M(a)\neq M(b)$.
\item [{$a\notin R$,~$b\in R$}] Analogous.
\item [{$a,b\notin R$}] $M(a)=(t,a)$; $M(b)=(t,b)$. Thus $M(a)=M(b)$
implies $a=b$. \end{description}
\begin{thm}
$M\circ E=\id_{S}$.\end{thm}
\begin{proof}
Let $x\in S$. Then $Ex\in R$; $M(Ex)=E^{-1}Ex=x$.\end{proof}
\begin{obvious}
$E=M^{-1}|_{S}$.
\end{obvious}

\subsection{Existence of primary filtrator}
\begin{thm}
For every poset~$\mathfrak{Z}$ there exists a poset $\mathfrak{A}\supseteq\mathfrak{Z}$
such that $(\mathfrak{A},\mathfrak{Z})$ is a primary filtrator.\end{thm}
\begin{proof}
Take $S=\mathfrak{Z}$, $B=\mathfrak{F}$, $E=\uparrow$. By the above
there exists an injection $M$ defined on~$\mathfrak{F}$ such that
$M\circ\uparrow=\id_{\mathfrak{Z}}$.

Take $\mathfrak{A}=\im M$. Order ($\sqsubseteq'$) elements of~$\mathfrak{A}$
in such a way that $M:\mathfrak{F}(\mathfrak{Z})\rightarrow\mathfrak{A}$
become order isomorphism. If $x\in\mathfrak{Z}$ then $x=\id_{\mathfrak{Z}}x=M\uparrow x\in\im M=\mathfrak{A}$.
Thus $\mathfrak{A}\supseteq\mathfrak{Z}$.

If $x\sqsubseteq y$ for elements $x$, $y$ of $\mathfrak{Z}$, then
$\uparrow x\sqsubseteq\uparrow y$ and thus $M\uparrow x\sqsubseteq'M\uparrow y$
that is $x\sqsubseteq'y$, so $\mathfrak{Z}$ is a subposet of~$\mathfrak{A}$,
that is $(\mathfrak{A},\mathfrak{Z})$ is a filtrator.

It remains to prove that $M$ is an isomorphism between filtrators
$(\mathfrak{F}(\mathfrak{Z}),\mathfrak{P})$ and $(\mathfrak{A},\mathfrak{Z})$.
That $M$ is an order isomorphism from $\mathfrak{F}(\mathfrak{Z})$
to $\mathfrak{A}$ is already known. It remains to prove that $M$
maps $\mathfrak{P}$ to~$\mathfrak{Z}$. We will instead prove that
$M^{-1}$ maps $\mathfrak{Z}$ to~$\mathfrak{P}$. Really, $\uparrow x=M^{-1}x$ for every $x\in\mathfrak{Z}$.\end{proof}


%%!TEX TS-program = latex
\documentclass[11pt]{article} %DIF >
\usepackage{etex}
\usepackage[utf8]{inputenc}
\input ../AuxFiles/PreambleNotes.tex

\begin{document}
\onehalfspace

\vspace*{\fill}
\begingroup
\centering

\Large {\scshape Introduction to Statistics}\\

(Lectures 7-8: Estimation)

\endgroup
\vspace*{\fill}

\newpage

\section{Estimation}

{\scshape Overview:} \noindent Lectures 7-8 will focus on \emph{estimation}. As we have mentioned before, a strategy in the estimation problem is an \emph{estimator}. The loss function used for this problem is the squared distance between the estimated value and the true parameter. This loss gives rise to the popular \emph{mean squared error} performance criterion.

We will analyze two commonly used estimation strategies---Maximum Likelihood (ML) and the Ridge estimator. We will do this in the context of the homoskedastic Normal regression model with (known variance); in which the conditional distribution of $Y|X$ is modeled as
\begin{equation} \label{equation:Regression-model}
Y|X \sim \mathcal{N}(X \beta, \sigma^2 \mathbb{I}_n ),
\end{equation}
 where $Y$ is the $n \times 1$ vector of outcome variables, $X$ is the $n \times k$ matrix of regressors, and $\beta \in \mathbb{R}^{k}$ is the parameter of interest. We denote by $P_{X}$ the distribution of the regressors, and we assume that this distribution is known and does not depend on $\beta$.

 We will show that if $k \leq n$ and $X$ has full-column rank, the Maximum Likelihood estimator for the model in (\ref{equation:Regression-model}) is the OLS estimator
\[ \widehat{\beta}_{\textrm{OLS}} \equiv (X’X)^{-1} X’Y,  \]

\noindent (which we introduced in the last problem set). We will compute the risk of this estimator and we will show that this estimator is minimax. We also show that, despite the minimaxity, when $k \geq 3$ the estimator is dominated and we present an ``empirical Bayes’’ estimator that dominates it.

%show this estimator is ``best'' among all \emph{unbiased} estimators and we will connect this result to the \emph{Cram\'er-Rao} bound for the variance of estimators in parametric models. We will also argue that ``best’’ among the class of unbiased estimators need not imply admissibility with respect to all estimators.

We also show that the \emph{Ridge estimator} is a Bayes decision rule for the prior distribution $\beta \sim \mathcal{N}_{k}(0 , (\sigma^2/ \lambda) \mathbb{I}_k )$. The Ridge estimator is well-defined regardless of the number of covariates and it is admissible by construction. The Ridge estimator provides a simple example of ``shrinkage’’ or ``regularization’’. A practical concern is that the Ridge estimator is biased, and the bias depends on the choice of the prior hyperparameters. Moreover, we show that the minimax risk of the Ridge estimator equals infinity.

One result that we will not cover in the notes, but that is important to keep in mind is that despite the inadmissibility of the OLS estimator for the \emph{full} vector of coefficients $\beta$, OLS is admissible for the problem in which we are interested in estimating only one regression coefficient but controlling for other variables.

\subsection{Mean-squared estimation error}
In an estimation problem the action space equals the parameter space. In the context of the linear regression model, we will assume that parameter space simply refers to all possible values that $\beta$ can take, which we take to be any vector in $\mathbb{R}^{k}$. This means that we are assuming that both $\sigma^2$ and $\mathbb{P}_{X}$ are known.

The loss function for this problem is the so-called \emph{quadratic loss}, defined as:
\[ \mathcal{L}(a, \beta) = || a- \beta ||^2 = (a-\beta)’(a-\beta) =  \sum_{j=1}^{k} (a_j  - \beta_j)^2,  \]
where $\beta_j$ represents the $j$-th coordinate of the $k$-dimensional vector $\beta$.

Decision rules in the estimation problem are called estimators. Thus, an estimator---which we will denote as $\widehat{\beta}$---is a map from data $(Y,X)$ to the parameter space. Its risk is given by
\begin{equation} \label{equation:MSE}
\mathbb{E}_{\mathbb{P}_{\beta}}[ \mathcal{L}(\widehat{\beta}, \beta) ] = \mathbb{E}_{\mathbb{P}_{\beta}}[ (\widehat{\beta}- \beta)’(\widehat{\beta}-\beta) ].
\end{equation}
Equation (\ref{equation:MSE}) is referred to as \emph{the mean-squared estimation error} at $\beta$, and denoted $\textrm{MSE}(\widehat{\beta}, \beta)$.

It will be convenient to express the mean squared error in terms of the ``bias’’ and ``variance’’ of $\widehat{\beta}$. Assuming that $\overline{\beta}  \equiv \mathbb{E}_{\beta}[ \widehat{\beta} ]$ is finite, we define the \underline{bias} of $\widehat{\beta}$ at $\beta$ as
$$ B_\beta(\widehat{\beta}) = \overline{\beta} - \beta.$$
If the covariance matrix of $\widehat{\beta}$---denoted $\textrm{V}_{\beta} (\widehat{\beta})$---is also finite, the mean squared-error can be written as
\begin{eqnarray*}
\mathbb{E}_{\mathbb{P}_{\beta}}[ (\widehat{\beta}- \beta)’(\widehat{\beta}-\beta) ] &=& \mathbb{E}_{\mathbb{P}_{\beta}}[ (\widehat{\beta}-\overline{\beta} +\overline{\beta} - \beta)’(\widehat{\beta}-\overline{\beta} +\overline{\beta} -\beta) ],  \\
&=& (\overline{\beta}-\beta)’(\overline{\beta}-\beta) +  \mathbb{E}_{\mathbb{P}_{\beta}}[ (\widehat{\beta}-\overline{\beta} )’(\widehat{\beta}-\overline{\beta}) ]  \\
&=& ||B_\beta(\widehat{\beta})  ||^2 + \textrm{tr} \left( \textrm{V}_{\beta} (\widehat{\beta}) \right),
\end{eqnarray*}
where $\textrm{tr}(\cdot)$ is the trace operator.  The decomposition is fairly straightforward, but it highlights the fact that the bias and variance of the estimator fully determine its risk whenever the loss is quadratic. Also, the correlation between any of the components of $\widehat{\beta}$ is not relevant for the risk calculation. In the next subsections we will compute the mean-squared estimation error of some popular estimators for model \eqref{equation:Regression-model}, including Bayes and minimax estimators.

\subsection{The Maximum Likelihood Estimator and its mean-squared error}

According to model (\ref{equation:Regression-model}), the distribution of $Y|X$ is a multivariate normal with parameters $X\beta$ and $\sigma^2 \mathbb{I}_n$ and thus has a p.d.f given by
\begin{equation}\label{equation:Regression-pdf}
f( Y | \beta , X ) = \frac{1}{(2 \pi \sigma^2)^{n/2}} \exp \left( -\frac{1}{2 \sigma^2} (Y - X \beta)’ (Y-X\beta) \right).
\end{equation}
This is true, regardless of whether we have very many covariates or not.

Assume that the distribution of $X$ has a density $f_{X}(X)$ that does not depend on $\beta$. For a fixed realization of the data, define the \underline{likelihood function}, $L(\beta; (Y,X))$, as the value attained by the p.d.f. of the joint distribution of $(Y,X)$ at different values of the parameter vector $\beta$.\footnote{For a textbook definition of the likelihood function for parametric models see Chapter 10.3 of Brucen Hansen's book url{https://www.ssc.wisc.edu/~bhansen/probability/Probability.pdf}} Thus,
\[ L(\beta, (Y,X)) \equiv f(Y | \beta, X )f_X(X). \]
The \underline{maximum likelihood estimator} of $\beta$ is then defined as the value of $\beta$ that maximizes the likelihood; that is
\[ \widehat{\beta}_{\textrm{ML}} \equiv \textrm{argmax}_{\beta \in \mathbb{R}^{k}} L(\beta, (Y,X)).   \]

\noindent The likelihood function implied by (\ref{equation:Regression-pdf}) is decreasing in $(Y-X\beta)’(Y-X\beta)$, a term which is usually referred to as the sum of squared residuals. Therefore, maximizing the likelihood is equivalent to solving the problem
\[ \textrm{min}_{\beta \in \mathbb{R}^{k}} (Y-X\beta)’(Y-X\beta).   \]
The first-order conditions for the program above, which are necessary and sufficient, yield
\[ X’(Y-X\widehat{\beta}_{\textrm{ML}}) = \textbf{0}_{k \times 1} \iff X’Y = (X’X) \widehat{\beta}_{\textrm{ML}}. \]

\noindent If $k > n$, there are infinitely many solutions that maximize the likelihood.  If $k \leq n$, and $X$ has full-column rank there is a unique solution to this problem given by the ordinary least-squares (OLS) estimator:
\begin{equation} \label{equation:OLS}
\widehat{\beta}_{\textrm{ML}} = \widehat{\beta}_{\textrm{OLS}} = (X’X)^{-1} X’Y.
\end{equation}

\noindent {\scshape Mean-squared Error of the Maximum Likelihood Estimator:} We have shown that we can express the mean-squared error of any estimator in terms of its bias and variance. Under the assumptions we have made for the distribution of $Y|X$, it is possible to show that the bias of the maximum likelihood estimator is zero, since:
\begin{equation} \label{equation:bias}
\textrm{B}_{\beta}(\widehat{\beta}_{\textrm{ML}}) \equiv \mathbb{E}_{\mathbb{P}_{\beta}} [ (X’X)^{-1} X’Y ] - \beta = \mathbf{0}_{k \times 1}.
\end{equation}
for any $\beta$. Thus, we say that the OLS estimator of $\beta$ (which is the ML estimator for problem \eqref{equation:Regression-model}) is \underline{unbiased}.\footnote{An estimator $\widehat{\beta}$ is unbiased if $\mathbb{E}_{\beta} [\widehat{\beta}] = \beta$ for all $\beta$ in the parameter space.}

The variance of the ML estimator is
\begin{eqnarray*} \mathbb{V}_{\beta} (\widehat{\beta}_{\textrm{ML}}) &=&  \mathbb{E}_{\mathbb{P}_{\beta}} [ (\widehat{\beta}_{\textrm{ML}}-\beta) (\widehat{\beta}_{\textrm{ML}} -\beta )^{\prime}  ] \\
&=&  \mathbb{E}_{f_{X}} \left[  \mathbb{E}_{\mathbb{P}_{\beta}} [ (\widehat{\beta}_{\textrm{ML}}-\beta) (\widehat{\beta}_{\textrm{ML}} -\beta )^{\prime}  | X  ]  \right] \\
&=&  \mathbb{E}_{f_{X}} \left[  (X’X)^{-1} X’ \mathbb{E}_{\mathbb{P}_{\beta}} [ (Y-X\beta) (Y-X\beta)’] X (X’X)^{-1} \right] \\
&=& \sigma^2 \mathbb{E}_{f_{X}} \left[ (X’X)^{-1} \right].  \label{equation:variance}
\end{eqnarray*}
This means that the mean squared error at $\beta$---henceforth denoted $\textrm{MSE}(\beta ; \widehat{\beta}_{\textrm{ML}})$---equals
\[ \sigma^2 \textrm{tr} \left (\mathbb{E}_{f_{X}} \left[ (X’X)^{-1} \right] \right). \]

\noindent for any $\beta$. Consequently, an interesting property of the ML/OLS estimator is that its risk function is constant over the parameter space.

The decision-theoretic approach to statistics that was presented last class is devoted to the problem of finding rules that, in one way or another, are optimal. Thus, in such a framework there is only room for maximum likelihood estimation insofar as such procedure has good risk properties (at least within some class of decision rules).

It is possible to provide such justification for the maximum likelihood estimator of model \eqref{equation:Regression-model}. In particular, it is possible to show that the OLS estimator minimizes risk among the class of all unbiased estimators, provided some regularity conditions are met. This is a version of the celebrated \emph{Gauss-Markov} theorem. There is a proof of this statement in Section 4.8 of Bruce Hansen's book.\footnote{https://www.ssc.wisc.edu/~bhansen/econometrics/Econometrics.pdf}. I encourage you to glance at this section. My focus on these lecture notes is to relate the OLS estimator to the Bayes and Minimax procedures we have discussed previously.

\newpage

%%TAKE A MORE CAREFUL LOOK AT THE PROOF BELOW. I THINK THE ARGUMENT IS SIMPLER THAN IN HANSEN,
%%BUT THE MAIN ASSUMPTION IS NOT VERY INTUITIVE.
%\begin{proposition}(Optimality of OLS among conditionally unbiased estimators) Consider the regression model in (\ref{equation:Regression-model}). Let $\widehat{\beta}$ be an estimator of $\beta$ for which
%\[ \frac{\partial}{\partial \beta} \int_{\mathbb{R}^{n}} \widehat{\beta} f(Y | \beta; X ) dY = \int_{\mathbb{R}^{n}} \widehat{\beta} \Big ( \frac{\partial}{\partial \beta} f(Y | \beta , X) \Big) dY,   \]
%at any $\beta$ in the parameter space. If $\widehat{\beta}$ is conditionally unbiased , then
%\[ \mathbb{V}_{\beta} ( \widehat{\beta} ) - \mathbb{V}_{\beta} (\widehat{\beta}_{\textrm{OLS}})  \]
%is positive semi-definite. This implies that the mean squared error of $\widehat{\beta}$ is larger than that of the OLS everywhere on the parameter space.
%\end{proposition}
%
%\begin{proof}
%We would like show  that for any $c \in \mathbb{R}^{k}$, $c \neq 0$
%\[ c’ \left( \mathbb{V}_{\beta} ( \widehat{\beta} ) - \mathbb{V}_{\beta} (\widehat{\beta}_{\textrm{OLS}}) \right) c \geq 0. \]
%
%To do this, we start by computing the variance of $c’\widehat{\beta} - c’ \widehat{\beta}_{\textrm{OLS}}$,
%which by definition of variance, has to be nonegative:
%\begin{eqnarray*}
%\mathbb{V}_{\beta} \left( c’\widehat{\beta} - c’ \widehat{\beta}_{\textrm{OLS}} \right)&=&  \mathbb{V}_{\beta} (c’\widehat{\beta}) + \mathbb{V}_{\beta} (c’\widehat{\beta}_{\textrm{OLS}}) - 2 \textrm{cov}_{\beta} ( c’\widehat{\beta} , c’ \widehat{\beta}_{\textrm{OLS}}  ),  \\
%&&  \textrm{(since $\textrm{Var}(X-Y) = \textrm{Var}(X) + \textrm{Var}(Y)- 2 \textrm{cov}(X,Y)$)}\\
%&=& c’ \mathbb{V}_{\beta}(\widehat{\beta}) c + c’ \mathbb{V}_{\beta}(\widehat{\beta}_{\textrm{OLS}}) c - 2 \mathbb{E}_{\mathbb{P}_{\beta}} \left[  c’ (\widehat{\beta}-\beta) (\widehat{\beta}_{\textrm{OLS}}-\beta)’ c \right] \\
%&& \textrm{(using the fact that both estimators are unbiased)}. \\
%&=& c’ \mathbb{V}_{\beta}(\widehat{\beta}) c + c’ \mathbb{V}_{\beta}(\widehat{\beta}_{\textrm{OLS}}) c - 2 c’ \mathbb{E}_{\mathbb{P}_{\beta}} \left[  \widehat{\beta} (\widehat{\beta}_{\textrm{OLS}}-\beta)’  \right] c.
%\end{eqnarray*}
%Where the last term is the covariance between $\widehat{\beta}$ and the OLS residual. Using the definition of OLS
%\[ \widehat{\beta}_{\textrm{OLS}}- \beta = (X’X)^{-1} X’(Y-X\beta),  \]
%and using the definition of the the Gaussian p.d.f. $f(Y | \beta,X)$ we can see verify that
%\[\widehat{\beta}_{\textrm{OLS}}- \beta =  \sigma^2  (X’X)^{-1} \frac{\partial}{\partial \beta} \ln f(Y | \beta, X ).  \]
%Consequently
%\begin{eqnarray*}
%\mathbb{E}_{\mathbb{P}_{\beta}} \left[  \widehat{\beta} (\widehat{\beta}_{\textrm{OLS}}-\beta)’ | X  \right] &=& \sigma^2 \mathbb{E}_{\mathbb{P}_{\beta}} \left[  \widehat{\beta} \left( \frac{\partial}{\partial \beta} \ln f(Y | \beta, X )\right)’ \Big | X \right]  (X’X)^{-1}, \\
%&=& \sigma^2 \left( \int_{\mathbb{R}^{n}} \widehat{\beta}  \left( \frac{\partial}{\partial \beta} \ln f(Y | \beta, X )\right)’  f(Y | \beta , X) dY \right) (X’X)^{-1} \\
%&=& \sigma^2 \left( \int_{\mathbb{R}^{n}} \widehat{\beta}  \left( \frac{\partial}{\partial \beta} f(Y | \beta, X )\right)’ dY \right) (X’X)^{-1}\\
%&& \textrm{(where we have used the chain rule and $\partial \ln x/ \partial x = 1/x$)}\\
%&=& \sigma^2 \frac{\partial }{\partial \beta }\left( \int_{\mathbb{R}^{k}} \widehat{\beta}   f(Y | \beta, X )\right)'  (X’X)^{-1}\\
%&& (\textrm{by assumption}) \\
%&=& \sigma^2 \left( \frac{\partial }{\partial \beta } \mathbb{E}_{\mathbb{P}_{\beta}}[\widehat{\beta} | X]  \right)' (X’X)^{-1}\\
%&=& \sigma^2 (X’X)^{-1}\\
%&& \textrm{(since $\widehat{\beta}$ is unbiased).} \\
%&=& \mathbb{V}_{\beta}(\widehat{\beta}_{\textrm{OLS}}).
%\end{eqnarray*}
%Hence,
%\[ 0 \leq \mathbb{V}_{\beta} \left( c’\widehat{\beta} - c’ \widehat{\beta}_{\textrm{OLS}} \right) = c’ \mathbb{V}_{\beta}(\widehat{\beta}) c - c’ \mathbb{V}_{\beta}(\widehat{\beta}_{\textrm{OLS}}) c,  \]
%for every $\beta$. The result then follows.
%\end{proof}

\noindent

\subsection{Bayesian and Minimax estimation of $\beta$}

\subsubsection{Bayes Estimation of $\beta$ under a Normal Prior}
In this subsection we analyze the Bayes estimator of the parameter $\beta$. We have already showed that any Bayes rule can be obtained by minimizing posterior loss for each data realization. Let $\pi$ denote a prior over the parameter $\beta$. In our set-up the posterior loss of an action $a$ is
\[ \mathbb{E}_{\pi} [ \: || a - \beta ||^2 \: | \: (Y,X) \: ]. \]
Using the same argument that we used to decompose the mean squared-error in terms of bias and variance we can show that
\begin{eqnarray*}
\mathbb{E}_{\pi} [ \: || a - \beta ||^2 \: | \: (Y,X) \: ] &=& \mathbb{E}[ \: || a - \mathbb{E}_{\pi} [ \: \beta \: | \:  (Y,X) \:  ] + \mathbb{E}_{\pi} [ \beta \: | \: (Y,X) ] -  \beta ||^2 \: | \: (Y,X) \: ],  \\
&=& || a - \mathbb{E}_{\pi} [ \beta \: | \: (Y,X) ] ||^2  + \textrm{tr} \left( \mathbb{V}_{\pi} ( \beta \: | \: (Y,X) ) \right).
\end{eqnarray*}
This shows that, regardless of the specific prior we pick, the Bayes estimator is the posterior mean of $\beta$
\[ \widehat{\beta}_{\textrm{Bayes}} =   \mathbb{E}_{\pi} [ \: \beta \: | \: (Y,X) \: ].  \]
Thus, reporting the posterior mean of $\beta$ is a Bayes decision rule.

{\scshape Posterior Mean under a Normal Prior:}  Consider then the following prior on $\beta$:

\begin{equation} \label{equation:prior}
\beta  \sim \pi (\beta) \equiv \mathcal{N}_{k}( \beta_0 \: , \: \sigma^2 V^{-1}).
\end{equation}

\noindent The prior assumes that all the coefficients are approximately normal with values close to the vector $\beta_0$ and covariance matrix given by $\sigma^2 V^{-1}$. There is no magical recipe to select a prior. More often than not, the selection of a prior trades-off interpretation and convenience in its implementation.

We derive the posterior distribution of $\beta$. One way of deriving this posterior distribution is by an application of Bayes Theorem
\begin{equation*}
\pi(\beta \: | \: \sigma^2, y, X ) = \frac{f(Y  \: | \: \beta, X) \pi(\beta ) }{\int_{\Theta} f(Y, \: | \: \beta, X) \pi (\beta ) d \beta}.
\end{equation*}

\noindent The posterior is thus proportional to the likelihood times the prior, both of which are Gaussian.

\noindent Consequently, $f(Y | X, \beta, \sigma^2 )\:  \pi(\beta | \sigma^2 )$ is---up to a constant that does not depend on $\beta$---proportional to
\begin{equation} \label{equation:posterior}
\exp \left( -\frac{1}{2\sigma^2} (Y-X\beta)^{\prime} (Y-X \beta) \right)  \exp \left(-\frac{1}{2 \sigma^2} (\beta-\beta_0)^{\prime} V (\beta-\beta_0) \right).
\end{equation}

\noindent The expression above equals:
\[ \exp \left( -\frac{1}{2\sigma^2} Y^{\prime} Y \right) \exp \left(  - \frac{1}{2 \sigma^2 } \beta \left(V + X’X \right) \beta + \frac{1}{\sigma^2}(Y^{\prime} X+V\beta_0) \beta \right). \]
Completing the square and ignoring all the terms that do not have $\beta$ on them, gives the posterior distribution as a constant times the exponential of:
\begin{equation*}
-\frac{1}{2 \sigma^2} \left( \beta -  \left( V + X^{\prime} X \right)^{-1} (X^{\prime} y+ V \beta_0) \right) \left(V+ X’X \right) \left(\beta -  ( V^{-1} + X^{\prime} X )^{-1} (X^{\prime} Y+ V \beta_0) \right).
\end{equation*}
This implies that:
\begin{equation}
\beta | Y, X \sim \mathcal{N}_{k} \left(  \left( V + X’X \right)^{-1} (X^{\prime} Y+V\beta_0) \: , \:   \sigma^2 \left(V + X^{\prime} X \right)^{-1}   \right).
\end{equation}
This means that the Bayesian Estimator of $\beta$ given the Gaussian prior $\pi(\beta)$ is:

\[ \widehat{\beta}_{\textrm{Bayes}} \equiv \left( V + X^{\prime} X \right)^{-1} \left( X^{\prime} Y + V \beta_0 \right). \]

\noindent The posterior mean estimator is well defined regardless the number of
covariates.\footnote{If $V$ is positive definite, then the matrix $V+ X’X$ is
  positive definite as well, and thus invertible.} The posterior mean estimator
for $V= \lambda \mathbb{I}_{k}$ and $\beta_0 = 0$ is called the \underline{Ridge estimator}:

\[  \widehat{\beta}_{\textrm{Ridge}} \equiv (X’X + \lambda \mathbb{I}_{k})^{-1} X’Y, \]

\noindent which, by construction, is admissible.\footnote{See Chapters 29.5-29.7 of Bruce Hansen's book for a definition of Ridge regression and some of its properties.}

{\scshape Posterior Mean as a Penalized/Regularized OLS estimator:} \noindent Equation (\ref{equation:posterior}) is decreasing as a function of the \emph{penalized} sum of squared residuals
\[ (y-X\beta)^{\prime} (y-X \beta) + (\beta-\beta_0)^{\prime} V (\beta-\beta_0). \]

\noindent Under the Gaussian posterior the posterior mean and posterior mode are the same. Therefore, another way of deriving the posterior mean estimator is by finding a solution to the problem:
\[ \min_{\beta} (y-X\beta)^{\prime} (y-X \beta) + (\beta-\beta_0)^{\prime} V (\beta-\beta_0) \]
The F.O.C are
\[ X’(Y-X\beta) + V(\beta - \beta_0) = \textbf{0}_{n \times k} \iff  X’Y + V \beta_0 = (X’X + V) \beta. \]
\noindent Solving for $\beta$ gives the estimator $\widehat{\beta}_{\textrm{Bayes}}$.

The Ridge estimator is biased (conditionally and unconditionally):
\begin{eqnarray*}
\mathbb{E}_{\mathbb{P}_{\beta}} [ \widehat{\beta}_{\textrm{Ridge}} | X ]  &=& (X’X + \lambda \mathbb{I}_{k})^{-1} X’X\beta,  \\
&=& (X’X + \lambda \mathbb{I}_{k})^{-1} (X’X + \lambda \mathbb{I}_{k}) \beta - \lambda (X’X + \lambda \mathbb{I}_{k})^{-1}  \beta, \\
&=& (\mathbb{I}_k-\lambda (X’X + \lambda \mathbb{I}_k)^{-1}) \beta.
\end{eqnarray*}
\noindent The magnitude of the bias depends on $\lambda$ and $\beta$.

The variance of the Ridge estimator can be computed using the law of total variance:
\[ \mathbb{V}_{\beta} (\widehat{\beta}_{\textrm{Ridge}}) =  \mathbb{V}_{f_X} \left(  \mathbb{E}_{\mathbb{P}_{\beta}} [ \widehat{\beta}_{\textrm{Ridge}} | X ]  \right)  + \mathbb{E}_{f_X}  \left[  \mathbb{V}_{\beta} (\widehat{\beta}_{\textrm{Ridge}} | X)  \right],     \]
where
\[ \mathbb{V}_{\beta} (\widehat{\beta}_{\textrm{Ridge}} | X) = \sigma^2( X’X + \lambda \mathbb{I}_k)^{-1} X’X ( X’X + \lambda \mathbb{I}_k)^{-1}.  \]

\noindent Now that we have computed the bias and variance: what can we say about the mean-squared error (MSE) of the Ridge estimator vis-\`a-vis that of OLS?

Since the Ridge estimator is admissible, it is not possible for the MSE of OLS to be smaller everywhere in the parameter space (otherwise, the Ridge estimator would not be admissible!). This means there are points in the parameter space in which the MSE of the Ridge estimator has to be below that of OLS.

This observation is true for any distribution of covariates for which the risk is well-defined. In particular, for distributions that are degenerate at some parameter value this means that variance of the Ridge estimator has to be smaller than that of the ML/OLS estimator.

Finally, it is worth mentioning that although---by construction---the Ridge estimator minimizes average risk (with respect to the prior in \eqref{equation:prior}), its worst-case risk is infinity.  To see this, it suffices to look at the expression for the bias of the Ridge estimator, and notice that if $|| \beta ||$ is unbounded, so is the bias.

\subsection{Minimax Estimation of $\beta$. }
In the context of the regression model, a minimax estimator solves the following problem.
\[ \inf_{\widehat{\beta}} \sup_{\beta} MSE(\widehat{\beta},\beta).    \]
In this subsection we will show that the OLS estimator is minimax when $k<n$. This is, we will show that:
\begin{equation} \label{equation:minimax_OLS}
\inf_{\widehat{\beta}} \sup_{\beta} \textrm{MSE}(\widehat{\beta},\beta) = \sup_{\beta} \textrm{MSE}(\widehat{\beta}_{\textrm{OLS}},\beta)
\end{equation}
The argument is constructive, and will actually use the Ridge estimator (and its risk) to establish the minimaxity of OLS. Note first that for any estimator (not only OLS):
\[\inf_{\widehat{\beta}} \sup_{\beta} \textrm{MSE}(\widehat{\beta},\beta) \leq \sup_{\beta} \textrm{MSE}(\widehat{\beta}_{\textrm{OLS}},\beta).\]
Thus, it is sufficient to show that
\[\inf_{\widehat{\beta}} \sup_{\beta} \textrm{MSE}(\widehat{\beta},\beta) \geq \sup_{\beta} \textrm{MSE}(\widehat{\beta}_{\textrm{OLS}},\beta).\]
The following inequality relating Bayes risk and minimax risk holds for any Bayesian estimator:
\[  \inf_{\widehat{\beta}} \sup_{\beta} \textrm{MSE}(\widehat{\beta},\beta)       \geq  \inf_{\widehat{\beta}} \mathbb{E}_{\pi} \left[ \textrm{MSE}(\widehat{\beta},\beta) \right].   \]
Let $\pi_{\lambda}$ denote the prior in \eqref{equation:prior} and let $\widehat{\beta}_{\lambda}$ denote the Ridge estimator corresponding to that prior. The equation in the previous display then implies that for any $\lambda$:
 \[ \inf_{\widehat{\beta}} \sup_{\beta} \textrm{MSE}(\widehat{\beta},\beta)       \geq  \mathbb{E}_{\pi_{\lambda}} \left[ \textrm{MSE}(\widehat{\beta}_{\lambda},\beta) \right]. \]
Take a sequence $\lambda_{m} \rightarrow 0$. Under any such sequence $\widehat{\beta}_{\lambda_m}$ converges (for every data realization) to $\widehat{\beta}_{\textrm{OLS}}$. It can be shown that under some regularity conditions on $f_{X}$ it then follows:
\[   \mathbb{E}_{\pi_{\lambda}} \left[  \textrm{MSE}(\widehat{\beta}_{\lambda},\beta) \right]  \rightarrow  \textrm{MSE}(\widehat{\beta}_{\textrm{OLS}},\beta).  \]

\noindent Since the MSE of $\widehat{\beta}_{\textrm{OLS}}$ is constant, then the result follows. The argument used to establish the minimaxity of OLS is standard. A textbook reference for it can be found in Theorem 1.12 of Section 5 in  \cite{lehmann1998theory}.

\subsection{Suboptimality of the OLS estimator}

Even though the OLS estimator is minimax, it is well known that if $k \geq 3$, then the estimator is dominated. The result is usually attributed to Charles Stein and his 1956 paper ``Inadmissibility of the usual estimator of the mean of a multivariate distribution'', which focuses on a multivariate normal model and not explicitly on the OLS. The estimator that dominates the usual mean is also often attributed to Willard James and Charles Stein in 1961 and referred to as the James-Stein estimator.

This section we provide a superficial presentation of the estimator that dominates OLS. For simplicity, we will consider the case in which $f_{X}$ is degenerate at a particular point.

Assume that $X’X = \mathbb{I}_n$. Start with a Bayesian estimator for $\beta$ under the normal prior $\beta \sim \mathcal{N}_{k}(0,  v \mathbb{I}_k ) = \mathcal{N}_{k}(0,  \sigma^2 (v/\sigma^2) \mathbb{I}_k ) $. We have shown that such Bayes estimator is
\begin{eqnarray*}
\widehat{\beta}_{\textrm{Bayes}} &=& \left(  \mathbb{I}_k + \frac{\sigma^2}{v} \mathbb{I}_k  \right)^{-1} \widehat{\beta} _{\textrm{OLS}}, \\
&=& \left( \frac{v}{v+\sigma^2} \right) \widehat{\beta}_{\textrm{OLS}}, \\
&=& \left(1 -  \frac{\sigma^2}{v+\sigma^2} \right) \widehat{\beta}_{\textrm{OLS}}
\end{eqnarray*}
The hyperparameter $v$ ``shrinks’’ the OLS estimator towards the prior mean. Instead of picking $v$ \emph{a priori}, it is possible to use data to estimate it. Such an approach is usually called ``empirical Bayes’’.

The distribution of the data conditional on the parameter is
\[ \widehat{\beta}_{\textrm{OLS}} \sim \mathcal{N}_{k}(\beta, \sigma^2 \mathbb{I}_k).  \]
This conditional distribution, along with the prior, specify a full joint distribution over $(\widehat{\beta}_{\textrm{OLS}}, \beta)$. The marginal distribution of the data is
\[\widehat{\beta}_{\textrm{OLS}} \sim \mathcal{N}_{k}\left ( 0 , (\sigma^2  + v)  \mathbb{I}_k  \right). \]
We can use this statistical model to estimate the ``shrinkage’’ factor that appears in the Bayes estimator. Note that
\[ ||\widehat{\beta}_{\textrm{OLS}} ||^2 \sim (\sigma^2 + v) \chi^2_{k}, \]
Therefore, standard results for the mean of the inverse of a chi-square distribution yield
\[\mathbb{E} \left[ \frac{k-2 }{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right] = \frac{1}{v+\sigma^2}, \]
provided $k \geq 3$. An unbiased estimator for the ``shrinkage’’ factor that appears in the formula of $\widehat{\beta}_{\textrm{Bayes}}$ is thus:
\[\left(1 -  \frac{\sigma^2 (k-2)}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right). \]
We now show that the ``empirical Bayes’’ estimator
\[\widehat{\beta}_{JS} \equiv \left(1 -  \frac{\sigma^2 (k-2)}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right) \widehat{\beta}_{\textrm{OLS}}, \]
dominates OLS.

\begin{proposition}
Suppose $X’X = \mathbb{I}_{k}$ and $\sigma^2=1$. If $k \geq 3$, the James-Stein estimator dominates the ML/OLS estimator.
\end{proposition}
\begin{proof}
\[ || \widehat{\beta}_{\textrm{JS}} - \widehat{\beta}_{\textrm{OLS}} ||^2  = || \widehat{\beta}_{\textrm{JS}} - \beta ||^2 + || \beta -\widehat{\beta}_{\textrm{OLS}} ||^2 + 2 ( \widehat{\beta}_{\textrm{JS}} - \beta  )’ (\beta -\widehat{\beta}_{\textrm{OLS}}) \]
implies
\[|| \widehat{\beta}_{\textrm{JS}} - \beta ||^2 = || \widehat{\beta}_{\textrm{JS}} - \widehat{\beta}_{\textrm{OLS}} ||^2 - || \beta -\widehat{\beta}_{\textrm{OLS}} ||^2 + 2 ( \widehat{\beta}_{\textrm{JS}} - \beta  )’ (\widehat{\beta}_{\textrm{OLS}}- \beta). \]
Taking expectation on both sides yields
\begin{eqnarray*}
\mathbb{E}_{\mathbb{P}_{\beta}} \left[ || \widehat{\beta}_{\textrm{JS}} - \beta ||^2 \right] & = & \mathbb{E}_{P_{\beta}} \left[  || \widehat{\beta}_{\textrm{JS}} - \widehat{\beta}_{\textrm{OLS}}||^2 \right] \\
&-& \mathbb{E}_{\mathbb{P}_{\beta}} \left[ || \widehat{\beta}_{\textrm{OLS}}- \beta ||^2 \right] \\
&+& 2 \mathbb{E}_{\mathbb{P}_{\beta}} \left[ (\widehat{\beta}_{\textrm{JS}} -\beta)’( \widehat{\beta}_{\textrm{OLS}}- \beta)  \right].
\end{eqnarray*}
Algebra shows that
\[\widehat{\beta}_{JS} - \widehat{\beta}_{\textrm{OLS}} \equiv \left(  \frac{(k-2)}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right) \widehat{\beta}_{\textrm{OLS}}. \]
Therefore,
\[  \mathbb{E}_{\mathbb{P}_{\beta}} \left[ ||\widehat{\beta}_{JS} -\beta ||^2 \right] =  \mathbb{E}_{\mathbb{P}_{\beta}}  \left[ \frac{(k-2)^2}{|| \widehat{\beta}_{\textrm{OLS}} ||^2} \right] - k + 2 \mathbb{E}_{\mathbb{P}_{\beta}} \left[ (\widehat{\beta}_{\textrm{JS}} -\beta)’( \widehat{\beta}_{\textrm{OLS}}- \beta)  \right]. . \]
Also
\begin{eqnarray*}
\mathbb{E}_{\mathbb{P}_{\beta}} \left[ (\widehat{\beta}_{\textrm{JS}} - \beta)’( \widehat{\beta}_{\textrm{OLS}}- \beta)  \right] &=& \sum_{j=1}^{k} \textrm{Cov}( \widehat{\beta}^j_{\textrm{JS}},\widehat{\beta}^j_{OLS} ).
\end{eqnarray*}
and the last term can be shown to equal
\begin{eqnarray*}
\sum_{j=1}^{k} \mathbb{E}_{\mathbb{P}_{\beta}} \left[ \frac{\partial \widehat{\beta}^j_{JS} }{\partial \widehat{\beta}^j_{\textrm{OLS}}} \right] &=& K - \mathbb{E}_{\mathbb{P}_{\beta}} \left[ \frac{k  (k-2)}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right] + \mathbb{E}_{\mathbb{P}_{\beta}} \left[ \frac{2(k-2)}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right]’ \\
&=&  K - \mathbb{E}_{\mathbb{P}_{\beta}} \left[ \frac{(k-2)^2}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right]
\end{eqnarray*}
Therefore,
\[  \mathbb{E}_{\mathbb{P}_{\beta}} \left[ ||\widehat{\beta}_{JS} -\beta ||^2 \right] = k - \mathbb{E}_{\mathbb{P}_{\beta}} \left[ \frac{(k-2)^2}{||\widehat{\beta}_{\textrm{OLS}} ||^2} \right] \leq k =\mathbb{E}_{\mathbb{P}_{\beta}} \left[ ||\widehat{\beta}_{OLS} -\beta ||^2 \right].    \]
\end{proof}

\newpage

\bibliographystyle{../AuxFiles/ecta}
\bibliography{../AuxFiles/BibMaster}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Written By Michael Brodskiy
% Class: Analytic Geometry & Calculus III (Math-292)
% Professor: V. Cherkassky
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[12pt]{article}
\usepackage{alphalph}
\usepackage[utf8]{inputenc}
\usepackage[russian,english]{babel}
\usepackage{titling}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{enumitem}
\usepackage{amssymb}
\usepackage[super]{nth}
\usepackage{everysel}
\usepackage{ragged2e}
\usepackage{geometry}
\usepackage{fancyhdr}
\geometry{top=1.0in,bottom=1.0in,left=1.0in,right=1.0in}
\newcommand{\subtitle}[1]{%
  \posttitle{%
    \par\end{center}
    \begin{center}\large#1\end{center}
    \vskip0.5em}%

}
\usepackage{hyperref}
\hypersetup{
colorlinks=true,
linkcolor=blue,
filecolor=magenta,
urlcolor=blue,
citecolor=blue,
}

\urlstyle{same}

\title{Lecture III Notes}
\date{\today}
\author{Michael Brodskiy\\ \small Professor: V. Cherkassky}

\begin{document}

\maketitle

\section{Cross Product $-$ 12.4}

\subsection{Cross Product Properties}

\begin{enumerate}

  \item $|\vec{a} \text{ x } \vec{b}| = |\vec{a}||\vec{b}|sin(\theta)$

    \begin{enumerate}

      \item The cross product of two parallel vectors is 0

    \end{enumerate}

\end{enumerate}

\section{Lines \& Planes $-$ 12.5}

Points on a line may be found using the formula $\vec{r} = \vec{r_o} + t\vec{v}$, where $t$ is any value from $-\infty \text{ to } \infty$, $v$ is a direction vector, found from two given points, and $r_o$ is a vector with values from a point itself (say you are given point $P = (1, 3 -2)$, $\vec{r_o}$ would be $<1, 3, -2>)$.\\

Using this formula, one would find the parametric vector equations of a line.

\subsection{Line Formula}

\begin{enumerate}

  \item $t=\frac{x-x_o}{a}=\frac{y-y_o}{b}=\frac{z-z_o}{c}$

\end{enumerate}

For a segment, the formula is $\vec{r} = (1-t)\vec{r_o} + t\vec{r_1}$, where $0<t<1$\\

For two segments, it can be determined whether there is skew by dividing each respective component. If the values are all equal, the lines are parallel. If not, they may either intersect or have no correlation (skew). It can be determined whether the segments intersect by solving the system of equations and finding a solution.\\

To find the vector equation of a plane, one needs two vectors, $\vec{r}$ and $\vec{r_o}$, and a vector orthogonal to the plane, $\vec{n}$. To verify this, one may use the equation $\vec{n}\vec{r}=\vec{n}\vec{r_o}$. The equation of the line itself is equal to $ax+by+cz=d$, where $a, b, c, \text{ and } d$ are constants. This equation may be rewritten as $z=Ax+ By + C$, where $A= -\frac{a}{c}, B= -\frac{b}{c}, \text{ and } C= -\frac{d}{c}$

\end{document}


\SecDef{cycles-invariants}{Proving Absence of Low Degree Invariants}

Recently, a nonlinear invariant attack was introduced by Todo \etal{} in \cite{NonlinInv}. They show that, for any SPN-based cipher, if there exists a quadratic invariant for the S-Box and  the linear layer is orthogonal, then it is possible to construct a nonlinear invariant for the full round of the cipher. Together with Christof Beierle and Alex Biryukov, we studied and generalized such linear layers in~\cite{OurNLI}. I describe the results in \ChapRef{linear}. In this chapter, I keep the focus on invariants of nonlinear functions.

In \cite{NonlinInv} it was noted that all nonlinear invariants of an S-Box can be obtained from its cycle decomposition. Indeed, an invariant must take a constant value on each cycle. If an S-Box $S$ has $t$ cycles, then there are precisely $2^t$ invariants corresponding to $2^t$ possible assignments of the invariant values to all $t$ cycles. A special kind of invariants was considered in~\cite{NonlinInv}, where the output value of the invariant is allowed to be a negation of the input value., i.e. a function $g$ such that $g(S(x)) = g(x) \oplus 1$. This case is only possible if all cycles of $S$ have even length.

Quadratic invariants are interesting because they are preserved by an orthogonal linear layer. However, one can argue that any low-degree invariant is an interesting property, even if no orthogonal layer is used in the analyzed cipher. First, the algebraic degree can be seen as a measure of simplicity. Second, invariants may be used to generate equation systems of a cipher, and low-degree equations are generally easier to solve. Third, low-degree invariants are also an evidence that the analyzed component is non-ideal.

\subsection{Low Degree Invariants from Cycle Decomposition}

\subsubsection{A naive approach}
Let $S$ be a permutation of $\field{n}$ with $t$ cycles. The most straightforward approach of finding all invariants $g\colon \field{n} \to \field{}$ of $S$ of degree at most $d$ is to generate all $2^t$ invariants and check their algebraic degree. Generating the ANF of an $n$-bit Boolean function requires $n2^n$ operations. As a result, the complexity of this approach is $n2^{t+n}$. It is possible to compute a single ANF coefficient of a monomial degree $d+1$ in $2^{d+1}$ evaluations of the function. If the coefficient is equal to one, then the considered invariant has degree at least $d+1$ and can be excluded. Otherwise, other monomials must be checked. In the best case (when the first chosen coefficient is equal to 1), the complexity is $2^{t+d+1}$.

\subsubsection{An improved approach}
The method can be improved by replacing the enumeration of all $2^t$ invariants by solving a linear algebra problem. Consider an affine subspace $A$ of dimension $d+1$. Any possible invariant $g$ of degree at most $d$ must sum to zero over this subspace:
$$
\bigoplus_{x \in A} g(x) = 0.
$$
As $g$ must be constant on each cycle of $S$, $g$ can be defined by $t$ unknowns - values of the invariant on each of the cycles. Let $c_1, \ldots, c_t \subseteq \field{n}$ be cycles of $S$. Denote by $g_1, \ldots, g_t \in \field{}$ the unknowns, where $g_i$ corresponds to the value of $g$ on all elements from $c_i$. It follows that the sum of $g$ over $A$ can be expressed as a linear combination of $g_i$:
\eq{
&\bigoplus_{x \in A} g(x) = \bigoplus_{1 \le i \le t} \lambda_i g_i = 0,~\text{where} \\
&\lambda_i = \pabs{A \cap c_i} \mod 2.
}

The improved approach is to generate enough linear equations on the unknowns $g_i$ and solve the system.

\paragraph{Complexity.}
Generating one equation requires $2^{d+1}$ operations, assuming that the map $x \mapsto (i: x \in c_i)$ is efficient. This assumption can be implemented by a precomputation, which requires $\OO(2^n)$ memory. This is a downside of this approach.
$\OO(t)$ equations are needed, and can be generated in time $\OO(t2^{d+1})$. Solving the linear equation system requires $\OO(t^3)$. Thus, the final time complexity is $\OO(t2^{d+1} + t^3)$. In particular, all degrees (lower bounds) of invariants can be computed in time $\OO(t2^n + t^3n)$ using memory $\OO(2^n)$. On practice, the algorithm is reasonably efficient for S-Boxes of sizes up to 32 bits and with a non-extremely large number of cycles. A parallelization is possible but requires a significant amount of memory.

\begin{remark}
Generating equations requires special care. If we choose a random affine subspace of dimension $d$, it is likely that only large cycles will be covered by the generated equation and we obtain no constraints on small cycles. Therefore, when generating an affine subspace, we ensure that multiple elements from the subspace lie in distinct cycles. This can be done by choosing basis vectors accordingly. For example, we can choose a cycle uniformly at random and then choose an element of the cycle uniformly at random.
\end{remark}

\begin{remark}
The algorithm has a one-sided error possibility. If the generated linear system has no solutions (besides the constant ones $g=0$ and $g=1$), then surely there are no invariants of degree at most $d$. In this way, the algorithm provides a tool for \emph{provable} security against low-degree invariants.

However, a solution to the system is not guaranteed to have degree at most $d$, only that it sums to 0 on generated affine spaces of dimension $d+1$. I argue that we do not restrict the affine subspaces to be cosets of cubes, and an unrestricted affine subspace corresponds to a monomial coefficient in a random basis. In this way, even an invariant with a sparse ANF can be disproved to have a low degree with high probability, since in a random basis it may have a non-sparse ANF.
\end{remark}

The pseudo-code of the proposed algorithm is given in \AlgRef{cycle-invariants}.

\begin{algorithm}
    \AlgDef{cycle-invariants}{Low-Degree Invariants from Cycle Decomposition.}
    \begin{algorithmic}[1]
    \Require partition $c_1,\ldots,c_t \subseteq \field{n}$ of $\field{n}$ corresp. to the cycles of $S\colon \field{n} \to \field{n}$;
        \Statex~~~~~ an integer $d, 1 \le d \le n-1$.
    \Ensure an invariant $g\colon \field{n} \to \field{}$ of $S$ with possibility that $\deg{g} \le d$;
        \Statex~~~~~~~ or \textsf{No invariants with $\deg{g} \le d$}.

    % \Statex
    \State $E \gets$ an empty equation system
    \For{$i \in \{0,\ldots,t+\epsilon\}$}
        \State $u \gets$ a random element from $c_{1+(i\mod t)}$
        \State $V \gets$ a random linear subspace of $\field{n}$ of dimension $d+1$
        \Statex \hspace{0.55cm}such that several $v \in V$ belong to distinct cycles.
        \State $A \gets u \oplus V$
        \For{$j \in \{1,\ldots,t\}$}
            \State $\lambda_j \gets 0$
        \EndFor
        \For{$x \in A$}
            \State $\lambda_k \gets \lambda_k \oplus 1$, where $k$ is such that $x \in c_k$
        \EndFor
        \State $E \gets E \cup (\bigoplus_{1 \le k \le t} \lambda_k g_k = 0)$
    \EndFor
    \If{$E$ has non-trivial solutions}
        \State $(g_1,\ldots,g_t) \gets$ a non-trivial solution of $E$
        \State $g \gets \pround{x \mapsto (g_i~\text{such that}~x \in c_i)}$
        \State \Return $g$
    \Else
        \State \Return \textsf{No invariants with $\deg{g} \le d$}
    \EndIf
    \end{algorithmic}
\end{algorithm}

\subsection{Cycle Decomposition of G from NORX8}

States consisting of four equal columns lie on the cycles of $F_{col}$ and $F_{diag}$ functions that correspond directly to cycles of the $G$ function. Indeed, such states always consist of four copies of a single column and applying $F_{col}$ or $F_{diag}$ to such state is equivalent to applying $G$ to the corresponding column. For instance, it is possible to enumerate all cycles of the $G$ function for NORX8, where $G$ permutes $(\field{8})^4$. All these cycles of $G$ can be transformed into cycles of $F_{col}$ or $F_{diag}$ by simply making 4 copies of the column. These cycles then are also cycles of $F$, except that all even cycles will split into two cycles each, because we need to consider cycles of $G^2$. I provide the cycle decomposition of $G\colon (\field{8})^4 \to (\field{8})^4$ from NORX8 in \TabRef{cycles} by providing starting points and lengths of the cycles.

\begin{table}[ht]
\TabDef{cycles}{Cycles of $G$ from NORX8. Starting points are of the form $(a,b,c,d) \in (\field{8})^4$ (see \FigRef{G}).}
\begin{center}
\setlength{\tabcolsep}{.2cm}
\begin{tabular}{c|lcc|l}

Starting point & Cycle length & ~ & Starting point & Cycle length \\

\midrule

$(\hex{00},\hex{00},\hex{00},\hex{01})$ & 3294443807 & ~ & $(\hex{00},\hex{00},\hex{2b},\hex{65})$ & 399843 \\
$(\hex{00},\hex{00},\hex{00},\hex{08})$ & 621984749 & ~ & $(\hex{00},\hex{02},\hex{1c},\hex{06})$ & 52972 \\
$(\hex{00},\hex{00},\hex{00},\hex{56})$ & 212798071 & ~ & $(\hex{00},\hex{00},\hex{5c},\hex{28})$ & 23344 \\
$(\hex{00},\hex{00},\hex{00},\hex{07})$ & 56236016 & ~ & $(\hex{00},\hex{00},\hex{00},\hex{d5})$ & 8301 \\
$(\hex{00},\hex{00},\hex{00},\hex{06})$ & 55712043 & ~ & $(\hex{00},\hex{00},\hex{b8},\hex{d2})$ & 6339 \\
$(\hex{00},\hex{00},\hex{00},\hex{02})$ & 21461014 & ~ & $(\hex{00},\hex{05},\hex{94},\hex{d3})$ & 2124 \\
$(\hex{00},\hex{00},\hex{02},\hex{29})$ & 9062510 & ~ & $(\hex{01},\hex{66},\hex{26},\hex{d2})$ & 848 \\
$(\hex{00},\hex{00},\hex{04},\hex{52})$ & 7374122 & ~ & $(\hex{00},\hex{4e},\hex{63},\hex{c1})$ & 595 \\
$(\hex{00},\hex{00},\hex{00},\hex{46})$ & 7328319 & ~ & $(\hex{00},\hex{9d},\hex{2b},\hex{c3})$ & 137 \\
$(\hex{00},\hex{00},\hex{08},\hex{4e})$ & 5608893 & ~ & $(\hex{03},\hex{4f},\hex{69},\hex{6c})$ & 78 \\
$(\hex{00},\hex{00},\hex{01},\hex{f7})$ & 2463170 & ~ & $(\hex{00},\hex{00},\hex{00},\hex{00})$ & 1 \\

\end{tabular}
\end{center}
\end{table}

\subsection{Low-Degree Invariants in G from NORX8}

I applied the proposed approach to the function $G$ from NORX8 permuting $\field{32}$. In the previous section I described the cycle decomposition of $G$. There are 22 cycles and thus there are $2^{22}$ invariants of $G$. Note that there are cycles with odd length, therefore there exist no ``switching'' invariants of $G$.

The algorithm used around 36 gigabytes of memory and ran for 25 hours on a single 3.5GHz core. It generated 100 linear equations over 22 unknowns for each subspace dimension in $\seg{1}{32}$. Solving the systems took negligible time.

The results are as follows. For all dimensions $d \le 31$, there are no non-trivial invariants of degree $d-1$. For the dimension $d = 32$, there a space of dimension 21 of invariants of degree $d-1=31$. It follows that the other coset of this space has invariants of degree 32. That is, a half of the invariants have degree $32$ and the other half are of degree 31. Note that these are the maximum possible degrees of invariants, since any pair of invariants of degree 32 must sum to an invariant of lower degree.

Furthermore, I searched for low-degree invariants of iterated $G$, i.e. $G^l$ for $1 \le l \le 16$ and for particular values of $l$ that lead to an increased number of cycles. Each cycle of $G$ of length $c$ splits into $gcd(l,c)$ cycles of $G^l$ of lengths $c/gcd(l,c)$. Thus, the number of cycles grows and the number of invariants too. For example, $G^8$ has 54 cycles and thus, $2^{54}$ invariants. The results show that for $1 \le l \le 16$ and for $l \in \pset{24,30,32,36,51,59}$, all non-trivial invariants of $G^l$ have degree at least 30. More detailed results are given in \TabRef{iter-invariants}. Evaluation of $G^l$ for a single $l$ took the time proportional to the number of cycles in $G^l$.

\begin{table}[ht]
    \centering
    \begin{tabular}{c|cccc}
    \toprule

    function & $d\le{29}$ & $d\le{30}$ & $d\le{31}$ & $d\le{32}=\#cycles$\\

    \midrule

$G^{1}$ & 1 & 1 & 21 & 22 \\
$G^{2}$ & 1 & 2 & 31 & 32 \\
$G^{3}$ & 1 & 5 & 37 & 38 \\
$G^{4}$ & 1 & 9 & 41 & 42 \\
$G^{5}$ & 1 & 2 & 33 & 34 \\
$G^{6}$ & 1 & 19 & 51 & 52 \\
$G^{7}$ & 1 & 2 & 33 & 34 \\
$G^{8}$ & 1 & 21 & 53 & 54 \\
$G^{9}$ & 1 & 23 & 55 & 56 \\
$G^{10}$ & 1 & 19 & 51 & 52 \\
$G^{11}$ & 1 & 1 & 21 & 22 \\
$G^{12}$ & 1 & 33 & 65 & 66 \\
$G^{13}$ & 1 & 2 & 33 & 34 \\
$G^{14}$ & 1 & 17 & 49 & 50 \\
$G^{15}$ & 1 & 17 & 49 & 50 \\
$G^{16}$ & 1 & 45 & 77 & 78 \\
$G^{24}$ & 1 & 45 & 77 & 78 \\
$G^{30}$ & 1 & 39 & 71 & 72 \\
$G^{32}$ & 1 & 45 & 77 & 78 \\
$G^{36}$ & 1 & 69 & 101 & 102 \\
$G^{51}$ & 1 & 85 & 117 & 118 \\
$G^{59}$ & 1 & 163 & 195 & 196 \\

    \bottomrule
    \end{tabular}
    \TabDef{iter-invariants}{Upper-bounds on dimensions of the spaces of the invariants of degree at most $d$ of the function $G$ from NORX8 iterated multiple times.}
\end{table}

I also verified correctness of the algorithm on a toy S-Box permutation of $\field{16}$ with an invariant of degree $11$. The algorithm successfully recovered this invariant.

I conclude that invariants of the mapping $G\colon (\field{8})^4 \to (\field{8})^4$ from NORX8 have maximum degree. Furthermore, invariants of $G$ iterated for several rounds have also close to maximum degree. I remark that this observation does not rule out a possibly simple structure or property of those invariants. Indeed, the invariant subspace of dimension $2^{8w}$ shown in \SecRef{symmetries} corresponds to an invariant function of quite large degree $8w$. On the other hand, a low-degree invariant could correspond to a bit-level property of $G$, whereas the invariant subspace from \SecRef{symmetries} corresponds to a high-level structural property of the NORX's permutation.


\documentclass[letterpaper]{article}

\usepackage[utf8]{luainputenc}
\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Notes}
\date{4 mars, 2015}
\maketitle
\begin{align*}
  \log x\\
  \sum\limits_{k=1}^\infty{\frac{(-1)^{k+1}(x-1)^k}{k}}
\end{align*}
radius of convergence is 1 and interval of convergence is $(0,2]$ from last time

in general the $n$th derivative looks like $f^{(n)}=\frac{(-1)^{n+1}(n-1)!}{x^n}$.

we restrict our $[a,b]$ such that $0<a\le b$.

now $|f^{(n_1)}(x)|=|f^{(n+1)}(a)|=\frac{(n)!}{a^n}$

now with taylors thm
$|R_n(x)|\le \frac{n!}{a^n}(\frac{|x-1|^n}{(n+1)!}$

take the limit of n to inf and
\begin{align*}
  \lim |R_n|&=\lim \frac{|x-1|^n}{a^nn}\to 0
\end{align*}

and so $\log x=\sum\limits_{k=1}^\infty{\frac{(-1)^{k+1}}{k!}(x-1)^k}$

\subsection*{example}
$f(x)=\begin{cases}e^{1/x^2}&x\ne 0\\0&x=0\end{cases}$

\begin{align*}
  f^{(n)}(x)&=\frac{q_n(x)}{x^{3n}}e^{-1/x^2}&x&\ne 0
\end{align*}
with $q_n(x)$ a polynomial of degree less than or equal to 2n
first derivative at zero is zero because $\lim_{x\to 0}\frac{f(x)-0}{x-0}=\frac{e^{-1/x^2}}{x}$ which is zero with l'hopitals rule. same with inductive step fr $n$th degree. but then the power series at 0 converges to $P_n(x)=0$!

\section*{can we approximate a continuous function by a polynomial?}

yes, but....

not usually with power series.

\section*{thm}
if $f$ is continuous on $[a,b]$ then for any $\epsilon>0$ there is a polynomial $p_\epsilon$ such that $||f-p_\epsilon||_\infty<\epsilon$

the book does three proofs of this. bernstein, chebychev, and a general proof. stone-weierstrass thm.

\subsection*{corollary}
if $f$ is continuous on $[0,1]$ and $\int_0^1f(x)x^n\;\mathrm{d}x$ for all $n$ theen $f(x)=0$.

\subsubsection*{proof}
$\int_0^1{|f(x)|^2\;\mathrm{d}x}=\lim\int_0^1f(x)p_n(x)\;\mathrm{d}x$ with $p_n(x)$ being the polynomial approximation from weierstrass approximation theorem.

\begin{align*}
  \lim a_n\int f(x)x^n+a_{n-1}\int\dots+a_0\int f(x)x^0\;\mathrm{d}x=0
\end{align*}
\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}

\title{MAT257 Notes}
\author{Jad Elkhaleq Ghalayini}
\date{November 14 2018}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{mathtools}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{cancel}

\usepackage[margin=1in]{geometry}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{definition}{Definition}
\newtheorem*{corollary}{Corollary}
\newtheorem{exercise}{Exercise}
\newtheorem{claim}{Claim}

\DeclareMathOperator{\Int}{Int}
\DeclareMathOperator{\grad}{grad}
\DeclareMathOperator{\Ker}{Ker}
\DeclareMathOperator{\Ima}{Im}

\newcommand{\reals}[0]{\mathbb{R}}
\newcommand{\nats}[0]{\mathbb{N}}
\newcommand{\ints}[0]{\mathbb{Z}}
\newcommand{\rationals}[0]{\mathbb{Q}}
\newcommand{\brac}[1]{\left(#1\right)}
\newcommand{\sbrac}[1]{\left[#1\right]}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\eval}[3]{\left.#3\right|_{#1}^{#2}}
\newcommand{\ip}[2]{\left\langle#1,#2\right\rangle}
\newcommand{\prt}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\mb}[1]{\mathbf{#1}}

\begin{document}

\maketitle

\section{Manifolds}

Consider the following definitions of a manifold:
\begin{definition}
  A \(k\)-dimensional manifold manifold is a set \(M \subseteq \reals^n\) satisfying the following equivalent conditions
  \begin{enumerate}

    \item For all \(a \in M\), there is a \(\mc{C}^r\) mapping \(f: U \to \reals^{n - k}\) defined on an open neighborhood \(U\) of \(a\) such that \(M \cap U = f^{-1}(0)\). \(f\) has rank \(n - k\) at every \(x \in M \cap U\).

    \item Every point of \(M\) has an open neighborhood \(U\) in \(\reals^n\) such that \(M \cap U\) is the graph of a \(\mc{C}^r\) function \(z = g(y)\), \(z = (z_1,...,z_{n - k}), y = (y_1,...,y_k)\) where
    \[(y, z) \neq (x_{\sigma(1)},...,x_{\sigma(n)})\]
    where \(\sigma\) is a permutation of \(1,...,n\).

    \item For all \(a \in M\), there is an open neighborhood \(U\) of \(a\), an open set \(V \subset \reals^n\) and a \(\mc{C}^r\) \underline{diffeomorphism} (a \(\mc{C}^r\) mapping with a \(\mc{C}^r\) inverse) \(h: U \to V\) such that
    \[h(M \cap U) = V \cap (\reals^k \times \{\mb{0} \in \reals^{n - k}\}) =
    \{y = (y_1,...,y_n) \in V : y_{k + 1} = ... = y_n\}\]

  \end{enumerate}
\end{definition}
\begin{claim}
  The definitions given for a manifold are equivalent
\end{claim}
\begin{proof}
\begin{itemize}

  \item 1 \(\implies\) 2: By the implicit function theorem

  \item 2 \(\implies\) 1: Let \(h(y, z) = z - g(y)\).

  \item 2 \(\implies\) 3: Let \(h(y, z) = (y, z - g(y))\)

  \item 3 \(\implies\) 1: Let \(f(x) = (h_{k + 1}(x),...,h_n(x))\)

\end{itemize}
\end{proof}

To illustrate the first definition in action, consider the example of the unit sphere \(S^{n - 1} \subset \reals^n\), given by
\[S^{n - 1} = \{x = (x_1,...,x_n) : f(x) - 1 = 0)\} \text{ where } f(x) = x_1^2 + ... + x_n^2\]
We have
\[\forall x \in S^{n - 1}, \grad f(x) = 2(x_1,...,x_n) = 2x \neq 0\]
so \(f\) has rank 1 at every point of \(S^{n - 1}\). Hence \(S^{n - 1}\) is a manifold.

\subsection{Tangent Spaces}

We now discuss the tangent space \(TM_a\) or \(M_a\) at a point \(a \in M\). If we have a function \(f: \reals^n \to \reals^p\), then \(Df(a)\) is a linear transformation, a function from \(\reals^n\) to \(\reals^p\). But we're going to consider it as a function from \(\reals^n_a\), the copy of \(\reals^n\) at \(a\), to \(\reals^p_{f(a)}\), the copy of \(\reals^p\) at \(f(a)\). To be clear, these copies of \(\reals^n, \reals^p\) are vector spaces we attach to the points \(a, f(a)\), with these points identified with the origin.

\begin{enumerate}

  \item The definition of the tangent space according to the first definition is \(\Ker Df(a) = Df(a)^{-1}(0)\), a vector subspace of \(\reals^n_a\)

  \item Write \(x = (y, z), a = (b, c)\). Then we can think of the tangent space as an affine space through the point \((b, c)\) by writing
  \[z - c = Dg(b)(y - b)\]
  But we want to think of it as a vector space with \(a\) as the origin, a vector subspace of \(\reals^n_a\). So we should write it as
  \[Z = Dg(b)(Y)\]
  where \(Z \in \reals_{f(a)}^p\), \(Y \in \reals_a^n\). This is because \(f(y, g(y)) = 0\), so
  \[Df(a)\begin{pmatrix} I \\ Dg(b) \end{pmatrix} = 0 \implies \Ker Df(a) \supseteq \Ima\begin{pmatrix} I \\ Dg(b) \end{pmatrix}\]
  But in fact, they're equal, because their dimensions are the same.

  So the kernel of \(Df(a)\), and the image of \(\begin{pmatrix} I \\ Dg(b) \end{pmatrix}\), are both equal to the set of points such that
  \[Z = Dg(b)(Y)\]

  \item In terms of the third definition, we can write the tangent space as
  \[Dh(a)^{-1}(\reals^k \times \{\mb{0} \in \reals^{n - k}\})\]

\end{enumerate}

\subsection{Definition by Coordinate Charts}

Let's try defining a (\(k\)-dimensional) manifold in another way: we want every point \(a \in M\) to have a ``neighborhood'' around it in \(M\) which ``looks like'' \(\reals^k\).

A \(k\)-dimensional manifold \(M \subset \reals^n\) is a set such that for all \(a \in M\) there is an open neighborhood \(U\) of \(a\) (in \(\reals^n\), \textbf{not} in \(M\)) and an open set \(W \subset \reals^k\) and a \(\mc{C}^r\) mapping \(\varphi: W \to \reals^n\) such that
\begin{itemize}
  \item \(\varphi\) is a bijection
  \item \(\varphi(W) = M \cap U\)
  \item \(\varphi\) has rank \(k\) at every point of \(W\), i.e. \(D\varphi(x)\) has rank \(k\) for every \(x \in W\).
  \item We'd like to say ``\(\varphi^{-1}: \varphi(W) = M \cap U \to W\)'' is continuous. But we don't know how to say it, because we haven't given \(M\) a topology. So we can't talk about a function defined on a piece of \(M\) being continuous. So the way we're going to say it is we're going to build the definition of the topology into the phrase that we'll say instead of this.

  We require instead that
  \((\varphi^{-1})^{-1}(\Omega)\), i.e. \(\varphi(\Omega)\), is \(\varphi(W)\) intersected with an open set of \(\reals^n\) for all \(\Omega\) open in \(W\).
\end{itemize}
Note this first and last condition on \(\varphi\) are equivalent to requiring that \(\varphi\) is a homeomorphism between the neighborhood \(M \cap U\) and \(W\).

\end{document}


\subsection{Plant Characterisation}
\label{sec:sim:characterisation}

\begin{figure}[b!]
    \centering
    \includegraphics[width=\imagewidth]{images/characterisation}
    \caption{Calculated tangent of measured step response data.}
    \label{fig:characterisation}
\end{figure}

We decided  to  use the step response conducted in the fairly linear region of
\SI{2}{\volt}-\SI{10}{\volt} for further processing.

Using  MATLAB, the measured step  response  is  smoothed,  the  derivative  is
computed  to find the point of inflection, and the tangent is calculated.  The
results   of  this  are  visualised  in   figure   \ref{fig:characterisation}.

Regarding the parameter $K_s$, a common pitfall is to  say  that  $K_s$ is the
total  change in the step response. This  statement  is  in  fact  false,  but
happens to yield the correct value as long as the input step  function  has an
amplitude of $1$, which unfortunately is the only case studied in the majority
of theory lectures. The parameter $K_s$ is in fact the total change in  output
divided by the total change in input:

\begin{equation}
    K_s = \frac{\Delta K_{out}}{\Delta K_{in}}
\end{equation}

The calculated values of $T_u$, $T_g$ and $K_s$ turn out to be:

\begin{align*}
    T_u &= 0.1655 \\
    T_g &= 5.9602 \\
    K_s &= 197.42
\end{align*}