Methods for Bayesian inference
often have a tradeoff between
efficiency and generality.
%Consider sampling algorithms:
%on one end of the spectrum we have
%Metropolis-Hastings
%which often scales poorly,
%but is extremely flexible, applying
%to almost any model.
%On the other end,
%we have Gibbs sampling
%which is more efficient, but
%is restricted to conjugate models.
In variational inference,
general methods like black-box variational inference \citep{Ranganath2014}
work in principle on any nonconjugate model,
but are less efficient
than specialized methods on certain models,
such as the variational autoencoder \citep[VAE; ][]{Kingma2014, Rezende2014}.
Specialized
methods have the ability to
exploit structure in a model
that lends itself to efficient computation,
whereas general algorithms are often
unaware of this structure and lose out on the potential
benefits.

The hope is to design an inference algorithm
that is both efficient and general,
and approaches to this problem have typically come in one of two forms.
The first is to make a general algorithm
aware of model-specific structure,
and to leverage it whenever possible.
The second is to take a specialized
algorithm and broaden its scope
by addressing its assumptions.
In this chapter, we take the latter approach,
taking an algorithm
that is designed for conjugate models
and addressing how it fails in nonconjugate models.
Our resulting algorithm combines two powerful ideas
in approximate inference: variational message
passing \citep[VMP; ][]{Winn2005} and recognition networks.

%Variational inference for nonconjugate
%graphical models is difficult, but important.
%Many powerful nonconjugate probabilistic models such as
%non-linear dynamical
%systems, deep exponential families,
%and the variational autoencoder
%have exploded in popularity
%because of efficient and scalable inference algorithms.
%Designing such algorithms is tricky however,
%as the desire for an algorithm's generality can detract
%from its efficiency.

Conjugacy in models naturally lends itself
to efficient inference algorithms.
In many simple conjugate models,
we can compute the posterior distribution analytically.
But even in the general case, conjugacy is important
for efficient approximate inference.
This is best observed in mean-field variational inference
for graphical models, also called variational message passing.
VMP is a simple, coordinate-ascent optimization of the
variational lower bound,
but its main drawback is its restriction
to conjugate-exponential models.

In many nonconjugate models,
however, efficient inference algorithms still exist.
Consider the VAE
and its structured variant \citep[SVAE; ][]{Johnson2016}.
Both of these models
are highly nonconjugate, due to the inclusion
of a neural network observation model.
However, they address the nonconjugacy
by using neural networks, called recognition networks,
to help learn previously intractable posterior distributions.
Although VAE and SVAE utilize these networks
in subtly different ways, their purpose is the same: using
neural networks as function approximators
to make a previously intractable optimization feasible.

We thus propose neural variational message passing (NVMP),
a framework for nonconjugate variational inference
that generalizes use of recognition networks beyond
VAE and SVAE to a larger class
of nonconjugate models.
As the name suggests,
NVMP is an extension of the
VMP framework, and while NVMP
generalizes VMP and SVAE,
we still retain the modularity and
simplicity of VMP.
We accomplish this by
isolating the failure modes of VMP in nonconjugate models
and replacing previously intractable
computations with approximations
from recognition networks.

The chapter is structured as follows. In \autoref{sec:background},
we derive the VMP algorithm, starting
with exponential families and conjugacy.
In \autoref{sec:nvmp},
we identify where VMP fails in nonconjugate
models and introduce the NVMP framework,
where we patch VMP's issues
with recognition networks.
We also provide
modifications to the NVMP framework
that integrate some newer ideas
in variational inference.
Finally, in \autoref{sec:examples},
we provide several examples of how NVMP can
be easily applied to nonconjugate models, such
as sparse Bayesian linear regression
and a nonlinear dynamical system.

%VMP's original goal of modular
%inference
%in conjugate models lended itself to an intuitive software
%implementation, wherein models are specified
%graphically and inference is performed
%automatically afterwards.
%Likewise, NVMP enables a practical software
%implementation, where models need only be specified
%as one would write down any generative model.
%Any conjugate structure
%in the model is automatically exploited via VMP,
%and nonconjugacy is approximated
%by user-supplied recognition networks.

\subsection{Related Work}
The relevant works that inform this chapter are
\citet[VMP; ][]{Winn2005}, \citet[VAE; ][]{Kingma2014}, and \citet[SVAE; ][]{Johnson2016}.
VMP is a framework for variational inference
in CE models, but does
address nonconjugacy in some select situations,
relying on the Jaakola-Jordan trick \citep{Jaakkola1997}
and Monte-Carlo approximation.
In this work, we provide a more general way of extending VMP
to nonconjugate models, based on ideas
from VAE and SVAE. VAE and SVAE,
on the other hand, are models
where nonconjugacy is isolated to the observation model,
whereas in NVMP, we are not limited to this class of models.
Also related is \citet{Knowles2011},
which is also a generalization of VMP.
It addresses nonconjugacy in VMP
by introducing a conjugate approximation
to the ELBO
that has the same gradients.
We also use a separate, tractable approximation
to the ELBO but rely on neural network
function approximation rather than matching gradients.

Several other works also
isolate the conjugacy in a model,
and address the nonconjugate component
by various means.
\citet{Khan2015} utilizes VMP
in conjugate components of the model, but stochastic
gradients in the nonconjugate components. This can in fact
be interpreted as a generalization of \citet{Knowles2011}
but differs from NVMP in our use of recognition networks.
Perhaps the most similar work to ours
is \citet{Lin2018}, which also utilizes recognition networks
alongside VMP.
The generalization of SVAE and VMP they provide is
is different from ours, however.
\citet{Lin2018} uses recognition networks
with integrated graphical model structure.
whereas
we utilize recognition networks as local
function approximators.
Furthermore recognition
networks in \citet{Lin2018} output model parameters,
amortizing inference in the process, in contrast with
NVMP recognition networks, which output potentials.

\section{Background}
\label{sec:background}

\subsection{Exponential families and conjugacy}

\begin{definition}
A probability distribution $\p(\x \given \theta)$ is in the \textbf{exponential family}
if it can be written as
\begin{equation}
    \log \p(\x \given \theta) = \left\langle t_\x(\x),\eta_\x(\theta)\right\rangle - \log Z_\x(\eta_\x(\theta))
\end{equation}
where
\begin{enumerate}
    \item $\eta_\x(\theta)$ is the \emph{natural parameter} function
    \item $t_\x(\x)$ is the \emph{sufficient statistic} function
    \item $\log Z_\x(\eta_\x(\theta))$ is the \emph{log partition} function
\end{enumerate}
\end{definition}

For a pair of random variables $\x$ and $\z$ and their
joint distribution $\p(\x, \z) = \p(\x \given \z)p(\z)$, we often say
they are conjugate if the posterior $\p(\z|\x)$ is in the same
family as the prior $\p(\z)$. This definition
is useful when considering pairs of distributions,
but we aim to define conjugacy in a more global sense.

\begin{definition}
\label{def:conjugacy}
Consider a probabilistic model over several variables $\p(\x_1, \ldots, \x_\numdata) = \prod_{i = \n}^\numdata p(\x_\n | \parents_{\x_\n})$ where $\parents_{\x_\n}$ are the parents of $\x_\n$.
We assume that each of these likelihoods $\p(\x_\n | \parents_{\x_\n})$
is exponential family with sufficient statistics $t_{\x_\n}(\x_\n)$.
We define a model to be \textbf{conjugate-exponential} (CE)
if for each variable $\x_\n$ we can rewrite
the log-joint distribution as
\begin{equation}\label{eq:natparam}
    \log \p(\x_1, \ldots, \x_\numdata) \propto \left\langle t_{\x_\n}(\x_\n), \eta^*_{\x_\n}(\backslash \x_\n)\right\rangle
\end{equation}
where $\not{\x_\n}$ corresponds to all variables but $\x_\n$.
All occurrences of $\x_\n$ must be in $t_{\x_\n}(\x_\n)$ and the
joint distribution natural parameter $\eta^*_{\x_\n}(\not{\x_\n})$
contains all terms that are multiplied by the sufficient statistics of $\x_\n$.
\end{definition}
The previous, pairwise notion of conjugacy is captured by this definition
by virtue of the fact that if two distributions
have matching sufficient statistics function, they are of the same family.

We now outline two properties of CE models
that will be important for VMP later on.
Consider a variable in a CE model $\x_\n$ with
parents $\parents_{\x_\n}$, children $\children_{\x_\n}$
and likelihood $\p(\x_\n | \parents_{\x_\n})$.
\begin{property}
\label{prop:multilinear}
The natural parameter of $\x_\n$, $\eta_{\x_\n}(\parents_{\x_\n})$
can be expressed as a multilinear combination
of its parents' sufficient statistics, i.e.
\begin{equation}
    \eta_{\x_\n}(\parents_{\x_\n}) = f_{\x_\n}\left(\{t_p(p)\}_{p \in \parents_{\x_\n}}\right)
\end{equation}
where $f_{\x_\n}$ is some multilinear function for $\x_\n$.
\end{property}

%Consider a variable $\x_\n$ in a CE model.
%Extracting the joint distribution natural parameter
%for a particular variable $\x_\n$
%can be done via a gradient with respect to $t_{\x_\n}(\x_\n)$.
%\begin{property}
%\label{prop:gradient}
%\begin{equation}
%    \eta^*_{\x_\n}(\not{\x_\n}) = \nabla_{t_{\x_\n}(\x_\n)} \log p(\x_\n, \not{\x_\n})
%\end{equation}
%This follows directly from \autoref{def:conjugacy}.
%\end{property}

\begin{property}
\label{prop:jointparam}
In a CE model,
the joint distribution natural parameter, $\eta^*_{\x_\n}(\not{\x_\n})$,
can be decomposed into summation of terms from $\x_\n$'s Markov blanket
(its children $\children_{\x_\n}$, parents $\parents_{\x_\n}$ and coparents).
\begin{equation}
\begin{split}
    &\eta^*_{\x_\n}(\not{\x_\n}) \\
    &= f_{\x_\n}\left(\{t_p(p)\}_{p \in \parents_{\x_\n}}\right)\\
    &+ \sum_{c \in \children_{\x_\n}} \nabla_{t_{\x_\n}(\x_\n)}\log p(c | \x_\n, \parents_{\x_\n} \not{\x_\n})\\
    &= f_{\x_\n}\left(\{t_p(p)\}_{p \in \parents_{\x_\n}}\right)
    + \sum_{c \in \children_{\x_\n}} \eta^c_{\x_\n}(\x_\n, \parents_c \backslash \x_\n)
\end{split}
\end{equation}
where $\parents_c \backslash \x_\n$ are the coparents of $\x_\n$ with respect to $c$.
The parents contribute one term to the natural parameter $f_{\x_\n}\left(\{t_p(p)\}_{p \in \parents_{\x_\n}}\right)$, along with one
term from each child $\eta^c_{\x_\n}(\x_\n, \parents_c \backslash \x_\n)$.

\end{property}
In general, these properties highlight the \emph{linearity} of terms
in CE models. The joint distribution natural parameter
is the sum of a multilinear function,
and several gradient terms.
This linearity is critical for the VMP derivation.

\subsection{Mean field variational inference}

Consider a latent variable model $\p(\observed, \locals, \globals) = \p(\globals)\p(\locals | \globals)\p(\observed | \locals, \globals)$
where $\observed$ are observed data, $\locals$ are
the local latent variables, and $\globals$ are the global latent variables.
In mean-field variational inference,
the analytically intractable
posterior $\p(\locals, \globals|\observed)$ is approximated
by a variational distribution $\q(\locals, \globals)$
which factorizes into individual distributions (factors), i.e.
$\q(\locals, \theta) = \q(\globals)\prod_{\n = 1}^\numdata \q(\z_n)$.

Variational inference proceeds by
maximizing the evidence lower bound (ELBO) $\L[\q]$.
\begin{equation}
    \L[\q] \triangleq \E_\q\left[\log \frac{\p(\observed, \locals, \globals)}{\q(\locals, \globals)}\right]
\end{equation}
With a mean-field factorization of $\q$, this optimization
breaks down into an individual optimization
over each variational factor.
We will focus on a particular local latent variable $\z_\n$.
Although VMP does not distinguish between local latents
and global latents, the distinction will play a part in \autoref{sec:extensions}.
For $\z_\n$'s variational factor $\q(\z_\n)$,
the ELBO is maximized by setting
\begin{equation}
    \log \q^*(\z_\n) \propto \E_{\q(\not{\z_\n})} \left[
    \log \p(\observed, \locals, \globals)\right]
\end{equation}
where $\q(\not{\z_\n})$ are all the variational factors excluding $\q(\z_\n)$.
The dependence of $\q^*(\z_\n)$ on the other factors leads
to a coordinate-ascent algorithm, where
each $\q(\z_\n)$ is updated with $q(\not{\z_\n})$ held constant,
iterating until
the ELBO has converged.

\begin{figure}[htp!]
    \centering
    \includegraphics[width=0.4\textwidth]{img/nvmp/vmp-4}
    \caption{The messages that need to be computed in order to update
    node $\z_\n$ in variational message passing. Specifically, we require
    messages computed over the Markov blanket of $\z_\n$.}
    \label{fig:vmp}
\end{figure}
\begin{algorithm}[htp!]
\begin{algorithmic}
   \STATE {\bfseries Input:} data $\dataset$, initial variational factors $\q(\z_\n)$ for each latent variable $z$
   \REPEAT
   \FOR{each factor $\q(\z_\n)$}
   \STATE Compute parent messages $\{\vmpmessage{p}{\z_\n}\}_{p \in \parents_{\z_\n}}$ (\autoref{eq:parentmessage})
   \STATE Compute coparent messages $\{\vmpmessage{v}{c}\}_{c \in \children_{\z_\n}, v \in \parents_c \backslash z}$ (\autoref{eq:parentmessage})
   \STATE Compute child messages $\{\vmpmessage{c}{\z_\n}\}_{c \in \children_{\z_\n}}$
   (\autoref{eq:childmessage})
   \STATE $\eta^\q_{\z_\n} \leftarrow f_{\z_\n}\left(\{\vmpmessage{p}{\z_\n}\}_{p \in \parents_{\z_\n}}\right) + \sum_{c \in \children_{\z_\n}} \vmpmessage{c}{\z_\n}$
   \ENDFOR
   \UNTIL{ELBO has converged}
\end{algorithmic}
   \caption{Variational message passing}
   \label{alg:vmp}
\end{algorithm}

In a conjugate-exponential model, we can utilize the definition
of conjugacy to further simplify the optimization of $\q^*(\z_\n)$.
\begin{equation}
\begin{split}
    \log \q^*(\z_\n) &\propto \E_{\q(\not{\z_\n})} \left[
    \log \p(\observed, \locals, \globals)\right]\\
    &\propto \E_{\q(\not{\z_\n})} \left[
    \left\langle t_{\z_\n}(\z_\n), \eta^*_{\z_\n}(\not{\z_\n})\right\rangle\right]\\
    &=
    \left\langle t_{\z_\n}(\z_\n), \E_{\q(\not{\z_\n})}\left[\eta^*_{\z_\n}(\not{\z_\n})\right]\right\rangle
\end{split}
\end{equation}
Thus, $\q^*(\z_\n)$
is in the same family as $\p(\z_\n | \parents_{\z_\n})$,
as the sufficient statistic functions match,
and has natural parameter $\eta^\q_{\z_\n} \triangleq \E_{\q(\not{\z_\n})}\left[\eta^*_{\z_\n}(\not{\z_\n})\right]$.

\subsection{Variational message passing}

Variational message passing provides a modular way of
computing
an optimal latent factor $\q^*(\z_\n)$
in terms of its local graphical model structure.
Specifically, we utilize \autoref{prop:jointparam}
to further pass in the expectation.
\begin{equation}
\begin{split}
    \eta^\q_{\z_\n} &= \E_{\q(\not{\z_\n})}\left[\eta^*_{\z_\n}(\not{\z_\n})\right] \\
                 &= f_{\z_\n}(\{\E_{\q(\not{\z_\n})}\left[t_p(p)\right]\}_{p \in \parents_{\z_\n}}) \\
                 &+ \sum_{c \in \children_{\z_\n}}
         \nabla_{t_{\z_\n}(\z_\n)} \E_{\q(\not{\z_\n})}\left[\log \p(c | \z_\n, \parents_{c} \backslash \z_\n)\right]
\end{split}
\end{equation}

This decomposition leads to a message passing algorithm, where
the natural parameter $\eta^\q_{\z_\n}$
can be computed as the sum of messages from the children and parents.

\textbf{Parent message:}
The message from a parent $p$ to $\z_\n$ is
\begin{equation} \label{eq:parentmessage}
    \vmpmessage{p}{\z_\n} = \E_{\q(p)}\left[t_p(p)\right]
\end{equation}

\textbf{Child message:}
The message from child $c$ to $\z_\n$ is

\begin{equation} \label{eq:childmessage}
    \vmpmessage{c}{\z_\n} = \nabla_{t_{\z_\n}(\z_\n)} \E_{\q(c, \parents_{c} \not{\z_\n})}\left[\log \p(c | \z_\n, \parents_{c} \not{\z_\n})\right]
\end{equation}

Computing the gradient in the child message involves
computing messages from the coparents of $\z_\n$ with respect to $c$ in order to evaluate the expectation.
After all relevant messages
have been computed, the natural parameter is $\q^*(\z_\n)$ calculated as
\begin{equation}
    \label{eq:vmpparam}
   \eta^\q_{\z_\n} = f_{\z_\n}\left(\{\vmpmessage{p}{\z_\n}\}_{p \in \parents_{\z_\n}}\right) + \sum_{c \in \children_{\z_\n}} \vmpmessage{c}{\z_\n}
\end{equation}

An important note is that if a node corresponds to observed data,
expectations over the node are replaced with the empirical quantity.
The complete VMP algorithm is described in \autoref{alg:vmp} and
a diagram of the necessary messages for updating a node
is shown in \autoref{fig:vmp}.

\section{Neural variational message passing}
\label{sec:nvmp}

In this section, we examine
the assumptions that the VMP derivation makes
and identify where recognition networks
can assist in inference.

\iffalse
VMP relies on the CE assumption that
for each variable $\z_\n$, the sufficient statistics
of its likelihood $t_{\z_\n}(\z_\n)$ are the same
sufficient statistics present in the joint distribution
over all variables.
This property ensures that the optimal
variational distribution for a latent variable
will be in the same family of that latent variable's likelihood.
However, suppose there are statistics in the joint density
that do not match a simple exponential family. Consider
the variational autoencoder, which has a neural network
in the generative model. Neural networks will produce
naturally complicated statistics of latent variables
in the joint density, so attempting to rewrite terms
to match, say a Gaussian posterior distribution,
will be impossible.
\fi

In NVMP we first
assume variational posteriors that match
the likelihood, that is $\q(\z_\n)$ will
be the family as $\p(\z_\n | \parents_{\z_\n})$.
This step
is quite common in variational inference
as even in conjugate
models, the true posterior of a latent variable may not belong
in the same family as its likelihood.
Where NVMP differs is in how latent variational factors
are optimized.
NVMP uses VMP messages whenever possible,
but when VMP messages are not possible due to
nonconjugacy, they are replaced with NVMP messages,
$m^{\text{NVMP}}_{p \rightarrow \z},m^{\text{NVMP}}_{c \rightarrow \z}$.
We now derive the form of these modified messages.

\textbf{Parent message:}
Consider \autoref{prop:multilinear}:
if $f_{\z_\n}$ is not multilinear in the sufficient statistics of one of
its parents
parents $p$,
we can no longer compute the natural parameter
from $\vmpmessage{p}{\z_\n}$.
However, we can perform a Monte Carlo approximation of $f_{\z_\n}$.
Thus, in NVMP, if we $f_{\z_\n}$ is no longer
multilinear in the sufficient statistics of its parents,
we approximate $f_{\z_\n}$ using samples from $\q(p)$.
\footnote{Note that we can use multiple samples from $\q(p)$ for a better Monte Carlo
approximation.}

\begin{equation}
    m^{\text{NVMP}}_{p \rightarrow z} = p^\prime
\end{equation}

\textbf{Child message:}
Nonconjugate models result in intractable child messages.
The entire strategy of re-arranging
the log-likelihood
in terms of sufficient statistics $t_{\z_\n}(\z_\n)$ to eventually
take a gradient is infeasible.
However, adapting the approach of SVAE allows us to
compute the child message for an approximation of the ELBO.
In conjugate-exponential models, utilizing \autoref{prop:jointparam},
we are able to rewrite
\begin{equation}
\log \p(c | \z_\n, \parents_c \not{\z_\n}) \propto \left\langle t_{\z_\n}(\z_\n), \eta_{\z_\n}^c(c, \parents_c \not{\z_\n})\right\rangle
\end{equation}
and $\E_{\q(\backslash \z_\n)}\left[\eta_{\z_\n}^c(c, \parents_c \not{\z_\n})\right]$
would be the corresponding message $\vmpmessage{c}{\z_\n}$.
If we are unable to arrange $\log \p(c | \z_\n, \parents_c \not{\z_\n})$ in this way, we approximate this log-likelihood with a factor $\psi(c, \z_\n, \parents_{c} \not{\z_\n}; \recparam)$, as done in \citet{Johnson2016}.
\begin{equation}
\label{eq:rectrick}
\begin{split}
\log \p(c | \z_\n, \parents_c \not{\z_\n}) &\approx \psi(c, \z_\n, \parents_{c} \not{\z_\n}; \recparam) \\
\end{split}
\end{equation}
We choose a particular form of $\psi$ that results in an easy optimization
of an approximate ELBO.
\begin{equation}
\begin{split}
    \psi(c, \z_\n, \parents_{c} \not{\z_\n}; \recparam) &\triangleq \left\langle t_{\z_\n}(\z_\n), r^{c \rightarrow \z_\n}_{\recparam}(c, \parents_{c} \not{\z_\n})\right\rangle
\end{split}
\end{equation}
where $r_{\recparam}$ is a recognition network parameterized by $\recparam$.
Following the original steps in VMP
and computing the gradient of the approximate log-likelihood
with respect to $t_{\z_\n}(\z_\n)$ results in a new message.
\begin{equation}
    \E_{\q(c)\q(\parents_c \backslash \z_\n)} \left[r^{c \rightarrow \z_\n}_{\recparam}(c, \parents_{c}\not{\z_\n})\right]
\end{equation}
This expectation can be approximated via a Monte Carlo sample $c' \sim \q(c)$
and NVMP messages from the coparents (which are also Monte Carlo samples).
\footnote{As in the parent message case, our Monte Carlo approximation can
be improved with multiple samples.}

\begin{equation}
    \nvmpmessage{c}{\z_\n} = r^{c\rightarrow \z_\n}_{\recparam}\left(
    c', \{\nvmpmessage{v}{\z_\n}\}_{v \in \parents_c \not{\z_\n}}\right)
\end{equation}

The NVMP child message for child $c$ passes Monte Carlo samples
of $c$ and of $\z_\n$'s coparents into a recognition network.
Note that this strategy assigns a recognition
network to \emph{each} nonconjugate edge
in a graph. However, in practice,
it would be prudent to share parameters
across networks
whenever possible,
especially for edges where nodes are sampled
independently (e.g. IID data).
Finally, we can compute the parameter for $\q^*(\z_\n)$
as before, but instead of using $f_{\z_\n}$, we use $\z_\n$'s
exponential family natural parameter function $\eta_{\z_\n}(\parents_{\z_\n})$.
\begin{equation}
    \label{eq:nvmpparam}
   \eta^\q_{\z_\n} = \eta_{\z_\n}\left(\{\nvmpmessage{p}{\z_\n}\}_{p \in \parents_{\z_\n}}\right) + \sum_{c \in \children_{\z_\n}} \nvmpmessage{c}{\z_\n}
\end{equation}
These message definitions are a drop-in replacement
for messages in VMP, allowing us to re-use \autoref{alg:vmp},
using NVMP messages whenever possible. The only
difference is that we use \autoref{eq:nvmpparam} to compute
the natural parameter instead of \autoref{eq:vmpparam}.
Importantly, introducing recognition networks
into the VMP framework does mean we are no longer
directly optimizing the ELBO. However, we will
show that we are still optimizing a lower
bound on the ELBO, and if the recognition networks
have high enough capacity, the bound is tight.

\subsection{Learning recognition network weights}
The final step is learning recognition network
weights and other model parameters.
In the inner loop of NVMP, we
iterate message passing to convergence.
This outputs a set of optimal variational
factors, conditioned on the recognition networks.
We then take a gradient step
to maximize the ELBO.
In a nonconjugate model, the ELBO isn't
directly computable due to intractable expectations.
Therefore,
recognition networks and model parameters are optimized
via stochastic gradients of the ELBO using samples from $q$.
In a latent variable model, this would correspond to
taking gradients of
\begin{equation}
    \L[\q] \approx \frac{1}{L}\sum_{l = 1}^L \log \frac{\p(\observed, \locals^{(l)}, \globals^{(l)})}{\q(\locals^{(l)}, \globals^{(l)})}
\end{equation}
The reparametrization trick should be utilized whenever possible,
to reduce the variance of the gradients \citep{Kingma2014}.
Furthermore, the gradients with respect recognition networks
and model parameters computed using optimal variational
distributions, meaning we must backpropagate through
the iterative process to compute the gradients.

\subsection{Lower-bounding the ELBO}
The inclusion of recognition networks
in the local optimization for each factor
induces a global
optimization on a lower bound the ELBO.

\begin{theorem}
\label{thm:lowerbound}
The NVMP algorithm optimizes a lower bound on the ELBO.
\begin{proof}
 (sketch) Each individual coordinate ascent update optimizes its own lower bound on the ELBO. Thus, holistically, NVMP optimizes a lower bound on the ELBO by optimizing individual lower bounds. We refer you to \autoref{chap:nvmp-appendix} for the full proof.
\end{proof}
\end{theorem}

\subsection{Extensions to NVMP}
\label{sec:extensions}

NVMP is a fully-specified template
for nonconjugate inference, but in practice,
some modifications are useful. In this section,
we outline ways some recent
developments in variational inference which naturally fit
into the NVMP framework.

Stochastic variational inference (SVI), or stochastic VMP, is
a strategy to adapt mean-field variational inference
to large datasets \citep{Hoffman2013}. As datasets
have grown larger, we typically require
algorithms that operate on mini-batches
of data to scale and SVI adds this functionality to VMP.
SVI enables computing natural gradients of the ELBO for parameters in our model.We detail integrating SVI into NVMP in the supplement.

%\subsection{Amortized inference}
%\label{sec:amortized}

Amortized inference enables neural networks
to directly act as posterior distributions, as in VAE, as opposed to outputting approximate messages, as in SVAE. When we want a black-box approximate posterior, learning such a recognition network can be useful in downstream inference tasks. We detail how
amortized inference networks can be used with NVMP in the supplement.

\section{Examples and experiments}
\label{sec:examples}

In this section, to show the simplicity by
which NVMP can be used for nonconjugate inference,
we provide several example models
where NVMP messages can be used, and some simple
experiments. We do not aim to outperform
similar frameworks, but rather to show the versatility of NVMP, as NVMP works on models ranging from the VAE to Bayesian logistic regression.
The graphical models are pictured in \autoref{fig:graphicalmodels},
and in all experiments, we utilized the stochastic variant
of NVMP, and did not use amortized inference.

\subsection{Bayesian linear regression with a sparse prior}

In the Bayesian lasso variant of sparse linear regression.
we put an exponential prior on the variance of our weights $w$ \citep{Park2008}.

\begin{figure*}
    \centering
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics{tikz/representation/nvmp/lr-prior}
        \caption{Bayesian linear regression with a sparse prior}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics{tikz/representation/nvmp/blr}
        \caption{Bayesian logistic regression}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics{tikz/representation/nvmp/vae}
        \caption{Variational autoencoder}
    \end{subfigure}
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics{tikz/representation/nvmp/svae-gmm}
        \caption{SVAE-GMM}
    \end{subfigure}
    \begin{subfigure}[t]{0.19\textwidth}
        \centering
        \includegraphics{tikz/representation/nvmp/nlds}
        \caption{Nonlinear dynamical system}
    \end{subfigure}
    \caption{Five nonconjugate models in which NVMP messages can be used for approximate inference.}
    \label{fig:graphicalmodels}
\end{figure*}

\begin{align*}
    \sigma_w^2 &\sim \text{Exponential}([1, \ldots, 1]) \\
    w | \sigma^2_w &\sim \N(0, \text{diag}\left(\sigma_w^2)\right) \\
    y_\n | w, \x_\n &\sim \N(w^T\x_\n, \sigma^2_y)
\end{align*}

Exact posterior inference in this model is not tractable, due
to the exponential prior, so approximation is necessary.
Since $y_\n$ is conjugate to $\x_\n$ and $w$, we still use
the VMP messages $\vmpmessage{\x_\n}{y_\n}, \vmpmessage{y_\n}{w}, \vmpmessage{w}{y_\n}$,
but otherwise we use NVMP messages $\nvmpmessage{w}{\sigma_w^2}, \nvmpmessage{\sigma_w^2}{w}$. We require one recognition
network $r_{\recparam}^{w \rightarrow \sigma^2_w}(w)$, which produces the message $\nvmpmessage{w}{\sigma_w^2}$.

\textbf{Experiment} We performed NVMP with this model on the
Boston housing regression dataset \citep{Lichman2013}.
We utilized a 2-layer 500-wide feedforward network with Tanh activations
as $r_{\recparam}^{w \rightarrow \sigma^2_w}(w)$.
Regular linear regression learns weights with $\|w\|_1 = 25.56$
and testing mean squared error $33.45$.
Lasso regression learns weights with $\|w\|_1 = 3.27$
and testing mean squared error $41.07$.
NVMP learns a weight vector with $\|w\|_1 = 7.31$
and testing mean squared error $35.60$.
The Bayesian lasso learns a sparsity
in between unregularized regression and lasso,
consistent with findings in \citet{Park2008},
which uses a Gibbs sampler to perform inference in the model.

\subsection{Bayesian logistic regression}

Bayesian logistic regression is a model for binary classification
of data and labels $\{\x_\n, y_\n\}_{i = 1}^N$.
\begin{align*}
    w &\sim \N(0, I) \\
    y_\n | w, \x_\n &\sim \text{Bernoulli}(\sigma(w^T\x_\n))
\end{align*}

The approximated messages are $\nvmpmessage{\x_\n}{y_\n}, \nvmpmessage{w}{y_\n}, \nvmpmessage{y_\n}{w}$ with recognition network $r_{\recparam}^{y_\n \rightarrow w}(\x_\n, y_\n)$.

\textbf{Experiment} We ran a simple experiment, performing
NVMP with this model on MNIST restricted to the 3 and 8 digit.
We utilized a 2-layer 500-wide feedforward network with Tanh activations
as $r_{\recparam}^{y_\n \rightarrow w}(\x_\n, y_\n)$.
Traditional logistic regression achieves a test set accuracy of 95.6\%
and our method achieves 95.1\%. This is drop in accuracy is likely due to the
prior, as running NVMP without the prior
also results in an accuracy of 95.6\%.

\iffalse
\subsection{Variational autoencoder}
The variational autoencoder \citep{Kingma2014, Rezende2014} generates data
by first sampling data from a simple Gaussian prior
and then passing it through a neural network.

\begin{align*}
    \x_\n &\sim \N(0, I) \\
    y_\n | \x_\n &\sim \N(\mu_\gamma(\x_\n), \Sigma_\gamma(\x_\n))
\end{align*}
where $\mu_\gamma$ and $\Sigma_\gamma$ are a neural network parameterized
by weights $\gamma$.

With NVMP, we use approximate messages
$\nvmpmessage{\x_\n}{y_\n}$, and
$\nvmpmessage{y_\n}{\x_\n}$, with a
single recognition network $r_\recparam^{y_\n \rightarrow \x_\n}(y_\n)$.

It is important to note the difference between the
inference algorithm for VAE as originally specified
and inference with NVMP. The recognition network in VAE outputs
the parameters of $q(\x_\n | y_\n)$ directly,
whereas in NVMP, the parameters of $q(\x_\n | y_\n)$
are the sum of the output of the recognition network $r_\recparam^{y_\n \rightarrow \x_\n}(y_\n)$
and the natural parameter of the prior on $\x_\n$.
We omit running an experiment for this model, due to NVMP's similarity
to the AEVB algorithm. In fact, using amortized inference
will result in the exact VAE inference algorithm.
\fi

\subsection{SVAE-GMM}
The structured variational autoencoder Gaussian mixture model \citep[SVAE-GMM; ][]{Johnson2016} learns representations
that have a latent cluster structure.
It accomplishes this by attaching a neural network observation model
to a Gaussian mixture model.
\begin{align*}
    \{\mu_k, \Sigma_k\}_{k = 1}^K &\sim \NIW(\Psi, \nu, \mu_0, \kappa) \\
    \pi &\sim \text{Dirichlet}(\alpha) \\
    \z_\n | \pi &\sim \text{Categorical}(\pi) \\
    \x_\n | \z_\n, \{\mu_k, \Sigma_k\} &\sim \N(\mu_{\z_\n}, \Sigma_{\z_\n}) \\
    y_\n | \x_\n &\sim \N(\mu_\gamma(\x_\n), \Sigma_\gamma(\x_\n))
\end{align*}
where $\mu_\gamma$ and $\Sigma_\gamma$ are a neural network parameterized by $\gamma$.

Our NVMP messages are $\nvmpmessage{\x_\n}{y_\n}, \nvmpmessage{y_\n}{\x_\n}$
and the rest remain VMP messages. We also omit an experiment
for this model, as stochastic NVMP
results in the same inference algorithm as described in \citet{Johnson2016},
without the prior on neural network observation model weights.

\subsection{Nonlinear dynamical system}
Consider a dynamical system that evolves
according to a neural
network transition function, but is observed with Gaussian noise:
\begin{align*}
    x_{t + 1} | x_t &\sim \N(\mu_\gamma(x_t), \Sigma_\gamma(x_t)) \\
    y_t | x_t &\sim \N(x_t, \sigma^2_yI)
\end{align*}
where $\mu_\gamma$ and $\Sigma_\gamma$ are a neural network parameterized by $\gamma$.

In this model, we require NVMP messages $\nvmpmessage{x_{t + 1}}{x_t}, \nvmpmessage{x_t}{x_{t + 1}}$ with a recognition network $r^{x_{t + 1} \rightarrow x_t}_\recparam(x_{t + 1})$.

\begin{figure}
    \centering
    \includegraphics[width=0.3\textwidth]{img/nvmp/nlds}
    \caption{A learned ``circular'' dynamics system using NVMP.}
    \label{fig:nlds}
\end{figure}

\textbf{Experiment} We simulated a nonlinear dynamical system
with 2-dimensional data and a transition function
that moves points along a circle centered at the origin.
We utilized 2-layer 500-wide feedforward networks with Tanh activations
for both $r^{x_{t + 1} \rightarrow x_t}_\recparam(x_{t + 1})$ and $\mu_\gamma, \Sigma_\gamma$.
Pictured in \autoref{fig:nlds} is the learned neural network dynamical system.
It learns the circular pattern the data is generated from, indicating
that NVMP successfully fit a transition function.

%\subsection{Recurrent switching linear dynamical system}
%The recurrent switching linear dynamical system, introduced
%in \cite{rslds},
%We consider a switching linear dynamical system
%where we have hidden states that evolve Markovian
%with a non-linear dependence on observation.
%
%Our model for dataset $\{x^{(i)}_0, x^{(i)}_1, \ldots, x^{(i)}_T\}_{i = 1}^N$
%is
%\begin{align*}
%    \pi^0 &\sim \mathrm{Dir}(\alpha) \\
%    \{\pi_k\}_{k = 1}^K &\sim \mathrm{Dir}(\alpha) \\
%    \{A_k, \Sigma_k\}_{k = 1}^K &\sim \MNIW(\Psi, M_0, V, \nu) \\
%    x_0 &\sim \N(\mu_0, \Sigma_0) \\
%    s_0 &\sim \mathrm{Cat}(\pi^0) \\
%    s_{t + 1} | s_t, x_t &\sim \mathrm{Cat}(f(\pi_{s_t}, g_\gamma(x_t)) \\
%    x_{t + 1} | s_{t + 1}, x_t, \{A_k, \Sigma_k\}_{k = 1}^K &\sim \N(A_{s_{t + 1}}x_t, \Sigma_{s_{t + 1}})
%\end{align*}
%
%This model is fully conjugate other than the non-linear state transition.
%Specifically, we have a neural network $g_\gamma$ which outputs
%potentials to the state transition distribution. $f$ is a function
%that normalizes the sum of the potentials and the Markovian transition
%probabilities $\pi_{s_t}$.
%
%We can learn this model with NVMP by approximating
%the message $m_{s_{t + 1} \rightarrow s_t} = r_\recparam(\eta_{s_t}, x_t)$.
%In this scenario, we share neural network weights $\recparam$
%between all nonconjugate messages, since there is one for each time
%step in this model.

\section{Discussion}
%We have written a software package,
%built on top of Tensorflow. The package
%was designed to make modeling as easy as possible.
%NVMP enables specifying models purely in
%the forward direction, like in Edward and Stan.
%Rather than using a black-box method
%or sampling afterwards,
%NVMP requires recognition networks
%as inputs to the algorithm
%wherever necessary. After this, inference
%is completely automated and efficient, relying on
%GPU computations through the Tensorflow computational graph.

A key limitation of NVMP
(and other models that utilize recognition networks)
is that gradients with respect to model parameters
must only include continuous random variables,
as backpropagation through discrete random
variables is not possible. However, integrating
the Gumbel-Softmax trick or modifications
such as those in
\citet{Rolfe2016}, \citet{Vahdat2018dvaeplus}, \citet{Vahdat2018dvaesharp} and \citet{Van2016} into NVMP is a promising
step forward in addressing this concern.

\nvmpack


\documentclass{article}
\usepackage[minionint,mathlf,textlf]{MinionPro} % To gussy up a bit
\usepackage[margin=1in]{geometry}
\usepackage{graphicx} % For .eps inclusion
%\usepackage{indentfirst} % Controls indentation
\usepackage[compact]{titlesec} % For regulating spacing before section titles
\usepackage{adjustbox} % For vertically-aligned side-by-side minipages
\usepackage{array, mathrsfs, mhchem, amsmath} % For centering of tabulars with text-wrapping columns
\usepackage{hyperref, chemfig}
\usepackage[autolinebreaks,framed,numbered]{mcode}
\pagenumbering{gobble}
\setlength\parindent{0 cm}
\begin{document}
\large

\section*{Conversion of a discrete model to an SDE}

The techniques we will introduce today require a stochastic differential equation of the following form:
\[ dx = \mu(x,t) \, dt + \sigma(x,t) \, dW_t, \]

where $\mu(x,t)$ reflects the rate of change of the deterministic system and $\sigma(x,t)$ is the standard deviation of the magnitude of the oscillations. For Poisson processes we assume $\sigma(x,t)=\sqrt{\mu(x,t)}$, but how do we find $\sigma(x,t)$ more generally? For example, what happens if our system has multiple interdependent variables?\\

The following example to demonstate this technique is based on McKane and Newman (2005). They studied a variant of the Lotka-Volterra predator-prey model. You may have heard of a popular application of this model to the sale of lynx and hare pelts to the Hudson Bay Company in the 1800s and early 1900s (see figure below). The story of its origin is even more interesting than that, however. Lotka originally developed the model in the context of autocatalytic chemical reactions. At the time (1910) there was a general resistance to the notion that a chemical reaction system could show oscillations, even in the presence of unlimited reagents; when Lotka published his model he noted that no such reaction scheme was known to chemistry. When Belousov and Zhabotinsky discovered such a chemical reaction system in the 1950s/60s, the general attitude toward the possibility of oscillatory chemical systems was so poor that they fought to get their work published. Volterra also did not originally study pelts, but rather took an interest due to his son-in-law's involvment in the fishing industry.\\

For a predator $x$ and prey $y$, the deterministic system is:
\[ \frac{dx}{dt} = p_1 x y - d_1 x \hspace{2 cm} \frac{dy}{dt} = (b - d_2) y - (p_1+p_2) x y \]
where $b$ is the birth rate of prey, $d_1$ and $d_2$ are the rates of death by natural causes, $p_1+p_2$ is the rate at which $x$ kills $y$, and $p_1< p_1 + p_2$ is the birth rate of $x$ due to the predation. Using fixed point analysis discussed earlier in the course, we could show that $x$ and $y$ oscillate around the fixed point but that any deviation will change the amplitude of their trajectory, just like the simple harmonic oscillator we studied earlier. Common sense tells us that this is not the way predator-prey cycles work in the real world: the real system likely approaches a limit cycle.\\

A modification of this approach attempts to limit the prey population by introducing a carrying capacity:
\[ \frac{dx}{dt} = p_1 x y - d_1 x \hspace{2 cm} \frac{dy}{dt} = by(1-\frac{y}{K}) - d_2 y - (p_1+p_2) x y \]
Now $y$ will not grow larger than $K$. Unfortunately, this system doesn't approach a limit cycle: it shows damped oscillations. Like an example on your problem set seven, McKane and Newman were able to show that noise drives oscillations in the latter system. They used both a discrete model and its Langevin equivalent, providing a great opportunity to show the interconversion.\\

Let's begin with the discrete model. The authors suppose that the environment permits up to $N$ organisms: physically we can envision that water or some other shared resource is limiting for growth. Some of the $N$ ``sites" are empty ($E$), others occupied by predators ($X$) or prey ($Y$). Then the ``reactions" and their propensities are given by the table below. Note that the system size $\Omega = N$ and the propensity is $N r(\mathbf{x},N)$. While it seems a bit odd, we divide the number of each animal by $N$ to get its ``concentration," determine the rate of the reaction from this value, then multiply through by $N$ again to convert back to appropriate units. ``Reactions" with two reactants therefore have a factor of $1/N$ in their propensity.
\begin{center}
\begin{tabular}{c|c|c|c}
Reaction & Stoichiometry column $s$ & Propensity & $ss^T$ \\ \hline
\ce{Y + E ->[b] Y + Y} & $s_1 = (0,1)^T$ & $P_1=\frac{by(N-x-y)}{N}$ & $s_1 s_1^T = \begin{pmatrix} 0 & 0\\ 0 & 1\end{pmatrix}$\\
\ce{X ->[d_1] E} & $s_2 = (-1,0)^T$ & $P_2 =d_1x$ & $s_2 s_2^T = \begin{pmatrix} 1 & 0\\ 0 & 0\end{pmatrix}$\\
\ce{Y ->[d_2] E} & $s_3 = (0,-1)^T$ & $P_3 =d_2y$ & $s_3 s_3^T = \begin{pmatrix} 0 & 0\\ 0 & 1\end{pmatrix}$\\
\ce{X + Y ->[p_1] X + X} & $s_4 = (1,-1)^T$ & $P_4=\frac{p_1xy}{N}$ & $s_4 s_4^T = \begin{pmatrix} 1 & -1\\ -1 & 1\end{pmatrix}$\\
\ce{X + Y ->[p_2] X + E} & $s_5 = (0,-1)^T$ & $P_5=\frac{p_2xy}{N}$ & $s_5 s_5^T = \begin{pmatrix} 0 & 0\\ 0 & 1\end{pmatrix}$\\
\end{tabular}
\end{center}
At this point we have everything we need for a Gillespie simulation, but let's see what this system would look like in Langevin form. The mean change in $(x,y)^T$ would be:
\[ \mu(x,y,t) = \sum_{i=1}^5 s_i P_i = \begin{pmatrix} \frac{p_1 x y}{N}- d_1x\\ \frac{by(N-x-y)}{N} - d_2 y - \frac{(p_1+p_2)xy}{N} \end{pmatrix} \]
Similarly, the variance in change in $(x,y)^T$ is:
\[ \sigma^2(x,y,t) = \sum_{i=1}^5 s_i  s_i^TP_i = \begin{pmatrix} d_1 x + \frac{p_1xy}{N} & -\frac{p_1 x y}{N}\\ -\frac{p_1 x y}{N} & \frac{by(N-x-y)}{N} + d_2 y + \frac{(p_1+p_2)xy}{N} \end{pmatrix} \]
The Langevin form is therefore:
\[ d\begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} \frac{p_1 x y}{N}- d_1x\\ \frac{by(N-x-y)}{N} - d_2 y - \frac{(p_1+p_2)xy}{N} \end{pmatrix} dt + \begin{pmatrix} d_1 x + \frac{p_1xy}{N} & -\frac{p_1 x y}{N}\\ -\frac{p_1 x y}{N} & \frac{by(N-x-y)}{N} + d_2 y + \frac{(p_1+p_2)xy}{N} \end{pmatrix} \begin{pmatrix} dW_t^1\\ dW_t^2 \end{pmatrix} \]
where $W_t^1$ and $W_t^2$ are Wiener processes. To simulate the system, we use the following update strategy:
\[ \Delta \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} \frac{p_1 x y}{N}- d_1x\\ \frac{by(N-x-y)}{N} - d_2 y - \frac{(p_1+p_2)xy}{N} \end{pmatrix} \Delta t + \begin{pmatrix} d_1 x + \frac{p_1xy}{N} & -\frac{p_1 x y}{N}\\ -\frac{p_1 x y}{N} & \frac{by(N-x-y)}{N} + d_2 y + \frac{(p_1+p_2)xy}{N} \end{pmatrix} \begin{pmatrix} \eta_1 \\ \eta_2 \end{pmatrix} \sqrt{\Delta t} \]
where $\eta_1, \eta_2 \sim \mathcal{N}(0,1)$ are random numbers drawn independently. Both types of simulations give approximately equivalent results (Langevin on the left, Gillespie on the right, data on the sale of lynx and hare pelts reproduced from Pineda-Krch et al. (2007):
\begin{center}
\includegraphics[width=1\textwidth]{mckane_newman.pdf}
\end{center}
By determining the power spectrum of e.g. the predator from these equations, McKane and Newman showed that there is a resonant frequency for this system: random noise that happens to occur at that frequency gets amplified.

\section*{Fokker-Planck equation}
Two more handy tricks, both concerning the probability density function of states $p(x,t)$, are worth knowing about should you ever need them. For a differential equation of the form
\[ dx = \mu(x,t) \, dt + \sigma(x,t) \, dW_t, \]
an approximate expression for the time evolution of $p(x,t)$ is given by the Fokker-Planck equation:
\[ \frac{\partial p(x,t)}{\partial t} = - \frac{\partial }{\partial x} \left[ \mu(x,t) p(x,t) \right] +  \frac{1}{2} \frac{\partial^2 }{\partial x^2} \left[ \sigma^2(x,t) p(x,t) \right]  \]
Why is this so? Consider the continuous version of the chemical master equation (aka the forward Kolmogorov equation), with $r(a,b)$ the rate at which the system changes by an amount $a$ from an initial state $b$:
\[ \frac{\partial p(x,t)}{\partial t} = \int \left[p(s,t) r(x-s,s) - p(x,t) r(s-x, x) \right] \, ds \]
Introducing a new variable $y = x - s$, this becomes:
\[ \frac{\partial p(x,t)}{\partial t} = \int \left[p(x-y,t) r(y,x-y) - p(x,t) r(y, x) \right] \, dy \]
Consider the Taylor expansion of the term on the left in the integrand:
\[ p(x-y,t) r(y,x-y) = p(x,t) r(y,x) - y \frac{\partial }{\partial x} \left[ p(x,t) r(y,x)  \right] + \frac{y^2}{2!} \frac{\partial^2}{\partial x^2} \left[ p(x,t) r(y,x)  \right] + \ldots \]
Keeping the first three terms (e.g. assuming that $p$ is slowly varying), this becomes:
\begin{eqnarray*}
\frac{\partial p(x,t)}{\partial t} & = & \int \left[ - y \frac{d}{dx} \left[ p(x,t) r(y,x)  \right] + \frac{y^2}{2!} \frac{d^2}{dx^2} \left[ p(x,t) r(y,x)  \right] \right] \, dy\\
& = & -  \frac{\partial}{\partial x} p(x,t)  \left[ \int y r(y,x) \, dy  \right] + \frac{1}{2} \frac{\partial^2}{\partial x^2} \left[ p(x,t) \int y^2 r(y,x) \, dy \right]
\end{eqnarray*}
which is nothing other than the Fokker-Planck equation with
\begin{eqnarray*}
\mu(x,t) =  \int y r(y,x) \, dy \hspace{3 cm} \sigma^2(x,t) =  \int y^2 r(y,x) \, dy
\end{eqnarray*}
These integrals are sometimes called the first and second \textit{jump moments}: their equality to $\mu(x,t)$ and $\sigma^2(x,t)$ can be seen by calculating $\left< \Delta x\right>$ and $\left< \Delta x^2\right>$ from the Langevin equation (see lecture notes from Scott if interested).

\subsection*{Drift in population genetics}

Suppose we start a culture of bacteria by mixing two strains, one red and one green but otherwise identical, in a 1:1 ratio to contain $N$ cells total. We allow the culture to grow exponentially then dilute it back to just $N$ cells\footnote{If we assume that all cells leave the same number of descendants during the growth phase, then dilution is the equivalent of picking balls from an $N$-ball urn with replacement.}. We will assume that no mutations occur during the experiment so that every cell has an equal chance of leaving offspring.\\

We understand intuitively that the ratio of green to red cells in our culture will fluctuate with each dilution step and will eventually reach either all red or all green, where it will remain: the endpoints will on average be reached faster if the population is small. To understand better how fast this happens, let's rewrite this as a stochastic differential equation in $x=n/N$, the fraction of green cells.\\

Let $n$ be the number of green cells after the last round of diluion. The probability that we have $m$ green cells after the next dilution is:
\[ P(m \, | \, n, N) = \begin{pmatrix} N \\ m \end{pmatrix} \left(\frac{n}{N} \right)^m \left(1 - \frac{n}{N} \right)^{N-m} \]
i.e., the number of green cells is binomially distributed with parameters $N$ and $x=n/N$. Using known properties of the binomial distribution:
\[ \left< m \right> = Nx = n \hspace{ 3 cm} \left< m^2 \right> -\left< m \right>^2 = Nx(1-x) - n^2= \frac{n}{N} \left( N - n \right) - n^2 \]
Therefore the first and second moments of change in number of green cells in one round are:
\[ \mu = \left< m - n \right> = 0 \hspace{ 3 cm} \sigma^2 = \left< (m - n)^2 \right> = \left< m^2 \right> - 2 n \left< m \right> + \left< m \right>^2 = \frac{n}{N} \left( N - n \right) \]
Now we have everything needed to write our SDE:
\[ dx =dW_t \, \sqrt{\frac{x(1-x)}{N}} \]
The formula applies until $x$ reaches $0$ or $1$, at which point it will remain there\footnote{Calculations of expected persistence times and related statistics are beyond the scope of this course but of obvious relevance to conservation biology and population genetics.}. The change in $x$ was indeed a random walk as we suspected, but it may not have been obvious that the step sizes were changing in this way. Plugging into the Fokker-Planck equation, we see that:
\begin{eqnarray*}
\frac{\partial p(x,t)}{\partial t} & = &  \frac{1}{2N} \frac{\partial^2}{\partial x^2} \left[ x(1-x) p(x,t) \right]
\end{eqnarray*}

 \section*{Wright's formula}
The stationary solutions of the Fokker-Planck equation satisfy:
\begin{eqnarray*}
2 \mu(x,t) p_{ss}(x,t) & = & \frac{\partial }{\partial x} \left[ \sigma^2(x,t) p_{ss}(x,t) \right]\\
\frac{2 \mu(x,t)}{\sigma^2(x,t)} & = & \frac{1}{\sigma^2(x,t) p_{ss}(x,t)} \frac{\partial }{\partial x} \left[ \sigma^2(x,t) p_{ss}(x,t) \right] = \frac{\partial}{\partial x} \ln \left[ \sigma^2(x,t) p_{ss}(x,t) \right]\\
2\int^x \frac{\mu(s,t)}{\sigma^2(s,t)} \, ds & = & \ln  \left[ \sigma^2(x,t) p_{ss}(x,t) \right] + C\\
p_{ss}(x,t) & \propto & \frac{1}{\sigma^2(x,t)} \exp \left[ 2\int^x \frac{\mu(s,t)}{\sigma^2(s,t)} \, ds \right]
\end{eqnarray*}
The probability distribution should of course be normalized. This formula for finding stationary solutions is sometimes attributed to Wright (1938). The good news is that you can find stationary solutions using regular calculus, even for an SDE. You will apply this approach in problem set eight to study the spread of disease.\\

Although we will go no further in this introductory course, I hope this discussion has piqued your interest in the study of stochastic differential equations or, if nothing else, conveyed a respect for their proper husbandry.

\end{document}

lectures/Lecture 24 - Fokker Planck Equation and Wright's Formula/lecture 24 notes.tex


\documentclass[simplex.tex]{subfiles}
% NO NEED TO INPUT PREAMBLES HERE
% packages are inherited; you can compile this on its own
\begin{document}
\subsection{Non-Parametric Shape Clustering}

We have been developing a non-parametric method for clustering
based on energy statistics. We were able to formulate the problem
in a precise mathematical form, which amounts to solve a quadratic
optimization problem with quadratic constraints, which is NP-hard.
Using the relation between energy statistics and kernel functions, we
implemented an algorithm in the same spirit as kernel k-means, but
consistent with energy statistics. Below we show some preliminary results,
where we cluster one-dimensional normal data, and also two-dimensional
circles with noise. The results of our method are shown in the column
$\mathcal{E}$-clustering. We compare with standard k-means and gaussian
mixture models (GMM). We compute the accuracy (which is
between $[0,1]$, the higher
the better) based on the true labels.
In this example we can see that $\mathcal{E}$-clustering attains the
same accuracy as GMM and k-means for normally distributed data, while
it is superior for data that are not normally distributed.
Now that we have a precise mathematical formulation and also a working
clustering
algorithm, we will start to perform more consistent numerical tests.

\begin{figure}[h!]
\begin{cframed}
\centering
\begin{subfigure}[t]{0.45\textwidth}
\centering
\includegraphics[width=\textwidth]{../../figs/hist_gauss1.pdf}\\
\begin{tabular}{ l l l}
\hline
$\mathcal{E}$-clustering &  GMM~~ & $k$-means \\
\hline
0.823 &
0.798 &
0.826 \\
0.852 &
0.852 &
0.853 \\
0.842 &
0.842 &
0.839 \\
\hline
\end{tabular}
\end{subfigure}
\begin{subfigure}[t]{0.45\textwidth}
\centering
\includegraphics[width=\textwidth]{../../figs/circles.pdf}
\begin{tabular}{ l l l}
\hline
$\mathcal{E}$-clustering & GMM~~ & $k$-means \\
\hline
0.74 &
0.516 &
0.536 \\
0.706 &
0.516 &
0.503 \\
0.633 &
0.553 &
0.52 \\
\hline
\end{tabular}
\end{subfigure}
\caption{
(a) Two clusters in one dimension where
$x \sim \tfrac{1}{2}\left( \mathcal{N}(\mu_1, \Sigma_1) +
\mathcal{N}(\mu_2, \Sigma_2)\right)$, with $\mu_1 = 0$, $\mu_2=2$,
$\Sigma_1 = \Sigma_2 = 1$,
and $500$ points on each cluster.
(b) We generate two circles with random noise.
The radius of the circles are $1$ and $0.2$. The gaussian noise has size $0.2$.
We have 200 points on each class. The tables below each figure show
the accuracy, based on the true labels, for three experiments.
We compare our method with GMM and k-means. We can see that for gaussian
data the three of them achieve the same accuracy, which is close to Bayes
error, while for the data in
(b) energy statistics based clustering is superior.
}
\label{fig:nonpar}
\end{cframed}
\end{figure}
\clearpage

\end{document}

%%% Jan
%We developed a prototype algorithm for 2-dimensional shape clustering, which
%is invariant under affine transformations. This employs the procrustes
%distance between two objects, which requires feature extraction to obtain
%landmark points; see Fig.~\ref{fig:nonpar} for an example.
%We intend to apply a variant of this method to analyze neural synapses,
%since we have such datasets from our collaborators. However, to this particular
%dataset, the preprocessing techniques may be highly sophisticated, and a new
%metric to compare different objects may be necessary. Moreover, these shapes are
%3-dimensional. We are currently working on this project
%with our collaborators, extending our existing  techniques to this
%particular dataset.
%
%We also started working on a related, and more general, project.
%We intend to develop non-parametric clustering algorithms with statistical
%guarantees. We will use an energy-statistics based approach. Given
%two datasets $X$ and $Y$, there is an energy function $\mathcal{E}(X,Y)$
%test statistic which allows us to infer if $X$ and $Y$ have the same
%distribution. Our results thus far suggest that this can be written
%as a quadratic optimization problem
%with quadratic constraints:
%\begin{equation}
%\max_{x,z\in \mathbb{R}^N} x^T \Delta z \qquad \mbox{s.t. $x_i^2=1$, $x+z = 0$}
%\end{equation}
%where $\Delta$ is a dissimilarity data matrix.
%There is not enough literature on
%this interesting problem, so this will very likely lead to new methods which
%can have interesting applications, in particular to neuroscience datasets.
%
%\begin{figure}[h!]
%\begin{cframed}
%\centering
%\begin{subfigure}[t]{0.45\textwidth}
%\includegraphics[width=\textwidth]{../../figs/nonpar57.png}
%\label{fig:nonpar57}
%\caption{
%  MNIST digits with extracted landmarks and alignment.
%  }
%\end{subfigure}
%\begin{subfigure}[t]{0.45\textwidth}
%\includegraphics[width=\textwidth]{../../figs/nonPar357.png}
%\label{fig:nonpar357}
%\caption{
%Classification error against the size of each
%cluster (the three classes have the same number of points) is
%shown in blue. The red line is standard
%K-means with Euclidean distance for comparison.}
%\end{subfigure}
%\caption{
%  MNIST handwritten digits and classification error results.
%}
%\label{fig:nonpar}
%\end{cframed}
%\end{figure}
%\clearpage
%
%%%% Feb
%We have been mostly focused on developing
%non-parametric clustering methods.
%To this purpose, we are exploring ideas from energy statistics,
%which is non-parametric,
%robust, and rotational invariant, thus it incorporates the main ingredients
%that we are looking for. The main difficulty is to formulate an
%algorithm based on this, i.e. to identify the correct test
%statistic, or to formulate it as a feasible optimization problem.
%Consider $K$-Means clustering problem which is
%$
%\min_{\{\mathcal{C}_k\}}
%\sum_{k=1}^K \sum_{x \in \mathcal{C}_k} \| x - \mu_k  \|^2,
%$
%where $\mathcal{C}_k$ is the $k$th cluster and $\mu_k$ the mean of its points.
%We showed that this problem is equivalent to
%\begin{equation}
%\max_{G} \textnormal{Tr}\left( G^T K G \right) \qquad
%\mbox{s.t. \, $G \ge 0$, $G^T G=I$, $G G^T e_1 = e_1$}.
%\end{equation}
%where $e_1 = (1,1,\dotsc,1)^T$.
%This is a Quadratically Constrained Quadratic Problem (QCQP), which is usually
%NP-hard.
%Analogously, consider the energy function
%$\mathcal{E}(F,G) = 2\mathbb{E} \| X - Y\| - \mathbb{E}\| X - X'\| - \mathbb{E}
%\| Y - Y'\|$ between $X,X' \sim F$ and $Y,Y' \sim G$. We showed that this can
%be written as $\mathcal{E}(A,B) = e_1^T \Delta e_1 $, where $\Delta$ is a
%dissimilarity
%matrix between the two sets of data points
%$A \stackrel{iid}{\sim} F$
%and $B \stackrel{iid}{\sim} G$.
%Consequently, a simple two-class clustering problem would be
%\begin{equation}
%\max_{x,z\in \mathbb{R}^N} x^T \Delta z \qquad
%\mbox{s.t. $x_i^2=1$, $x+z = 0$},
%\end{equation}
%which is also a QCQP problem. We are currently investigating this problem and
%trying to generalize it correctly for more classes.
%A simple check of the energy function as a test statistic
%is shown in Fig.~\ref{fig:nonpar}.
%Under the null $F = G$,
%$T$ converges to a quadratic form of normally distributed random variables. This
%seems to be the case in the first (blue) histogram, while it is definitely not
%the case in the other (red and green) histograms.
%For the blue histogram a single test gives
%$T \approx 0.32$ (small), for the red
%histogram $T \approx 4000 $ (large), and for the green
%histogram $T \approx 105$ (large), with
%only a few points. Thus, energy statistics based approach is
%able to distinguish between
%different distributions, even when the clusters have the same mean, which is a
%property that $K$-Means cannot resolve.
%
%%
%\begin{figure}[h!]
%\begin{cframed}
%\centering
%%\begin{subfigure}[t]{0.45\textwidth}
%\includegraphics[width=\textwidth]{../../figs/energy_hists.pdf}
%%\label{fig:nonpar57}
%\caption{
%Distribution of test statistic
%$T\equiv\tfrac{n m }{n+m}\mathcal{E}(A,B)$
%for an ensamble obtained from two distributions:
%$A \stackrel{iid}{\sim} \mathcal{N}(\mu_A,\sigma_A^2)$ and
%$B \stackrel{iid}{\sim} \mathcal{N}(\mu_B, \sigma_B^2)$, where
%$|A|=n$ and $|B|=m$.
%Blue histogram: $\mu_A = \mu_B = 0$ and $\sigma_A = \sigma_B = 1$;
%Red histogram: $\mu_A = - \mu_B = 1$ and $\sigma_A = \sigma_B = 1$;
%Green histogram: $\mu_A = \mu_B = 0$, $\sigma_A = 1$ and $\sigma_B = 1.5$.
%}
%\label{fig:nonpar}
%\end{cframed}
%\end{figure}
%%
%\clearpage
%
%%% March
%Energy statistics provides a nonparametric test for equality of distributions.
%It is rotational invariant which is a highly desirable quality
%for clustering. For a two-class problem, $X,X' \sim \mu$ and
%$Y,Y' \sim \nu$, where $\mu,\nu$ are CDFs, it reads
%\begin{equation}\label{eq:energy}
%\mathcal{E}(X,Y) = 2\mathbb{E} \| X - Y \|
%- \mathbb{E} \| X - X' \| - \mathbb{E} \| Y - Y'\|.
%\end{equation}
%We are developing a clustering framework based on \ref{eq:energy}.
%Our criteria is that $\mathcal{E}$ should be a maximum when data points are
%correctly classified.
%%
%It is possible to show that there is a map from the data space of $X,Y$
%to the probability space of $\mu,\nu$ which is a Hilbert space whose
%inner product  can be obtained from a kernel
%function related to \ref{eq:energy},
%$\langle \mu, \nu \rangle
%= k(x,y)$.
%This enables us to formulate our clustering problem as follows:
%\begin{equation}\label{eq:opt}
%\textnormal{max} \left\{ \textnormal{Tr} L^{1/2} Z^T K Z L^{1/2} \right\}
%\quad
%\textnormal{s.t.} \quad
%\, Z_{ij}\in\{0,1\}, \sum_i Z_{ij} = N_j, \sum_j Z_{ij}=1,
%Z^T Z = L^{-1}
%\end{equation}
%where $N_j$ is the number of elements in the $j$th cluster,
%$L^{-1}=\textnormal{diag}(N_1,\dotsc,N_k)$, and $K$ is the Gram
%matrix obtained
%from the kernel. Let $\mathcal{X}$ be the pooled data matrix. If we
%replace $K \to \mathcal{X}^T \mathcal{X}$ we recover the well-known
%$k$-means problem, which in this formulation is related
%to spectral clustering and normalized cuts. Problem
%\ref{eq:opt} is
%NP-hard and a numerical implementation is prohibitive even
%for small data sets. We are investigating how to
%solve \ref{eq:opt} in
%a feasible way. As an evidence that \ref{eq:opt} is the correct optimization
%problem, and more importantly, it illustrates the power behind our proposal,
%in Fig.~\ref{fig:plots} we generate data and plot the objective in
%\ref{eq:opt} versus $n$, where $n$ is the number of points randomly shuffled
%from one class to the other. Therefore, for $n=0$ the function must be a
%maximum. We do this for the kernel related to \ref{eq:energy} (blue dots)
%and compare with the kernel related to $k$-means (red dots).
%Clearly, \ref{eq:opt} based on \ref{eq:energy} is able to distinguish
%between different cluster even for complex data sets that are
%not linearly separable. Moreover, in our formulation
%there are no free-parameters in the kernel.
%
%\begin{figure}[h!]
%\begin{cframed}
%\centering
%\begin{minipage}{0.49\textwidth}
%\includegraphics[scale=.37]{../../figs/plot1.pdf}
%\includegraphics[scale=.37]{../../figs/plot2.pdf}
%\end{minipage}
%\begin{minipage}{0.49\textwidth}
%\includegraphics[scale=.37]{../../figs/plot3.pdf}
%\includegraphics[scale=.37]{../../figs/plot4.pdf}
%\end{minipage}
%\caption{
%Two dimensional datasets and the objective function
%in \ref{eq:opt} as a function of $n$, where $n$ is the number of shuffled
%points from its correct class to the wrong class. Blue dots
%are for \ref{eq:energy} and red dots for $k$-means. A good function must
%be monotonically decreasing. We can clearly see that \ref{eq:energy} is
%way more powerful than $k$-means.
%}
%\label{fig:plots}
%\end{cframed}
%\end{figure}
%\clearpage
%\end{document}


\documentclass[12pt]{cdblatex}
\usepackage{eqtns}

\begin{document}

\section*{PhysRevD.62.044034 equation (18)}

\begin{cadabra}
   from shared import *
   import cdblib

   jsonfile = 'eqtn18.json'
   cdblib.create (jsonfile)

   # --------------------------------------------------------------------------
   # RBar pt.1 = split into two terms

   defGammaBar := GammaBar^{a}_{b c} ->
                  (1/2) gBar^{a e} (   \partial_{b}{gBar_{e c}}
                                     + \partial_{c}{gBar_{b e}}
                                     - \partial_{e}{gBar_{b c}}).

   defRiemBar := RBar^{a}_{b c d} ->
                 \partial_{c}{GammaBar^{a}_{b d}} + GammaBar^{a}_{e c} GammaBar^{e}_{b d}
               - \partial_{d}{GammaBar^{a}_{b c}} - GammaBar^{a}_{e d} GammaBar^{e}_{b c}.

   defRBar := RBar_{a b} -> RBar^{c}_{a c b}.

   RBar := RBar_{a b}.                                                  # cdb(eq18.000,RBar)

   substitute   (RBar, defRBar)                                         # cdb(eq18.001,RBar)
   substitute   (RBar, defRiemBar)                                      # cdb(eq18.002,RBar)
   substitute   (RBar, $GammaBar^{a}_{b a} -> 0$)                       # cdb(eq18.003,RBar)  # follows from det g = 1
   canonicalise (RBar)
\end{cadabra}

\clearpage

\begin{dgroup*}[spread=5pt]
   \begin{dmath*}
      \cdb{eq18.000}
         = \Cdb*{eq18.001}
         = \Cdb*{eq18.002}
         = \Cdb*{eq18.003}
   \end{dmath*}
\end{dgroup*}

From here the computations will be splt into two threads, one for each of the two terms in the above result.

\clearpage

\begin{cadabra}
   # --------------------------------------------------------------------------
   # get tmpA & tmpB from RBar

   GammaBar^{a}_{b c}::Weight(label=numG).
   \partial{#}::WeightInherit(label=all, type=multiplicative).

   tmpA := @(RBar).                                                     # cdb(tmp18.101,tmpA)
   keep_weight (tmpA, $numG=1$)                                         # cdb(tmp18.102,tmpA)  # the derivative terms

   tmpB := @(RBar).
   keep_weight (tmpB, $numG=2$)                                         # cdb(tmp18.103,tmpB)  # the quadrtaic terms
\end{cadabra}

\begin{dgroup*}
   \begin{dmath*} \cdb*{tmp18.101} \end{dmath*}
   \begin{dmath*} \cdb*{tmp18.102} \end{dmath*}
   \begin{dmath*} \cdb*{tmp18.103} \end{dmath*}
\end{dgroup*}

\clearpage

\begin{cadabra}
   # --------------------------------------------------------------------------
   # tmpA pt. 1

   substitute   (tmpA, defGammaBar)                                     # cdb(tmp18.201,tmpA)
   distribute   (tmpA)                                                  # cdb(tmp18.202,tmpA)
   product_rule (tmpA)                                                  # cdb(tmp18.203,tmpA)
   substitute   (tmpA, $\partial_{a}{gBar^{a b}} -> - GammaBar^{b}$)    # cdb(tmp18.204,tmpA)
\end{cadabra}

\begin{dgroup*}[spread=5pt]
   \begin{dmath*}
      \cdb{tmp18.102}
         = \Cdb*{tmp18.201}
         = \Cdb*{tmp18.202}
         = \Cdb*{tmp18.203}
         = \Cdb*{tmp18.204}
   \end{dmath*}
\end{dgroup*}

Notice that this result contains two terms containg second derivatives of $\bar g_{ij}$.
This pair of terms will now be replaced with an expression built from the first derivatives
of $\Gamma^{i}$.

\clearpage

\begin{cadabra}
   # --------------------------------------------------------------------------
   # tmpC

   defGi := GammaBar^{i} -> - \partial_{j}{gBar^{i j}}.

   # lower the indices on gBar^{b c}

   defLowerIndices := \partial_{a}{gBar^{b c}} -> - gBar^{i b} gBar^{j c} \partial_{a}{gBar_{i j}}.

   substitute (defGi, defLowerIndices)

   tmpC := gBar_{a i} \partial_{b}{GammaBar^{i}}
         + gBar_{b i} \partial_{a}{GammaBar^{i}}.                       # cdb(tmp18.301,tmpC)

   saveC := @(tmpC).

   substitute   (tmpC, defGi)                                           # cdb(tmp18.302,tmpC)
   product_rule (tmpC)                                                  # cdb(tmp18.303,tmpC)
   distribute   (tmpC)                                                  # cdb(tmp18.304,tmpC)
   canonicalise (tmpC)                                                  # cdb(tmp18.305,tmpC)
   substitute   (tmpC, $gBar_{a b} gBar^{b c} -> gBar_{a}^{c}$)         # cdb(tmp18.306,tmpC)
   eliminate_kronecker (tmpC)                                           # cdb(tmp18.307,tmpC)

   # foo is the target expression to be moved to the lhs

   foo := gBar^{i j} \partial_{a i}{gBar_{b j}}
        + gBar^{i j} \partial_{b i}{gBar_{a j}} -> X_{a b}.

   # bah helps when rebuilding the equation

   bah := X_{a b} ->
          gBar^{i j} \partial_{a i}{gBar_{b j}}
        + gBar^{i j} \partial_{b i}{gBar_{a j}}.

   substitute (tmpC, foo)                                               # cdb(tmp18.308,tmpC)

   # |\clearpage|
   # rearrange to move the target to the lhs

   tmpE := @(tmpC).
   tmpF := @(tmpC).

   X_{a b}::Weight(label=numX).

   # get the two pieces of the equation
   keep_weight (tmpE, $numX=0$)                                         # cdb(tmp18.309,tmpE)
   keep_weight (tmpF, $numX=1$)                                         # cdb(tmp18.310,tmpF)

   substitute  (tmpF, bah)                                              # cdb(tmp18.311,tmpF)

   # now rebuild with terms reorderd
   tmpG := @(saveC) - @(tmpE).                                          # cdb(tmp18.312,tmpG)

   defTmpSub := @(tmpF) -> @(tmpG).                                     # cdb(tmp18.313,defTmpSub)
\end{cadabra}

\clearpage

\begin{dgroup*}[spread=5pt]
   \begin{dmath*}
      \cdb{tmp18.301}
         = \Cdb*{tmp18.302}
         = \Cdb*{tmp18.303}
         = \Cdb*[\hskip2cm\hfill]{tmp18.304}
         = \Cdb*[\hskip2cm\hfill]{tmp18.305}
         = \Cdb*{tmp18.306}
         = \Cdb*{tmp18.307}
         = \Cdb*{tmp18.308}
   \end{dmath*}
\end{dgroup*}

\begin{dgroup*}[spread=5pt]
   \begin{dmath*}
      \cdb{tmp18.311}
         = \Cdb*{tmp18.310}
         = \Cdb*{tmp18.312}
   \end{dmath*}
\end{dgroup*}

This result will now be applied to the earlier equation {\tt (tmp18.204)}.

\clearpage

\begin{cadabra}
   # --------------------------------------------------------------------------
   # tmpA pt.2 eliminate second partial derivatives of gBar

   canonicalise   (tmpA)                                                # cdb(tmp18.401,tmpA)

   substitute     (tmpA, defTmpSub)                                     # cdb(tmp18.402,tmpA)
   tmpA = product_sort (tmpA)
   rename_dummies (tmpA)
   canonicalise   (tmpA)                                                # cdb(tmp18.403,tmpA)

   foo := gBar^{d e} \partial_{c}{gBar_{e f}} -> - gBar_{e f} \partial_{c}{gBar^{d e}}.
   bah := \partial_{d}{gBar^{d f}} -> - GammaBar^{f}.

   substitute     (tmpA, foo)                                           # cdb(tmp18.404,tmpA)
   substitute     (tmpA, bah)                                           # cdb(tmp18.405,tmpA)

   foo := gBar_{e f} \partial_{a}{gBar^{c f}}  -> - \partial_{a}{gBar_{e f}} gBar^{c f}.

   substitute     (tmpA, foo)                                           # cdb(tmp18.406,tmpA)

   foo := gBar_{b d} gBar^{d e}  -> gBar_{b}^{e}.

   substitute     (tmpA, foo)                                           # cdb(tmp18.407,tmpA)
   eliminate_kronecker (tmpA)                                           # cdb(tmp18.408,tmpA)
   tmpA = product_sort (tmpA)
   rename_dummies (tmpA)
   canonicalise   (tmpA)                                                # cdb(tmp18.409,tmpA)
\end{cadabra}

\clearpage

\begin{dgroup*}[spread=5pt]
   \begin{dmath*}
      \cdb{tmp18.102}
         = \Cdb*{tmp18.401}
         = \Cdb*[\hskip2cm\hfill]{tmp18.402}
         = \Cdb*[\hskip2cm\hfill]{tmp18.403}
         = \Cdb*[\hskip2cm\hfill]{tmp18.404}
         = \Cdb*[\hskip2cm\hfill]{tmp18.405}
         = \Cdb*[\hskip2cm\hfill]{tmp18.406}
         = \Cdb*[\hskip2.5cm\hfill]{tmp18.407}
         = \Cdb*[\hskip2.5cm\hfill]{tmp18.408}
         = \Cdb*{tmp18.409}
   \end{dmath*}
\end{dgroup*}

\clearpage

\begin{cadabra}
   # --------------------------------------------------------------------------
   # tmpB

   substitute     (tmpB, defGammaBar)                                   # cdb(tmp18.501,tmpB)
   distribute     (tmpB)                                                # cdb(tmp18.502,tmpB)
   tmpB = product_sort (tmpB)                                           # cdb(tmp18.503,tmpB)
   rename_dummies (tmpB)                                                # cdb(tmp18.504,tmpB)
   canonicalise   (tmpB)                                                # cdb(tmp18.505,tmpB)
\end{cadabra}

\begin{dgroup*}[spread=5pt]
   \begin{dmath*}
      \cdb{tmp18.103}
         = \Cdb*{tmp18.501}
         = \Cdb*{tmp18.502}
         = \Cdb*{tmp18.503}
         = \Cdb*{tmp18.504}
         = \Cdb*{tmp18.505}
   \end{dmath*}
\end{dgroup*}

\clearpage

\begin{cadabra}
   # --------------------------------------------------------------------------
   # RBar pt.2 = Rebuild Rab from tmpA and tmpB

   RBar := @(tmpA) + @(tmpB).

   canonicalise   (RBar)                                                # cdb(eq18.601,RBar)

   foo := \partial_{a}{gBar^{c d}} -> - gBar^{c i} gBar^{d j} \partial_{a}{gBar_{i j}}.

   substitute     (RBar, foo)
   distribute     (RBar)
   RBar = product_sort (RBar)
   rename_dummies (RBar)
   canonicalise   (RBar)                                               # cdb(eq18.602,RBar)

   foo := \partial_{a}{gBar_{b c}} -> GammaBar_{b c a} + GammaBar_{c b a}.

   substitute     (RBar, foo)                                          # cdb(eq18.603,RBar)
   distribute     (RBar)
   RBar = product_sort (RBar)
   rename_dummies (RBar)
   canonicalise   (RBar)                                               # cdb(eq18.604,RBar)

   foo := GammaBar_{d e f} gBar^{d e} -> 0.

   substitute     (RBar, foo)                                          # cdb(eq18.605,RBar)

   defRab := RBar_{a b} -> @(RBar).
\end{cadabra}

\clearpage

\begin{dgroup*}[spread=5pt]
   \begin{dmath*}
      \cdb{eq18.000}
         = \Cdb*{eq18.003}
         = \Cdb*[\hskip2cm\hfill]{eq18.601}
         = \Cdb*{eq18.602}
         = \Cdb*{eq18.603}
         = \Cdb*{eq18.604}
         = \Cdb*{eq18.605}
   \end{dmath*}
\end{dgroup*}

\clearpage

\begin{cadabra}
   # --------------------------------------------------------------------------
   # Check against prd62.

   foo := @(RBar).                                      # cdb(eq18.lcb,foo)
   bah  = cdblib.get('prd62.eq18.rhs','prd62.json')     # cdb(eq18.prd,bah)

   diff := @(foo) - @(bah).

   distribute     (diff)
   diff = product_sort (diff)
   rename_dummies (diff)
   map_sympy      (diff, "simplify")
   canonicalise   (diff)                                # cdb(eq18.chk,diff)
\end{cadabra}

% \clearpage

\begin{dgroup*}
   \begin{dmath*} \cdb*{eq18.lcb} \end{dmath*}
   \begin{dmath*} \cdb*{eq18.prd} \end{dmath*}
   \begin{dmath*} \cdb*{eq18.chk} \end{dmath*}
\end{dgroup*}

\end{document}


\documentclass{article}
\usepackage[hmargin=1in,vmargin=1.5in]{geometry}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{bm}
\newcommand{\A}{\bm A}
\newcommand{\x}{\bm x}
\newcommand{\y}{\bm y}
\newcommand{\z}{\bm z}
\renewcommand{\a}{\bm a}
\renewcommand{\b}{\bm b}
\renewcommand{\c}{\bm c}
\renewcommand{\d}{\bm d}
\renewcommand{\v}{\bm v}
\newcommand{\p}{\bm p}
\newcommand{\q}{\bm q}
\newcommand{\1}{\bm 1}
\title{Homework 2}
\author{Xinyi Gu, Songchen Tan}
\date{\today}
\begin{document}
\maketitle
\section{}
When we convert the polyhedron in the geometric form $\A\x\ge\b$ to standard form $\A'\x'=\b',\x'\ge0$, we will add a new variable for each inequality, therefore the extreme point in the standard form is not related to the extreme point in the geometric form. For example, $1x_1+0x_2\ge0$ is a valid polyhedron in $\mathbb R^2$ with no extreme points, and its corresponding standard form $x_1^+-x_1^--x_3=0, \x'=(x_1^+,x_1^-,x_2^+,x_2^-,x_3)\ge0$ has an extreme point $(0, 0, 0, 0, 0)$, which does not correspond to an extreme point in the geometric form.

\section{}

Dual problem:

\begin{align*}
    \max \quad & 3p_2 + 6 p_3  \\
    \text{s.t.} \quad & p_1 \leq 0 \\
    & p_2 \geq 0\\
    & p_3 \quad \text{free}\\
    & 2p_1 + 3p_2 - p_3 \geq 1 \\
    & 3p_1 + p_2 - p_3 \leq -1 \\
    & -p_1 + 4p_2 + 2p_3 \leq 0 \\
    & p_1 - 2p_2 + p_3 = 0
\end{align*}

\section{}
We denote $z=(x',y')'$, $v=(c',d')'$ such that we can reformulate the problem into minimizing $v'z$ subject to $(A, B)z=b$, $(D, 0)z\le f$ and $(0, G)z\le g$.

The dual problem is therefore maximizing $p'b+q'f+r'g$ where $p\in\mathbb R^{p_1}, q\in\mathbb R^{p_2}, r\in\mathbb R^{p_3}$, subject to

$$
\begin{cases}
    q\le 0\\
    r\le 0\\
    p'A+q'D=c\\
    p'B+r'G=d
\end{cases}
$$

Using the given data, we obtained that the optimal solution $\approx -236.41$ for both the primal and dual problem. (See code \verb|3.jl|.)

\section{}
\subsection*{(a)}

We denote the polynomial coefficients $a_0, \cdots, a_d$, so that the minimization problem is defined as

$$
\min_{a_i, i = 0,\cdots, d}\max_{j=1,\cdots,N}|f(x_j) - (a_0+a_1x_j^1+\cdots+a_dx_j^d)|
$$

Given $a_i$, let $m>0$ be the maximum, then we can express the problem as a LO over variables $\{a_i|i=0,\cdots,d\} \cup \{m\}$ that minimizes $m$ subject to

$$
\begin{cases}
    m \ge 0\\
    m \ge + f(x_j) - (a_0 + a_1x_j^1+\cdots+a_dx_j^d) & \forall j=1, \cdots, N\\
    m \ge - f(x_j) + (a_0 + a_1x_j^1+\cdots+a_dx_j^d) & \forall j=1, \cdots, N\\
\end{cases}
$$

\subsection*{(b)}
Dual problem:

Two primal constraints in (a), set dual variables as $p^{+}_j$ and $p^{-}_j$.

\begin{align*}
    \max \quad & \sum^{N}_{j=1} f(x_j)(p^{+}_j - p^{-}_j) & \\
    \text{s.t.} \quad & \sum^{N}_{j=1}(p^{+}_j + p^{-}_j) = 1 \\
    & \sum^{N}_{j=1}x^{i}_j (p^{+}_j - p^{-}_j) = 0, & i = 0, 1, \cdots, d \\
    & p^{+}_j, p^{-}_j \geq 0, & j = 1, \cdots, N
\end{align*}

\subsection*{(c)}
complementary slackness conditions:

$$
\begin{cases}
    (m - f(x_j) + (a_0 + a_1x_j^1+\cdots+a_dx_j^d)) p^{+}_j = 0 & \forall j=1, \cdots, N\\
    (m + f(x_j) - (a_0 + a_1x_j^1+\cdots+a_dx_j^d)) p^{-}_j = 0 & \forall j=1, \cdots, N\\
\end{cases}
$$

When $p^{+}_j, p^{-}_j > 0$, $m = |f(x_j) - p(x_j)|$, get maximum approximation error when $x_j$. From (b), we know that there are d+2 constraints in dual problem, results at most d+2 positive valunes in the optimal basic solutionmeans, means at most d+2 $x_j$ points get the maximum approximation error.

\subsection*{(d)}

The approximation Chebyshev polynomials and the approximation errors are shown below. See Fig. 1.

\begin{figure}[!ht]
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
        \centering
        \includegraphics[width=\textwidth]{polynomial.png}
        \caption{Polynomials}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \centering
        \includegraphics[width=\textwidth]{error.png}
        \caption{Errors}
    \end{subfigure}
       \caption{The Chebyshev polynomial and error for $d=0,1,2,3,4$}
\end{figure}

\subsection*{(e)}

We notice that the Chebyshev polynomials are visually closer to the original function $e^x$ in the given interval $[-1, 1]$. This is because Chebyshev polynomials consider all points in the interval and Taylor polynomials only consider the function value and derivatives at 0. See Fig. 2 in P4.

\begin{figure}[!ht]
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
        \centering
        \includegraphics[width=\textwidth]{compare-0.png}
        \caption{$\deg p=0$}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \centering
        \includegraphics[width=\textwidth]{compare-1.png}
        \caption{$\deg p=1$}
    \end{subfigure}
    \newline
    \begin{subfigure}[b]{0.48\textwidth}
        \centering
        \includegraphics[width=\textwidth]{compare-2.png}
        \caption{$\deg p=2$}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.48\textwidth}
        \centering
        \includegraphics[width=\textwidth]{compare-3.png}
        \caption{$\deg p=3$}
    \end{subfigure}
    \newline
    \begin{subfigure}[b]{0.48\textwidth}
        \centering
        \includegraphics[width=\textwidth]{compare-4.png}
        \caption{$\deg p=4$}
    \end{subfigure}
    \caption{Comparison of Chebyshev polynomials and Taylor polynomials}
\end{figure}

\section{}

We start the proof by showing that (b) $\Rightarrow$ (a) is easy. If (b) holds, for all $\x>0$,

$$
\a^T\x=\sum_{j=1}^na_jx_j
\le\sum_{j=1}^n\sum_{i=1}^m\lambda_i(\a_{i})_j
=\sum_{i=1}^m\lambda_i(\a_i)^T\x\le\max_i(\a_i)^T\x
$$

Then we consider the LO problem: $\min z-\a^T\x$, subject to $(\a_i)^Tx-z\le 0$ ($i=1,\cdots,m$) and $x\ge0$. If (a) holds, then $z\ge\max_i(\a_i)^T\x\ge\a^Tx$, so the objective function is bounded. This problem is also feasible since $z$ is free. Now we denote $\y=-\x$ and define the resulting formulation as the primal problem: $\min \a^T\y+z$, subject to $(\a_i)^Ty+z\ge 0$ and $\y\le 0$. The dual of this problem would be $\max 0$, subject to

$$
\begin{cases}
(\lambda_1,\cdots,\lambda_m)\begin{pmatrix}
    \a_1^T\\\a_2^T\\\vdots\\\a_m^T
\end{pmatrix}
\ge\a^T\\
\sum_{i=1}^m\lambda_i=1\\
\lambda_i\ge0
\end{cases}
$$

According to the duality theorem, this problem is feasible, so (b) holds.

\end{document}