text:
\section{Introduction}
    Many machine learning applications involve fitting a probability distribution over a vector of observations
    ${\bm{x}}$. This is often accomplished by specifying a parametric family of probability distributions indexed
    by a parameter vector ${\bm{\theta}}$.
    The resulting $p_\text{model}({\bm{x}}; {\bm{\theta}})$ can be approximately
    matched to the reference distribution $p_\text{data}({\bm{x}})$ by using a statistical estimator to find
    a good value of ${\bm{\theta}}$.

    Maximum likelihood estimation (MLE) is the most popular statistical estimator used to accomplish this task.
    Maximum likelihood estimation works by maximizing the probability of the observed data according to the model.
    For several models of interest, exact MLE is intractable, and must be approximated using techniques such
    as Markov chain Monte Carlo methods. As a result, one popular avenue of research is the design of alternative
    statistical estimators that have lower computation cost.

    Two recently proposed methods both employ a binary classifier that attempts to discern whether a given ${\bm{x}}$
    was drawn from the training data or sampled from a ``generator'' distribution. Both methods are primarily
    driven by a function we call the {\em distinguishability game value function}:

    \[ V(p_c, p_g) = %
        \mathbb{E}_{{\bm{x}} \sim p_d} \log p_c(y=1 \mid {\bm{x}}) +
\mathbb{E}_{{\bm{x}} \sim p_g} \log p_c(y=0 \mid {\bm{x}})  \]
    where $p_g({\bm{x}})$ is a distribution representing a source of ``fake'' samples, $p_d({\bm{x}})$ is the distribution
    over the training data (which we can usually access only approximately via an emprical distribution),
    and $p_c(y = 1 \mid {\bm{x}})$ is the classifier's probability that ${\bm{x}}$ comes from $p_d$ rather than $p_g$.

    The distinguishability game value function is (proportional to) the log likelihood of the classifier
    on the binary class labeling problem, where the training examples for the classifier are drawn from a uniform mixture of
    two components: real data labeled with $y=1$ and ``fake'' samples labeled with $y=0$.

    Of the two recently proposed methods, the first is {\em noise-contrastive estimation}~\citep{nce}.
    Noise-contrastive estimation (NCE) uses an arbitrary, fixed ``noise'' distribution for $p_g$.
    The goal of NCE is to learn a model $p_m({\bm{x}})$ that is used to define the classifier:
    \begin{equation}
        p_c(y=1\mid {\bm{x}}) = \frac{ p_m({\bm{x}}) } { p_m({\bm{x}}) + p_g({\bm{x}}) }.
            \label{eq:nce_c}
        \end{equation}
    Learning then proceeds by using a standard optimization algorithm to maximize $V$.

    More recently, the generative adversarial network (GAN) framework introduced a different approach
    using the same value function~\citep{gan}. Here, the goal is to learn $p_g$, and there is no
    explicit $p_m$. Instead, $p_c$ is parameterized directly. Rather than treating $V$ as an objective
    function to maximize, it is used to define a minimax game, with $p_g$ trained to minimize the
    objective and $p_c$ trained to maximize it.

    MLE, NCE, and GANs are all asymptotically consistent, which means that in the limit of
    infinitely many samples from $p_d$ being available, their criteria each have a unique stationary
    point that corresponds to the learned distribution matching the data distribution\footnote{
    By ``match'' we mean more formally that $p_d({\bm{x}}) = p_m({\bm{x}})$ for all ${\bm{x}}$ except a set of measure zero.
    }.
    For MLE and NCE, this stationary point is a global maximum of their  objective function, while
    for GANs it corresponds to a saddle point that is a local maximum for the classifier and a local
    minimum for the generator. Asymptotic consistency is proven in the space of unconstrained
    probability distribution functions; when we move to specific parametric families of distributions
    a variety of caveats apply: the given function family may not include the true training data distribution,
    different parameter values may encode the same function and thus introduce identifiability issues, and
    the optimizer may fail to find the global optimum.

    Because GANs and NCE both use the same value function, it is natural to wonder how they are related
    to each other, and to maximum likelihood. In this paper, we provide some initial answers to each
    question. We show

\begin{itemize}
    \item A modified version of NCE with a dynamic generator is equivalent to MLE.
    \item The existing theoretical work on GANs does not guarantee convergence on
        practical applications.
    \item Because GANs do the model estimation in the generator network, they
        cannot recover maximum likelihood using $V$.
\end{itemize}

Throughout this paper, we make some weak regularity assumptions.
Specifically, we will assume that all of our models, be they $p_m$ or $p_g$, parameterize
the probability distribution such that
$p({\bm{x}}) > 0$ for all ${\bm{x}}$.
In the case of continuous random variables,
we additionally assume that
that $p$ and $\nabla_{{\bm{\theta}}} p$ are continuous at all ${\bm{x}}$ and ${\bm{\theta}}$ points.

\section{Self-contrastive estimation}

The performance of NCE is highly dependent on the choice of noise distribution.
It is not difficult to discriminate data samples from totally unstructured noise,
so models trained with too simplistic of a noise distribution often underfit badly.
This has
motivated a variety of heuristic attempts to design better noise distributions.
~\citet{nce} suggest
that ``one could choose a noise distribution by first estimating a preliminary model
of the data, and then use this preliminary model as the noise distribution.''

Let use consider the extreme version of this approach, where the model is copied and used
as the new noise distribution after every step of learning. We call this approach
{\em self-contrastive estimation}. Here we show that self-contrastive estimation has the same expected gradient as
maximum likelihood estimation.

Let $\theta$ be a parameter of $p_m$. Then
\begin{equation}
    \frac{\partial}{\partial \theta} V(p_c, p_g)  = \frac{\partial}{\partial \theta}
    \left[
\mathbb{E}_{{\bm{x}} \sim p_d} \log p_c(y=1 \mid {\bm{x}}) +
\mathbb{E}_{{\bm{x}} \sim p_g} \log p_c(y=0 \mid {\bm{x}})
\right]
\label{eq:foo}
\end{equation}

Recall from Eq.~\ref{eq:nce_c} that in NCE, the classifier is defined by
\[ p_c(y= 1\ \mid {\bm{x}}) = \frac{p_m({\bm{x}})} {p_m({\bm{x}}) + p_g({\bm{x}}) }. \]
In the context of SCE, we have copied $p_m$ into $p_g$ before each step of learning.
It may therefore be tempting to make the assumption that $p_m = p_g$ and simplify
the classifier to $p_c(y = 1 \mid {\bm{x}}) = \frac{1}{2}$. However, this is incorrect,
because we have made a deep copy, rather than aliasing $p_g$ to $p_m$. It is crucial
that $p_g$ is not dependent on $\theta$; therefore when calculating derivatives
we must consider the effect of $\theta$ on $p_m$ but not $p_g$.
Substituting Eq.~\ref{eq:nce_c} into Eq.~\ref{eq:foo} we obtain
\[
    \frac{\partial}{\partial \theta} V(p_c, p_g)  = \frac{\partial}{\partial \theta}
    \left[
    \mathbb{E}_{{\bm{x}} \sim p_d} \log \frac{p_m({\bm{x}})}{p_m({\bm{x}})+p_g({\bm{x}})} +
    \mathbb{E}_{{\bm{x}} \sim p_g} \log \frac{p_g({\bm{x}})}{p_m({\bm{x}})+p_g({\bm{x}})}
\right]
\]
\[
\ = \frac{\partial}{\partial \theta}
\left[
\mathbb{E}_{{\bm{x}} \sim p_d} \left[
    \log p_m({\bm{x}}) - \log\left( p_m({\bm{x}})+p_g({\bm{x}}) \right)
\right]+
\mathbb{E}_{{\bm{x}} \sim p_g} \left[
    \log p_g({\bm{x}}) - \log\left(p_m({\bm{x}})+p_g({\bm{x}})\right)
    \right]
\right]
\]
\[
 = \frac{\partial}{\partial \theta}
\left[
\mathbb{E}_{{\bm{x}} \sim p_d} \left[
    \log p_m({\bm{x}}) - \log\left( p_m({\bm{x}})+p_g({\bm{x}}) \right)
\right]
- \mathbb{E}_{{\bm{x}} \sim p_g}  \log\left(p_m({\bm{x}})+p_g({\bm{x}})\right)
\right]
\]
\begin{equation}
 =
\mathbb{E}_{{\bm{x}} \sim p_d}
\frac{\partial}{\partial \theta}
\left[
    \log p_m({\bm{x}}) - \log\left( p_m({\bm{x}})+p_g({\bm{x}}) \right)
\right]
-
\mathbb{E}_{{\bm{x}} \sim p_g}
\frac{\partial}{\partial \theta}
\log\left(p_m({\bm{x}})+p_g({\bm{x}})\right)
\label{eq:bar}
\end{equation}

We can show that the term on the right vanishes, as follows.
\[
\mathbb{E}_{{\bm{x}} \sim p_g}
\frac{\partial}{\partial \theta}
\log\left(p_m({\bm{x}})+p_g({\bm{x}})\right)
\]
\[
=
\mathbb{E}_{{\bm{x}} \sim p_g}
\frac{
    \frac{\partial}{\partial \theta}
    p_m({\bm{x}})
}
{
p_m({\bm{x}})+p_g({\bm{x}})
}.
\]
Because we have assumed our distributions are strictly positive, we
can use the trick $\frac{\partial}{\partial \theta} p({\bm{x}})
= \frac{\partial}{\partial \theta} \exp \left( \log p({\bm{x}}) \right)
= p({\bm{x}}) \frac{\partial}{\partial \theta} \log p({\bm{x}})$:
\[
\mathbb{E}_{{\bm{x}} \sim p_g}
\frac{
    \frac{\partial}{\partial \theta}
    p_m({\bm{x}})
}
{
p_m({\bm{x}})+p_g({\bm{x}})
}
=
\mathbb{E}_{{\bm{x}} \sim p_g}
\frac{
    p_m({\bm{x}})
    \frac{\partial}{\partial \theta}
    \log p_m({\bm{x}})
}
{
p_m({\bm{x}})+p_g({\bm{x}})
}.
\]
Crucially, outside of the differentiation sign, we {\em are} allowed to exploit the
fact that $p_m({\bm{x}})$ and $p_g({\bm{x}})$ are equal in order to simplify $p_m / (p_m + p_g)$ into $\frac{1}{2}$.
We are left with
\[
    \frac{1}{2} \mathbb{E}_{{\bm{x}} \sim p_g} \log p_g({\bm{x}}).
\]
We can use the inverse of the $\exp(\log p({\bm{x}}))$ trick to observe that this is equal to
$\frac{\partial} {\partial \theta} \sum_{\bm{x}} p_g({\bm{x}})
= \frac{\partial} {\partial \theta} 1 = 0$. (In the continuous case one must use an integral
rather than a summation)

Plugging this result in Eq.~\ref{eq:bar}, we obtain:
\[
    \frac{\partial}{\partial \theta} V(p_c, p_g)  =
\mathbb{E}_{{\bm{x}} \sim p_d}
\frac{\partial}{\partial \theta}
\left[
    \log p_m({\bm{x}}) - \log\left( p_m({\bm{x}})+p_g({\bm{x}}) \right)
\right]
\]
Using the same tricks as previously, we can simplify this to
\[
    \frac{1}{2} \mathbb{E}_{{\bm{x}} \sim p_d}
\frac{\partial}{\partial \theta}
\log p_m({\bm{x}}).
\]
This is $\frac{1}{2}$ the log likelihood gradient, and the $\frac{1}{2}$ can of course be folded
into the learning rate.

Note that while the gradients are equivalent, the objective functions are not.
MLE maximizes a single objective function, while SCE changes the objective function at every step.
The MLE gradient for a specific value of $\bm{\theta}$ always matches the gradient at $\bm{\theta}$
of a different SCE objective constructed specifically for that $\bm{\theta}$. The value of the
SCE objective does not change over time; each new objective function always has value $-2\log 2$ as
a consequence of the model distribution never being distinguishable from the noise distribution.

\section{Interpreting the theory of GANs}

One can prove the asymptotic consistency of MLE, NCE, and GANs by observing that
each minimizes a convex divergence in function space. In practice, one cannot
optimize directly over probability distribution functions. Instead, one must optimize
over parameters indexing a parametric family of functions. Fortunately, estimators
that are consistent in function space often prove to perform well in parameter space.

In the case of GANs, a subtlety of the theory may have important implications for
the behavior of the algorithm in practice. The existing theory justifying GANs uses
convexity in function space to prove not only asymptotic consistency but also convergence.

Estimating a model by maximimizing the MLE or NCE objective function is guaranteed to
converge for smooth functions that are bounded from above regardless of whether these
objective functions are convex. It is possible for optimization to get stuck in a local
maximum in parameter space, but the optimization process will at least arrive at some
critical point.

In the case of GANs, the generator is updated while holding the discriminator fixed,
and vice versa. In function space this corresponds to performing subgradient descent
on a convex problem, and is guaranteed to converge.

In the non-convex case, the existing theory does not specify what will happen.

To reach the equilibrium point, $p_g$ should be trained to minimize
    $\max_{p_c} V(p_c, p_d)$. Instead, it takes successive steps partially
    minimizing $V(p_c, p_d)$ using the current value of $p_c$ at each step.
    Because by definition $V(p_c, p_d) \leq \max_{p_c} V(p_c, p_d)$ this
    corresponds to taking steps that partially {\em minimize a lower bound}.
    In the non-convex case, it is conceivable that this could merely make
    the bound looser rather than decrease the underlying objective function
    as desired. This may result in the learning procedure oscillating rather
    than converging.

    Such a process could explain the underfitting that has
    been observed with GANs thus far. No deep generative model has been
    demonstrated to be able to memorize a rich, complicated training set.
    For many models, this could be explained by inaccuracies in the approximation
    of the gradient or too strong of simplifying assumptions for variational
    learning. For GANs, the failure to memorize the training set is surprising
    because the gradient can be computed with backpropagation and there are no
    variational approximations. Non-convergence of gradient-based learning
    for continuous games stands out as a candidate explanation for why this happens.

    To be clear, we do not have any positive results identifying non-convergence as the problem.
    We are merely identifying one way in which the existing theoretical results for
    GANs {\em fail to guarantee} good performance in practice. We suggest that future
    work could attempt to positively identify non-convergence in GAN learning or
    apply better algorithms for computing the equilibrium of the game.

\section{GANs cannot implement maximum likelihood}

GANs work by learning in the {\em generator}, while NCE works by learning in the discriminator
(via a generative model that is used to implicitly define the generator).
This turns out to have important results for learning. Namely, each step of learning in a GAN pair consists
of decreasing an expectation of a function of samples from the generator:
\[
\mathbb{E}_{{\bm{x}} \sim p_g} f({\bm{x}})
\]
where $f({\bm{x}}) = \log p_c\left(y=0 \mid {\bm{x}} \right)$.

For a parameter $\theta$ of $p_g$, we find
\[
\frac{\partial}{\partial \theta} \mathbb{E}_{{\bm{x}} \sim p_g} f({\bm{x}})
\]
\[
    = \int f({\bm{x}}) \frac{\partial}{\partial \theta} p_g({\bm{x}})
\]
\[
    = \int f({\bm{x}}) p_g({\bm{x}}) \frac{\partial} {\partial \theta} \log p_g({\bm{x}}).
\]

From this vantage point it is clear that to obtain the maximum likelihood derivatives, we need
\[
    f({\bm{x}}) = - \frac{p_d({\bm{x}})} {p_g({\bm{x}})}.
\]
(We could also add an arbitrary constant and still obtain the correct result)
Suppose our discriminator is given by $p_c(y=1 \mid {\bm{x}}) = \sigma\left( a({\bm{x}}) \right)$ where $\sigma$ is the
logistic sigmoid function. Suppose further that our discriminator has converged to its optimal value for
the current generator,
\[ p_c(y = 1 \mid {\bm{x}}) = \frac{ p_d({\bm{x}}) } {p_g({\bm{x}}) + p_d({\bm{x}})}.\]
Then $f(x) = - \exp\left( a( {\bm{x}} ) \right)$.
This is clearly different from the value given by the distinguishability game, which simplifies to
$f({\bm{x}}) = -\zeta\left( a( {\bm{x}} \right))$, where $\zeta$ is the softplus function. See this function
plotted alongside the GAN cost in Fig~\ref{fig:cost_diff}.

\begin{figure}
  \centering
  \includegraphics[width=\textwidth]{cost.pdf}
  \caption{The cost the generator pays for sampling a point is a function of the optimal discriminator's
    output for that point. This much is true for maximum likelihood, the minimax formulation
    of the distinguishability game, and a heuristic reformulation used in most experiments by
    ~\citet{gan}. Where the methods differ is the exact value of that cost. As we can see, the
    cost for maximum likelihood only has significant gradient through the discriminator if the
    discriminator is fooled with high confidence. Since this is an extremely rare event when sampling
    from an untrained generator, estimates
    of the maximum likelihood gradient based on this approach have high variance.
  }
  \label{fig:cost_diff}
\end{figure}

In other words, the discriminator gives us the necessary information to compute the maximum likelihood
gradient of the generator, but it requires that we abandon the distinguishability game.
In practice, the estimator based on $\exp \left( a({\bm{x}}) \right)$ has too high of variance.
For an untrained model, sampling from the generator almost always yields very low values of $\frac{p_d({\bm{x}})}{p_g({\bm{x}})}.$
The value of the expectation is dominated by the rare cases where the generator manages to sample
something that resembles the data by chance. Empirically, GANs have been able to overcome this problem but
it is not entirely clear why. Further study is needed to understand exactly what tradeoff GANs are
making.

\section{Discussion}

Our analysis has shown a close relationship between noise contrastive estimation and maximum likelihood.
We can now interpret noise-contrastive estimation as being a one-sided version of a distinguishability
game.
Our analysis has also shown
that generative adversarial networks are not as closely related to noise contrastive estimation as
previously believed. The fact that the primary model is the generator turns out to result in a departure
from maximum likelihood even in situations where NCE and MLE are equivalent. Further study is needed to
understand exactly what tradeoff is incurred when using adversarial learning.
Finally, the problem of non-convergence of independent SGD in
the non-convex case may explain the underfitting observed in GANs and suggests the application of better
algorithms for solving for the equilibrium strategies of the distinguishability game.

\subsubsection*{Acknowledgments}
We would like to thank Andrew Dai, Greg Corrado, David Sussillo, Jeff Dean
and David Warde-Farley
for their feedback on drafts of this article.

\small
\setlength{\bibsep}{5pt plus 0.3ex}

redpajama_set_name: RedPajamaArXiv


Today is filled with kindergarten drop-off, Bible study, kindergarten pick-up, and a trip to the Richmond airport to meet my sister and niece. I was hunting for a post to share with you today, and realized that I never posted this Paper Towel Roll Party Favor that I shared last Fall at Infarrantly Creative. Great for birthday party goodies, you could easily customize it to fit any holiday theme! I love thrifty party favors.
Today's craft starts with a paper towel roll.
(Crickets chirping) Did I lose you yet? I promise it gets better!
I laid the roll on some 12 x 12 scrapbook paper and cut it to fit the length and enough to go around and just barely overlap on the seams. Some good old Elmer's glue held the paper down just fine, but I placed a strip of Scotch tape on the bottom seam for extra hold.
I inked the edges of the tube with blue ink to coordinate with the paper and then hot glued one end shut.
After filling the favor with treats, I hot glued the other end shut and got to work on some embellishments. I lost my acrylic crown stamp, so I had to draw that on. If you are organized, you could just not lose your stamp and use it, unlike some people.
I closed the other end a different way to see which look I liked better. You can decide for yourself.
I had some solid colored papers, clear acrylic stamps, and gift ribbon in my craft stash, so I used those to layer and make a little extra "fluff" for the favor.
Again, these would be great for just about any holiday party. Preschoolers everywhere would love you, and you can't beat the price! These didn't cost me a dime.
What a great idea! Super cute and inexpensive...gotta love it!!
Love it, wonderful way to recycle!
Great use of materials most of us have around the house. I'm going to share this with my sister- her kids would love it!
What a cute and easy idea!!
I'd forgotten all about this! I love this project. Party favors are wonderful, I think. My boys always opted to have inexpensive parties at home and provide good favors for their friends rather than having expensive parties elsewhere. They would have loved these.
I'm thinking that the "mini" size candy bars would be great filler for these (not fun size, but the smaller ones). Those will probably be on sale next Tuesday--wouldn't it be a good idea to grab those when they're on clearance and store them in the freezer until the next birthday party?
I'd forgotten about this project, too. I like that it's easy, inexpensive and can be personalized for any holiday.
I'm laughing right now after reading Richella's suggestion to buy candy on Tuesday and freeze it for a future party. I KNOW I'd raid my freezer stash over and over and even eat some of that candy frozen! I guess most people would have some restraint but I get desperate for sugar at times.
did you put something inside?
I remember for one of my birthday parties my mom and I used toilet paper rolls (In Argentina ppl wouldn't think anything bad of it) and did something like but with animal faces, it was amazing!
Cute party favor idea! Will have to pass this idea along to my daughter-in-laws for my grand-kids!! nice blog btw!
I'm confused, which isn't really a surprise because I am not a mom, nor am i creative with this kind of thing.
Is this supposed to be a party favor, like something someone can take home and use, or just something neat to decorate at a party?
Just wondering, because if I unwrapped a gift and found out it was just a toilet paper roll I'd feel jipped.
Really cute idea and great way to upcycle!
genius and will be a memory the kids will have for a long timvisiting via TTJ and so glad to find your post! would love to have you stop by for my shabby chic giveaway!
Really cute! Looks like so much fun to make and to give! Love the colors and embellishments you used!