  This paper studies a curious phenomenon in learning energy-based model (EBM)
using MCMC. In each learning iteration, we generate synthesized examples by
running a non-convergent, non-mixing, and non-persistent short-run MCMC toward
the current model, always starting from the same initial distribution such as
uniform noise distribution, and always running a fixed number of MCMC steps.
After generating synthesized examples, we then update the model parameters
according to the maximum likelihood learning gradient, as if the synthesized
examples are fair samples from the current model. We treat this non-convergent
short-run MCMC as a learned generator model or a flow model. We provide
arguments for treating the learned non-convergent short-run MCMC as a valid
model. We show that the learned short-run MCMC is capable of generating
realistic images. More interestingly, unlike traditional EBM or MCMC, the
learned short-run MCMC is capable of reconstructing observed images and
interpolating between images, like generator or flow models. The code can be
found in the Appendix.

Machine Learning;Machine Learning