title: Direct Evolutionary Optimization of Variational Autoencoders With Binary Latents
abstract: Many types of data are generated at least partly by discrete causes that are sparsely
  active. To model such data, we here investigate a deep generative model in the
  form of a variational autoencoder (VAE) which can learn a sparse, binary code
  for its latents. Because of the latents’ discrete nature, standard VAE training is
  not possible. The goal of previous approaches has therefore been to amend (i.e.,
  typically anneal) discrete priors in order to train discrete VAEs analogously to
  conventional ones. Here, we divert much more strongly from conventional VAE
  training: We ask if it is also possible to keep the discrete nature of the latents
  fully intact by applying a direct, discrete optimization for the encoding model. In
  doing so, we (1) sidestep standard VAE mechanisms such as sampling approximation, reparameterization trick and amortization, and (2) observe a much sparser
  encoding compared to autoencoders that use annealed discrete latents. Direct optimization of VAEs is enabled by an evolutionary algorithm in conjunction with
  truncated posteriors as variational distributions, i.e. by a combination of methods
  which is here for the first time applied to a deep model. We first show how the discrete variational method (A) ties into gradient ascent for network weights, and how
  it (B) uses the decoder network to select binary latent states for training. Sparse
  codes have prominently been applied to image patches, where latents encode edge-like structure. For our VAEs, we maintain this prototypical application domain
  and observe the emergence of much sparser codes compared to more conventional
  VAEs. To allow for a broad comparison to other approaches, the emerging encoding was then evaluated on denoising and inpainting tasks, which are canonically
  benchmarks for image patch models. For datasets with many, large images of single objects (ImageNet, CIFAR etc) deep generative models with dense codes seem
  preferable. For image patches, however, we observed advantages of sparse codes
  that give rise to state-of-the-art performance in ‘zero-shot’ denoising and inpainting benchmarks. Sparse codes can consequently make VAEs competitive on tasks
  where they have previously been outperformed by non-generative approaches.
authors: Enrico Guiraud Jakob Drefs Filippos S Panagiotou Jorg Lucke
hIndex: Enrico Guiraud 4