Clustering high-dimensional data, such as images or biological measurements, is a long-standing problem and has been studied extensively. Recently, Deep Clustering gained popularity due to the non-linearity of neural networks, which allows for flexibility in fitting the specific peculiarities of complex data. Here we introduce the Mixture-of-Experts Similarity Variational Autoencoder (MoE-Sim-VAE), a novel generative clustering model. The model can learn multi-modal distributions of high-dimensional data and use these to generate realistic data with high efficacy and efficiency. MoE-Sim-VAE is based on a Variational Autoencoder (VAE), where the decoder consists of a Mixture-of-Experts (MoE) architecture. This specific architecture allows for various modes of the data to be automatically learned by means of the experts. Additionally, we encourage the lower dimensional latent representation of our model to follow a Gaussian mixture distribution and to accurately represent the similarities between the data points. We assess the performance of our model on synthetic data, the MNIST benchmark data set, and a challenging real-world task of defining cell subpopulations from mass cytometry (CyTOF) measurements on hundreds of different datasets. MoE-Sim-VAE exhibits superior clustering performance on all these tasks in comparison to the baselines and we show that the MoE architecture in the decoder reduces the computational cost of sampling specific data modes with high fidelity.
Summary:
The paper proposes to expand the VAE architecture with a
mixture-of-experts latent representation, with a
mixture-component-specific decoder that can specialize in a specific
cluster.  Importantly, the method can take advantage of a similarity
matrix to help with the clustering.
Overall, I recommend a weak accept.  The method seems reasonable, and
the paper is well-written, but the results are only marginally better
than other methods, and there are several weaknesses with the proposed
architecture and experimental setup.
Positives:
* The idea of a more expressive variational distribution seems good,
  although it is not novel.
* The ability to have multiple decoder networks seems reasonable.
* The ability to incorporate domain knowledge (in the form of a
  similarity matrix S) is a plus.
* The experiments are thorough, although the method is generally only
  slightly better than competing methods.
Negatives:
* It's not clear if the similarity matrix S is already solving the
  clustering problem - in which case, why do we need the rest of the
  model?  For example, in your experiments you often used UMAP to
  cluster data.  How does using UMAP by itself work?  (Along these
  lines, it was not clear if your GMM experiments clustered data in
  the original space, or in the UMAP'd space - please clarify this).
  A good ablation would be to somehow remove the S matrix, to see if
  the model can accurately cluster samples.
* There is little variance in the generated samples.  
* There is not a one-to-one mapping of clusters to labels, so it is
  hard to use this method to generate a specific type of data (for
  example, it is hard to generate a specific digit).  This is a big
  difference from, say, a conditional sampler as learned by a GAN.
  This also arises in Fig. 3, where it is clear that latent cluster
  assignments do not match human-interpretable cluster assignments.  I
  suppose this is to be expected, but taken with the previous point
  (little variance in generated samples) I think it seriously weakens
  the paper's claim that this is an "accurate an efficient data
  generation method."
* The method does not do well when the number of clusters is large.
  Regular GMMs seem to outperform it.
* I felt that this paper made excessive use of the appendix.  The
  paper is not self-contained enough, effectively violating the length
  restrictions.  Please make an effort to move key results back in to
  the main body of the paper.
Experiments to run:
An ablation regarding the similarity matrix S.
Clarification of whether GMM experiments are run in data-space, or
UMAP'd space.
MIXAE features prominently in your related works, but is not compared
to in your experiments.  It sounds like a natural comparison.  Please
run this experiment, or explain why it is not a comparable method.