Mixture-of-Experts Variational Autoencoder for clustering and generating from similarity-based representations
Clustering high-dimensional data, such as images or biological measurements, is a long-standing problem and has been studied extensively. Recently, Deep Clustering gained popularity due to the non-linearity of neural networks, which allows for flexibility in fitting the specific peculiarities of complex data. Here we introduce the Mixture-of-Experts Similarity Variational Autoencoder (MoE-Sim-VAE), a novel generative clustering model. The model can learn multi-modal distributions of high-dimensional data and use these to generate realistic data with high efficacy and efficiency. MoE-Sim-VAE is based on a Variational Autoencoder (VAE), where the decoder consists of a Mixture-of-Experts (MoE) architecture. This specific architecture allows for various modes of the data to be automatically learned by means of the experts. Additionally, we encourage the latent representation of our model to follow a Gaussian mixture distribution and to accurately represent the similarities between the data points. We assess the performance of our model on synthetic data, the MNIST benchmark data set, and a challenging real-world task of defining cell subpopulations from mass cytometry (CyTOF) measurements on hundreds of different datasets. MoE-Sim-VAE exhibits superior clustering performance on all these tasks in comparison to the baselines and we show that the MoE architecture in the decoder reduces the computational cost of sampling specific data modes with high fidelity.
The proposed method of mixture-of-experts variational autoencoders is valuable and insightful. On the other hand the work could be improved and clarified at some points: - in the abstract it is claimed that the method works for high-dimensional data.However, it should be better explained why this is the case. The method is largely based on density estimation with a mixture of Gaussians which is known to have limitations in higher dimensions (see e.g. classical textbooks like Bishop 1995) - the similarity matrix and the similarity values should be carefully defined. Is there also an underlying similarity function assumed? - a main shortcoming is that there is no discussion or experimental comparison with methods like spectral clustering and kernel spectral clustering. Given that the paper and the proposed method relates to similarity-based representations it would be important to know how it compares to such methods. Though e.g. in Table 1 the authors compare with about 10 other methods it would be more relevant that among some of these would have been spectral clustering and kernel spectral clustering, because of the similarity-based representations. - in section 4.1 the MNIST data are taken with k=10. Though it is nicely explained and illustrated on this data set, it is possibly somewhat misleading as an example. The reason is that this is a classification problem with 10 classes, therefore the choice k=10 is obvious. It would be more important to consider benchmark problems for clustering, instead of classification, for which the choice of k is also an important model selection issue and for which k is unknown (how should k be selected then?). - is each cluster always be assumed to be a Gaussian (which seems to be a strong assumption in general, and possibly not always realistic)? Could other components be used in the mixture?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Mixture-of-Experts Variational Autoencoder for clustering and generating from similarity-based representations
Clustering high-dimensional data, such as images or biological measurements, is a long-standing problem and has been studied extensively. Recently, Deep Clustering gained popularity due to the non-linearity of neural networks, which allows for flexibility in fitting the specific peculiarities of complex data. Here we introduce the Mixture-of-Experts Similarity Variational Autoencoder (MoE-Sim-VAE), a novel generative clustering model. The model can learn multi-modal distributions of high-dimensional data and use these to generate realistic data with high efficacy and efficiency. MoE-Sim-VAE is based on a Variational Autoencoder (VAE), where the decoder consists of a Mixture-of-Experts (MoE) architecture. This specific architecture allows for various modes of the data to be automatically learned by means of the experts. Additionally, we encourage the latent representation of our model to follow a Gaussian mixture distribution and to accurately represent the similarities between the data points. We assess the performance of our model on synthetic data, the MNIST benchmark data set, and a challenging real-world task of defining cell subpopulations from mass cytometry (CyTOF) measurements on hundreds of different datasets. MoE-Sim-VAE exhibits superior clustering performance on all these tasks in comparison to the baselines and we show that the MoE architecture in the decoder reduces the computational cost of sampling specific data modes with high fidelity.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The proposed method of mixture-of-experts variational autoencoders is valuable and insightful. On the other hand the work could be improved and clarified at some points: - in the abstract it is claimed that the method works for high-dimensional data.However, it should be better explained why this is the case. The method is largely based on density estimation with a mixture of Gaussians which is known to have limitations in higher dimensions (see e.g. classical textbooks like Bishop 1995) - the similarity matrix and the similarity values should be carefully defined. Is there also an underlying similarity function assumed? - a main shortcoming is that there is no discussion or experimental comparison with methods like spectral clustering and kernel spectral clustering. Given that the paper and the proposed method relates to similarity-based representations it would be important to know how it compares to such methods. Though e.g. in Table 1 the authors compare with about 10 other methods it would be more relevant that among some of these would have been spectral clustering and kernel spectral clustering, because of the similarity-based representations. - in section 4.1 the MNIST data are taken with k=10. Though it is nicely explained and illustrated on this data set, it is possibly somewhat misleading as an example. The reason is that this is a classification problem with 10 classes, therefore the choice k=10 is obvious. It would be more important to consider benchmark problems for clustering, instead of classification, for which the choice of k is also an important model selection issue and for which k is unknown (how should k be selected then?). - is each cluster always be assumed to be a Gaussian (which seems to be a strong assumption in general, and possibly not always realistic)? Could other components be used in the mixture?
assistant
The proposed method of mixture-of-experts variational autoencoders is valuable and insightful. On the other hand the work could be improved and clarified at some points: - in the abstract it is claimed that the method works for high-dimensional data.However, it should be better explained why this is the case. The method is largely based on density estimation with a mixture of Gaussians which is known to have limitations in higher dimensions (see e.g. classical textbooks like Bishop 1995) - the similarity matrix and the similarity values should be carefully defined. Is there also an underlying similarity function assumed? - a main shortcoming is that there is no discussion or experimental comparison with methods like spectral clustering and kernel spectral clustering. Given that the paper and the proposed method relates to similarity-based representations it would be important to know how it compares to such methods. Though e.g. in Table 1 the authors compare with about 10 other methods it would be more relevant that among some of these would have been spectral clustering and kernel spectral clustering, because of the similarity-based representations. - in section 4.1 the MNIST data are taken with k=10. Though it is nicely explained and illustrated on this data set, it is possibly somewhat misleading as an example. The reason is that this is a classification problem with 10 classes, therefore the choice k=10 is obvious. It would be more important to consider benchmark problems for clustering, instead of classification, for which the choice of k is also an important model selection issue and for which k is unknown (how should k be selected then?). - is each cluster always be assumed to be a Gaussian (which seems to be a strong assumption in general, and possibly not always realistic)? Could other components be used in the mixture?