DIMENSIONALITY REDUCTION FOR REPRESENTING THE KNOWLEDGE OF PROBABILISTIC MODELS
Most deep learning models rely on expressive high-dimensional representations to achieve good performance on tasks such as classification. However, the high dimensionality of these representations makes them difficult to interpret and prone to over-fitting. We propose a simple, intuitive and scalable dimension reduction framework that takes into account the soft probabilistic interpretation of standard deep models for classification. When applying our framework to visualization, our representations more accurately reflect inter-class distances than standard visualization techniques such as t-SNE. We experimentally show that our framework improves generalization performance to unseen categories in zero-shot learning. We also provide a finite sample error upper bound guarantee for the method.
Authors propose a method of embedding training data examples into low-dimensional spaces such that mixture probabilities from a mixture model on these points are close to probability predictions from the original model in terms of KL divergence. Authors suggest two use-cases of such an approach: 1) data visualization, and 2) zero-shot learning. For the visualization use-case, authors compare against other dimensionality reduction methods with qualitative analysis on a synthetic problem, as well as evaluation metrics such as Neighborhood-Preservation Ratio and Clustering Distance Preservation Ratio. For zero-shot use-case, they take pre-trained models on two zero-shot tasks, and improve the accuracy by using probability outputs from pre-trained models as target.
Regarding the benefit of using the proposed method for visualization, the DRPR is making a strong assumption that representations of data points that belong to the same class form a uni-modal, Gaussian distribution (since authors don't experiment with distance functions other than L2). This inductive bias comes with a strong benefit when the assumption is true - as demonstrated in the toy dataset experiment - but when it is not true, the visualization would strongly distort the underlying structure of the model. And I don't believe this is a realistic assumption, because there has to be a reason that most deep-learning based classification models in the literature don't always use a model like (3) or Prototypical Networks instead of typical fully-connected + softmax layer, unless the data size is small and we need stronger inductive bias to improve the performance of the model.  That is, we usually don't think unimodality is the right assumption, even with learned representations. I suspect that the while DRPR might be good at visualizing relationships between class labels - especially which class can be easily confused with another - but would be worse at faithfully representing each data point, especially the ambiguity of class labels on individual ones. I would argue, however, that faithful representation of each data point is more important for scatter plots than relationship between classes, because the latter can be more effectively analyzed with other methods such as confusion matrices. As it is typical in most dimensionality reduction papers, I would encourage authors to consider more types of synthetic datasets which nonlinearity and multimodality are critical to be learned. I don't believe quantitative evaluation in Table 1 and 2 are very meaningful, because DRPR's objective function is much better aligned with these metrics than others. 
Zero-shot experiments show a promising lift over the baseline pre-trained models. The kind of bias we should be careful about, however, is that when we distillate one model into another, the performance generally improves even when the same exact model is both the teacher and the student: (Furlanello et al, ICML 2018 https://arxiv.org/abs/1805.04770 ). Therefore, it would be interesting to compare against distillation with baseline models themselves.
Pros:
* Extensive theoretical and empirical analysis
* Simple idea that generalizes to multiple use-cases, which implies robustness of the approach as a methodology
Cons:
* Unimodal assumption is likely not realistic, which would result in misleading visualization of data
* Visualization analysis focuses on how class-relationships are preserved rather than faithful representation of each data point, which is a wrong target
* Synthetic experiment is conducted on a single, too simplistic one; more examples are needed to understand the capabilities of the model in more detail
* The bias of knowledge distillation is not controlled