Supervised deep-embedding methods project inputs of a domain to a representational space in which same-class instances lie near one another and different-class instances lie far apart. We propose a probabilistic method that treats embeddings as random variables. Extending a state-of-the-art deterministic method, Prototypical Networks (Snell et al., 2017), our approach supposes the existence of a class prototype around which class instances are Gaussian distributed. The prototype posterior is a product distribution over labeled instances, and query instances are classified by marginalizing relative prototype proximity over embedding uncertainty. We describe an efficient sampler for approximate inference that allows us to train the model at roughly the same space and time cost as its deterministic sibling. Incorporating uncertainty improves performance on few-shot learning and gracefully handles label noise and out-of-distribution inputs. Compared to the state-of-the-art stochastic method, Hedged Instance Embeddings (Oh et al., 2019), we achieve superior large-and open-set classification accuracy. Our method also aligns class-discriminating features with the axes of the embedding space, yielding an interpretable, disentangled representation.
The paper proposes stochastic prototype embeddings (SPE) for few-shot learning. The method is an extension of Prototypical Networks (PN, [1]) with Gaussian embeddings. The idea is to take representation uncertainty into account when classifying objects which makes the model more robust to input and label noise. The authors propose an efficient sampling algorithm to train SPE which outperforms naive Monte Carlo sampling. They conduct a range of experiments on few-shot learning tasks on a synthetic dataset, Omniglot and N-digit MNIST and compare to Prototypical Networks and previous stochastic embedding state-of-the-art method HIB [3]; SPE was shown to outperform prior work in most settings. The authors also plot interpretable disentangled representations learned in 2D embedding space. 
However, in my opinion there are a few weaknesses in the paper in its current state: (1) the clarity of sections 3.1-3.2 could be significantly improved as currently the proposed probabilistic model framework is confusing and not well-defined; (2) the experimental results are provided only for embedding spaces of dimensionality 2-3 which significantly hinders performance of Prototypical Networks compared to having a much higher dimensional embedding space, so it would help to see comparisons of SPE and PN using high-dimensional embeddings; (3) the central idea and proposed model seem to be very close to those of [2] which is mentioned in the related work, so, please, list differences with this prior work in the updated version.
For these reasons, I recommend a weak reject for the paper. I explain these issues in more detail below.
(1) SPE model assumes that each embedding z is Gaussian-distributed p(z|x) = N(z; mu_x, sigma^2_x I) where parameters of the distribution mu_x and sigma_x are outputted by an embedding neural network. In equation (3) the author define each class prototype as rho_y = z_i + eps where z_i are instances of class y and eps is noise ~ N(0, sigma^2_eps I), so p(rho_y | z_i) = N(rho_y | z_i, sigma^2_eps I) for every z_i from class y. This definition is confusing to me since rho_y is redefined for each z_i, so it is not clear what the generative process for rho_y is. In the Introduction section it is mentioned “...each class instance is assumed to be a Gaussian perturbation of the prototype” which suggests z_i = rho_y + eps for every z_i, so graphical model would be rho_y -> z_i instead of z_i -> rho_y implied in section 3.1. Please, clarify in the rebuttal which the graphical model is implied in SPE. Is it x_i -> z_i -> rho_y? How is then the distribution of rho_y defined given all z_i from class y?
Further, in the equation (4) the likelihood of rho_y given all x_i from class y is proportional to product of p(rho_y | x_i) so the expression was factorized over the condition which is the reflection of the authors’ assumption of consistency of the prototype. p(rho_y | x_1, … x_n) could be also written as proportional to the product of p(x_i | rho) times p(rho) using Bayes rule. It would help if authors could explain the differences between the two possible expressions for p(rho_y | x_1 … x_n) and why the former is chosen.
Also, in equation (6) the right-hand-side is p(z | y, S) = p(z | rho_y, S) = N(z | mu_y, \hat{sigma}^2_y I): why is the variance is \hat{sigma}^2_y (so extra sigma^2_eps added to sigma^2_y) instead of just sigma^2_y defined in equation (5)? The noise term with sigma^2_eps seems to be already included in sigma^2_y.
(2) In all presented experiments, the dimensionality of the embedding is 2-3 instead of high dimensional embedding spaces in the original Prototypical Network paper which causes a significant drop in performance: e.g., on Omniglot 1-shot 5-class classification, the accuracy dropped from 98.8% to 75.7%. SPE outperformed PN, but an important aspect of the PN model, embedding space dimensionality, was changed from the original in a way that hindered the performance very much, so SPE’s advantage is unclear. It is mentioned in section 4.2 that “We also compared PN and SPE using a 64D embedding, but with high dimensional embeddings, both methods are near ceiling on this data set, resulting in comparable performance between the two methods. (See Appendix D for additional results, broken down by condition.)” but the results on 64D embeddings are not added and discussed in the appendix. If the performance on high dimensional embeddings in the standard setting is similar for SPE and PC, it would help to see the 64D embedding experiment with corrupted support and query data, because if SPE outperformed PN on the corrupted data setting, it would be a fair advantage. It is interesting to see what the 2D embedding space looks like in the visualizations (Figures 2, 5, 7), and SPE is learning interpretable disentangles representations. However, for better classification performance it makes sense to use higher dimensional embedding space to allow learning more expressive representations.
For Figures 2c, 5 and 7, it would be helpful to see how the embedding space of PN looks like and compare it visually to the one of SPE, e.g. whether PN has disentangled representations. It seems like we could get a similar embedding space to, for example, Figure 2c and the label uncertainty would come from the roughly equal distance of object representation to different class prototypes. Please, add respective embedding visualizations for PN and explain what the advantages of SPE learned representations over PN’s representations are.
(3) The prior work of [2] referenced in the paper seems to have a very similar method (judging from Figure 2 and equations 5-6 which compute parameters of the prototype distribution, [2]). Please, list the differences with this prior work and provide experimental comparison if possible.
Other questions:
1. In Figure 2c, please plot and highlight the learned distribution of the class prototypes p(rho | S).
2. Section 4.1: did each object in the training set have only one label or multiple labels (since classes are overlapping)?
3. Experiments: please add HIB method to synthetic data and Omniglot experiments if possible. It would also be interesting to compare to PN on miniImageNet (section 3.2 of [1]). Another interesting experiment which could show advantage of SPE could be out-of-distribution data detection (e.g., comparing likelihoods p(z | y) for in-distribution representations z and out-of-distribution z).
[1] Snell, Jake, Kevin Swersky, and Richard Zemel. "Prototypical networks for few-shot learning." Advances in Neural Information Processing Systems. 2017.
[2] Fort, Stanislav. "Gaussian prototypical networks for few-shot learning on omniglot." arXiv preprint arXiv:1708.02735 (2017).
[3] Oh, Seong Joon, et al. "Modeling uncertainty with hedged instance embedding." arXiv preprint arXiv:1810.00319 (2018).