Learning at small or large scales of data is addressed by two strong but divided frontiers: few-shot learning and standard supervised learning. Few-shot learning focuses on sample efficiency at small scale, while supervised learning focuses on accuracy at large scale. Ideally they could be reconciled for effective learning at any number of data points (shot) and number of classes (way). To span the full spectrum of shot and way, we frame the variadic learning regime of learning from any number of inputs. We approach variadic learning by meta-learning a novel multi-modal clustering model that connects bayesian nonparametrics and deep metric learning. Our bayesian nonparametric deep embedding (BANDE) method is optimized end-to-end with a single objective, and adaptively adjusts capacity to learn from variable amounts of supervision. We show that multi-modality is critical for learning complex classes such as Omniglot alphabets and carrying out unsupervised clustering. We explore variadic learning by measuring generalization across shot and way between meta-train and meta-test, show the first results for scaling from few-way, few-shot tasks to 1692-way Omniglot classification and 5k-shot CIFAR-10 classification, and find that nonparametric methods generalize better than parametric methods. On the standard few-shot learning benchmarks of Omniglot and mini-ImageNet, BANDE equals or improves on the state-of-the-art for semi-supervised classification.
Update after Author Rebuttal
--------------
After reading the rebuttal, I'm pleased that the authors have made significant revisions, but I still think more work is needed. The "hard/soft" hybrid approach still lacks justification and perhaps wasn't compared to a soft/soft approach in a fair and fully-correct way (see detailed reply to authors). I also appreciate the efforts on revising clarity, but still find many clarity issues in the newest version that make the method hard to understand let alone reproduce. I thus stand by my rating of "borderline rejection" and urge the authors to prepare significant revisions for a future venue that avoid hybrids of hard/soft probabilities without justification. 
(Original review text below. Detailed replies to authors are in posts below their responses).
Review Summary
--------------
While the focus on variadic learning is interesting, I think the present version of the paper needs far more presentational polish as well as algorithmic improvements before it is ready for ICLR. I think there is the potential for some neat ideas here and I hope the authors prepare stronger versions in the future. However, the current version is unfortunately not comprehensible or reproducible.
Paper Summary
-------------
The paper investigates developing an effective ML method for the "variadic" regime, where the method might be required to perform learning from few or many examples (shots) and few or many classes (ways). The term "variadic" comes from use in computer science for functions that can a flexible number of arguments. There may also be unlabeled data available in the few shot case, creating semi-supervised learning opportunities.
The specific method proposed is called BANDE: Bayesian Nonparametric Deep Embedding. The idea is that each data point's feature vector x_i is transformed into an embedding vector h(x_i) using a neural network, and then clustering occurs in the embedding space via a single-pass of the DP-means algorithm (Kulis & Jordan 2012). Each cluster is assumed to correspond to one "class" in the eventual classification problem, though each class might have multiple clusters (and thus be multi-modal).  
Learning occurs in an episodic manner. After each episode (single-pass of DP-means), each point in a query set is embedded to its feature vector, then fed into each cluster's Gaussian likelihoods to produce a normalized cluster-assignment-probability vector that sums to one. This vector is then fed into a cross-entropy loss, where the true class's nearest cluster (largest probability value) is taken to be the true cluster. This loss is used to perform gradient updates of the embedding neural network.
There is also a "cumulative" version of the method called BANDE-C. This version keeps track of cluster means from previous episodes and allows new episodes to be initialized with these.
Experiments examine the proposed approach across image categorization tasks on Omniglot, mini-ImageNet, and CIFAR datasets.
Strengths
---------
* I like that many clusters are used for each true class label, which is better than rigid one-to-one assumptions.
Limitations
-----------
* Can only be used for classification, not regression
* The DP-means procedure does not account for the cluster-specific variance information that is used at other steps of the algorithm
Significance and Originality
----------------------------
To me, the method appears original. Any method that could really succeed across various variadic settings would be significant.
Presentation Concerns
---------------------
I have serious concerns about the presentation quality of this paper. Each section needs careful reorganization as well as rewording.
## P1: Algo. 1 contains numerous omissions that make it as written not correct.
* the number of clusters count variable "n" is not updated anywhere. As writting this algo can only update one extra cluster beyond the original n.
* the variable "c" is unbound in the else clause. You need a line that clarifies that c = argmin_{c in 1 ... n} d_ic
Would be careful about saying that "a single pass is sufficient"... you have *chosen* to do only one pass. When doing k-means, we could also make this choice. Certainly the DP-means objective could keep improving with multiple passes.
## P2: Many figures and tables lack appropriate captions/labels
Table 1: What metric is reported? Accuracy percentage? Not obvious from title/caption. Should also make very clear here how much labeled data was used.
Table 2: What metric is reported? Accuracy percentage? Not obvious from title/caption. Should also make how many labeled and unlabeled examples were used easier to find.
## P3: Descriptions of episodic learning and overall algorithm clarity
Readers unfamiliar with episodic learning are not helped with the limited coverage provided here in 3.1 and 3.2. When exactly is the "support" set used and the "query" set used? How do unlabeled points get used (both support and query appear fully labeled)? What is n? What is k? What is T? Why are some points in Q denoted with apostrophes but not others? Providing a more formal step-by-step description (perhaps with pseudocode) will be crucial.
In Sec. 3.2, the paragraph that starts with "The loss is defined" is very hard to read and parse. I suggest adding math to formally define the loss with equations. What parameters are being optimized? Which ones are fixed?
Additionally, in Sec. 3.2: "computed in the same way as standard prototypical networks"... what is the procedure exactly? If your method relies on a procedure, you should specify it in this paper and not make readers guess or lookup a procedure elsewhere.
## P4: Many steps of the algorithm are not detailed
The paper claims to set \lambda using a technique from another paper, but does not summarize this technique. This makes things nearly impossible to reproduce. Please add such details in the appendix.
Major Technical Concerns
------------------------
## Alg. 1 concerns: Requires two (not one) passes and mixes hard and soft assingments and different variance assumptions awkwardly
The BANDE algorithm (Alg. 1) has some unjustified properties. Hard assignment decisions which assume vanishing variances are used to find a closest cluster, but then later soft assignments with non-zero variances are used. This is a bit heuristic and lacks justification... why not use soft assignment throughout? The DP means procedure is derived from a specific objective function that assumes hard assignment. Seems weird to use it for convenience and then discard instead of coming up with the small fix that would make soft assignment consistent throughout.
Furthermore, The authors claim it is a one pass algorithm, but in fact as written in Alg. 1 it seems to require two passes: the first pass keeps an original set of cluster centers fixed and then creates new centers whenever an example's distance to the closest center exceeds \lambda. But then, the *soft* assignment step that updates "z" requires again the distance from each point to all centers be computed, which requires another pass (since some new clusters may exist which did not when the point was first visited). While the new soft values will be close to zero, they will not be *exactly* zero, and thus they matter. 
## Unclear if/how cluster-specific variance parameters learned
From the text on top of page 4, it seems that the paper assumes that there exist cluster-specific variances \sigma_c. However, these are not mentioned elsewhere, only a general (not cluster-specific) label variance \sigma and fixed unlabeled variance sigma_u are used.
## Experiments lack comparison to internal baselines
The paper doesn't evaluate sensitivity to key fixed hyperparameters (e.g. \alpha, \lambda) or compare variants of their approach (with and without soft clustering step, with and without multimodality via DP-means). It is difficult to tell which design choices of the method are most crucial.