In many real-world settings, a learning model must perform few-shot classification: learn to classify examples from unseen classes using only a few labeled examples per class. Additionally, to be safely deployed, it should have the ability to detect out-of-distribution inputs: examples that do not belong to any of the classes. While both few-shot classification and out-of-distribution detection are popular topics, their combination has not been studied. In this work, we propose tasks for outof-distribution detection in the few-shot setting and establish benchmark datasets, based on four popular few-shot classification datasets. Then, we propose two new methods for this task and investigate their performance. In sum, we establish baseline out-of-distribution detection results using standard metrics on new benchmark datasets and show improved results with our proposed methods.
This work investigates a new problem setting that combines few-shot image classification and out-of-distribution detection. The main procedure for the task still follows a standard few-shot classification task, but in each episode, the data from a different distribution may be presented together with the query images. The evaluation focuses on the performance of out-of-distribution detection, which relies on a scoring function to assign a high score for the normal query images while having a low score for the out-of-distribution images. The paper evaluates with three different scoring functions on four few-shot classification datasets and nine out-of-distribution datasets.
We recommend a weak rejection, although the proposed new problem setting might be interesting to the community, due to three major concerns. The first is insufficient description for reproducing the evaluation. For example, the text mentions “All results are evaluated using 1000 test episodes” without information of how the in-distribution data and out-of-distribution data (especially the OOS data) is chosen in each episode. Second, the experiments and discussions do not provide enough insights for readers to understand the impact of combining the two problems. Some questions should be addressed and are listed below. The last is the writing style which has a noticeable fraction of content not directly related to the proposed problem setting. Such a style can confuse readers and require additional passes of reading to understand. 
Some questions to be addressed for the second concern:
1. What’s the impact of doing out-of-distribution under the few-shot setting? Is it harder than a normal out-of-distribution detection setting? How does the “N-way X-shot” setting affect the difficulty of the problem?
2. The paper proposes to use -MinDist and LCBO for the scoring functions instead of the methods that are commonly seen in a standard out-of-distribution detection paper. Why not use those previous methods (ex: ODIN, Mahalanobis, ensemble strategies, etc.) for the evaluation? If those previous methods do not work well with the proposed new setting, what are the possible causes?
Examples of the third concern include: (1) The MAML in Section 2 has a whole paragraph that could be summarized in a few sentences. (2) Figure 3 draws the schematics of out-of-distribution detection, but its connection to the proposed setting is not clearly described. (3) The introduction mentioned semi-supervised learning and continual learning, which does not strengthen the argument of why few-shot learning and out-of-distribution detection should be combined. (4) Lastly, the FS-SSL paragraph in the experiment section has no conclusion.