Due to the significant costs of data generation, many prediction tasks within drug algorithm that is better suited to these settings. Our algorithm consists of learning 7 a deep network in combination with a kernel function and a differentiable kernel 8 algorithm. As the choice of kernel is critical, our algorithm learns to find the 9 appropriate kernel for each task during inference. It thus performs more effectively 10 with complex task distributions, outperforming current state-of-the-art algorithms 11 on both toy and novel, real-world benchmarks that we introduce herein. By 12 introducing novel benchmarks derived from biological assays, we hope that the 13 community will progress towards the development of FSR algorithms suitable for 14 use in noisy and uncertain environments such as drug discovery.
The submission is at the intersection of few-shot learning, kernel regression methods, and computational biology.
The main contributions are:
  - A few-shot learning algorithm combining several ideas and features:
    - Combining metric learning (shared across tasks) and kernel regression within each task
    - Learning a task representation by maximizing an estimate of the mutual information between the train (support) and valid (query) sets of a task
    - The addition of learned, synthetic "pseudo-examples"
  - Two datasets for few-shot regression, from real-life biological assays
The proposed algorithm outperforms (or is competitive with) mainstream few-shot learning methods on a synthetic 1D regression dataset, as well as the two proposed datasets.
This paper should be accepted, because it significantly expands the field of few-shot learning by proposing both a novel problem to tackle (few-shot regression from a high-dimensional, noisy input), with public datasets, and novel algorithm to solve it (combining several recent advances in different sub-fields of machine learning).
Overview
The overall problem is clearly stated, as well as its main challenges: noisiness of the data, different behaviors of the same input across tasks.
Despite the complexity of the proposed algorithm, its different pieces and their motivations are clearly motivated, introduced, and tested in ablation studies. I liked the clarity of section 2.
The "related work" section is clear and presents a good overall picture of the field. Additional papers that may be of interest:
  - Learned hallucination (Low-shot learning from imaginary data, Wang et al., CVPR 2018) seems to relate to the "pseudo-representations"
  - TADAM (Task dependent adaptive metric for improved few-shot learning, Oreshkin et al., NeurIPS 2018) is another example of task representation used in conjunction with metric learning (in the context of classification, not regression, though)
The proposed datasets are an interesting new benchmarking task, that naturally requires learning a task description, I hope it will get traction.
Questions:
  - The backpropagation through the kernel regressor (in order to train the parameters of the embedding function and other networks) seems unexpectedly straightforward. Is that only because there is a closed-form solution for the optimal regressor? Were the specific algorithms (KRR, GP) chosen because of that? Or could something like MetaOptNet (Meta-Learning with differentiable convex optimization, Lee et al., CVPR 2019) be used to relax that constraint?
  - What "generalization guarantees of kernel-based models" (l. 122) would be applicable here? Do they hold even when the kernel is applied on top of a learned embedding ($\phi_\theta$), or even when it depends on a trained model itself ($C_t$)?
  - What does "correlations > 0.8" mean exactly? (l. 258, 263)
  - In the "Binding" dataset, is it possible that the same protein is used in different bio-assays? In that case, are the different experiments "merged" into the same task, or be considered different tasks? Could it be possible that tasks involving the same protein would be in different (meta-)splits, or has that been taken care of during the data collection?
  - In the meta-test splits, how are examples split between the train/support and valid/query parts of each task?
  - How were the specific architectures of the different neural networks designed, or selected? Between $\phi$, v, r, $MLP_\rho$, the space of hyper-parameters seems huge, and the effects of these choices might be drastic.
Additional feedback
  - The caption of Figure 1 could (re-)introduce a definition of the notation. U and C_t for instance have not been introduced yet.
  - I'm not sure I agree that FSDKL only "share[s] characteristics with the metric learning framework", I see it more as being in that framework, but incorporating other elements as well (like other methods do, e.g., TADAM or RelationNet).
  - The horizontal axis of Figure 4 (a) and (c) are not clear until we see Table 4 of the appendix, and suggest an ordering of the different configurations, rather than 10 different categories. I'm not sure how to improve it though, maybe letters instead of numbers?
On notation:
  - On l. 155, is $\phi$ the same as $\phi_\theta$, or a different embedding function for x?
  - In Eq. 11, should $D^t_{trn}$ be $D^{t_j}_{trn}$ instead in both terms? Similarly, $D^{t_j}_{val}$ in the first term?
  - In Eq. 12, and l. 173, $\phi_x'$ suggests it is a different embedding $\phi'$ of the same $x$, if we want to convey that there are two inputs x and x' instead, $\phi_{x'}$ may be clearer.
  - In Fig. 4 (a) and 5, the vertical axis is labeled $\gamma_{mine}$ instead of (I assume) $\gamma_{task}$.