We propose a meta-learning approach that learns from multiple tasks in a transductive setting, by leveraging unlabeled information in the query set to learn a more powerful meta-model. To develop our framework we revisit the empirical Bayes formulation for multi-task learning. The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as a sum of local KL divergences between the variational posterior and the true posterior of each task. We derive a novel amortized variational inference that couples all the variational posteriors into a meta-model, which consists of a synthetic gradient network and an initialization network. The combination of local KL divergences and synthetic gradient network allows for backpropagating information from unlabeled data, thereby enabling transduction. Our results on the Mini-ImageNet and CIFAR-FS benchmarks for episodic few-shot classification outperform previous state-of-the-art methods.
The authors argue for the importance of transduction in few-shot learning approaches, and augment the empirical Bayes objective established in previous work (estimating the hyperpior $\psi$ in a frequentist fashion) so as to take advantage of the unlabelled test data.
Since the label is, by definition, unknown, a synthetic gradient is instead learned to parameterize the gradient of the variational posterior and tailor it for the transductive setting. The authors provide an analysis of the generalization ability of EB and term their method _synthetic information bottleneck_ (SIB) owing to parallels between its objective of that of the information bottleneck. SIB is tested on two standard few-shot image benchmarks in CIFAR-FS and MiniImageNet, exhibiting impressive performance and outperforming, in some cases by some margin, the baseline methods, in the 1- and 5-shot settings alike, in addition to a synthetic dataset.
The paper is technically sound and, for the most part, well-written, with the authors' motivations and explanation of the method conceived quite straightforwardly. The basic premise of using an estimated gradient to fashion an inductive few-shot learning algorithm into a transductive one is a natural and evidently potent one. The paper does, however, at times feel to be disjointed and, to an extent, lacking in focus. The respective advantages of EB and the repurposing of synthetic gradients to enable the transductive approach are clear to me, yet while they might indeed be obviously complementary, what is not obvious is the necessity of the pairing: it seems there is nothing prohibiting the substitution of the gradient for a learned surrogated just as well under a deterministic meta-initialization framework. As such, despite sporting impressive results on the image datasets, I am not convinced about how truly novel the method is when viewed as a whole. 
On a similar note, while the theoretical analysis provided in section 4 was not unappreciated, and indeed it was interesting to see such a connection between EB with information theory rigorously made, it does feel a little out of place within the main text, especially since it is not specific to the transductive setting considered, nor even to the meta-learning setting more broadly. Rather, more experiments, per Appendix C, highlighting the importance of transduction and therein the synthetic gradients and its formulation would be welcome. Indeed, it is stated that an additional loss for training the synthetic gradient network to mimic the true gradient is unnecessary; while I agree with this conclusion, I likewise do not think it would hurt to explore use of the more explicit formulation.
Considering the authors argue specifically for the importance of transduction in the zero-shot learning regime, I think it would be reasonable to expect experiments substantiating this, and the strength of their method in this regard, on non-synthetic datasets. As far as the toy problem is concerned, I am slightly confused as to the choice of baseline, both in the regard to its training procedure and as to why this was deemed more suitable than one purposed for few-shot learning, so that we might go beyond simple verification to getting some initial sense for the performance of SIB. Moreover, it is not clear from the description as to how $\lambda$ is implemented here. As it stands, Section 5, for me, offers little in the way of valuable insights. The experiments section on the whole, results aside, feels somewhat rushed; the synthetic gradients being a potential limiting factor for instance feels "tacked on" and seems to warrant more than just a passing comment.
Minor errors
- Page 7: the "to" in "let $p_\psi(w)$ to be a Gaussian" is extraneous
- Page 8: "split" not "splitted".
- Further down on the same page, "scale" in "scale each feature dimension" should be singular and Drop" is misspelled as "dropp".
- Page 9: "We report results **with using** learning rate..."
- _Equation 17_ includes the indicator function $k \neq i$ but $i$ is not defined within the context.
EDIT: changed score