Using variational Bayes with neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks. This results in a rich prior capable of few-shot learning on new tasks. The posterior can go beyond the mean field approximation and yields good uncertainty on the performed experiments. Analysis on toy tasks show that it can learn from significantly different tasks while finding similarities among them. Experiments on Mini-Imagenet reach state of the art with 74.5% accuracy on 5 shot learning. Finally, we provide two new benchmarks, each showing a failure mode of existing meta learning algorithms such as MAML and prototypical Networks.
The authors state that their goal with this paper is manifold:
They want to learn a prior over neural networks for multiple tasks. The posterior should go beyond mean field inference and yield good results.  The authors claim in their paper that they learn an 'expressive transferable prior over the weights of a network' for multi-task settings, which they denote with the unfortunate term 'deep prior'.
In sec. 2.1 the authors introduce the idea of a hierarchical probabilistic model of weights for a neural network p(W|a) conditioned on task latent variables p(a). They realize that one might want to generate those weights with a function which conditions on variable "z" and has parameters "a". They continue their argument in Sec 2.2 that since the weight scoring can be canceled out in the ELBO, the score of the model does not depend on weights "w" explicitly anymore.
This, of course, is wrong, since the likelihood term in the ELBO still is an expectation over the posterior of q(w|z)q(z). 
However, the authors also realize this and continue their argumentation as follows:
In this case -according to the authors- one may drop the entire idea about learning distributions over weights entirely.
The math says: p(y|x ; a) = int_z p(z) int_w p(w|z ; a) p(y|x, w)dw dz.
So the authors claim that a model p(y|x, z) which only conditions on 'z' is the same as the full Bayesian Model with marginalized weights. They then suggest to just use any neural network with parameters "a" to model this p(y|x, z ;a) directly with z being used as an auxiliary input variable to the network with parameters "a" and claim this is doing the same. This is of course utterly misleading, as the parameter "a" in the original model indicated a model mapping a low dimensional latent variable to weights, but now a maps to a neural network mapping a latent variable and an input vector x to an output vector y. As such, these quantities are different and the argument does not hold. Also a point estimate of said mapping will not be comparable to the marginalized p(y|x).
What is more concerning is that the authors claim this procedure is equivalent to learning a distribution over weights and call the whole thing a deep prior, while this paper contains no work on trying to perform the hard task of successfully parametrizing a high-dimensional conditional distribution over weights p(w|z) (apart from a trivial experiment generating all of them at once  from a neural network for a single layer in a failed experiment) but claims to succeed in doing so by circumventing it entirely. 
In their experiments, the authors also do not actually successfully try to really learn a full distribution over the weights of a neural network. This alone suffices to realize that the paper appears to be purposefully positioned in a highly misleading way and makes claims about weight priors that are superficially discussed in various sections but never actually executed on properly in the paper.
This is a disservice to the hard work many recent and older papers are doing in actually trying to derive structured hierarchical weight distributions for deep networks, which this paper claims is a problem they find to be 'high dimensional and noisy', which is exactly why it is a valid research avenue to begin with that should not be trivially subsumed by work such as this.
When reducing this paper to the actual components it provides, it is a simple object: A deterministic neural network with an auxiliary, task-dependent latent variable which provides extra inputs to model conditional densities.
Such ideas have been around for a while and the authors do not do a good job of surveying the landscape of such networks with additional stochastic input variables.
One example is "Learning Stochastic Feedforward Neural Networks" by Tang and Salakhutdinov, NIPS 2013, a more recent one is "Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables" by Depeweg et al 2017.
An obvious recent example of multi-task/meta/continual learning comparators would be "VARIATIONAL CONTINUAL LEARNING" by Nguyen et al. and other work from the Cambridge group that deals with multi-task and meta-learning and priors for neural networks.
Another weakness of the paper is that the main driver of success in the paper's experiment regarding classification is the prototypical network idea, rather than anything else regarding weight uncertainty which seems entirely disentangled from the core theoretical statements of the paper.
All in all, I find this paper unacceptably phrased with promises it simply does not even attempt to keep and a misleading technical section that would distort the machine learning literature without actually contributing to a solution to the technical problems it claims to tackle (in relation to modeling weight uncertainty/priors on NN). Paired with the apparent disinterest of the authors to cite recent and older literature executing strongly related underlying ideas combining neural networks with auxiliary latent variables, I can only recommend that the authors significantly change the writing and the attribution of ideas in this paper for a potential next submission focusing on multi-task learning and clarify and align the core ideas in the theory sections and the experiment sections.