Uncertainty in Multitask Transfer Learning
Using variational Bayes neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks. This results in a rich prior capable of few-shot learning on new tasks. The posterior can go beyond the mean field approximation and yields good uncertainty on the performed experiments. Analysis on toy tasks show that it can learn from significantly different tasks while finding similarities among them. Experiments on Mini-Imagenet reach state of the art with 74.5% accuracy on 5 shot learning. Finally, we provide two new benchmarks, each showing a failure mode of existing meta learning algorithms such as MAML and prototypical Networks.
 The paper presents a method for training a probabilistic model for Multitask Transfer Learning. The key idea is to introduce a latent variable "z" per task which to capture the commonality in the task instances. Since this leads to an intractable likelihood the authors use the standard ELBO with a Variational Distribution over "z" defined as a Gaussian + Inverse Autoregressive Flow. For classification, the authors also show that they can combine the model with the main idea in Prototypical Networks. The experiments evaluate on three different task, the comparison against MAML on the toy problem is quite interesting. However, the results on the Mini-Imagenet suggest that the main contributors to the better performance are the Prototypical Networks idea and the improved ResNet. Additionally, the authors compare against MAML only on the toy task and not on their synthetic dataset. I think that the experiments need better comparisons (there have been published an improved version of MAML, or even just add results from your own implementation of MAML with the same ResNet on the 3rd task as well). A major issue is that the model presented is not really a Hierarchical Bayesian model as being strongly presented. It is much more a practical variational algorithm, which is not bad by no means, but I find its "interpretation" as a Hierarchical Bayesian method as totally unnecessary and making the paper significantly harder to read and follow than it needs to be. This is true for both the base model and the model + ProtoNet. I think that the manuscript itself requires more work as well as a better comparison of the method to baseline algorithms. Section 2.2: The authors start by introducing a "Hierarchical Bayes" model over the parameters of a Neural Network for multi-task learning. By defining the model parameters to be an implicit function of some low-dimensional noise and the hyper-parameter they shift the inference to the noise variable "z". One issue, which I won*t discuss further, is that this defines a degenerate distribution over the parameters (a fact well known in the GAN literature), which seem counter-intuitive to call "Bayesian". Later, since the parameters "w" has vanished from the equation the authors conclude that now they can change the whole graphical models such that there is actually no distribution over the parameters of a Neural Network, while the hyper-parameter IS now the parameters of a Neural Network and the latent variable is an input to it. Mathematically, the transformation is valid, however, this no longer corresponds to the original graphical model that was described earlier. The procedure described here is essentially a Variational Model with latent variable "z" for each task and the method performs a MAP estimation of the parameters of the Generative Model by doing Variational Inference (VAE to be exact) on the latent "z". There is nothing bad about this model, however, the whole point of using a "Hierarchical Bayes" for the parameters of the Network serves no purpose and is significantly different to the actual model that is proposed. In section 2, the prior term p(a) in equation 7 and Algorithm 1 is missing. Section 3: The authors argue that they add yet another level of hierarchy in the Graphical Model with a further latent variable "v", which is unclear fundamentally why you need it as it can be subsumed inside "z" (from a probabilistic modelling perspective they play similar roles). Additionally, they either do not include a prior or on "v" or there is a mistake in the equation for p(S|z) at the bottom of page 4. The main motivation for this comes from the literature where for instance if we have a linear regression and "v" represents the weights of the last linear layer with a Gaussian Prior than the posterior over "v" has an analytical form. After this whole introduction into the special latent variable "v", the authors actually use the idea from Prototypical Networks. They introduce a valid leave-one producer for training. However, the connection to the latent variable "v" which was argued to be the third level of a Hierarchical Bayes model is now lost, as the context c_k is no longer a separate latent variable (it has no prior and in the original Prototypical Network although the idea can be interpreted in a probabilistic framework it is never presented as a Hierarchical Bayes).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Uncertainty in Multitask Transfer Learning
Using variational Bayes neural networks, we develop an algorithm capable of accumulating knowledge into a prior from multiple different tasks. This results in a rich prior capable of few-shot learning on new tasks. The posterior can go beyond the mean field approximation and yields good uncertainty on the performed experiments. Analysis on toy tasks show that it can learn from significantly different tasks while finding similarities among them. Experiments on Mini-Imagenet reach state of the art with 74.5% accuracy on 5 shot learning. Finally, we provide two new benchmarks, each showing a failure mode of existing meta learning algorithms such as MAML and prototypical Networks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper presents a method for training a probabilistic model for Multitask Transfer Learning. The key idea is to introduce a latent variable "z" per task which to capture the commonality in the task instances. Since this leads to an intractable likelihood the authors use the standard ELBO with a Variational Distribution over "z" defined as a Gaussian + Inverse Autoregressive Flow. For classification, the authors also show that they can combine the model with the main idea in Prototypical Networks. The experiments evaluate on three different task, the comparison against MAML on the toy problem is quite interesting. However, the results on the Mini-Imagenet suggest that the main contributors to the better performance are the Prototypical Networks idea and the improved ResNet. Additionally, the authors compare against MAML only on the toy task and not on their synthetic dataset. I think that the experiments need better comparisons (there have been published an improved version of MAML, or even just add results from your own implementation of MAML with the same ResNet on the 3rd task as well). A major issue is that the model presented is not really a Hierarchical Bayesian model as being strongly presented. It is much more a practical variational algorithm, which is not bad by no means, but I find its "interpretation" as a Hierarchical Bayesian method as totally unnecessary and making the paper significantly harder to read and follow than it needs to be. This is true for both the base model and the model + ProtoNet. I think that the manuscript itself requires more work as well as a better comparison of the method to baseline algorithms. Section 2.2: The authors start by introducing a "Hierarchical Bayes" model over the parameters of a Neural Network for multi-task learning. By defining the model parameters to be an implicit function of some low-dimensional noise and the hyper-parameter they shift the inference to the noise variable "z". One issue, which I won*t discuss further, is that this defines a degenerate distribution over the parameters (a fact well known in the GAN literature), which seem counter-intuitive to call "Bayesian". Later, since the parameters "w" has vanished from the equation the authors conclude that now they can change the whole graphical models such that there is actually no distribution over the parameters of a Neural Network, while the hyper-parameter IS now the parameters of a Neural Network and the latent variable is an input to it. Mathematically, the transformation is valid, however, this no longer corresponds to the original graphical model that was described earlier. The procedure described here is essentially a Variational Model with latent variable "z" for each task and the method performs a MAP estimation of the parameters of the Generative Model by doing Variational Inference (VAE to be exact) on the latent "z". There is nothing bad about this model, however, the whole point of using a "Hierarchical Bayes" for the parameters of the Network serves no purpose and is significantly different to the actual model that is proposed. In section 2, the prior term p(a) in equation 7 and Algorithm 1 is missing. Section 3: The authors argue that they add yet another level of hierarchy in the Graphical Model with a further latent variable "v", which is unclear fundamentally why you need it as it can be subsumed inside "z" (from a probabilistic modelling perspective they play similar roles). Additionally, they either do not include a prior or on "v" or there is a mistake in the equation for p(S|z) at the bottom of page 4. The main motivation for this comes from the literature where for instance if we have a linear regression and "v" represents the weights of the last linear layer with a Gaussian Prior than the posterior over "v" has an analytical form. After this whole introduction into the special latent variable "v", the authors actually use the idea from Prototypical Networks. They introduce a valid leave-one producer for training. However, the connection to the latent variable "v" which was argued to be the third level of a Hierarchical Bayes model is now lost, as the context c_k is no longer a separate latent variable (it has no prior and in the original Prototypical Network although the idea can be interpreted in a probabilistic framework it is never presented as a Hierarchical Bayes).
assistant
 The paper presents a method for training a probabilistic model for Multitask Transfer Learning. The key idea is to introduce a latent variable "z" per task which to capture the commonality in the task instances. Since this leads to an intractable likelihood the authors use the standard ELBO with a Variational Distribution over "z" defined as a Gaussian + Inverse Autoregressive Flow. For classification, the authors also show that they can combine the model with the main idea in Prototypical Networks. The experiments evaluate on three different task, the comparison against MAML on the toy problem is quite interesting. However, the results on the Mini-Imagenet suggest that the main contributors to the better performance are the Prototypical Networks idea and the improved ResNet. Additionally, the authors compare against MAML only on the toy task and not on their synthetic dataset. I think that the experiments need better comparisons (there have been published an improved version of MAML, or even just add results from your own implementation of MAML with the same ResNet on the 3rd task as well). A major issue is that the model presented is not really a Hierarchical Bayesian model as being strongly presented. It is much more a practical variational algorithm, which is not bad by no means, but I find its "interpretation" as a Hierarchical Bayesian method as totally unnecessary and making the paper significantly harder to read and follow than it needs to be. This is true for both the base model and the model + ProtoNet. I think that the manuscript itself requires more work as well as a better comparison of the method to baseline algorithms. Section 2.2: The authors start by introducing a "Hierarchical Bayes" model over the parameters of a Neural Network for multi-task learning. By defining the model parameters to be an implicit function of some low-dimensional noise and the hyper-parameter they shift the inference to the noise variable "z". One issue, which I won*t discuss further, is that this defines a degenerate distribution over the parameters (a fact well known in the GAN literature), which seem counter-intuitive to call "Bayesian". Later, since the parameters "w" has vanished from the equation the authors conclude that now they can change the whole graphical models such that there is actually no distribution over the parameters of a Neural Network, while the hyper-parameter IS now the parameters of a Neural Network and the latent variable is an input to it. Mathematically, the transformation is valid, however, this no longer corresponds to the original graphical model that was described earlier. The procedure described here is essentially a Variational Model with latent variable "z" for each task and the method performs a MAP estimation of the parameters of the Generative Model by doing Variational Inference (VAE to be exact) on the latent "z". There is nothing bad about this model, however, the whole point of using a "Hierarchical Bayes" for the parameters of the Network serves no purpose and is significantly different to the actual model that is proposed. In section 2, the prior term p(a) in equation 7 and Algorithm 1 is missing. Section 3: The authors argue that they add yet another level of hierarchy in the Graphical Model with a further latent variable "v", which is unclear fundamentally why you need it as it can be subsumed inside "z" (from a probabilistic modelling perspective they play similar roles). Additionally, they either do not include a prior or on "v" or there is a mistake in the equation for p(S|z) at the bottom of page 4. The main motivation for this comes from the literature where for instance if we have a linear regression and "v" represents the weights of the last linear layer with a Gaussian Prior than the posterior over "v" has an analytical form. After this whole introduction into the special latent variable "v", the authors actually use the idea from Prototypical Networks. They introduce a valid leave-one producer for training. However, the connection to the latent variable "v" which was argued to be the third level of a Hierarchical Bayes model is now lost, as the context c_k is no longer a separate latent variable (it has no prior and in the original Prototypical Network although the idea can be interpreted in a probabilistic framework it is never presented as a Hierarchical Bayes).