Optimization as a Model for Few-Shot Learning
Though deep neural networks have shown great success in the large data domain, they generally perform poorly on few-shot learning tasks, where a model has to quickly generalize after seeing very few examples from each class. The general belief is that gradient-based optimization in high capacity models requires many iterative steps over many examples to perform well. Here, we propose an LSTM-based meta-learner model to learn the exact optimization algorithm used to train another learner neural network in the few-shot regime. The parametrization of our model allows it to learn appropriate parameter updates specifically for the scenario where a set amount of updates will be made, while also learning a general initialization of the learner network that allows for quick convergence of training. We demonstrate that this meta-learning model is competitive with deep metric-learning techniques for few-shot learning.
This paper describes a new approach to meta learning by interpreting the SGD update rule as gated recurrent model with trainable parameters. The idea is original and important for research related to transfer learning. The paper has a clear structure, but clarity could be improved at some points. Pros: - An interesting and feasible approach to meta-learning - Competitive results and proper comparison to state-of-the-art - Good recommendations for practical systems Cons: - The analogy would be closer to GRUs than LSTMs - The description of the data separation in meta sets is hard to follow and could be visualized - The experimental evaluation is only partly satisfying, especially the effect of the parameters of i_t and f_t would be of interest - Fig 2 doesn*t have much value Remarks: - Small typo in 3.2: *This means each coordinate has it* -> its > We plan on releasing the code used in our evaluation experiments. This would certainly be a major plus.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Optimization as a Model for Few-Shot Learning
Though deep neural networks have shown great success in the large data domain, they generally perform poorly on few-shot learning tasks, where a model has to quickly generalize after seeing very few examples from each class. The general belief is that gradient-based optimization in high capacity models requires many iterative steps over many examples to perform well. Here, we propose an LSTM-based meta-learner model to learn the exact optimization algorithm used to train another learner neural network in the few-shot regime. The parametrization of our model allows it to learn appropriate parameter updates specifically for the scenario where a set amount of updates will be made, while also learning a general initialization of the learner network that allows for quick convergence of training. We demonstrate that this meta-learning model is competitive with deep metric-learning techniques for few-shot learning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper describes a new approach to meta learning by interpreting the SGD update rule as gated recurrent model with trainable parameters. The idea is original and important for research related to transfer learning. The paper has a clear structure, but clarity could be improved at some points. Pros: - An interesting and feasible approach to meta-learning - Competitive results and proper comparison to state-of-the-art - Good recommendations for practical systems Cons: - The analogy would be closer to GRUs than LSTMs - The description of the data separation in meta sets is hard to follow and could be visualized - The experimental evaluation is only partly satisfying, especially the effect of the parameters of i_t and f_t would be of interest - Fig 2 doesn*t have much value Remarks: - Small typo in 3.2: *This means each coordinate has it* -> its > We plan on releasing the code used in our evaluation experiments. This would certainly be a major plus.
assistant
This paper describes a new approach to meta learning by interpreting the SGD update rule as gated recurrent model with trainable parameters. The idea is original and important for research related to transfer learning. The paper has a clear structure, but clarity could be improved at some points. Pros: - An interesting and feasible approach to meta-learning - Competitive results and proper comparison to state-of-the-art - Good recommendations for practical systems Cons: - The analogy would be closer to GRUs than LSTMs - The description of the data separation in meta sets is hard to follow and could be visualized - The experimental evaluation is only partly satisfying, especially the effect of the parameters of i_t and f_t would be of interest - Fig 2 doesn*t have much value Remarks: - Small typo in 3.2: *This means each coordinate has it* -> its > We plan on releasing the code used in our evaluation experiments. This would certainly be a major plus.