Model-agnostic meta-learning (MAML) is known as a powerful meta-learning method. However, MAML is notorious for being hard to train because of the existence of two learning rates. Therefore, in this paper, we derived a sufficinent condition of inner learning rate α and meta-learning rate β for a simplified MAML to locally converge to local minima from any point in the vicinity of the local minima. We find that the upper bound of β depends on α. Moreover, we show that the threshold of β increases as α approaches its own upper bound. This result is verified by experiments on various few-shot tasks and architectures; specifically, we perform sinusoid regression and classification of Omniglot and MiniImagenet datasets with a multilayer perceptron and a convolutional neural network. Based on this outcome, we present a guideline for determining the learning rates: first, search for the largest possible α; next, tune β based on the chosen value of α.
This paper studies the two learning rates \alpha and \beta used in Model-Agnostic Meta-Learning (MAML) algorithm by [Finn et al., 2017]. MAML is known to be difficult to train, and part of the reason why is the need to tune the two learning rates. Under simplifications, the paper derives some necessary conditions on \alpha and \beta for the MAML iterates to converge to local minima, and then verifies the theory by experiments on synthetic and real-world data.
Overall, I think this paper should be rejected. In my opinion, this paper analyzes a simplified setting which is far from the original MAML setting and yet derives conditions that are not very meaningful or useful. Although it is interesting to see some match between theory and experiments (especially Fig 4), the theory part seems to have room for improvement, and I will detail the reasons in the following.
I believe that the analysis is done on an overly simplified setting and easily breaks without such simplifications.
- The paper assumes that training set is equal to test set and these sets do not change over iterations, which is essentially equivalent to assuming full gradient access; this is different from the stochastic setting of MAML. 
- The "necessary condition" is derived in the case where there is only one task, and there is no discussion on generalizing to some other tasks. This means that the setting is too simplified so that it is not even a meta-learning setup.
- In the extension to multiple tasks, the paper derives "sufficient conditions" to a set of "necessary conditions" for convergence of MAML to local minima. This means that, if we consider the sets of events A = {convergence of MAML to local min}, B = {necessary conditions for A}, and C = {sufficient conditions for B}, A is a subset of B and C is a subset of B and nothing can be specified between A and C. In the extreme case, A and C may be even disjoint subsets of B. Thus, at least in theory, the conditions for multiple tasks do not tell us anything about convergence.
- The analysis relies on a number of "A is approximately equal to B" arguments without careful handling of errors, e.g., (3) and (4), and Tg = 0.
I also have concerns about the correctness of the analysis in the single task case.
- In Section 3.1.2 and eq (12), the paper reparametrizes \theta to v and analyzes updates on v. How is (12) obtained from the original update rule of \theta? In fact, the paper analyzes steepest GD but doesn't provide the explicit update rule; for example, is it steepest GD with respect to which norm? It'd be helpful to have the details in the main text.
- Even if (12) is true, there are pathological cases lying in a set of measure zero, where (12) converges even when (13) is not satisfied. For simplicity, consider v(t+1) = G v(t), where G is a real symmetric matrix. Assume all but one eigenvalue \lambda_1 are greater than 1 and |\lambda_1| < 1. Let v_1 be the corresponding eigenvalue for \lambda_1. Then, if v(0) = v_1, the sequence v(t) converges to zero. This means that the "necessary condition" derived in (16) may not actually be a necessary condition for convergence.
There are also some points in the main text that doesn't describe the setting clearly; if the reader has no prior knowledge of MAML algorithm, they may get confused.
- In the update rule (1) and (2), it'd be better to mention that \nabla_\theta L_\tau (\theta) is an *estimate* of the true gradient using the training data and test data, respectively. In their current status, the gradients in (1) and (2) will read as the full gradients of the population where the training/test data points are sampled from.
- Section 2.1 only introduces the case where the update (1) is done only once in the inner loop. However, later Section 2.2 and Section 3, the paper says "only one step is taken for update...". Without the prior knowledge that there are versions in which multiple updates (1) are done, the readers can easily get confused.
Minor comments
- The word "minima" used throughout should better be corrected to "local minima," as mere minima may be understood as "global minima."
- In the abstract, there is a phrase "in contrast to the case of using the normal gradient descent method." Which setting do you mean by the "normal gradient descent"?
- After equations (3) and (4), the paper says "The above is known as the first-order approximation...", but as far as I'm concerned, the first-order approximation version of MAML is (3) without the hessian term, not (4). This statement can potentially be misleading.
- Column vectors / row vectors are mixed up. In (3), the partial derivatives are row vectors, but in (4) g_\tau(\theta)'s are column vectors. However, right above (3), g_\tau(\theta) is a row vector this time.
- Did (5) come from the approximation (4)? If so, as \tilde L(\theta) is defined to be L(\theta'), the two sides of (5) are "approximately equal" not "equal."
- In Eq (19): P_\tau (\theta - \theta^*) -> (\theta-\theta^*).
- At the end of page 6, what do you mean by "not a sum but a mean"? Isn't sum of n things equal to the mean, except for a factor of 1/n?