Many (but not all) approaches self-qualifying as "meta-learning" in deep learning and reinforcement learning fit a common pattern of approximating the solution to a nested optimization problem. In this paper, we give a formalization of this shared pattern, which we call GIMLI, prove its general requirements, and derive a general-purpose algorithm for implementing similar approaches. Based on this analysis and algorithm, we describe a library of our design, unnamedlib, which we share with the community to assist and enable future research into these kinds of meta-learning approaches. We end the paper by showcasing the practical applications of this framework and library through illustrative experiments and ablation studies which they facilitate.
Summary:
The authors present a PyTorch based framework for performing second-order
reverse mode autodiff for meta-learning.
First, the authors present a formalization of a general prototypical
meta-learning setting.
They then provide an algorithm that solves this problem via gradient based
optimization.
Finally, perhaps the main contribution is a specific PyTorch implementation
of said algorithm.
The type of meta-learning setting the authors consider is one where a gradient
based inner loop optimizer finds $\theta^\star$ by performing a finite number of steps.
The inner loop optimizer is parameterized through $\varphi$ that consists of
two parts $\varphi^\text{loss}$ and $\varphi^\text{opt}$.
The parameters $\varphi^\text{loss}$ are somehow part of the loss used for
training in the inner loop. Example: Regularization paramter.
The parameters $\varphi^\text{opt}$ do not occur in the loss but in the
optimizer step. Example: Learning rate.
Example of an inner loop step:
$\theta^{k+1} := \theta^k - \alpha (\nabla L(\theta^k) + \lambda \nabla R(\theta^k))$
where $\theta$ are the parameters of a neural network, $L$ is the training loss,
$R$ is the regularizer, $\alpha$ is the learning rate, $\lambda$ is the regularization parameter.
In this example we would have $\varphi = (\alpha \lambda)^T$.
The authos assume $\theta^K$, the output of the inner loop after $K$ steps to
be differentiable wrt $\varphi$.
Furthermore, the meta-learning loss is assumed to be differentiable wrt $\theta$
so that a gradient of the meta-learning loss wrt to $\varphi$ can be computed.
The authors also assume the meta-learning loss to be sufficient smooth in $\varphi$
such that a gradient based optimization can even be used for meta learning to
a local optimum.
The authors explicitly write down the reverse mode auto differentiation of
the inner loop and show how to, in that way, compute the gradient of the
meta-learning loss wrt to $\varphi$.
The reverse (adjoint) mode auto differentiation of the above example inner loop step
is the following step (iterated over in reverse down from $k = K$ to $k = 1$:
$\bar \theta^k := (I - \alpha (\nabla^2 L(\theta^k) + \lambda \nabla^2 R(\theta^k)))^T \bar \theta^{k+1}$
$\bar \alpha := \bar \alpha - (\nabla l(\theta^k) + \lambda \nabla R(\theta^k))^T \bar \theta^{k+1}$
$\bar \lambda := \bar \lambda - \alpha \nabla R(\theta^k)^T \bar \theta^{k+1}$
where $\bar \alpha$ accumulates the gradient of $\theta$ wrt $\alpha$ and
$\bar \lambda$ accumulates the gradient of $\theta$ wrt $\lambda$.
The authors give some implementation details specific to some frameworks necessary
for implementing such "gradient of an inner loop".
The authors present experiments where they show how to meta-learn learning rates
with their framework.
They also how their framework can be used to quickly implement a MAML type
meta-learning optimizer ablation study comparing various combinations of
architecture, optimizer and inner loop steps etc..
Recommendation:
I propose to reject the paper.
In my eyes the only contribution is the implementation of a meta-learner in
PyTorch based on well known methods.
The provided unifying formalization is theoretically inaccurate (see below)
and to me come across as merely a motivation for their framework
(but no value added compared to existing literature).
There is no new insight provided on the software engineering level either as
far as I can see.
Detailed comments:
- Page 1: ...provides tooling for analysing the provable requirements...
	seems like a complicated way of saying something that could be said simple
- Page 2: Without loss of generality, ...
	You are assuming a parametric model. Not sure what generality this phrase
	refers to.
- Page 2: A formalization as $\theta^\star = argmin(\theta, L(\theta, \varphi))$
	is inaccurate in the sense that it does not acknowledge the existing of
	multiple optima.
	I would recommend not to use the $argmin$ operator here, since $argmin$ for
	something like a neural network would for example either return a global
	optimum (which no optimizer used in practice finds, and is not ment here)
	or would take on a set value with multiple local minima for example.
	In the same context, the authors should mention the issues about uniqueness
	of optima (we are not even really finding optima when training neural networks),
	implicit functions / implicit differentiation
	In the context of using stochastic optimizers one should also at least
	mention something about the differentiability of outputs of such optimizers
	and how they potentially depend on randomness of mini-batches
	(what if different randomness is used with the same or a perturbed
	hyper parameter?)
- Page 3: You mention the potential statefulness of the optimizer.
	Why not explicitly carry it in the math notation?
	Probably things would get cluttered but saying it should be covered within
	$\varphi$ does not seem reasonable to me.
- Page 3: While this may seem like a fairly trivial formalization...
	Yes, but also nesting this in an outer loop is fairly trivial in the sense
	that it is a well known approach.
- Page 4: there exist continuous hyperparam...
	If they are not continuous then they should not even occur in this
	formalization so saying there exist... does not make much sense to me here
	$\alpha \subseteq \varphi^\text{opt}$ implies that $\varphi^\text{opt}$ is
	a set from notation although we are treating it as a vector everywhere else
- Page 4: All of section 2.4 seems somewhat trivial to me, but I guess that is
	highly subjective.
- Page 5: in the definition of stop operator perhaps use $:\Leftrightarrow$
- Page 5: Perhaps explicitly mention how your approach differs from a reverse
	mode differentiation of training or if it does not differ, say this.
- Page 14: When talking about _S_GD (instead of just GD) perhaps mention
	something about non-existence of mini-batch randomness / being deterministic