TRANSFERRING KNOWLEDGE ACROSS LEARNING PROCESSES
In complex transfer learning scenarios new tasks might not be tightly linked to previous tasks. Approaches that transfer information contained only in the final parameters of a source model will therefore struggle. Instead, transfer learning at a higher level of abstraction is needed. We propose Leap, a framework that achieves this by transferring knowledge across learning processes. We associate each task with a manifold on which the training process travels from initialization to final parameters and construct a meta-learning objective that minimizes the expected length of this path. Our framework leverages only information obtained during training and can be computed on the fly at negligible cost. We demonstrate that our framework outperforms competing methods, both in meta-learning and transfer learning, on a set of computer vision tasks. Finally, we demonstrate that Leap can transfer knowledge across learning processes in demanding reinforcement learning environments (Atari) that involve millions of gradient steps.
\documentclass[10pt]{article}
\usepackage{geometry}[1in]
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{enumerate}
\usepackage{indentfirst}
\begin{document}
	
	\section*{SUMMARY}
	
	The article proposes Leap, a novel meta-learning objective aimed at outperforming state-of-the-art approaches when dealing with collections of tasks that exhibit substantial between-task diversity.
	
	Similarly to prior work such as MAML [1] or Reptile [2], the goal of Leap is to learn an initialization $\theta_{0}$ for the model parameters, shared across tasks, which leads to good and data-efficient generalization performance when fine-tuning the model on a set of held-out tasks. In a nutshell, what sets Leap apart from MAML or Reptile is its cost function, which explicitly accounts for the entire path traversed by the model parameters during task-specific fine-tuning -- i.e., ``inner loop'' optimization --, rather than mainly focusing on the final value attained by the model parameters after fine-tuning. More precisely, Leap looks for an initialization $\theta_{0}$ of the model parameters such that the energy of the path traversed by $\gamma_{\tau}(\theta) = (\theta, f_{\tau}(\theta))$ while fine-tuning $\theta$ to optimize the loss $f_{\tau}(\theta)$ of a task $\tau$ is minimized, on average, across $\tau \sim p(\tau)$. Thus, it could be argued that Leap extends Reptile, which can be informally understood as seeking an initialization $\theta_{0}$ that minimizes the average squared Euclidean distance between $\theta_{0}$ and the model parameters after fine-tuning on each task $\tau \sim p(\tau)$ [2, Section 5.2], by using a distance function between initial and final model parameters that accounts for the geometry of the loss surface of each task during optimization.  
	
	The final algorithm introduced in the paper considers however a variant of the aforementioned cost function, motivated by its authors on the basis of stabilising learning and eliminating the need for Hessian-vector products. The resulting approach is then evaluated on image recognition tasks (Omniglot plus a set of six additional computer vision datasets) as well as reinforcement learning tasks (Atari games).
	
	\section*{HIGH-LEVEL ASSESSMENT}
	
	The article proposes an interesting extension of existing work in meta-learning. In a slightly different context (meta-optimization), recent work [3] pointed out the existence of a ``short-horizon bias'' which could arise when using meta-learning objectives that apply only a small number of updates during ``inner-loop'' optimization. This observation is well-aligned with the motivation of this article, in which the authors attempt to complement successful methods like MAML or Reptile to perform well also in situations where a large number of gradient descent-based updates are applied during task-specific fine-tuning. Consequently, I believe the article is timely and relevant.
	
	Unfortunately, I have some concerns with the current version of the manuscript regarding (i) the proposed approach and the way it is motivated, (ii) the underlying theoretical results and, perhaps most importantly, (iii) the experimental evaluation. In my opinion, these should ideally be tackled prior to publication. Nonetheless, I believe that the proposed approach is promising and that these concerns can be either addressed or clarified. Thus I look forward to the rebuttal.
	
	\section*{MAJOR POINTS}
	
	\subsection*{1. Issues regarding proposed approach and its motivation/derivation}
	
	\textbf{1.a} Section 2.1 argues in favour of studying the path traversed by $\gamma_{\tau}(\theta) = (\theta, f_{\tau}(\theta))$ rather than the path traversed by the model parameters $\theta$ alone. However, this could in turn exacerbate the difficulty in dealing with collections of tasks for which the loss functions have highly diverse scales. For instance, taking the situation to the extreme, one could define an equivalence class of tasks $[\tau] = \left\{\tau \mid f_{\tau}(\theta) = g(\theta) + \mathrm{constant} \right\}$ such that any two tasks $\tau_{1}, \tau_{2} \in [\tau]$ would essentially represent the same underlying task, but could lead to arbitrarily different values of the Leap cost function. 
	
	Given that Leap is a model-agnostic approach, like MAML or Reptile, and thus could be potentially applied in many different settings and domains, I believe the authors should study and discuss (theoretically or experimentally) the robustness of Leap with respect to between-task variation in the scale of the loss functions and, in case the method is indeed sensitive to those, propose an effective scheme to normalize them.
	
	\textbf{1.b} The current version of the manuscript motivates defining the cost function in terms of $\gamma_{\tau}(\theta) = (\theta, f_{\tau}(\theta))$ rather than the model parameters $\theta$ alone in order to ``avoid information loss'', making it seem that this modification is ``optional'' or, at least, not critical. Nevertheless, taking a closer look at the Leap objective and the meta-updates it induces, I believe it might actually be essential for the correctness of the approach. I elaborate this view in what follows. Let us write the Leap objective for a task $\tau$ as
	\[
	F_{\tau}(\theta_{0},\widetilde{\theta}_{0}) = \underbrace{\sum_{i=0}^{K_{\tau} - 1}{\left\vert\left\vert u^{(i+1)}_{\tau}(\widetilde{\theta}_{0}) - u^{(i)}_{\tau}(\theta_{0}) \right\vert\right\vert^{2}}}_{C_{\tau, 1}(\theta_{0},\widetilde{\theta}_{0})} + \underbrace{\sum_{i=0}^{K_{\tau} - 1}{\left( f_{\tau}\left(u^{(i+1)}_{\tau}(\widetilde{\theta}_{0})\right) - f_{\tau}\left(u^{(i)}_{\tau}(\theta_{0})\right) \right)^{2}}}_{C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0})},
	\]
	where $\widetilde{\theta}_{0}$ denotes a ``frozen'' or ``detached'' copy of $\theta_{0}$ and $u^{(i)}_{\tau}$ maps $\theta_{0}$ to $\theta_{i}$, the model parameters after applying $i$ gradient descent updates to $f_{\tau}$ according to Equation (1) in the manuscript. Then, differentiating $C_{\tau, 1}(\theta_{0},\widetilde{\theta}_{0})$ and $C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0})$ with respect to $\theta_{0}$ separately yields:
	\begin{align*}
	\nabla_{\theta_{0}} C_{\tau, 1}(\theta_{0},\widetilde{\theta}_{0}) &= -2 \sum_{i=0}^{K_{\tau} - 1}{J_{i}^{T}\left(\theta_{i+1} - \theta_{i} \right)} = -2 \alpha \sum_{i=0}^{K_{\tau} - 1}{J_{i}^{T} g_{i}} \\
	\nabla_{\theta_{0}} C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0}) &= -2 \sum_{i=0}^{K_{\tau} - 1}{\left(f_{\tau}(\theta_{i+1}) -  f_{\tau}(\theta_{i})\right) J_{i}^{T}g_{i}} = -2 \sum_{i=0}^{K_{\tau} - 1}{\Delta f^{i}_{\tau} J_{i}^{T}g_{i}}
	\end{align*}
	where $J_{i} = J_{\theta_{0}}u^{(i)}_{\tau}(\theta_{0})$ denotes the Jacobian of $u^{(i)}_{\tau}$ with respect to $\theta_{0}$, $g_{i} = \left. \nabla_{\theta} f_{\tau}(\theta)\right\rvert_{\theta=\theta_{i}}$ denotes the gradient of the loss function $f_{\tau}$ evaluated at $\theta_{i}$ and $\Delta f^{i}_{\tau} = f_{\tau}(\theta_{i+1}) -  f_{\tau}(\theta_{i})$ stands for the change in the loss function after the $i$-th update. To simplify the exposition, a constant ``inner-loop'' learning rate and no preconditioning were assumed, i.e., $\alpha_{i} = \alpha$ and $S_{i} = I$.
	
	Furthermore, the article claims that all Jacobian terms are approximated by identity matrices (i.e., $J_{i} = I$) as suggested in Section 5.2 of [1], leading to the following approximations:
	\begin{align*}
		\nabla_{\theta_{0}} C_{\tau, 1}(\theta_{0},\widetilde{\theta}_{0}) \approx -2 \alpha \sum_{i=0}^{K_{\tau} - 1}{ g_{i}} \\
		\nabla_{\theta_{0}} C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0}) \approx -2 \sum_{i=0}^{K_{\tau} - 1}{\Delta f^{i}_{\tau} g_{i}}
	\end{align*}
	
	Interestingly, it can be seen that the contribution to the meta-update of the energy of the path traversed by the model parameters $\theta$, $g_{\mathrm{Leap},1} =\nabla_{\theta_{0}} C_{\tau, 1}(\theta_{0},\widetilde{\theta}_{0})$, actually points in exactly the opposite direction than the meta-update of Reptile, given by $g_{\mathrm{Reptile}} = \sum_{i=0}^{K_{\tau} - 1}{g_{i}}$ (e.g. Equation (27) in [2]). In summary, if the Leap objective was defined in terms of $\theta$ rather than $(\theta, f_{\tau}(\theta))$, minimising the Leap cost function should maximise Reptile's cost function and viceversa. It is only the term $g_{\mathrm{Leap},2} =\nabla_{\theta_{0}} C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0})$ that presumably ``re-aligns'' $g_{\mathrm{Reptile}}$ and $g_{\mathrm{Leap}} = g_{\mathrm{Leap},1} + g_{\mathrm{Leap},2}$. Indeed, 
	\[
	g_{\mathrm{Leap}} = 2 \sum_{i=0}^{K_{\tau} - 1}{\left(-\Delta f^{i}_{\tau} - \alpha \right) g_{i}}
	\]
	will have positive inner product with $g_{\mathrm{Reptile}}$ if each gradient update yields a sufficient decrease in the loss $f_{\tau}$, that is, $\Delta f^{i}_{\tau} < -\alpha$.
	
	Moreover, I also wonder if this is the reason why the authors introduce the ``regularization'' term $\mu_{\tau}^{i}$, which as it currently stands in the manuscript, does not seem to relate in a particularly intuitive manner to the original objective of minimising the energy of $\gamma(t)$. By introducing $\mu_{\tau}^{i}$, the term $C_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0})$ becomes
	\[
		C^{\prime}_{\tau, 2}(\theta_{0},\widetilde{\theta}_{0}) = \sum_{i=0}^{K_{\tau} - 1}{-\mathrm{sign}  \left( f_{\tau}\left(u^{(i+1)}_{\tau}(\widetilde{\theta}_{0})\right) - f_{\tau}\left(u^{(i)}_{\tau}(\theta_{0})\right) \right) \left( f_{\tau}\left(u^{(i+1)}_{\tau}(\widetilde{\theta}_{0})\right) - f_{\tau}\left(u^{(i)}_{\tau}(\theta_{0})\right) \right)^{2}},
	\]
	leading to $g^{\prime}_{\mathrm{Leap},2} = 2 \sum_{i=0}^{K_{\tau} - 1}{\vert \Delta f^{i}_{\tau} \vert g_{i}}$ and 
	\[
	g^{\prime}_{\mathrm{Leap}} = 2 \sum_{i=0}^{K_{\tau} - 1}{\left(\vert \Delta f^{i}_{\tau} \vert - \alpha \right) g_{i}}.
	\]
	In turn, this relaxes the sufficient condition under which Leap and Reptile lead to meta-updates with positive inner product, namely, it changes the condition $\Delta f^{i}_{\tau} < -\alpha$ by a less restrictive counterpart $\vert \Delta f^{i}_{\tau} \vert \ge \alpha$.
	
	If these derivations happen to be correct, then I believe the way Leap is currently motivated in the article could be argued to be slightly misleading. What seems to be its main inspiration, accounting for the path that the model parameters traverse during fine-tuning, does not seem to be what drives the meta-updates towards the ``correct'' direction. Instead, the component of the objective due to the path traversed by the loss function values appears to be more important or, at least, not optional. Furthermore, I believe the regularization term $\mu_{\tau}^{i}$ should be better motivated, as the current version of the manuscript does not seem to justify its need clearly enough.
	
	Finally, under the assumption that the above is not mistaken, I wonder whether further tweaks to the meta-update, such as $g^{\prime\prime}_{\mathrm{Leap}} = 2 \sum_{i=0}^{K_{\tau} - 1}{\mathrm{max}\left(\vert \Delta f^{i}_{\tau} \vert - \alpha, 0 \right) g_{i}}$, could perhaps turn out to be helpful as well.
	\subsection*{2. Theoretical results}
	
	\textbf{2.a} Theorem 1 currently claims that the Pull-Forward algorithm converges to a local minimum of Equation (5). However, due to the non-convexity of the objective function, only convergence to a stationary point is established.
	
	\textbf{2.b} Most importantly, I am not entirely certain that the proof of Theorem 1 is complete in its current form. As I understand it, using the notation introduced by the authors in Appendix A, the following identities hold:
	\begin{align*}
		F(\psi_{s};\Psi_{s}) &= \mathbb{E}_{\tau,i} \vert\vert h_{\tau}^{i} - z_{\tau}^{i} \vert\vert^{2} \\
		F(\psi_{s+1};\Psi_{s}) &= \mathbb{E}_{\tau,i} \vert\vert h_{\tau}^{i} - x_{\tau}^{i} \vert\vert^{2} \\
		F(\psi_{s};\Psi_{s+1}) &= \mathbb{E}_{\tau,i} \vert\vert y_{\tau}^{i} - z_{\tau}^{i} \vert\vert^{2} \\
		F(\psi_{s+1};\Psi_{s+1}) &= \mathbb{E}_{\tau,i} \vert\vert y_{\tau}^{i} - x_{\tau}^{i} \vert\vert^{2}.
	\end{align*}
	
	The bulk of the proof is then devoted to show that $\mathbb{E}_{\tau,i} \vert\vert y_{\tau}^{i} - z_{\tau}^{i} \vert\vert^{2} = F(\psi_{s};\Psi_{s+1}) \ge \mathbb{E}_{\tau,i} \vert\vert y_{\tau}^{i} - x_{\tau}^{i} \vert\vert^{2} = F(\psi_{s+1};\Psi_{s+1})$. However, I do not immediately see how to make the final ``leap'' from $F(\psi_{s+1};\Psi_{s+1}) \le F(\psi_{s};\Psi_{s+1})$ to the actual claim of the Theorem, $F(\psi_{s+1};\Psi_{s+1}) \le F(\psi_{s};\Psi_{s})$.
	
	\subsection*{3. Experimental evaluation}
	
	\textbf{3.a} The experimental setup of Section 4.1 closely resembles experiments described in articles that introduced continual learning approaches, such as [4]. However, rather than including [4] as a baseline, the current manuscript compares against meta-learning approaches typically used for few-shot learning, such as MAML and Reptile. Consequently, I would argue the combination of experimental setup and selection of baselines is not entirely fair or, at least, it is incomplete.
	
	To this end, I would suggest to (i) include [4] (or a related continual learning approach) as an additional baseline in the experiments currently described in Section 4.1 as well as (ii) perform a new experiment to compare the performance of Leap to that of MAML and Reptile in few-shot classification tasks using OmniGlot and/or Mini-ImageNet as datasets.
	
	\textbf{3.b} The Multi-CV experiment described in Section 4.2 currently does not have strong baselines other than Leap. If possible, I would suggest including [5] in the comparison, as it is the article which inspired this particular experiment.
	
	\textbf{3.b} Likewise, the same holds for the experiment described in Section 4.3. In this case, I would suggest comparing to [4] for the same reason described above.
	
	\section*{MINOR POINTS}
	
	\begin{enumerate}
	
	\item In Section 2.1, it is claimed that "gradients that largely point in the same direction indicate a convex loss surface, whereas gradients with frequently opposing directions indicate an ill-conditioned loss landscape". Nevertheless, convex loss surfaces can in principle be ill-conditioned as well.
	
	\item Introducing a mathematical definition for the metric "area under the training curve" could make the experiment in Section 4.1 more self-contained.
	
	\item Several references are outdated, as they cite preprints that have since been accepted at peer-reviewed venues.
	
	\item The reinforcement learning experiments in Section 4.3 would benefit from additional runs with multiple seeds, and the subsequent inclusion of confidence intervals.
	
	\item I believe certain additional experiments could be insightful. For example, (i) studying how sensitive the performance of Leap is to parameter of the ``inner-loop'' optimizer (e.g. choice of 
	optimizer, learning rate, batch size) or (ii) describing how the introduction of $\mu_{\tau}^{i}$ affects the performance of Leap.
	
	\end{enumerate}
	
	\section*{TYPOS}
	
	\begin{enumerate}
	
	\item The first sentence entirely in page 6 appears to have a superfluous word.
	
	\item The Taylor series expansion in the proof of Theorem 1 is missing the $O(\bullet)$ terms (or a $\approx$ sign).
	
	\item Also in the proof of Theorem 1, if $c_{\tau}^{i} = (\delta_{\tau}^{i})^{2} - \alpha_{\tau}^{i}\xi_{\tau}^{i}\delta_{\tau}^{i}$, wouldn't $\omega = \underset{\tau, i}{\mathrm{sup}} \langle \hat{x}^{i}_{\tau} - \hat{z}^{i}_{\tau}, g(\hat{x}^{i}_{\tau}) - g(\hat{z}^{i}_{\tau})\rangle + \xi_{\tau}^{i}\delta_{\tau}^{i}$ instead?
	
	\end{enumerate}
        \section*{ANSWER TO REBUTTAL}
        Please see comments in the thread.
	
	\section*{REFERENCES}
	
	\begin{enumerate}[ {[}1{]} ]
		\item Finn et al. ``Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.'' International Conference on Machine Learning. 2017.
		\item Nichol et al. ``On First-Order Meta-Learning Algorithms.'' arXiv preprint. 2018
		\item Wu et al. ``Understanding Short-Horizon Bias in Stochastic Meta-Optimization.'' International Conference on Learning Representations. 2018.
		\item Schwarz et al. ``Progress \& Compress: A scalable framework for continual learning.''  International Conference on Machine Learning. 2018.
		\item Serr{\`a} et al. ``Overcoming Catastrophic Forgetting with Hard Attention to the Task.''  International Conference on Machine Learning. 2018.
	\end{enumerate}	
\end{document}