Learning To Learn Around A Common Mean
The problem of learning-to-learn (LTL) or meta-learning is gaining increasing attention due to recent empirical evidence of its effectiveness in applications. The goal addressed in LTL is to select an algorithm that works well on tasks sampled from a meta-distribution. In this work, we consider the family of algorithms given by a variant of Ridge Regression, in which the regularizer is the square distance to an unknown mean vector. We show that, in this setting, the LTL problem can be reformulated as a Least Squares (LS) problem and we exploit a novel metaalgorithm to efficiently solve it. At each iteration the meta-algorithm processes only one dataset. Specifically, it firstly estimates the stochastic LS objective function, by splitting this dataset into two subsets used to train and test the inner algorithm, respectively. Secondly, it performs a stochastic gradient step with the estimated value. Under specific assumptions, we present a bound for the generalization error of our meta-algorithm, which suggests the right splitting parameter to choose. When the hyper-parameters of the problem are fixed, this bound is consistent as the number of tasks grows, even if the sample size is kept constant. Preliminary experiments confirm our theoretical findings, highlighting the advantage of our approach, with respect to independent task learning.
This paper studies how to define an algorithm that, given an increasing number of tasks sampled from a defined environment,  will train on them and learn a model that will be well suited for any new task sampled from the same environment. The scenario  just described corresponds to the 'learning to lean' problem where a learning agent improves its learning performance with the number of tasks.  Specifically in this work the focus is on the 'ridge regression' family of algorithms and the environment consists in tasks that can be  solved by ridge regression with models around a common mean. In other words, we need a learning algorithm that besides solving regression problems, progressively learns how to approximate the environment model mean.   The transfer risk is a measure of how much the knowledge acquired over certain available tasks allow to improve future learning.  In the proposed setting the transfer risk corresponds to the difference between the estimated model environmental mean and the correct environmental mean. It can be well estimated with theoretical guarantees and can be explicitly minimized to 'learn to learn' the optimal regression model.  The novel contribution of the paper is in the choice of splitting the data of each tasks in two parts, one on which the regression problem is learned and the training error is minimized and a second one useful to estimate and minimize the transfer risk. Finally the proposed strategy is tested onto two synthetic dataset where the results confirm the paper thesis.  Ovearall the paper is well written but I find it dense in some points which makes it a little bit difficult to read. The proposed method is theoretically grounded and I did not find any evident flaw. I just remain with one doubt: what could be real-world cases where the assumptions done for the synthetic settings hold? And what would be the challenges to be tackled in case of extensions to non-convex learning problems?  After rebuttal ----------------- I've noticed that in the rebuttal some of the reviewers' questions are indicated as 'open problems' or deferred to future works, Anyway I do not have strong arguments against this paper. I align with the other reviewers's score by raising my rate from 6 to 7.