Learning feed-forward one-shot learners
One-shot learning is usually tackled by using generative models or discriminative embeddings. Discriminative methods based on deep learning, which are very effective in other learning scenarios, are ill-suited for one-shot learning as they need large amounts of training data. In this paper, we propose a method to learn the parameters of a deep model in one shot. We construct the learner as a second deep network, called a learnet, which predicts the parameters of a pupil network from a single exemplar. In this manner we obtain an efficient feed-forward one-shot learner, trained end-to-end by minimizing a one-shot classification objective in a learning to learn formulation. In order to make the construction feasible, we propose a number of factorizations of the parameters of the pupil network. We demonstrate encouraging results by learning characters from single exemplars in Omniglot, and by tracking visual objects from a single initial exemplar in the Visual Object Tracking benchmark.
This paper focuses on the problem of one-shot learning, where one wants to build a new classifier given just a single training image. The obvious complication in this problem is that with just one image it is very hard to predict whether the resulting network can generalize well in new categories, as the danger of overfitting is very high.  To deal with this problem the authors propose to learn a second "meta"-network, whose outputs will be the parameters of the classification network. As the potential number of parameters to be returned is very high, especially given the number of available training images, the authors propose to factorize the matrix-vector multiplications of a fully connected in a form (M * diag(W) * M') and learn only a diagonal matrix W, while having M and M' pre-trained. Similarly, for the convolutional layers a filter basis is learnt and at test time it is decided which filter should be applied on which channel.  Three different architectures are presented: a) a siamese architecture casting the one-shot classification as an image comparison problem, b) a siamese learnet, which predicts the shared parameters for some layers in a siamese network that compares images, and c) a learnet, which directly returns the parameters for the (not siamese) classification network. The method is evaluated on character recognition and on tracking.
The paper overall looks interesting with some nice ideas. Below I make a list of positive and negative elements and some remarks.  Positives --------------- + The proposed method and the factorization makes sense and is interesting. Could one explain the matrices M and M' as learning some latent, topic models and then using w(z) to select the right proportions of these topic models for the novel category? Alternatively, could there be a connection with sparce coding, where at test time one needs to recover the right set of basis vectors?  + The results support the method. In two benchmarks the obtained accuracies improve over reasonable (but not neceessarily the latest state-of-the-art) baselines.  + The writing is mostly clear, except for section 2.3 (see below).   Negatives & Remarks ----------------------  - The paper seems to overclaim and fails to cite some relevant papers. First and most importantly, there are some previous works, which also learn parameters dynamically, such as "A Dynamic Convolutional Layer for Short Range Weather Prediction" presented in CVPR 2015 and "DPPnet: Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction" in arXiv, 2015. Although the first paper is not for one-shot learning, for image question answering the model also needs to learn in one-shot. Also, regarding object tracking the proposed method seems very close to "Siamese Instance Search for Tracking", CVPR 2016. The paper needs to make more precise statements and incorporate missing references.  - Given that one wants discriminative classifiers, wouldn't it make more sense to consider triplet siamese networks that consider positive and negative pairs at the same time? How would such a network work compare to the standalone learnet?  - Some clearer explanation would be welcome is in Section 2.3, where the factorized convolutional layers are explained. What is it exactly meant by saying that "M and M' disentangle the feature channels, allowing w(z) to choose which filter to apply to each channel"? Is there an over-redundant number of channels, which is then reshuffled and the right set of channels are selected to be convolved with the right filters? Also, it is not clear why the factorization of the convolutional layers should be convolutional by itself? Namely, the convolution operates as normal between filters F and the input x. Why does F need to also rely on some convolutional operation as well? Could one directly predict the parameters of the convolutional filters?  - How is the weighted l1 distance in Table 1 computed?