We address the challenging problem of deep representation learning-the efficient adaption of a pre-trained deep network to different tasks. Specifically, we propose to explore gradient-based features. These features are gradients of the model parameters with respect to a task-specific loss given an input sample. Our key innovation is the design of a linear model that incorporates both gradient features and the activation of the network. We show that our model provides a local linear approximation to a underlying deep model, and discuss important theoretical insights. Moreover, we present an efficient algorithm for the training and inference of our model without computing the actual gradients. Our method is evaluated across a number of representation learning tasks on several datasets and using different network architectures. Strong results are obtained in all settings, and are well-aligned with our theoretical insights.
Summary: This paper proposes to use the gradients of specific layers of convolutional networks as features in a linearized model for transfer learning and fast adaptation. The method is theoretically backed by an appeal to the recently proposed neural tangent kernel and seems like it could be practically useful.
Edit: Post rebuttal, I am somewhat satisfied by the authors' response and find their ablation study somewhat compelling hence am increasing my score to a weak accept. However, I'm not completely convinced that their choice of layers to use as gradient features is reasonable. If this paper does end up getting accepted, I'd very strongly advocate for being extremely clear as to 
1) why the gradient features used are used, 
2) a good choice of gradient features in practice
3) inclusion of all of the relevant ablation studies in the final version.
I tend to weakly reject this paper currently despite liking the simplicitly and timeliness of the approach. Specifically, I would like further discussion of the choice of the layers to use as gradient features and an ablation study on supervised trained networks.
Originality: I believe that this is one of the first papers to explicitly use the Taylor approximation (neural tangent kernel) in a transfer learning setting, making the approach timely and potentially practically useful. 
Significance: The approach is a quite nice merging of theoretical insights with a neat practical implementation. Although the main methodological advance is a straightforward application of the thinking behind Jacobian vector products, the method is well described and ought to be practically useful. However, I have a bit of a concern as to its practical necessity in comparison to simple fine-tuning of the final softmax layer (referred to in Table 1 as \theta_2) on a new dataset. 
Clarity: While the approach described in Section 3 is quite generic – theoretically, the method should simply consist of training the final layer of the neural network (w_1) and weights from the Jacobian features around the Taylor expansion. However, the experimental approaches suggest that different layers were used in each experiment – see e.g. “[we] … compute our gradient based features from one, two, or all of the top-3 conv layers” (Section 4.2 for the BiGAN architectures) versus “our gradient based features are from the last 2 conv layers” (Table 2 for the AlexNet architectures). Why was the entire network (modulo the last feed forward layer) not simply used for the Jacobian features? 
Similarly, why was the entirety of the model (again modulo the last feedforward layer) not used for forwards model in the AlexNet experiments?
Based on the ablation study in Section 4.1, the authors find that “it suffices to set the very top layer as \theta_2 to enjoy a reasonably large performance gain.” When this is the case, it is a bit tough to distinguish the approach from standard final layer transferring approaches (Sharif Razavian et al, 2014) and indeed Bayesian final layer approaches that simply retrain an output layer (e.g. Perrone et al, 2018). Indeed this seems to be the case in Table 1, if that is so, then why not just re-train \theta_2 as well throughout the experiments as it will typically just be another stochastic convex optimization problem?
Could the authors quickly describe an implementation of the scalable Jacobian inner products in Section 3.3? It seems like outputs will have to be cached during the forwards passes, thereby requiring a somewhat significant amount of software engineering (and memory overhead) to be have to do this process for each new layer. Is this the correct understanding of how one would implement the procedure? A quick paste of PyTorch pseudo-code would be sufficient here.
Quality: The experiments seem to be relatively convincing – it seems exciting that a linear model + the network’s features itself can typically perform as well as fine-tuning and occasionally even better than fine-tuning itself. 
However, I am a bit concerned by the fact that the ablation studies themselves only utilize models trained in an un-supervised fashion. 
I’d instead like the authors to run an ablation study on supervised trained models (perhaps 8 layer conv nets or VGG16) in the same manner as in Table 1 and Section 4.1. Specifically, I’d like to see this done to see whether the gradients in fact are as interesting as features when they have been trained in a supervised fashion. 
Similarly, I’d like to see the features themselves used as a linear classifier (no network forwards passed) in the same two ablation studies. That is, could the authors use w_1^T J w_2 as the features for their linear classifier. If they have already done so and I’ve missed that in the tables somehow, I apologize. This experiment should help to test out how _useful_ the features defined by the Jacobian matrix are in comparison to the network’s forward pass itself.
Minor Comments: 
Introduction: “After learning, the the (sic) activation of the deep network are considered as generic features.” Not only is there a small typo, but you should either include a citation here or be more specific as to what “the activation” of the deep network is here.
Introduction: “And the accuracy of the rthe (sic) …” typo + please do not start sentences with and if at all possible.
Section 3.1 (beneath eq. 2): “liner” should be linear.
Table 3: “Self-supervise” should be “Self-supervised.”
References: 
Perrone et al, Scalable Hyperparameter Transfer Learning, NeurIPS, 2018. http://papers.nips.cc/paper/7917-scalable-hyperparameter-transfer-learning.pdf
Sharif Razavian et al, CNN Features off-the-Shelf: an Astounding Baseline for Image Recognition, CVPRW, 2014. https://arxiv.org/abs/1403.6382