SNOW is an efficient learning method to improve training/serving throughput as well as accuracy for transfer and lifelong learning of convolutional neural networks based on knowledge subscription. SNOW selects the top-K useful intermediate feature maps for a target task from a pre-trained and frozen source model through a novel channel pooling scheme, and utilizes them in the task-specific delta model. The source model is responsible for generating a large number of generic feature maps. Meanwhile, the delta model selectively subscribes to those feature maps and fuses them with its local ones to deliver high accuracy for the target task. Since a source model takes part in both training and serving of all target tasks in an inference-only mode, one source model can serve multiple delta models, enabling significant computation sharing. The sizes of such delta models are fractional of the source model, thus SNOW also provides model-size efficiency. Our experimental results show that SNOW offers a superior balance between accuracy and training/inference speed for various image classification tasks to the existing transfer and lifelong learning practices.
*** Increased to Accept from Weak Accept after author rebuttal and changes to the paper ***
This paper proposes a method, SNOW, for improving the speed of training and inference for transfer and lifelong learning. SNOW starts with a pre-trained, frozen source model, and trains delta models for target tasks which, at each layer, concatenate a small number of task-specific features with the top-K most useful subset of features in the corresponding layer in the source model. As long as the target tasks are sufficiently related to the source task, it allows for small delta models and a small additional parameter overhead in the form of one weight per source model feature map. While there are (i) some issues with the presentation of results for training efficiency, (ii) some question marks over the sensitivity of the model to hyper-parameters, and (iii) several grammatical errors / typos in the manuscript, if these can be addressed I recommend the paper for acceptance because it seems to strike a superior balance of efficiency (regarding memory usage and inference speed) and accuracy when compared to a number of baselines, and to my knowledge it is a novel approach.
Detailed comments:
* Section 2.2 - How is sigma (the exploratory noise added for feature selection during training) chosen and how sensitive is the approach to its value? It seems like it was fine-tuned, given that a different sigma is chosen for the Action dataset (several orders of magnitude difference). In practice, tuning sigma could significantly increase training time. 
* It seems like the performance of only one run was plotted per hyperparameter setting - it would be informative to see a mean and standard deviation especially since the approach seems like it could be unstable for the wrong hyperparameter settings.
* Related to the previous point, how much do the top-K feature selections change throughout training? One would have thought that this could cause instability during training for a high sigma. If sigma is too low, you could end up with suboptimal feature selection.
* Figure 4 graphs are a bit misleading because the throughput on the x-axis is reported per GPU and the larger models all need 2 or more GPUs. While this is mentioned in the main text, it is still optically deceptive and the results are GPU-dependent - presumably if the GPUs had a larger memory, the larger models would not seems as slow. I think it would be clearer to plot images/sec on the x-axis or to rerun the experiments just using a single GPU.
* It is stated that  “[d]etermining K […] has a critical impact on both size and target accuracy in the target models”, where K is the number of feature maps in the source model that the delta model subscribes to in each layer. How sensitive is the accuracy exactly? Can this be quantified or discussed in more detail?
* Furthermore, how sensitive is the performance to the number of target-model-specific features at each layer?
* Different learning rate schedules were used for SNOW and baselines - initial lr for SNOW is 1.0, while for all other models it is 0.1. Was it checked whether the baselines improve when they are run with an initial lr of 1.0? Was this hyperparameter more heavily tuned for SNOW than for the baselines?
* Since the source model is fixed, the applicability of the approach to lifelong learning is heavily dependent on the usefulness of the source model to subsequent tasks. If it is not, then one will have to incorporate large delta models. Furthermore, there can be no transfer between the tasks trained in the delta models.
 
Grammatical errors / suggestions:
* Page 1, first line: “hallmark” doesn’t make sense in this context - maybe “key objective” or “goal”?
* Page 1, 2nd paragraph, first line: “wee” -> “we”.
* Page 1, 2nd paragraph, line 6: “best top-K” -> either “K best” or “top K"
* Page 2, last paragraph, first line: “three folds” -> “threefold"
* Section 2.1, line 2: “pooing”->”pooling”. Same typo on Page 4, last line.
* Page 6, line 1: “training from the scratch” -> “training from scratch"
* Page 6, line 9: “more 6x than” -> “6x more than"
* Overall, the manuscript needs to be proofread a few times.