Beyond Fine Tuning: A Modular Approach to Learning on Small Data
In this paper we present a technique to train neural network models on small amounts of data. Current methods for training neural networks on small amounts of rich data typically rely on strategies such as fine-tuning a pre-trained neural network or the use of domain-specific hand-engineered features. Here we take the approach of treating network layers, or entire networks, as modules and combine pre-trained modules with untrained modules, to learn the shift in distributions between data sets. The central impact of using a modular approach comes from adding new representations to a network, as opposed to replacing representations via fine-tuning. Using this technique, we are able surpass results using standard fine-tuning transfer learning approaches, and we are also able to significantly increase performance over such approaches when using smaller amounts of data.
This paper proposes a method of augmenting pre-trained networks for one task with an additional inference path specific to an additional task, as a replacement for the standard “fine-tuning” approach. Pros: -The method is simple and clearly explained. -Standard fine-tuning is used widely, so improvements to and analysis of it should be of general interest. -Experiments are performed in multiple domains -- vision and NLP. Cons: -The additional modules incur a rather large cost, resulting in 2x the parameters and roughly 3x the computation of the original network (for the “stiched” network). These costs are not addressed in the paper text, and make the method significantly less practical for real-world use where performance is very often important. -Given these large additional costs, the core of the idea is not sufficiently validated, to me. In order to verify that the improved performance is actually coming from some unique aspects of the proposed technique, rather than simply the fact that a higher-capacity network is being used, some additional baselines are needed: (1) Allowing the original network weights to be learned for the target task, as well as the additional module. Outperforming this baseline on the validation set would verify that freezing the original weights provides an interesting form of regularization for the network. (2) Training the full module/stitched network from scratch on the *source* task, then fine-tuning it for the target task. Outperforming this baseline would verify that having a set of weights which never “sees” the source dataset is useful. -The method is not evaluated on ImageNet, which is far and away the most common domain in which pre-trained networks are used and fine-tuned for other tasks. I’ve never seen networks pre-trained on CIFAR deployed anywhere, and it’s hard to know whether the method will be practically useful for computer vision applications based on CIFAR results -- often improved performance on CIFAR does not translate to ImageNet. (In other contexts, such as more theoretical contributions, having results only on small datasets is acceptable to me, but network fine-tuning is far enough on the “practical” end of the spectrum that claiming an improvement to it should necessitate an ImageNet evaluation.) Overall I think the proposed idea is interesting and potentially promising, but in its current form is not sufficiently evaluated to convince me that the performance boosts don’t simply come from the use of a larger network, and the lack of ImageNet evaluation calls into question its real-world application. =============== Edit (1/23/17): I had indeed missed the fact that the Stanford Cars does do transfer learning from ImageNet -- thanks for the correction. However, the experiment in this case is only showing late fusion ensembling, which is a conventional approach compared with the *stitched network* idea which is the real novelty of the paper. Furthermore the results in this case are particularly weak, showing only that an ensemble of ResNet+VGG outperforms VGG alone, which is completely expected given that ResNet alone is a stronger base network than VGG (*ResNet+VGG > ResNet* would be a stronger result, but still not surprising). Demonstrating the stitched network idea on ImageNet, comparing with the corresponding VGG-only or ResNet-only finetuning, could be enough to push this paper over the bar for me, but the current version of the experiments here don*t sufficiently validate the stitched network idea, in my opinion.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Beyond Fine Tuning: A Modular Approach to Learning on Small Data
In this paper we present a technique to train neural network models on small amounts of data. Current methods for training neural networks on small amounts of rich data typically rely on strategies such as fine-tuning a pre-trained neural network or the use of domain-specific hand-engineered features. Here we take the approach of treating network layers, or entire networks, as modules and combine pre-trained modules with untrained modules, to learn the shift in distributions between data sets. The central impact of using a modular approach comes from adding new representations to a network, as opposed to replacing representations via fine-tuning. Using this technique, we are able surpass results using standard fine-tuning transfer learning approaches, and we are also able to significantly increase performance over such approaches when using smaller amounts of data.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes a method of augmenting pre-trained networks for one task with an additional inference path specific to an additional task, as a replacement for the standard “fine-tuning” approach. Pros: -The method is simple and clearly explained. -Standard fine-tuning is used widely, so improvements to and analysis of it should be of general interest. -Experiments are performed in multiple domains -- vision and NLP. Cons: -The additional modules incur a rather large cost, resulting in 2x the parameters and roughly 3x the computation of the original network (for the “stiched” network). These costs are not addressed in the paper text, and make the method significantly less practical for real-world use where performance is very often important. -Given these large additional costs, the core of the idea is not sufficiently validated, to me. In order to verify that the improved performance is actually coming from some unique aspects of the proposed technique, rather than simply the fact that a higher-capacity network is being used, some additional baselines are needed: (1) Allowing the original network weights to be learned for the target task, as well as the additional module. Outperforming this baseline on the validation set would verify that freezing the original weights provides an interesting form of regularization for the network. (2) Training the full module/stitched network from scratch on the *source* task, then fine-tuning it for the target task. Outperforming this baseline would verify that having a set of weights which never “sees” the source dataset is useful. -The method is not evaluated on ImageNet, which is far and away the most common domain in which pre-trained networks are used and fine-tuned for other tasks. I’ve never seen networks pre-trained on CIFAR deployed anywhere, and it’s hard to know whether the method will be practically useful for computer vision applications based on CIFAR results -- often improved performance on CIFAR does not translate to ImageNet. (In other contexts, such as more theoretical contributions, having results only on small datasets is acceptable to me, but network fine-tuning is far enough on the “practical” end of the spectrum that claiming an improvement to it should necessitate an ImageNet evaluation.) Overall I think the proposed idea is interesting and potentially promising, but in its current form is not sufficiently evaluated to convince me that the performance boosts don’t simply come from the use of a larger network, and the lack of ImageNet evaluation calls into question its real-world application. =============== Edit (1/23/17): I had indeed missed the fact that the Stanford Cars does do transfer learning from ImageNet -- thanks for the correction. However, the experiment in this case is only showing late fusion ensembling, which is a conventional approach compared with the *stitched network* idea which is the real novelty of the paper. Furthermore the results in this case are particularly weak, showing only that an ensemble of ResNet+VGG outperforms VGG alone, which is completely expected given that ResNet alone is a stronger base network than VGG (*ResNet+VGG > ResNet* would be a stronger result, but still not surprising). Demonstrating the stitched network idea on ImageNet, comparing with the corresponding VGG-only or ResNet-only finetuning, could be enough to push this paper over the bar for me, but the current version of the experiments here don*t sufficiently validate the stitched network idea, in my opinion.
assistant
This paper proposes a method of augmenting pre-trained networks for one task with an additional inference path specific to an additional task, as a replacement for the standard “fine-tuning” approach. Pros: -The method is simple and clearly explained. -Standard fine-tuning is used widely, so improvements to and analysis of it should be of general interest. -Experiments are performed in multiple domains -- vision and NLP. Cons: -The additional modules incur a rather large cost, resulting in 2x the parameters and roughly 3x the computation of the original network (for the “stiched” network). These costs are not addressed in the paper text, and make the method significantly less practical for real-world use where performance is very often important. -Given these large additional costs, the core of the idea is not sufficiently validated, to me. In order to verify that the improved performance is actually coming from some unique aspects of the proposed technique, rather than simply the fact that a higher-capacity network is being used, some additional baselines are needed: (1) Allowing the original network weights to be learned for the target task, as well as the additional module. Outperforming this baseline on the validation set would verify that freezing the original weights provides an interesting form of regularization for the network. (2) Training the full module/stitched network from scratch on the *source* task, then fine-tuning it for the target task. Outperforming this baseline would verify that having a set of weights which never “sees” the source dataset is useful. -The method is not evaluated on ImageNet, which is far and away the most common domain in which pre-trained networks are used and fine-tuned for other tasks. I’ve never seen networks pre-trained on CIFAR deployed anywhere, and it’s hard to know whether the method will be practically useful for computer vision applications based on CIFAR results -- often improved performance on CIFAR does not translate to ImageNet. (In other contexts, such as more theoretical contributions, having results only on small datasets is acceptable to me, but network fine-tuning is far enough on the “practical” end of the spectrum that claiming an improvement to it should necessitate an ImageNet evaluation.) Overall I think the proposed idea is interesting and potentially promising, but in its current form is not sufficiently evaluated to convince me that the performance boosts don’t simply come from the use of a larger network, and the lack of ImageNet evaluation calls into question its real-world application. =============== Edit (1/23/17): I had indeed missed the fact that the Stanford Cars does do transfer learning from ImageNet -- thanks for the correction. However, the experiment in this case is only showing late fusion ensembling, which is a conventional approach compared with the *stitched network* idea which is the real novelty of the paper. Furthermore the results in this case are particularly weak, showing only that an ensemble of ResNet+VGG outperforms VGG alone, which is completely expected given that ResNet alone is a stronger base network than VGG (*ResNet+VGG > ResNet* would be a stronger result, but still not surprising). Demonstrating the stitched network idea on ImageNet, comparing with the corresponding VGG-only or ResNet-only finetuning, could be enough to push this paper over the bar for me, but the current version of the experiments here don*t sufficiently validate the stitched network idea, in my opinion.