Beyond Fine Tuning: A Modular Approach to Learning on Small Data
In this paper we present a technique to train neural network models on small amounts of data. Current methods for training neural networks on small amounts of rich data typically rely on strategies such as fine-tuning a pre-trained neural network or the use of domain-specific hand-engineered features. Here we take the approach of treating network layers, or entire networks, as modules and combine pre-trained modules with untrained modules, to learn the shift in distributions between data sets. The central impact of using a modular approach comes from adding new representations to a network, as opposed to replacing representations via fine-tuning. Using this technique, we are able surpass results using standard fine-tuning transfer learning approaches, and we are also able to significantly increase performance over such approaches when using smaller amounts of data.
This paper proposed to perform finetuning in an augmentation fashion by freezing the original network and adding a new model aside it. The idea itself is interesting and complements existing training and finetuning approaches, although I think there are a few baseline approaches that can be compared against, such as: (1) Ensemble: in principle, the idea is similar to an ensembling approach where multiple networks are ensembled together to get a final prediction. The approach in Figure 1 should be compared with such ensemble baselines - taking multiple source domain predictors, possibly with the same modular setting as the proposed method, and compare the performance. (2) comparison with late fusion: if we combine the pretrained network and a network finetuned from the pretrained one, and do a late fusion? Basically, I think it is a valuable argument in section 3.2 (and Figure 4) that finetuning with a small amount of data may hurt the performance in general. This builds the ground for freezing a pretrained network and only augmenting it, not changing it. I agree with the authors on this argument, although currently other than Figure 4 there seem to be little empirical study that justifies it. It is worth noting that Figure 3 seems to suggest that some of the module filters are either not converging or are learning unuseful features - like the first two filters in 3(a). Overall I think it is an interesting idea and I would love to see it better developed, thus I am giving a weak accept recommendation, but with a low confidence as the experiments section is not very convincing.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Beyond Fine Tuning: A Modular Approach to Learning on Small Data
In this paper we present a technique to train neural network models on small amounts of data. Current methods for training neural networks on small amounts of rich data typically rely on strategies such as fine-tuning a pre-trained neural network or the use of domain-specific hand-engineered features. Here we take the approach of treating network layers, or entire networks, as modules and combine pre-trained modules with untrained modules, to learn the shift in distributions between data sets. The central impact of using a modular approach comes from adding new representations to a network, as opposed to replacing representations via fine-tuning. Using this technique, we are able surpass results using standard fine-tuning transfer learning approaches, and we are also able to significantly increase performance over such approaches when using smaller amounts of data.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposed to perform finetuning in an augmentation fashion by freezing the original network and adding a new model aside it. The idea itself is interesting and complements existing training and finetuning approaches, although I think there are a few baseline approaches that can be compared against, such as: (1) Ensemble: in principle, the idea is similar to an ensembling approach where multiple networks are ensembled together to get a final prediction. The approach in Figure 1 should be compared with such ensemble baselines - taking multiple source domain predictors, possibly with the same modular setting as the proposed method, and compare the performance. (2) comparison with late fusion: if we combine the pretrained network and a network finetuned from the pretrained one, and do a late fusion? Basically, I think it is a valuable argument in section 3.2 (and Figure 4) that finetuning with a small amount of data may hurt the performance in general. This builds the ground for freezing a pretrained network and only augmenting it, not changing it. I agree with the authors on this argument, although currently other than Figure 4 there seem to be little empirical study that justifies it. It is worth noting that Figure 3 seems to suggest that some of the module filters are either not converging or are learning unuseful features - like the first two filters in 3(a). Overall I think it is an interesting idea and I would love to see it better developed, thus I am giving a weak accept recommendation, but with a low confidence as the experiments section is not very convincing.
assistant
This paper proposed to perform finetuning in an augmentation fashion by freezing the original network and adding a new model aside it. The idea itself is interesting and complements existing training and finetuning approaches, although I think there are a few baseline approaches that can be compared against, such as: (1) Ensemble: in principle, the idea is similar to an ensembling approach where multiple networks are ensembled together to get a final prediction. The approach in Figure 1 should be compared with such ensemble baselines - taking multiple source domain predictors, possibly with the same modular setting as the proposed method, and compare the performance. (2) comparison with late fusion: if we combine the pretrained network and a network finetuned from the pretrained one, and do a late fusion? Basically, I think it is a valuable argument in section 3.2 (and Figure 4) that finetuning with a small amount of data may hurt the performance in general. This builds the ground for freezing a pretrained network and only augmenting it, not changing it. I agree with the authors on this argument, although currently other than Figure 4 there seem to be little empirical study that justifies it. It is worth noting that Figure 3 seems to suggest that some of the module filters are either not converging or are learning unuseful features - like the first two filters in 3(a). Overall I think it is an interesting idea and I would love to see it better developed, thus I am giving a weak accept recommendation, but with a low confidence as the experiments section is not very convincing.