Top-down training for neural networks
Vanishing gradients pose a challenge when training deep neural networks, resulting in the top layers (closer to the output) in the network learning faster when compared with lower layers closer to the input. Interpreting the top layers as a classifier and the lower layers a feature extractor, one can hypothesize that unwanted network convergence may occur when the classifier has overfit with respect to the feature extractor. This can lead to the feature extractor being under-trained, possibly failing to learn much about the patterns in the input data. To address this we propose a good classifier hypothesis: given a fixed classifier that partitions the space well, the feature extractor can be further trained to fit that classifier and learn the data patterns well. This alleviates the problem of under-training the feature extractor and enables the network to learn patterns in the data with small partial derivatives. We verify this hypothesis empirically and propose a novel top-down training method. We train all layers jointly, obtaining a good classifier from the top layers, which are then frozen. Following re-initialization, we retrain the bottom layers with respect to the frozen classifier. Applying this approach to a set of speech recognition experiments using the Wall Street Journal and noisy CHiME-4 datasets we observe substantial accuracy gains. When combined with dropout, our method enables connectionist temporal classification (CTC) models to outperform joint CTC-attention models, which have more capacity and flexibility.
This paper studies the common experimental finding that low level features trained end-to-end in a deep model converge (get *locked in place*) earlier than higher level features, which may result in problematic undertraining. The focus of the study is not on skip connections, but really on getting adequate training in deeper networks. They posit a *good classifier hypothesis* where, once a deep network converged, they fix the top layers (the *good classifier*) and train only the lower ones. They propose a *top-down training strategy* to search where to make the cut for the *top layers* of the *good classifier*, based on the validation set. (+) The experimental results seem encouraging and supporting the author*s claim (consistently improve over baseline on WSJ and CHiME-4). (-) No WER (not even without a language model) results on WSJ make it harder to (i) compare to other work (is it just that in this case the authors didn*t optimize properly in the first place?), (ii) compare the relative gains between with and without the method in WER. (-) For an experimental (no theorem) optimization paper, there should be experiments on at least another domain. And in particular one would have expected more analysis of the experimental optimization results. (-) (minor) There is no discussion of the link with target propagation or other synthetic gradients. Overall, I think this could be an interesting paper, but more work is needed to prove the effectiveness of the method, and to analyze experimentally in more details some of the claims from this paper.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Top-down training for neural networks
Vanishing gradients pose a challenge when training deep neural networks, resulting in the top layers (closer to the output) in the network learning faster when compared with lower layers closer to the input. Interpreting the top layers as a classifier and the lower layers a feature extractor, one can hypothesize that unwanted network convergence may occur when the classifier has overfit with respect to the feature extractor. This can lead to the feature extractor being under-trained, possibly failing to learn much about the patterns in the input data. To address this we propose a good classifier hypothesis: given a fixed classifier that partitions the space well, the feature extractor can be further trained to fit that classifier and learn the data patterns well. This alleviates the problem of under-training the feature extractor and enables the network to learn patterns in the data with small partial derivatives. We verify this hypothesis empirically and propose a novel top-down training method. We train all layers jointly, obtaining a good classifier from the top layers, which are then frozen. Following re-initialization, we retrain the bottom layers with respect to the frozen classifier. Applying this approach to a set of speech recognition experiments using the Wall Street Journal and noisy CHiME-4 datasets we observe substantial accuracy gains. When combined with dropout, our method enables connectionist temporal classification (CTC) models to outperform joint CTC-attention models, which have more capacity and flexibility.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper studies the common experimental finding that low level features trained end-to-end in a deep model converge (get *locked in place*) earlier than higher level features, which may result in problematic undertraining. The focus of the study is not on skip connections, but really on getting adequate training in deeper networks. They posit a *good classifier hypothesis* where, once a deep network converged, they fix the top layers (the *good classifier*) and train only the lower ones. They propose a *top-down training strategy* to search where to make the cut for the *top layers* of the *good classifier*, based on the validation set. (+) The experimental results seem encouraging and supporting the author*s claim (consistently improve over baseline on WSJ and CHiME-4). (-) No WER (not even without a language model) results on WSJ make it harder to (i) compare to other work (is it just that in this case the authors didn*t optimize properly in the first place?), (ii) compare the relative gains between with and without the method in WER. (-) For an experimental (no theorem) optimization paper, there should be experiments on at least another domain. And in particular one would have expected more analysis of the experimental optimization results. (-) (minor) There is no discussion of the link with target propagation or other synthetic gradients. Overall, I think this could be an interesting paper, but more work is needed to prove the effectiveness of the method, and to analyze experimentally in more details some of the claims from this paper.
assistant
This paper studies the common experimental finding that low level features trained end-to-end in a deep model converge (get *locked in place*) earlier than higher level features, which may result in problematic undertraining. The focus of the study is not on skip connections, but really on getting adequate training in deeper networks. They posit a *good classifier hypothesis* where, once a deep network converged, they fix the top layers (the *good classifier*) and train only the lower ones. They propose a *top-down training strategy* to search where to make the cut for the *top layers* of the *good classifier*, based on the validation set. (+) The experimental results seem encouraging and supporting the author*s claim (consistently improve over baseline on WSJ and CHiME-4). (-) No WER (not even without a language model) results on WSJ make it harder to (i) compare to other work (is it just that in this case the authors didn*t optimize properly in the first place? ), (ii) compare the relative gains between with and without the method in WER. (-) For an experimental (no theorem) optimization paper, there should be experiments on at least another domain. And in particular one would have expected more analysis of the experimental optimization results. (-) (minor) There is no discussion of the link with target propagation or other synthetic gradients. Overall, I think this could be an interesting paper, but more work is needed to prove the effectiveness of the method, and to analyze experimentally in more details some of the claims from this paper.