Deep Convolutional Neural Network Design Patterns
Recent research in the deep learning field has produced a plethora of new architectures. At the same time, a growing number of groups are applying deep learning to new applications. Some of these groups are likely to be composed of inexperienced deep learning practitioners who are baffled by the dizzying array of architecture choices and therefore opt to use an older architecture (i.e., Alexnet). Here we attempt to bridge this gap by mining the collective knowledge contained in recent deep learning research to discover underlying principles for designing neural network architectures. In addition, we describe several architectural innovations, including Fractal of FractalNet network, Stagewise Boosting Networks, and Taylor Series Networks (our Caffe code and prototxt files are available at https://github.com/iPhysicist/CNNDesignPatterns). We hope others are inspired to build on our preliminary work.
The authors have grouped recent work in convolutional neural network design (specifically with respect to image classification) to identify core design principles guiding the field at large. The 14 principles they produce (along with associated references) include a number of useful and correct observations that would be an asset to anyone unfamiliar with the field. The authors explore a number of architectures on CIFAR-10 and CIFAR-100 guided by these principles. The authors have collected a quality set of references on the subject and grouped them well which is valuable for young researchers. Clearly the authors explored a many of architectural changes as part of their experiments and publicly available code base is always nice. Overall the writing seems to jump around a bit and the motivations behind some design principles feel lost in the confusion. For example, *Design Pattern 4: Increase Symmetry argues for architectural symmetry as a sign of beauty and quality* is presented as one of 14 core design principles without any further justification. Similarly *Design Pattern 6: Over-train includes any training method where the network is trained on a harder problem than necessary to improve generalization performance of inference* is presented in the middle of a paragraph with no supporting references or further explanation. The experimental portion of this paper feels scattered with many different approaches being presented based on subsets of the design principles. In general, these approaches either are minor modifications of existing networks (different FractalNet pooling strategies) or are novel architectures that do not perform well. The exception being the Fractal-of-Fractal network which achieves slightly improved accuracy but also introduces many more network parameters (increased capacity) over the original FractalNet. Preliminary rating: It is a useful and perhaps noble task to collect and distill research from many sources to find patterns (and perhaps gaps) in the state of a field; however, some of the patterns presented do not seem well developed and include principles that are poorly explained. Furthermore, the innovative architectures motivated by the design principles either fall short or achieve slightly better accuracy by introducing many more parameters (Fractal-Of-Fractal networks). For a paper addressing the topic of higher level design trends, I would appreciate additional rigorous experimentation around each principle rather than novel architectures being presented.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Deep Convolutional Neural Network Design Patterns
Recent research in the deep learning field has produced a plethora of new architectures. At the same time, a growing number of groups are applying deep learning to new applications. Some of these groups are likely to be composed of inexperienced deep learning practitioners who are baffled by the dizzying array of architecture choices and therefore opt to use an older architecture (i.e., Alexnet). Here we attempt to bridge this gap by mining the collective knowledge contained in recent deep learning research to discover underlying principles for designing neural network architectures. In addition, we describe several architectural innovations, including Fractal of FractalNet network, Stagewise Boosting Networks, and Taylor Series Networks (our Caffe code and prototxt files are available at https://github.com/iPhysicist/CNNDesignPatterns). We hope others are inspired to build on our preliminary work.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors have grouped recent work in convolutional neural network design (specifically with respect to image classification) to identify core design principles guiding the field at large. The 14 principles they produce (along with associated references) include a number of useful and correct observations that would be an asset to anyone unfamiliar with the field. The authors explore a number of architectures on CIFAR-10 and CIFAR-100 guided by these principles. The authors have collected a quality set of references on the subject and grouped them well which is valuable for young researchers. Clearly the authors explored a many of architectural changes as part of their experiments and publicly available code base is always nice. Overall the writing seems to jump around a bit and the motivations behind some design principles feel lost in the confusion. For example, *Design Pattern 4: Increase Symmetry argues for architectural symmetry as a sign of beauty and quality* is presented as one of 14 core design principles without any further justification. Similarly *Design Pattern 6: Over-train includes any training method where the network is trained on a harder problem than necessary to improve generalization performance of inference* is presented in the middle of a paragraph with no supporting references or further explanation. The experimental portion of this paper feels scattered with many different approaches being presented based on subsets of the design principles. In general, these approaches either are minor modifications of existing networks (different FractalNet pooling strategies) or are novel architectures that do not perform well. The exception being the Fractal-of-Fractal network which achieves slightly improved accuracy but also introduces many more network parameters (increased capacity) over the original FractalNet. Preliminary rating: It is a useful and perhaps noble task to collect and distill research from many sources to find patterns (and perhaps gaps) in the state of a field; however, some of the patterns presented do not seem well developed and include principles that are poorly explained. Furthermore, the innovative architectures motivated by the design principles either fall short or achieve slightly better accuracy by introducing many more parameters (Fractal-Of-Fractal networks). For a paper addressing the topic of higher level design trends, I would appreciate additional rigorous experimentation around each principle rather than novel architectures being presented.
assistant
The authors have grouped recent work in convolutional neural network design (specifically with respect to image classification) to identify core design principles guiding the field at large. The 14 principles they produce (along with associated references) include a number of useful and correct observations that would be an asset to anyone unfamiliar with the field. The authors explore a number of architectures on CIFAR-10 and CIFAR-100 guided by these principles. The authors have collected a quality set of references on the subject and grouped them well which is valuable for young researchers. Clearly the authors explored a many of architectural changes as part of their experiments and publicly available code base is always nice. Overall the writing seems to jump around a bit and the motivations behind some design principles feel lost in the confusion. For example, *Design Pattern 4: Increase Symmetry argues for architectural symmetry as a sign of beauty and quality* is presented as one of 14 core design principles without any further justification. Similarly *Design Pattern 6: Over-train includes any training method where the network is trained on a harder problem than necessary to improve generalization performance of inference* is presented in the middle of a paragraph with no supporting references or further explanation. The experimental portion of this paper feels scattered with many different approaches being presented based on subsets of the design principles. In general, these approaches either are minor modifications of existing networks (different FractalNet pooling strategies) or are novel architectures that do not perform well. The exception being the Fractal-of-Fractal network which achieves slightly improved accuracy but also introduces many more network parameters (increased capacity) over the original FractalNet. Preliminary rating: It is a useful and perhaps noble task to collect and distill research from many sources to find patterns (and perhaps gaps) in the state of a field; however, some of the patterns presented do not seem well developed and include principles that are poorly explained. Furthermore, the innovative architectures motivated by the design principles either fall short or achieve slightly better accuracy by introducing many more parameters (Fractal-Of-Fractal networks). For a paper addressing the topic of higher level design trends, I would appreciate additional rigorous experimentation around each principle rather than novel architectures being presented.