Inductive Bias of Deep Convolutional Networks through Pooling Geometry
Our formal understanding of the inductive bias that drives the success of convolutional networks on computer vision tasks is limited. In particular, it is unclear what makes hypotheses spaces born from convolution and pooling operations so suitable for natural images. In this paper we study the ability of convolutional networks to model correlations among regions of their input. We theoretically analyze convolutional arithmetic circuits, and empirically validate our findings on other types of convolutional networks as well. Correlations are formalized through the notion of separation rank, which for a given partition of the input, measures how far a function is from being separable. We show that a polynomially sized deep network supports exponentially high separation ranks for certain input partitions, while being limited to polynomial separation ranks for others. The network*s pooling geometry effectively determines which input partitions are favored, thus serves as a means for controlling the inductive bias. Contiguous pooling windows as commonly employed in practice favor interleaved partitions over coarse ones, orienting the inductive bias towards the statistics of natural images. Other pooling schemes lead to different preferences, and this allows tailoring the network to data that departs from the usual domain of natural imagery. In addition to analyzing deep networks, we show that shallow ones support only linear separation ranks, and by this gain insight into the benefit of functions brought forth by depth - they are able to efficiently model strong correlation under favored partitions of the input.
This paper addresses the question of which functions are well suited to deep networks, as opposed to shallow networks. The basic intuition is convincing and fairly straightforward. Pooling operations bring together information. When information is correlated, it can be more efficiently used if the geometry of pooling regions matches the correlations so that it can be brought together more efficiently. Shallow networks without layers of localized pooling lack this mechanism to combine correlated information efficiently. The theoretical results are focused on convolutional arithmetic circuits, building on prior theoretical results of the authors. The results make use of the interesting technical notion of separability, which in some sense measures the degree to which a function can be represented as the composition of independent functions. Because separability is measured relative to a partition of the input, it is an appropriate mechanism for measuring the complexity of functions relative to a particular geometry of pooling operations. Many of the technical notions are pretty intuitive, although the tensor analysis is pretty terse and not easy to follow without knowledge of the authors’ prior work. In some sense the comparison between deep and shallow networks is somewhat misleading, since the shallow networks lack a hierarchical pooling structure. For example, a shallow convolutional network with RELU and max pooling does not really make sense, since the max occurs over the whole image. So it seems that the paper is really more of an analysis of the effect of pooling vs. not having pooling. For example, it is not clear that a deep CNN without pooling would be any more efficient than a shallow network, from this work. It is not clear how much the theoretical results depend on the use of a model with product pooling, and how they might be extended to the more common max pooling. Even if theoretical results are difficult to derive in this case, simple illustrative examples might be helpful. In fact, if the authors prepare a longer version of the paper for a journal I think the results could be made more intuitive if they could add a simple toy example of a function that can be efficiently represented with a convolutional arithmetic circuit when the pooling structure fits the correlations, and perhaps showing also how this could be represented with a convolutional network with RELU and max pooling. I would also appreciate a more explicit discussion of how the depth of a deep network affects the separability of functions that can be represented. A shallow network doesn’t have local pooling, so the difference between deep and shallow if perhaps mostly one of pooling vs. not pooling. However, practitioners find that very deep networks seem to be more effective than “deep” networks with only a few convolutional layers and pooling. The paper does not explicitly discuss whether their results provide insight into this behavior. Overall, I think that the paper attacks an important problem in an interesting way. It is not so convincing that this really gets to the heart of why depth is so important, because of the theoretical limitation to arithmetic circuits, and because the comparison is to shallow networks that are without localized pooling.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Inductive Bias of Deep Convolutional Networks through Pooling Geometry
Our formal understanding of the inductive bias that drives the success of convolutional networks on computer vision tasks is limited. In particular, it is unclear what makes hypotheses spaces born from convolution and pooling operations so suitable for natural images. In this paper we study the ability of convolutional networks to model correlations among regions of their input. We theoretically analyze convolutional arithmetic circuits, and empirically validate our findings on other types of convolutional networks as well. Correlations are formalized through the notion of separation rank, which for a given partition of the input, measures how far a function is from being separable. We show that a polynomially sized deep network supports exponentially high separation ranks for certain input partitions, while being limited to polynomial separation ranks for others. The network*s pooling geometry effectively determines which input partitions are favored, thus serves as a means for controlling the inductive bias. Contiguous pooling windows as commonly employed in practice favor interleaved partitions over coarse ones, orienting the inductive bias towards the statistics of natural images. Other pooling schemes lead to different preferences, and this allows tailoring the network to data that departs from the usual domain of natural imagery. In addition to analyzing deep networks, we show that shallow ones support only linear separation ranks, and by this gain insight into the benefit of functions brought forth by depth - they are able to efficiently model strong correlation under favored partitions of the input.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper addresses the question of which functions are well suited to deep networks, as opposed to shallow networks. The basic intuition is convincing and fairly straightforward. Pooling operations bring together information. When information is correlated, it can be more efficiently used if the geometry of pooling regions matches the correlations so that it can be brought together more efficiently. Shallow networks without layers of localized pooling lack this mechanism to combine correlated information efficiently. The theoretical results are focused on convolutional arithmetic circuits, building on prior theoretical results of the authors. The results make use of the interesting technical notion of separability, which in some sense measures the degree to which a function can be represented as the composition of independent functions. Because separability is measured relative to a partition of the input, it is an appropriate mechanism for measuring the complexity of functions relative to a particular geometry of pooling operations. Many of the technical notions are pretty intuitive, although the tensor analysis is pretty terse and not easy to follow without knowledge of the authors’ prior work. In some sense the comparison between deep and shallow networks is somewhat misleading, since the shallow networks lack a hierarchical pooling structure. For example, a shallow convolutional network with RELU and max pooling does not really make sense, since the max occurs over the whole image. So it seems that the paper is really more of an analysis of the effect of pooling vs. not having pooling. For example, it is not clear that a deep CNN without pooling would be any more efficient than a shallow network, from this work. It is not clear how much the theoretical results depend on the use of a model with product pooling, and how they might be extended to the more common max pooling. Even if theoretical results are difficult to derive in this case, simple illustrative examples might be helpful. In fact, if the authors prepare a longer version of the paper for a journal I think the results could be made more intuitive if they could add a simple toy example of a function that can be efficiently represented with a convolutional arithmetic circuit when the pooling structure fits the correlations, and perhaps showing also how this could be represented with a convolutional network with RELU and max pooling. I would also appreciate a more explicit discussion of how the depth of a deep network affects the separability of functions that can be represented. A shallow network doesn’t have local pooling, so the difference between deep and shallow if perhaps mostly one of pooling vs. not pooling. However, practitioners find that very deep networks seem to be more effective than “deep” networks with only a few convolutional layers and pooling. The paper does not explicitly discuss whether their results provide insight into this behavior. Overall, I think that the paper attacks an important problem in an interesting way. It is not so convincing that this really gets to the heart of why depth is so important, because of the theoretical limitation to arithmetic circuits, and because the comparison is to shallow networks that are without localized pooling.
assistant
This paper addresses the question of which functions are well suited to deep networks, as opposed to shallow networks. The basic intuition is convincing and fairly straightforward. Pooling operations bring together information. When information is correlated, it can be more efficiently used if the geometry of pooling regions matches the correlations so that it can be brought together more efficiently. Shallow networks without layers of localized pooling lack this mechanism to combine correlated information efficiently. The theoretical results are focused on convolutional arithmetic circuits, building on prior theoretical results of the authors. The results make use of the interesting technical notion of separability, which in some sense measures the degree to which a function can be represented as the composition of independent functions. Because separability is measured relative to a partition of the input, it is an appropriate mechanism for measuring the complexity of functions relative to a particular geometry of pooling operations. Many of the technical notions are pretty intuitive, although the tensor analysis is pretty terse and not easy to follow without knowledge of the authors’ prior work. In some sense the comparison between deep and shallow networks is somewhat misleading, since the shallow networks lack a hierarchical pooling structure. For example, a shallow convolutional network with RELU and max pooling does not really make sense, since the max occurs over the whole image. So it seems that the paper is really more of an analysis of the effect of pooling vs. not having pooling. For example, it is not clear that a deep CNN without pooling would be any more efficient than a shallow network, from this work. It is not clear how much the theoretical results depend on the use of a model with product pooling, and how they might be extended to the more common max pooling. Even if theoretical results are difficult to derive in this case, simple illustrative examples might be helpful. In fact, if the authors prepare a longer version of the paper for a journal I think the results could be made more intuitive if they could add a simple toy example of a function that can be efficiently represented with a convolutional arithmetic circuit when the pooling structure fits the correlations, and perhaps showing also how this could be represented with a convolutional network with RELU and max pooling. I would also appreciate a more explicit discussion of how the depth of a deep network affects the separability of functions that can be represented. A shallow network doesn’t have local pooling, so the difference between deep and shallow if perhaps mostly one of pooling vs. not pooling. However, practitioners find that very deep networks seem to be more effective than “deep” networks with only a few convolutional layers and pooling. The paper does not explicitly discuss whether their results provide insight into this behavior. Overall, I think that the paper attacks an important problem in an interesting way. It is not so convincing that this really gets to the heart of why depth is so important, because of the theoretical limitation to arithmetic circuits, and because the comparison is to shallow networks that are without localized pooling.