Inductive Bias of Deep Convolutional Networks through Pooling Geometry
Our formal understanding of the inductive bias that drives the success of convolutional networks on computer vision tasks is limited. In particular, it is unclear what makes hypotheses spaces born from convolution and pooling operations so suitable for natural images. In this paper we study the ability of convolutional networks to model correlations among regions of their input. We theoretically analyze convolutional arithmetic circuits, and empirically validate our findings on other types of convolutional networks as well. Correlations are formalized through the notion of separation rank, which for a given partition of the input, measures how far a function is from being separable. We show that a polynomially sized deep network supports exponentially high separation ranks for certain input partitions, while being limited to polynomial separation ranks for others. The network*s pooling geometry effectively determines which input partitions are favored, thus serves as a means for controlling the inductive bias. Contiguous pooling windows as commonly employed in practice favor interleaved partitions over coarse ones, orienting the inductive bias towards the statistics of natural images. Other pooling schemes lead to different preferences, and this allows tailoring the network to data that departs from the usual domain of natural imagery. In addition to analyzing deep networks, we show that shallow ones support only linear separation ranks, and by this gain insight into the benefit of functions brought forth by depth - they are able to efficiently model strong correlation under favored partitions of the input.
This paper investigates the fact why deep networks perform well in practice and how modifying the geometry of pooling can make the polynomially sized deep network to provide a function with exponentially high separation rank (for certain partitioning.) In the authors* previous works, they showed the superiority of deep networks over shallows when the activation function is ReLu and the pooling is max/mean pooling but in the current paper there is no activation function after conv and the pooling is just a multiplication of the node values. Although for the experimental results they*ve considered both scenarios. Actually, the general reasoning for this problem is hard, therefore, this drawback is not significant and the current contribution adds a reasonable amount of knowledge to the literature. This paper studies the convolutional arithmetic circuits and shows how this model can address the inductive biases and how pooling can adjust these biases. This interesting contribution gives an intuition about how deep network can capture the correlation between the input variables when its size is polynomial but and correlation is exponential. It worth to note that although the authors tried to express their notation and definitions carefully where they were very successful, it would be helpful if they elaborate a bit more on their definitions, expressions, and conclusions in the sense to make them more accessible.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Inductive Bias of Deep Convolutional Networks through Pooling Geometry
Our formal understanding of the inductive bias that drives the success of convolutional networks on computer vision tasks is limited. In particular, it is unclear what makes hypotheses spaces born from convolution and pooling operations so suitable for natural images. In this paper we study the ability of convolutional networks to model correlations among regions of their input. We theoretically analyze convolutional arithmetic circuits, and empirically validate our findings on other types of convolutional networks as well. Correlations are formalized through the notion of separation rank, which for a given partition of the input, measures how far a function is from being separable. We show that a polynomially sized deep network supports exponentially high separation ranks for certain input partitions, while being limited to polynomial separation ranks for others. The network*s pooling geometry effectively determines which input partitions are favored, thus serves as a means for controlling the inductive bias. Contiguous pooling windows as commonly employed in practice favor interleaved partitions over coarse ones, orienting the inductive bias towards the statistics of natural images. Other pooling schemes lead to different preferences, and this allows tailoring the network to data that departs from the usual domain of natural imagery. In addition to analyzing deep networks, we show that shallow ones support only linear separation ranks, and by this gain insight into the benefit of functions brought forth by depth - they are able to efficiently model strong correlation under favored partitions of the input.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper investigates the fact why deep networks perform well in practice and how modifying the geometry of pooling can make the polynomially sized deep network to provide a function with exponentially high separation rank (for certain partitioning.) In the authors* previous works, they showed the superiority of deep networks over shallows when the activation function is ReLu and the pooling is max/mean pooling but in the current paper there is no activation function after conv and the pooling is just a multiplication of the node values. Although for the experimental results they*ve considered both scenarios. Actually, the general reasoning for this problem is hard, therefore, this drawback is not significant and the current contribution adds a reasonable amount of knowledge to the literature. This paper studies the convolutional arithmetic circuits and shows how this model can address the inductive biases and how pooling can adjust these biases. This interesting contribution gives an intuition about how deep network can capture the correlation between the input variables when its size is polynomial but and correlation is exponential. It worth to note that although the authors tried to express their notation and definitions carefully where they were very successful, it would be helpful if they elaborate a bit more on their definitions, expressions, and conclusions in the sense to make them more accessible.
assistant
This paper investigates the fact why deep networks perform well in practice and how modifying the geometry of pooling can make the polynomially sized deep network to provide a function with exponentially high separation rank (for certain partitioning.) In the authors* previous works, they showed the superiority of deep networks over shallows when the activation function is ReLu and the pooling is max/mean pooling but in the current paper there is no activation function after conv and the pooling is just a multiplication of the node values. Although for the experimental results they*ve considered both scenarios. Actually, the general reasoning for this problem is hard, therefore, this drawback is not significant and the current contribution adds a reasonable amount of knowledge to the literature. This paper studies the convolutional arithmetic circuits and shows how this model can address the inductive biases and how pooling can adjust these biases. This interesting contribution gives an intuition about how deep network can capture the correlation between the input variables when its size is polynomial but and correlation is exponential. It worth to note that although the authors tried to express their notation and definitions carefully where they were very successful, it would be helpful if they elaborate a bit more on their definitions, expressions, and conclusions in the sense to make them more accessible.