Answers.Body: <p>MaxPooling pools together information. Imagine you have 2 convolutional layers <span class="math-container">$(F_1, F_2)$</span> respectively, each with a 3x3 kernel and a stride of <span class="math-container">$1$</span>. Also, imagine your input is <span class="math-container">$I$</span> is of shape <span class="math-container">$(w,h)$</span>. Let's call a max-pooling layer <span class="math-container">$M$</span> is of size <span class="math-container">$(2,2)$</span>.</p>
  
  <p><strong>Note:</strong> I'm ignoring channels because, for these purposes, it's not necessary and can be extended to any amount of them.  </p>
  
  <p>Now you have two cases:  </p>
  
  <ol>
  <li><span class="math-container">$O_1 = F_2 \circ F_1 \circ I$</span>  </li>
  <li><span class="math-container">$O_2 = F_2 \circ M \circ F_1 \circ I$</span></li>
  </ol>
  
  <p>In these cases, <span class="math-container">$shape(O_1)=(w-4, h-4)$</span> and <span class="math-container">$shape(O_2)=\left(\frac{w-2}{2}-2, \frac{h-2}{2}-2 \right)$</span>. If we plug in dummy values, like <span class="math-container">$w,h = 64,64$</span>, we get the shapes become <span class="math-container">$(60,60)$</span> and <span class="math-container">$(29,29)$</span> respectively. As you can tell, these are very different!   </p>
  
  <p>Now, there is more of a difference than just the size of the outputs, each neuron holds a pooling of more information. Let's do it out:  </p>
  
  <ol>
  <li><p>Each output neuron of <span class="math-container">$F_1 \circ I$</span> has information from a <span class="math-container">$(3,3)$</span> receptive field.</p></li>
  <li><p>Each output neuron then of <span class="math-container">$F_2 \circ F_1 \circ I$</span> has information from a <span class="math-container">$(3,3)$</span> receptive field of <span class="math-container">$F_1 \circ I$</span>, which, if we eliminate reused nodes, is a <span class="math-container">$(5,5)$</span> receptive field from the initial <span class="math-container">$I$</span>. </p></li>
  <li><p>Each output neuron then of <span class="math-container">$M \circ F_1 \circ I$</span> has information from a <span class="math-container">$(2,2)$</span> receptive field of <span class="math-container">$F_1 \circ I$</span>, which, if we eliminate reused nodes, is a <span class="math-container">$(4,4)$</span> receptive field from the initial <span class="math-container">$I$</span>.</p></li>
  <li><p>Each output neuron then of <span class="math-container">$F_2 \circ M \circ F_1 \circ I$</span> has information from a <span class="math-container">$(4,4)$</span> receptive field of <span class="math-container">$M \circ F_1 \circ I$</span>, which, if we eliminate reused nodes, is a <span class="math-container">$(8,8)$</span> receptive field from the initial <span class="math-container">$I$</span>.</p></li>
  </ol>
  
  <p>So, let us discuss these: Using max-pooling reduces the feature space heavily by throwing out a lot of nodes whose features aren't as indicative (makes training models more tractable) along with it does extend the receptive field with no additional parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-26T16:28:54.477Z
Answers.Text: @mshlis thank you for your answer. Do you mind elaborating on points 2,3, and 4? Specifically, how did you get the numbers (4,4) (5,5) and (6,6)?
Answers.UserDisplayName: An Ignorant Wanderer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T00:06:29.000Z
Answers.Text: @user10796158 the convolution in point two's case is a 3x3 grid of nodes that touches local 3x3 grids of the layer even before that. To convince yourself draw it out and you get a grid thats 5x5 of the original. For the third point it works the same way just with a 2x2 and for the fourth point i forgot to include the stride of the maxpool, so its actually 8x8
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T18:00:19.493Z
Answers.LastActivityDate: 2019-07-27T00:07:22.763Z
Answers.LastEditDate: 2019-07-27T00:07:22.763Z
Answers.UserDisplayName: mshlis
Body: <p>I am reading Francois Chollet's <a href="http://faculty.neu.edu.cn/yury/AAI/Textbook/Deep%20Learning%20with%20Python.pdf" rel="nofollow noreferrer">Deep learning with Python</a>, and I came across a section about max-pooling that's really giving me trouble.</p>
  <p>I am unable to copy-paste the content, so I've included screenshots of the paragraph that's troubling me.</p>
  <p><a href="https://i.sstatic.net/0jDhk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0jDhk.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/x01ii.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/x01ii.png" alt="enter image description here" /></a></p>
  <p>I simply don't understand what he means when he talks about &quot;What's wrong with this setup?&quot; (towards the end).</p>
  <p><strong>How does removing the max-pooling layers &quot;reduce&quot; the amount of the initial image that we're looking at? What are the benefits of using max-pooling in convolutional neural networks, as opposed to just using convolution layers?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-26T02:33:47.993Z
Comments.Text: I think the "What's wrong with this setup" paragraph is trying to convey this: Lets say you have an extreme scenario of a 10^5 x 10^5 sized input image of a car, the conv net sees this image as slices of size 7x7 using this example, and in such a massive image, a slice like that hardly show anything. By maxpooling, you essentially "collect" a group of these slices, so the next time the conv net sees the image, it sees a larger window and can therefor identify over-arching features
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-26T16:17:12.607Z
Comments.Text: @Recessive thank you for your comment. Where do you think the 7 x 7 came from though? Like why did he mention 7 x 7 specifically? (how did he get that number?)
Comments.UserDisplayName: An Ignorant Wanderer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-26T18:10:35.063Z
Comments.Text: Essentially no proven benefits. Max pooling is not used in latest models any more.
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-27T00:20:51.820Z
Comments.Text: @user10796158 He's referring to the size of the filter for that particular conv layer, which is 7x7. You could increase the size of the filter to observe more of the image (say, 20x20 slices) and I don't believe there's actually anything wrong with that (aside from a large amount of computations). However, in spite of this, deepmind has found (https://arxiv.org/pdf/1804.04438.pdf) that the "what's wrong with this" is only true at initialisation, so it seems that regardless of maxpooling layers, the network will learn stability.
Comments.UserDisplayName: Recessive
Tags: <convolutional-neural-networks><convolution><pooling><max-pooling>
Title: What are the benefits of using max-pooling in convolutional neural networks?
UserDisplayName: An Ignorant Wanderer
ThreadText: # What are the benefits of using max-pooling in convolutional neural networks?
  
  (Asked by: username_0 on 2019-07-25)
  
  I am reading Francois Chollet's [Deep learning with Python](http://faculty.neu.edu.cn/yury/AAI/Textbook/Deep%20Learning%20with%20Python.pdf), and I came across a section about max-pooling that's really giving me trouble.
  
  I am unable to copy-paste the content, so I've included screenshots of the paragraph that's troubling me.
  
  [![enter image description here](https://i.sstatic.net/0jDhk.png)](https://i.sstatic.net/0jDhk.png)
  
  [![enter image description here](https://i.sstatic.net/x01ii.png)](https://i.sstatic.net/x01ii.png)
  
  I simply don't understand what he means when he talks about "What's wrong with this setup?" (towards the end).
  
  **How does removing the max-pooling layers "reduce" the amount of the initial image that we're looking at? What are the benefits of using max-pooling in convolutional neural networks, as opposed to just using convolution layers?**
  
  **Tags:** convolutional-neural-networks, convolution, pooling, max-pooling
  
  > **Comments:**
  > **username_1**: I think the "What's wrong with this setup" paragraph is trying to convey this: Lets say you have an extreme scenario of a 10^5 x 10^5 sized input image of a car, the conv net sees this image as slices of size 7x7 using this example, and in such a massive image, a slice like that hardly show anything. By maxpooling, you essentially "collect" a group of these slices, so the next time the conv net sees the image, it sees a larger window and can therefor identify over-arching features  
  > **username_0**: @username_1 thank you for your comment. Where do you think the 7 x 7 came from though? Like why did he mention 7 x 7 specifically? (how did he get that number?)  
  > **username_2**: Essentially no proven benefits. Max pooling is not used in latest models any more.  
  > **username_1**: @user10796158 He's referring to the size of the filter for that particular conv layer, which is 7x7. You could increase the size of the filter to observe more of the image (say, 20x20 slices) and I don't believe there's actually anything wrong with that (aside from a large amount of computations). However, in spite of this, deepmind has found (https://arxiv.org/pdf/1804.04438.pdf) that the "what's wrong with this" is only true at initialisation, so it seems that regardless of maxpooling layers, the network will learn stability.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-07-25. Score: 1
  
  MaxPooling pools together information. Imagine you have 2 convolutional layers $(F\_1, F\_2)$ respectively, each with a 3x3 kernel and a stride of $1$. Also, imagine your input is $I$ is of shape $(w,h)$. Let's call a max-pooling layer $M$ is of size $(2,2)$.
  
  **Note:** I'm ignoring channels because, for these purposes, it's not necessary and can be extended to any amount of them.
  
  Now you have two cases:
  
  1. $O\_1 = F\_2 \circ F\_1 \circ I$
  2. $O\_2 = F\_2 \circ M \circ F\_1 \circ I$
  
  In these cases, $shape(O\_1)=(w-4, h-4)$ and $shape(O\_2)=\left(\frac{w-2}{2}-2, \frac{h-2}{2}-2 \right)$. If we plug in dummy values, like $w,h = 64,64$, we get the shapes become $(60,60)$ and $(29,29)$ respectively. As you can tell, these are very different!
  
  Now, there is more of a difference than just the size of the outputs, each neuron holds a pooling of more information. Let's do it out:
  
  1. Each output neuron of $F\_1 \circ I$ has information from a $(3,3)$ receptive field.
  2. Each output neuron then of $F\_2 \circ F\_1 \circ I$ has information from a $(3,3)$ receptive field of $F\_1 \circ I$, which, if we eliminate reused nodes, is a $(5,5)$ receptive field from the initial $I$.
  3. Each output neuron then of $M \circ F\_1 \circ I$ has information from a $(2,2)$ receptive field of $F\_1 \circ I$, which, if we eliminate reused nodes, is a $(4,4)$ receptive field from the initial $I$.
  4. Each output neuron then of $F\_2 \circ M \circ F\_1 \circ I$ has information from a $(4,4)$ receptive field of $M \circ F\_1 \circ I$, which, if we eliminate reused nodes, is a $(8,8)$ receptive field from the initial $I$.
  
  So, let us discuss these: Using max-pooling reduces the feature space heavily by throwing out a lot of nodes whose features aren't as indicative (makes training models more tractable) along with it does extend the receptive field with no additional parameters.
  
  > **Comments:**
  > **username_0**: @username_3 thank you for your answer. Do you mind elaborating on points 2,3, and 4? Specifically, how did you get the numbers (4,4) (5,5) and (6,6)?  
  > **username_3**: @user10796158 the convolution in point two's case is a 3x3 grid of nodes that touches local 3x3 grids of the layer even before that. To convince yourself draw it out and you get a grid thats 5x5 of the original. For the third point it works the same way just with a 2x2 and for the fourth point i forgot to include the stride of the maxpool, so its actually 8x8  
  
  ---
  
  