Answers.Body: <p>The problem is in your diagram. Here are the steps to get to a 5x5 receptive field. Here is your diagram, redone slightly:
  <a href="https://i.sstatic.net/lEe45.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lEe45.jpg" alt="The 3x3 input field is summed up (by the weights), and passed through a nonlinear activation function"></a></p>
  
  <p>Notice that the new unit takes a weighted sum of the 9 pixels in the input, and then applies a rectified linear nonlinearity. Now, there are more of these, creating three new numbers computed from that part of the image. Each one slides over by one pixel:</p>
  
  <p><a href="https://i.sstatic.net/jl3lW.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jl3lW.jpg" alt="enter image description here"></a></p>
  
  <p>We repeat this process going down three pixels as well, and then finally, we have a new 3x3 input field:</p>
  
  <p><a href="https://i.sstatic.net/68ib4.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/68ib4.jpg" alt="enter image description here"></a></p>
  
  <p>Notice that the new unit on the right now gets input from a 5x5 input field. I hope this helps!</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-10T07:46:51.280Z
Answers.LastActivityDate: 2018-04-10T07:46:51.280Z
Answers.UserDisplayName: Gary Cottrell
Answers.Body: <p>The intention of the referred text is to reason out the disadvantage of equivalent-merged-single-convolution-layer over multiple [CONV -&gt; RELU]*N layers.</p>
  <p>In the given scenario, if 2 layers of 3x3 filters were to be replaced by an equivalent single layer then this equivalent layer would need a filter with a receptive field of size 5x5.</p>
  <p>Similarly, an equivalent layer filter would need its receptive field to be of size 7x7 to compress 3 layers of 3x3 filters. Note that the most obvious disadvantage would be missing out on modeling non-linearity.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-07T15:26:04.627Z
Answers.LastActivityDate: 2021-10-02T22:37:29.777Z
Answers.LastEditDate: 2021-10-02T22:37:29.777Z
Answers.UserDisplayName: sunil4data
Answers.Body: <p>It is really easy to visualize the growth in the receptive field of the input as you go deep into the CNN layers if you consider a small example.</p>
  <p>Let's take a simple example:</p>
  <p>The dimensions are in the form of <span class="math-container">$\text{channels} \times \text{height} \times \text{width}$</span>.</p>
  <ul>
  <li>The input image <span class="math-container">$I$</span> is a <span class="math-container">$3 \times 5 \times 5$</span> matrix</li>
  <li>The first convolutional layer's kernel <span class="math-container">$K_1$</span> has shape <span class="math-container">$3 \times 2 \times 2$</span> (we consider only 1 filter for simplicity)</li>
  <li>The second convolutional layer's kernel <span class="math-container">$K_2$</span> has shape <span class="math-container">$1 \times 2 \times 2$</span></li>
  <li>Padding <span class="math-container">$P = 0$</span></li>
  <li>Stride <span class="math-container">$S = 1$</span></li>
  </ul>
  <p>The output dimensions <span class="math-container">$O$</span> are calculated by the following formula taken from the lecture CS231n.</p>
  <p><span class="math-container">$$O= (I - K + 2P)/S + 1$$</span></p>
  <p>When you do a convolution of the input image with the first filter <span class="math-container">$K_1$</span>, you get an output of shape <span class="math-container">$1 \times 4 \times 4$</span> (this is the activation of the CONV1 layer). The receptive field for this is the same as the kernel size (<span class="math-container">$K_1$</span>), that is, <span class="math-container">$2 \times 2$</span>.</p>
  <p>When this layer (of shape <span class="math-container">$1 \times 4 \times 4$</span>) is convolved with the second kernel (CONV2) <span class="math-container">$K_2$</span> (<span class="math-container">$1 \times 2 \times 2$</span>), the output would be <span class="math-container">$1 \times 3 \times 3$</span>. The receptive field for this would be the <span class="math-container">$3 \times 3$</span> window of the input because you have already accumulated the sum of the <span class="math-container">$2 \times 2$</span> window in the previous layer.</p>
  <p>Considering your example of three CONV layers with <span class="math-container">$3 \times 3$</span> kernels is also similar. The first layer activation accumulates the sum of all the neurons in the <span class="math-container">$3 \times 3$</span> window of the input. When you further convolve this output with a kernel of <span class="math-container">$3 \times 3$</span>, it will accumulate all the outputs of the previous layers covering a bigger receptive field of the input.</p>
  <p>This observation comes in line with the argument that deeper layers learn more intricate features like facial expressions, abstract concepts, etc. because they cover a larger receptive field of our original input image.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-19T07:16:50.670Z
Answers.LastActivityDate: 2021-10-09T11:50:57.260Z
Answers.LastEditDate: 2021-10-09T11:50:57.260Z
Answers.UserDisplayName: Ayushi Agarwal
Body: <p>Below is a quote from <a href="https://cs231n.github.io/convolutional-networks/" rel="nofollow noreferrer">CS231n</a>:</p>
  <blockquote>
  <p>Prefer a stack of small filter CONV to one large receptive field CONV layer. Suppose that you stack three 3x3 CONV layers on top of each other (with non-linearities in between, of course). In this arrangement, each neuron on the first CONV layer has a 3x3 view of the input volume. A neuron on the second CONV layer has a 3x3 view of the first CONV layer, and hence by extension a 5x5 view of the input volume. Similarly, a neuron on the third CONV layer has a 3x3 view of the 2nd CONV layer, and hence a 7x7 view of the input volume. Suppose that instead of these three layers of 3x3 CONV, we only wanted to use a single CONV layer with 7x7 receptive fields. These neurons would have a receptive field size of the input volume that is identical in spatial extent (7x7), but with several disadvantages</p>
  </blockquote>
  <p>My visualized interpretation:</p>
  <p><a href="https://i.sstatic.net/KZy6R.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KZy6R.png" alt="image" /></a></p>
  <p>How can you see through the first CNN layer from the second CNN layer and see a 5x5 sized receptive field?</p>
  <p>There were no previous comments stating all the other hyperparameters, like input size, steps, padding, etc. which made this very confusing to visualize.</p>
  <hr />
  <p>Edited:</p>
  <p>I think I found the <a href="https://medium.com/@nikasa1889/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807" rel="nofollow noreferrer">answer</a>. BUT I still don't understand it. In fact, I am more confused than ever.</p>
  
Tags: <deep-learning><convolutional-neural-networks><convolution-arithmetic><receptive-field>
Title: How can 3 same size CNN layers in different ordering output different receptive field from the input layer?
UserDisplayName: Inkplay_
ThreadText: # How can 3 same size CNN layers in different ordering output different receptive field from the input layer?
  
  (Asked by: username_0 on 2018-01-23)
  
  Below is a quote from [CS231n](https://cs231n.github.io/convolutional-networks/):
  
  > Prefer a stack of small filter CONV to one large receptive field CONV layer. Suppose that you stack three 3x3 CONV layers on top of each other (with non-linearities in between, of course). In this arrangement, each neuron on the first CONV layer has a 3x3 view of the input volume. A neuron on the second CONV layer has a 3x3 view of the first CONV layer, and hence by extension a 5x5 view of the input volume. Similarly, a neuron on the third CONV layer has a 3x3 view of the 2nd CONV layer, and hence a 7x7 view of the input volume. Suppose that instead of these three layers of 3x3 CONV, we only wanted to use a single CONV layer with 7x7 receptive fields. These neurons would have a receptive field size of the input volume that is identical in spatial extent (7x7), but with several disadvantages
  
  My visualized interpretation:
  
  [![image](https://i.sstatic.net/KZy6R.png)](https://i.sstatic.net/KZy6R.png)
  
  How can you see through the first CNN layer from the second CNN layer and see a 5x5 sized receptive field?
  
  There were no previous comments stating all the other hyperparameters, like input size, steps, padding, etc. which made this very confusing to visualize.
  
  ---
  
  Edited:
  
  I think I found the [answer](https://medium.com/@nikasa1889/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807). BUT I still don't understand it. In fact, I am more confused than ever.
  
  **Tags:** deep-learning, convolutional-neural-networks, convolution-arithmetic, receptive-field
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2018-03-19. Score: 5 | Accepted answer
  
  It is really easy to visualize the growth in the receptive field of the input as you go deep into the CNN layers if you consider a small example.
  
  Let's take a simple example:
  
  The dimensions are in the form of $\text{channels} \times \text{height} \times \text{width}$.
  
  * The input image $I$ is a $3 \times 5 \times 5$ matrix
  * The first convolutional layer's kernel $K\_1$ has shape $3 \times 2 \times 2$ (we consider only 1 filter for simplicity)
  * The second convolutional layer's kernel $K\_2$ has shape $1 \times 2 \times 2$
  * Padding $P = 0$
  * Stride $S = 1$
  
  The output dimensions $O$ are calculated by the following formula taken from the lecture CS231n.
  
  $$O= (I - K + 2P)/S + 1$$
  
  When you do a convolution of the input image with the first filter $K\_1$, you get an output of shape $1 \times 4 \times 4$ (this is the activation of the CONV1 layer). The receptive field for this is the same as the kernel size ($K\_1$), that is, $2 \times 2$.
  
  When this layer (of shape $1 \times 4 \times 4$) is convolved with the second kernel (CONV2) $K\_2$ ($1 \times 2 \times 2$), the output would be $1 \times 3 \times 3$. The receptive field for this would be the $3 \times 3$ window of the input because you have already accumulated the sum of the $2 \times 2$ window in the previous layer.
  
  Considering your example of three CONV layers with $3 \times 3$ kernels is also similar. The first layer activation accumulates the sum of all the neurons in the $3 \times 3$ window of the input. When you further convolve this output with a kernel of $3 \times 3$, it will accumulate all the outputs of the previous layers covering a bigger receptive field of the input.
  
  This observation comes in line with the argument that deeper layers learn more intricate features like facial expressions, abstract concepts, etc. because they cover a larger receptive field of our original input image.
  
  
  ---
  
  ### Answer by username_1 on 2018-04-10. Score: 3
  
  The problem is in your diagram. Here are the steps to get to a 5x5 receptive field. Here is your diagram, redone slightly:
  [![The 3x3 input field is summed up (by the weights), and passed through a nonlinear activation function](https://i.sstatic.net/lEe45.jpg)](https://i.sstatic.net/lEe45.jpg)
  
  Notice that the new unit takes a weighted sum of the 9 pixels in the input, and then applies a rectified linear nonlinearity. Now, there are more of these, creating three new numbers computed from that part of the image. Each one slides over by one pixel:
  
  [![enter image description here](https://i.sstatic.net/jl3lW.jpg)](https://i.sstatic.net/jl3lW.jpg)
  
  We repeat this process going down three pixels as well, and then finally, we have a new 3x3 input field:
  
  [![enter image description here](https://i.sstatic.net/68ib4.jpg)](https://i.sstatic.net/68ib4.jpg)
  
  Notice that the new unit on the right now gets input from a 5x5 input field. I hope this helps!
  
  
  ---
  
  ### Answer by username_2 on 2018-07-07. Score: 0
  
  The intention of the referred text is to reason out the disadvantage of equivalent-merged-single-convolution-layer over multiple [CONV -> RELU]\*N layers.
  
  In the given scenario, if 2 layers of 3x3 filters were to be replaced by an equivalent single layer then this equivalent layer would need a filter with a receptive field of size 5x5.
  
  Similarly, an equivalent layer filter would need its receptive field to be of size 7x7 to compress 3 layers of 3x3 filters. Note that the most obvious disadvantage would be missing out on modeling non-linearity.
  
  
  ---
  
  