Answers.Body: <p>Answers:</p>
  
  <ol>
  <li>Generally its the former. The next layer would learn at each filter how to merge the channels of the previous layer, that is why in a 2D convolution the kernel is a 3-dimensional tensor. But the number of parameters is <span class="math-container">$nmc_ic_{i+1}$</span> at the <span class="math-container">$i^{th}$</span> layer (this is ignoring bias). lets assume all channels are <span class="math-container">$O(c)$</span> then the spatial complexity becomes <span class="math-container">$O(knmc^2)$</span> where <span class="math-container">$k$</span> is the number of layers.   </li>
  <li><em>a)</em> The first convolutional filter would have kernel size of (w,h,3,10) where w and h are the kernel sizes of the 2d convolution (often in practice is 3). So there are 10 filters of size (w,h,3), but the # of parameters is w*h*30 (once again for ease, ignoring bias). The second layer though, since it is working on a layer with 10 channels, will have kernel (w,h,10,10).<br>
  <em>b)</em> I think you need to go back and look at what a convolution does (in your setting specifically a 2D convolution). Each <strong>filter</strong> works on every <strong>channel</strong> of the previous layer. Each <strong>channel</strong> of the last convolutional layer refers to a <strong>single</strong> filter that convolved over the entire previous layer to that. </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-20T12:43:46.367Z
Answers.LastActivityDate: 2019-07-20T12:43:46.367Z
Answers.UserDisplayName: mshlis
Answers.Body: <h3>tl;dr</h3>
  <p>It helps to think that the <em>channels</em> dimension of a convolutional layer works like a <strong>fully connected</strong> layer (i.e. the layer computes the weighted sum over all channels).</p>
  <h3>For a single pixel...</h3>
  <p>Let's consider a single pixel (e.g. the top left pixel). This pixel has <span class="math-container">$C$</span> different values, where <span class="math-container">$C$</span> are the number of channels. In order to produce the result of a single filter, the layer takes the weighted sum of these <span class="math-container">$C$</span> pixels. It does this by actually having <span class="math-container">$C$</span> weights, multiplying the pixel values with their corresponding weights and summing them together.</p>
  <h3>Example</h3>
  <p>Let's say you have an <span class="math-container">$n \times m \times 10$</span> tensor as an input to a convolutional layer with <span class="math-container">$1$</span> filter and a <span class="math-container">$3 \times 3$</span> kernel. For creating its output the layer has <span class="math-container">$3 \times 3 \times 10 = 90$</span> weights, i.e. a different <span class="math-container">$3 \times 3$</span> kernel for each of the <span class="math-container">$10$</span> input channels. To create its output, the layer performs the convolution operation <strong>separately</strong> on each of the input channels (each with its corresponding weight matrix) and creates this way <span class="math-container">$10$</span> feature maps which are <strong>summed</strong> together.</p>
  <p>Now imagine the layer has <span class="math-container">$20$</span> filters instead of <span class="math-container">$1$</span>. Nothing changes, just that the same procedure is done <span class="math-container">$20$</span> times with <span class="math-container">$20$</span> different sets of weights So the total number of weights in the layer in this case is <span class="math-container">$3 \times 3 \times 10 \times 20 = 1800$</span>.</p>
  <h3>To answer your questions...</h3>
  <p><strong>(1)</strong> You have a grayscale image <span class="math-container">$n \times m \times 1$</span> and it passes through the first convolution layer (which has <span class="math-container">$10$</span> filters). This layer will perform the convolution operation on the input image <span class="math-container">$10$</span> times <strong>independently</strong> and produce <span class="math-container">$10$</span> feature maps, i.e. an output tensor of <span class="math-container">$n \times m \times 10$</span>. Now this in fed into the second convolution layer, which again has <span class="math-container">$10$</span> filters. <strong>For each</strong> filter the layer will perform the convolution operation on the <span class="math-container">$10$</span> input feature maps independently and will sum the corresponding pixels together to form a <span class="math-container">$n \times m \times 1$</span> feature map. It will perform this procedure <span class="math-container">$10$</span> times and generate an output tensor of <span class="math-container">$n \times m \times 10$</span>.</p>
  <p><strong>(2)</strong> Yes, each layer actually has <span class="math-container">$30$</span> filters, a total of <span class="math-container">$10$</span> for each of the R, G, B channels. The layer just sums the results of the R, G, B channels to produce a single feature map. For the second part, if I got it right it's pretty much what you said but it sums the maps instead of averaging them</p>
  <h3>Notes</h3>
  <p>I'd recommend checking <a href="http://cs231n.github.io/convolutional-networks/" rel="nofollow noreferrer">Stanford's CS231 notes on convolution layers</a>, which explain it in much detail and also have numerical examples to confirm if you've got it right.</p>
  <p>You could also check <a href="https://stats.stackexchange.com/a/335327/119015">this answer</a> for more details.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-20T15:34:48.060Z
Answers.LastActivityDate: 2019-07-20T15:34:48.060Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Djib2011
Body: <p>I am unsure about the following parts of the architecture and mechanics of convolution layers in CNNs. Possibly, this is implementation-dependent though.</p>
  
  <p><strong>First question:</strong></p>
  
  <p>Say I have 2 convolution layers with 10 filters each and the dimension of my input tensors is <span class="math-container">$n \times m \times 1$</span>  (so, grayscale images for example). Passing this input to the first convolution layer results in 10 feature maps (10 matrices of <span class="math-container">$n \times m$</span>, if we use padding), each produced by a different filter.</p>
  
  <p>Now, what does actually happen when this is passed to the second convolution layer? Are all 10 feature maps passed as one big <span class="math-container">$m \times n \times 10$</span> tensor or are the overlapping cells of the 10 feature maps averaged and a <span class="math-container">$m \times n \times 1$</span> tensor is passed to the next convolution layer? The former would result in an explosion of feature maps with increasing number of convolution layers and the spacial complexity would be in <span class="math-container">$\mathcal{O}\left((nm)^k\right)$</span>, where <span class="math-container">$k$</span> is the number of chained convolution layers. Averaging the feature maps before passing them to the next layer would keep the complexity linear. So, which is it? Or are both possibilities commonly used?</p>
  
  <p><strong>Second question</strong> (with two sub questions)<strong>:</strong></p>
  
  <p>a) This is a similar question. If I have an input volume of <span class="math-container">$n \times m \times 3$</span> (e.g. RGB images) and I have again 2 convolution layers with 10 filters, does each convolution layer have in actuality 30 filters? So 10 sets of 3 filters, one for each channel? Or do I have in fact only 10 filters and the filters are applied to all 3 channels?</p>
  
  <p>b) This is the same question as question (1) but for channels: Once I have convolved a filter (consisting of three channel filters? (a)) over the input tensor I end up with 3 feature maps. One for each channel. What do I do with these? Do I average them component-wise with each other? Or do I keep them separate until I have convolved all 10 filters across the input and THEN average the 10 feature maps of each channel? Or do I average all 30 feature maps of all three channels? Or do I just pass on 30 feature maps to the next convoloution layers which in turn knows which of these feature maps belong to which channel?</p>
  
  <p>Quite a few possibilities... None of the sources I consulted makes this explicit. Maybe because it depends on the individual implementation.</p>
  
  <p>Anyway, would be great if somebody could clear this confusion up a little!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-20T13:55:52.117Z
Comments.Text: Hi and welcome to this community! Please, next time ask **one question per post**! Here you're asking too many questions and this does not facilitate the life of possible answerers and makes your question too broad, so, in theory, it should be closed.
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><convolutional-neural-networks><definitions><implementation>
Title: Are feature maps merged or are they passed on as they are?
UserDisplayName: lo tolmencre
ThreadText: # Are feature maps merged or are they passed on as they are?
  
  (Asked by: username_0 on 2019-07-19)
  
  I am unsure about the following parts of the architecture and mechanics of convolution layers in CNNs. Possibly, this is implementation-dependent though.
  
  **First question:**
  
  Say I have 2 convolution layers with 10 filters each and the dimension of my input tensors is $n \times m \times 1$ (so, grayscale images for example). Passing this input to the first convolution layer results in 10 feature maps (10 matrices of $n \times m$, if we use padding), each produced by a different filter.
  
  Now, what does actually happen when this is passed to the second convolution layer? Are all 10 feature maps passed as one big $m \times n \times 10$ tensor or are the overlapping cells of the 10 feature maps averaged and a $m \times n \times 1$ tensor is passed to the next convolution layer? The former would result in an explosion of feature maps with increasing number of convolution layers and the spacial complexity would be in $\mathcal{O}\left((nm)^k\right)$, where $k$ is the number of chained convolution layers. Averaging the feature maps before passing them to the next layer would keep the complexity linear. So, which is it? Or are both possibilities commonly used?
  
  **Second question** (with two sub questions)**:**
  
  a) This is a similar question. If I have an input volume of $n \times m \times 3$ (e.g. RGB images) and I have again 2 convolution layers with 10 filters, does each convolution layer have in actuality 30 filters? So 10 sets of 3 filters, one for each channel? Or do I have in fact only 10 filters and the filters are applied to all 3 channels?
  
  b) This is the same question as question (1) but for channels: Once I have convolved a filter (consisting of three channel filters? (a)) over the input tensor I end up with 3 feature maps. One for each channel. What do I do with these? Do I average them component-wise with each other? Or do I keep them separate until I have convolved all 10 filters across the input and THEN average the 10 feature maps of each channel? Or do I average all 30 feature maps of all three channels? Or do I just pass on 30 feature maps to the next convoloution layers which in turn knows which of these feature maps belong to which channel?
  
  Quite a few possibilities... None of the sources I consulted makes this explicit. Maybe because it depends on the individual implementation.
  
  Anyway, would be great if somebody could clear this confusion up a little!
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, definitions, implementation
  
  > **Comments:**
  > **username_1**: Hi and welcome to this community! Please, next time ask **one question per post**! Here you're asking too many questions and this does not facilitate the life of possible answerers and makes your question too broad, so, in theory, it should be closed.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-07-20. Score: 3 | Accepted answer
  
  ### tl;dr
  
  It helps to think that the *channels* dimension of a convolutional layer works like a **fully connected** layer (i.e. the layer computes the weighted sum over all channels).
  
  ### For a single pixel...
  
  Let's consider a single pixel (e.g. the top left pixel). This pixel has $C$ different values, where $C$ are the number of channels. In order to produce the result of a single filter, the layer takes the weighted sum of these $C$ pixels. It does this by actually having $C$ weights, multiplying the pixel values with their corresponding weights and summing them together.
  
  ### Example
  
  Let's say you have an $n \times m \times 10$ tensor as an input to a convolutional layer with $1$ filter and a $3 \times 3$ kernel. For creating its output the layer has $3 \times 3 \times 10 = 90$ weights, i.e. a different $3 \times 3$ kernel for each of the $10$ input channels. To create its output, the layer performs the convolution operation **separately** on each of the input channels (each with its corresponding weight matrix) and creates this way $10$ feature maps which are **summed** together.
  
  Now imagine the layer has $20$ filters instead of $1$. Nothing changes, just that the same procedure is done $20$ times with $20$ different sets of weights So the total number of weights in the layer in this case is $3 \times 3 \times 10 \times 20 = 1800$.
  
  ### To answer your questions...
  
  **(1)** You have a grayscale image $n \times m \times 1$ and it passes through the first convolution layer (which has $10$ filters). This layer will perform the convolution operation on the input image $10$ times **independently** and produce $10$ feature maps, i.e. an output tensor of $n \times m \times 10$. Now this in fed into the second convolution layer, which again has $10$ filters. **For each** filter the layer will perform the convolution operation on the $10$ input feature maps independently and will sum the corresponding pixels together to form a $n \times m \times 1$ feature map. It will perform this procedure $10$ times and generate an output tensor of $n \times m \times 10$.
  
  **(2)** Yes, each layer actually has $30$ filters, a total of $10$ for each of the R, G, B channels. The layer just sums the results of the R, G, B channels to produce a single feature map. For the second part, if I got it right it's pretty much what you said but it sums the maps instead of averaging them
  
  ### Notes
  
  I'd recommend checking [Stanford's CS231 notes on convolution layers](http://cs231n.github.io/convolutional-networks/), which explain it in much detail and also have numerical examples to confirm if you've got it right.
  
  You could also check [this answer](https://stats.stackexchange.com/a/335327/119015) for more details.
  
  
  ---
  
  ### Answer by username_2 on 2019-07-20. Score: 2
  
  Answers:
  
  1. Generally its the former. The next layer would learn at each filter how to merge the channels of the previous layer, that is why in a 2D convolution the kernel is a 3-dimensional tensor. But the number of parameters is $nmc\_ic\_{i+1}$ at the $i^{th}$ layer (this is ignoring bias). lets assume all channels are $O(c)$ then the spatial complexity becomes $O(knmc^2)$ where $k$ is the number of layers.
  2. *a)* The first convolutional filter would have kernel size of (w,h,3,10) where w and h are the kernel sizes of the 2d convolution (often in practice is 3). So there are 10 filters of size (w,h,3), but the # of parameters is w\*h\*30 (once again for ease, ignoring bias). The second layer though, since it is working on a layer with 10 channels, will have kernel (w,h,10,10).  
     *b)* I think you need to go back and look at what a convolution does (in your setting specifically a 2D convolution). Each **filter** works on every **channel** of the previous layer. Each **channel** of the last convolutional layer refers to a **single** filter that convolved over the entire previous layer to that.
  
  
  ---
  
  