Answers.Body: <p>You can think of applying a point-wise, i.e. <span class="math-container">$1\times 1$</span>, convolution with <span class="math-container">$F$</span> filters to a <span class="math-container">$H\times W\times C$</span> image (or feature maps, in general) as the application of a <em>linear fully-connected layer</em> (with <span class="math-container">$F$</span> units) to each spatial location of the input, resulting in <span class="math-container">$H\times W\times F$</span> tensor.</p>
  <p>In general, you can see the application of a convolutional layer (regardless the kernel size) as a matrix multiplication between the <em>kernel matrix</em>, say <span class="math-container">$K$</span>, and the <em>flattened image matrix</em>, say <span class="math-container">$I$</span>, where: <span class="math-container">$K$</span> is a block diagonal matrix (you have zeros outside the diagonal) build by kind of stacking the kernel (the weights of the conv layer) <span class="math-container">$H$</span> times as follows:</p>
  <p><a href="https://i.sstatic.net/hpINY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hpINY.png" alt="enter image description here" /></a></p>
  <p>then <span class="math-container">$I$</span> is obtained by flattening the input image. After the multiplication you reshape the results back to a 3d-tensor shape.</p>
  <p>In the example you have a <span class="math-container">$1\times 2$</span> kenel (reddish vector), and a <span class="math-container">$3\times 3$</span> image. So, you should consider for the matrix multiplication a stack of <span class="math-container">$2\times 3$</span> kenels (<span class="math-container">$k\times W$</span>, in general) and a reshaped image of size <span class="math-container">$(H\times W,1)$</span>.</p>
  <blockquote>
  <p>what happens under the hood of a NN framework, such as PyTorch, when the number of input channels is equal to the number of output channels?</p>
  </blockquote>
  <p>For practical implementations I don't think the matrix formulation of conv layers is beneficial, especially if you consider GPUs which have many threads and you can basically let each thread access a location of the input tensor, compute the respective index in the kernel, do the product and finally sum. Now if <span class="math-container">$k=1$</span> (as in pointwise conv), the matrix <span class="math-container">$K$</span> is built from <span class="math-container">$1\times W$</span> matrices, stacked <span class="math-container">$H$</span> times, resulting in <span class="math-container">$(W\times H)\times H$</span> kernel matrix, which is also mostly sparse. Maybe with ad-hoc algorithms for sparse multiplication this operation may be convenient.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-16T14:51:21.340Z
Answers.LastActivityDate: 2023-07-16T14:51:21.340Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <p>It is useless to use a <code>1x1</code> convolution if the number of output channel is equal to the number of input channel, as it allows no additionnal expressivity for the network and just adds useless computations. You can however add a ReLu between the two layers to make it useful as the linearity will no longer hold.</p>
  <p>If you use a <code>1x1</code> convolution (or more precisely, a <code>1x1xc</code> convolution where <code>c</code> is the number of input channels), then each output channel will be a linear combination of the input channels. The coefficients of these linear combinations corresponds to the coefficients of the different <code>1x1</code> kernels.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-11T10:22:48.060Z
Answers.Text: This answers one of the questions but does not answer the main question which is what does that mean implementation-wise? Does going from N channels to M channels involve a matrix multiplication between the input of shape (height*width, N) and a weights matrix of shape (N, M) ? Apologies if the question above was not clear.
Answers.UserDisplayName: alexandrosangeli
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-11T09:56:30.867Z
Answers.LastActivityDate: 2023-07-11T09:56:30.867Z
Answers.UserDisplayName: Lelouch
Body: <p>I know in order to shrink or expand the number of channels a 1x1 convolution is performed.</p>
  <p>I need to clarify the following: is the 1x1 convolution(s) just a matrix multiplication between the image with shape (h<em>w, 3) (RGB) and a matrix that holds the learnable weights with shape (3, 1)? Which will result in a new matrix of shape (h</em>w, 1) (in this case the number of channels shrunk from 3 to 1).</p>
  <p>If the above is correct, what happens under the hood of a NN framework, such as PyTorch, when the number of input channels is equal to the number of output channels? Does a matrix multiplication take place between the input (h*w, 3) and a matrix with learnable weights (num_channels, num_channels)? Doesn't this introduce unnecessary (and unwanted) operations?</p>
  
Tags: <convolutional-neural-networks><computer-vision><image-processing>
Title: How is the number of channels in a convolutional layer shrinked or expanded?
UserDisplayName: alexandrosangeli
ThreadText: # How is the number of channels in a convolutional layer shrinked or expanded?
  
  (Asked by: username_0 on 2023-07-08)
  
  I know in order to shrink or expand the number of channels a 1x1 convolution is performed.
  
  I need to clarify the following: is the 1x1 convolution(s) just a matrix multiplication between the image with shape (h*w, 3) (RGB) and a matrix that holds the learnable weights with shape (3, 1)? Which will result in a new matrix of shape (h*w, 1) (in this case the number of channels shrunk from 3 to 1).
  
  If the above is correct, what happens under the hood of a NN framework, such as PyTorch, when the number of input channels is equal to the number of output channels? Does a matrix multiplication take place between the input (h\*w, 3) and a matrix with learnable weights (num\_channels, num\_channels)? Doesn't this introduce unnecessary (and unwanted) operations?
  
  **Tags:** convolutional-neural-networks, computer-vision, image-processing
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-07-16. Score: 0
  
  You can think of applying a point-wise, i.e. $1\times 1$, convolution with $F$ filters to a $H\times W\times C$ image (or feature maps, in general) as the application of a *linear fully-connected layer* (with $F$ units) to each spatial location of the input, resulting in $H\times W\times F$ tensor.
  
  In general, you can see the application of a convolutional layer (regardless the kernel size) as a matrix multiplication between the *kernel matrix*, say $K$, and the *flattened image matrix*, say $I$, where: $K$ is a block diagonal matrix (you have zeros outside the diagonal) build by kind of stacking the kernel (the weights of the conv layer) $H$ times as follows:
  
  [![enter image description here](https://i.sstatic.net/hpINY.png)](https://i.sstatic.net/hpINY.png)
  
  then $I$ is obtained by flattening the input image. After the multiplication you reshape the results back to a 3d-tensor shape.
  
  In the example you have a $1\times 2$ kenel (reddish vector), and a $3\times 3$ image. So, you should consider for the matrix multiplication a stack of $2\times 3$ kenels ($k\times W$, in general) and a reshaped image of size $(H\times W,1)$.
  
  > what happens under the hood of a NN framework, such as PyTorch, when the number of input channels is equal to the number of output channels?
  
  For practical implementations I don't think the matrix formulation of conv layers is beneficial, especially if you consider GPUs which have many threads and you can basically let each thread access a location of the input tensor, compute the respective index in the kernel, do the product and finally sum. Now if $k=1$ (as in pointwise conv), the matrix $K$ is built from $1\times W$ matrices, stacked $H$ times, resulting in $(W\times H)\times H$ kernel matrix, which is also mostly sparse. Maybe with ad-hoc algorithms for sparse multiplication this operation may be convenient.
  
  
  ---
  
  ### Answer by username_2 on 2023-07-11. Score: 0
  
  It is useless to use a `1x1` convolution if the number of output channel is equal to the number of input channel, as it allows no additionnal expressivity for the network and just adds useless computations. You can however add a ReLu between the two layers to make it useful as the linearity will no longer hold.
  
  If you use a `1x1` convolution (or more precisely, a `1x1xc` convolution where `c` is the number of input channels), then each output channel will be a linear combination of the input channels. The coefficients of these linear combinations corresponds to the coefficients of the different `1x1` kernels.
  
  > **Comments:**
  > **username_0**: This answers one of the questions but does not answer the main question which is what does that mean implementation-wise? Does going from N channels to M channels involve a matrix multiplication between the input of shape (height*width, N) and a weights matrix of shape (N, M) ? Apologies if the question above was not clear.  
  
  ---
  
  