Answers.Body: <h1>Fully convolution networks</h1>
  <p>A <a href="https://arxiv.org/pdf/1411.4038.pdf" rel="noreferrer"><strong>fully convolution network (FCN)</strong></a> is a neural network that only performs convolution (and subsampling or upsampling) operations. Equivalently, an FCN is a CNN without fully connected layers.</p>
  <h2>Convolution neural networks</h2>
  <p>The typical <em>convolution neural network (CNN)</em> is not fully convolutional because it often contains <em>fully connected layers</em> too (which do not perform the convolution operation), which are <em>parameter-rich</em>, in the sense that they have many parameters (compared to their equivalent convolution layers), although <a href="https://arxiv.org/pdf/1411.4038.pdf" rel="noreferrer">the fully connected layers can also be viewed as convolutions with kernels that cover the entire input regions</a>, which is the main idea behind converting a CNN to an FCN. See <a href="https://www.youtube.com/watch?v=XdsmlBGOK-k" rel="noreferrer">this video</a> by Andrew Ng that explains how to convert a fully connected layer to a convolutional layer.</p>
  <h2>An example of an FCN</h2>
  <p>An example of a fully convolutional network is the <a href="https://arxiv.org/pdf/1505.04597.pdf" rel="noreferrer"><strong>U-net</strong></a> (called in this way because of its U shape, which you can see from the illustration below), which is a famous network that is used for <strong>semantic segmentation</strong>, i.e. classify pixels of an image so that pixels that belong to the same class (e.g. a person) are associated with the same label (i.e. person), aka pixel-wise (or dense) classification.</p>
  <p><a href="https://i.sstatic.net/938D2.png" rel="noreferrer"><img src="https://i.sstatic.net/938D2.png" alt="enter image description here" /></a></p>
  <h3>Semantic segmentation</h3>
  <p>So, in semantic segmentation, you want to associate a label with each pixel (or small patch of pixels) of the input image. Here's a more suggestive illustration of a neural network that performs semantic segmentation.</p>
  <p><a href="https://i.sstatic.net/CX1w0.png" rel="noreferrer"><img src="https://i.sstatic.net/CX1w0.png" alt="enter image description here" /></a></p>
  <h3>Instance segmentation</h3>
  <p>There's also <strong>instance segmentation</strong>, where you also want to differentiate different instances of the same class (e.g. you want to distinguish two people in the same image by labeling them differently). An example of a neural network that is used for instance segmentation is <a href="https://arxiv.org/pdf/1703.06870.pdf" rel="noreferrer">mask R-CNN</a>. The blog post <a href="https://glassboxmedicine.com/2020/01/21/segmentation-u-net-mask-r-cnn-and-medical-applications/" rel="noreferrer">Segmentation: U-Net, Mask R-CNN, and Medical Applications</a> (2020) by Rachel Draelos describes these two problems and networks very well.</p>
  <p>Here's an example of an image where instances of the same class (i.e. person) have been labeled differently (orange and blue).</p>
  <p><a href="https://i.sstatic.net/Xi6po.jpg" rel="noreferrer"><img src="https://i.sstatic.net/Xi6po.jpg" alt="enter image description here" /></a></p>
  <p>Both semantic and instance segmentations are <em>dense</em> classification tasks (specifically, they fall into the category of <a href="https://en.wikipedia.org/wiki/Image_segmentation" rel="noreferrer">image segmentation</a>), that is, you want to classify each pixel or many small patches of pixels of an image.</p>
  <h2><span class="math-container">$1 \times 1$</span> convolutions</h2>
  <p>In the U-net diagram above, you can see that there are only convolutions, copy and crop, max-pooling, and upsampling operations. There are no fully connected layers.</p>
  <p>So, how do we associate a label to each pixel (or a small patch of pixels) of the input? How do we perform the classification of each pixel (or patch) without a final fully connected layer?</p>
  <p>That's where the <span class="math-container">$1 \times 1$</span> convolution and upsampling operations are useful!</p>
  <p>In the case of the U-net diagram above (specifically, the top-right part of the diagram, which is illustrated below for clarity), <em>two</em> <span class="math-container">$1 \times 1 \times 64$</span> kernels are applied to the input volume (not the images!) to produce <em>two</em> feature maps of size <span class="math-container">$388 \times 388$</span>. They used two <span class="math-container">$1 \times 1$</span> kernels because there were two classes in their experiments (cell and not-cell). <a href="https://glassboxmedicine.com/2020/01/21/segmentation-u-net-mask-r-cnn-and-medical-applications/" rel="noreferrer">The mentioned blog post</a> also gives you the intuition behind this, so you should read it.</p>
  <p><a href="https://i.sstatic.net/rSiWZ.png" rel="noreferrer"><img src="https://i.sstatic.net/rSiWZ.png" alt="enter image description here" /></a></p>
  <p>If you have tried to analyze the U-net diagram carefully, you will notice that the output maps have different spatial (height and weight) dimensions than the input images, which have dimensions <span class="math-container">$572 \times 572 \times 1$</span>.</p>
  <p>That's fine because our general goal is to perform <strong>dense</strong> classification (i.e. classify patches of the image, where the patches can contain only one pixel), although I said that we would have performed pixel-wise classification, so maybe you were expecting the outputs to have the same exact spatial dimensions of the inputs. However, note that, in practice, you could also have the output maps to have the same spatial dimension as the inputs: you would just need to perform a different upsampling (deconvolution) operation.</p>
  <h3>How <span class="math-container">$1\times 1$</span> convolutions work?</h3>
  <p>A <span class="math-container">$1 \times 1$</span> convolution is just the typical 2d convolution but with a <span class="math-container">$1\times1$</span> kernel.</p>
  <p>As you probably already know (and if you didn't know this, now you know it), if you have a <span class="math-container">$g \times g$</span> kernel that is applied to an input of size <span class="math-container">$h \times w \times d$</span>, where <span class="math-container">$d$</span> is the depth of the input volume (which, for example, in the case of grayscale images, it is <span class="math-container">$1$</span>), the kernel actually has the shape <span class="math-container">$g \times g \times d$</span>, i.e. the third dimension of the kernel is equal to the third dimension of the input that it is applied to. This is always the case, except for 3d convolutions, but we are now talking about the typical 2d convolutions! See <a href="https://ai.stackexchange.com/a/18669/2444">this answer</a> for more info.</p>
  <p>So, in the case we want to apply a <span class="math-container">$1\times 1$</span> convolution to an input of shape <span class="math-container">$388 \times 388 \times 64$</span>, where <span class="math-container">$64$</span> is the depth of the input, then the actual <span class="math-container">$1\times 1$</span> kernels that we will need to use have shape  <span class="math-container">$1\times 1 \times 64$</span> (as I said above for the U-net). The way you reduce the depth of the input with <span class="math-container">$1\times 1$</span> is determined by the number of <span class="math-container">$1\times 1$</span> kernels that you want to use. This is exactly the same thing as for any 2d convolution operation with different kernels (e.g. <span class="math-container">$3 \times 3$</span>).</p>
  <p>In the case of the U-net, the spatial dimensions of the input are reduced in the same way that the spatial dimensions of any input to a CNN are reduced (i.e. 2d convolution followed by downsampling operations). The main difference (apart from not using fully connected layers) between the U-net and other CNNs is that the U-net performs upsampling operations, so it can be viewed as an encoder (left part) followed by a decoder (right part).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T23:35:14.593Z
Answers.Text: Thanks for your detailed reply, I really appreciate it!
Answers.UserDisplayName: r4bb1t
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-09T19:57:20.487Z
Answers.Text: Could `Xception` be considered an _FCN_? As it seems it can support any image size due to the final phase being global max pooling (Hence the final size before the `Dense` layers is independent of input size).
Answers.UserDisplayName: Royi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-09T20:20:53.220Z
Answers.Text: @Royi If it doesn't use the fully connected layers, but only convolutional and poolings, yes. I don't remember all the other details of the inception now.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-09T20:31:19.090Z
Answers.Text: It does use Fully Connected. But just after `GlobalMaxPooling` namely the size is only a function of the channels and not the 2D dimensions. In this case I abuse FCN as any net which supports arbitrary input size.
Answers.UserDisplayName: Royi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-27T22:14:38.207Z
Answers.Text: Search "1x1" on this page for another take on 1x1 convolutions:
  https://cs231n.github.io/convolutional-networks/
Answers.UserDisplayName: davidvandebunte
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T13:25:16.283Z
Answers.LastActivityDate: 2020-06-18T12:35:16.737Z
Answers.LastEditDate: 2020-06-18T12:35:16.737Z
Answers.UserDisplayName: nbro
Body: <p>I was surveying some <a href="https://arxiv.org/pdf/1512.08086.pdf" rel="noreferrer">literature</a> related to Fully Convolutional Networks and came across the following phrase, </p>
  
  <blockquote>
    <p>A fully convolutional network is achieved by replacing the parameter-rich fully connected layers in standard CNN architectures by convolutional layers with <span class="math-container">$1 \times 1$</span> kernels.</p>
  </blockquote>
  
  <p>I have two questions.</p>
  
  <ol>
  <li><p>What is meant by <em>parameter-rich</em>? Is it called parameter rich because the fully connected layers pass on parameters without any kind of "spatial" reduction? </p></li>
  <li><p>Also, how do <span class="math-container">$1 \times 1$</span> kernels work? Doesn't <span class="math-container">$1 \times 1$</span> kernel simply mean that one is sliding a single pixel over the image? I am confused about this.</p></li>
  </ol>
  
Tags: <machine-learning><convolutional-neural-networks><computer-vision><image-segmentation><fully-convolutional-networks>
Title: What is a fully convolution network?
UserDisplayName: r4bb1t
ThreadText: # What is a fully convolution network?
  
  (Asked by: username_0 on 2020-06-12)
  
  I was surveying some [literature](https://arxiv.org/pdf/1512.08086.pdf) related to Fully Convolutional Networks and came across the following phrase,
  
  > A fully convolutional network is achieved by replacing the parameter-rich fully connected layers in standard CNN architectures by convolutional layers with $1 \times 1$ kernels.
  
  I have two questions.
  
  1. What is meant by *parameter-rich*? Is it called parameter rich because the fully connected layers pass on parameters without any kind of "spatial" reduction?
  2. Also, how do $1 \times 1$ kernels work? Doesn't $1 \times 1$ kernel simply mean that one is sliding a single pixel over the image? I am confused about this.
  
  **Tags:** machine-learning, convolutional-neural-networks, computer-vision, image-segmentation, fully-convolutional-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-12. Score: 29 | Accepted answer
  
  # Fully convolution networks
  
  A [**fully convolution network (FCN)**](https://arxiv.org/pdf/1411.4038.pdf) is a neural network that only performs convolution (and subsampling or upsampling) operations. Equivalently, an FCN is a CNN without fully connected layers.
  
  ## Convolution neural networks
  
  The typical *convolution neural network (CNN)* is not fully convolutional because it often contains *fully connected layers* too (which do not perform the convolution operation), which are *parameter-rich*, in the sense that they have many parameters (compared to their equivalent convolution layers), although [the fully connected layers can also be viewed as convolutions with kernels that cover the entire input regions](https://arxiv.org/pdf/1411.4038.pdf), which is the main idea behind converting a CNN to an FCN. See [this video](https://www.youtube.com/watch?v=XdsmlBGOK-k) by Andrew Ng that explains how to convert a fully connected layer to a convolutional layer.
  
  ## An example of an FCN
  
  An example of a fully convolutional network is the [**U-net**](https://arxiv.org/pdf/1505.04597.pdf) (called in this way because of its U shape, which you can see from the illustration below), which is a famous network that is used for **semantic segmentation**, i.e. classify pixels of an image so that pixels that belong to the same class (e.g. a person) are associated with the same label (i.e. person), aka pixel-wise (or dense) classification.
  
  [![enter image description here](https://i.sstatic.net/938D2.png)](https://i.sstatic.net/938D2.png)
  
  ### Semantic segmentation
  
  So, in semantic segmentation, you want to associate a label with each pixel (or small patch of pixels) of the input image. Here's a more suggestive illustration of a neural network that performs semantic segmentation.
  
  [![enter image description here](https://i.sstatic.net/CX1w0.png)](https://i.sstatic.net/CX1w0.png)
  
  ### Instance segmentation
  
  There's also **instance segmentation**, where you also want to differentiate different instances of the same class (e.g. you want to distinguish two people in the same image by labeling them differently). An example of a neural network that is used for instance segmentation is [mask R-CNN](https://arxiv.org/pdf/1703.06870.pdf). The blog post [Segmentation: U-Net, Mask R-CNN, and Medical Applications](https://glassboxmedicine.com/2020/01/21/segmentation-u-net-mask-r-cnn-and-medical-applications/) (2020) by Rachel Draelos describes these two problems and networks very well.
  
  Here's an example of an image where instances of the same class (i.e. person) have been labeled differently (orange and blue).
  
  [![enter image description here](https://i.sstatic.net/Xi6po.jpg)](https://i.sstatic.net/Xi6po.jpg)
  
  Both semantic and instance segmentations are *dense* classification tasks (specifically, they fall into the category of [image segmentation](https://en.wikipedia.org/wiki/Image_segmentation)), that is, you want to classify each pixel or many small patches of pixels of an image.
  
  ## $1 \times 1$ convolutions
  
  In the U-net diagram above, you can see that there are only convolutions, copy and crop, max-pooling, and upsampling operations. There are no fully connected layers.
  
  So, how do we associate a label to each pixel (or a small patch of pixels) of the input? How do we perform the classification of each pixel (or patch) without a final fully connected layer?
  
  That's where the $1 \times 1$ convolution and upsampling operations are useful!
  
  In the case of the U-net diagram above (specifically, the top-right part of the diagram, which is illustrated below for clarity), *two* $1 \times 1 \times 64$ kernels are applied to the input volume (not the images!) to produce *two* feature maps of size $388 \times 388$. They used two $1 \times 1$ kernels because there were two classes in their experiments (cell and not-cell). [The mentioned blog post](https://glassboxmedicine.com/2020/01/21/segmentation-u-net-mask-r-cnn-and-medical-applications/) also gives you the intuition behind this, so you should read it.
  
  [![enter image description here](https://i.sstatic.net/rSiWZ.png)](https://i.sstatic.net/rSiWZ.png)
  
  If you have tried to analyze the U-net diagram carefully, you will notice that the output maps have different spatial (height and weight) dimensions than the input images, which have dimensions $572 \times 572 \times 1$.
  
  That's fine because our general goal is to perform **dense** classification (i.e. classify patches of the image, where the patches can contain only one pixel), although I said that we would have performed pixel-wise classification, so maybe you were expecting the outputs to have the same exact spatial dimensions of the inputs. However, note that, in practice, you could also have the output maps to have the same spatial dimension as the inputs: you would just need to perform a different upsampling (deconvolution) operation.
  
  ### How $1\times 1$ convolutions work?
  
  A $1 \times 1$ convolution is just the typical 2d convolution but with a $1\times1$ kernel.
  
  As you probably already know (and if you didn't know this, now you know it), if you have a $g \times g$ kernel that is applied to an input of size $h \times w \times d$, where $d$ is the depth of the input volume (which, for example, in the case of grayscale images, it is $1$), the kernel actually has the shape $g \times g \times d$, i.e. the third dimension of the kernel is equal to the third dimension of the input that it is applied to. This is always the case, except for 3d convolutions, but we are now talking about the typical 2d convolutions! See [this answer](https://ai.stackexchange.com/a/18669/2444) for more info.
  
  So, in the case we want to apply a $1\times 1$ convolution to an input of shape $388 \times 388 \times 64$, where $64$ is the depth of the input, then the actual $1\times 1$ kernels that we will need to use have shape $1\times 1 \times 64$ (as I said above for the U-net). The way you reduce the depth of the input with $1\times 1$ is determined by the number of $1\times 1$ kernels that you want to use. This is exactly the same thing as for any 2d convolution operation with different kernels (e.g. $3 \times 3$).
  
  In the case of the U-net, the spatial dimensions of the input are reduced in the same way that the spatial dimensions of any input to a CNN are reduced (i.e. 2d convolution followed by downsampling operations). The main difference (apart from not using fully connected layers) between the U-net and other CNNs is that the U-net performs upsampling operations, so it can be viewed as an encoder (left part) followed by a decoder (right part).
  
  > **Comments:**
  > **username_0**: Thanks for your detailed reply, I really appreciate it!  
  > **username_2**: Could `Xception` be considered an _FCN_? As it seems it can support any image size due to the final phase being global max pooling (Hence the final size before the `Dense` layers is independent of input size).  
  > **username_1**: @username_2 If it doesn't use the fully connected layers, but only convolutional and poolings, yes. I don't remember all the other details of the inception now.  
  > **username_2**: It does use Fully Connected. But just after `GlobalMaxPooling` namely the size is only a function of the channels and not the 2D dimensions. In this case I abuse FCN as any net which supports arbitrary input size.  
  > **username_3**: Search "1x1" on this page for another take on 1x1 convolutions: https://cs231n.github.io/convolutional-networks/  
  
  ---
  
  