Answers.Body: <p>All-convolutional neural network is a more general concept which can be (and is often) used without deconvolutional and unpolling layers, e.g. for an ordinary classification task. The idea is to replace the pooling and fully-connected layer with particular convolutional layers that do the same. Note that activation functions and dropout are not affected by this transformation. You can read more about it <a href="http://cs231n.github.io/convolutional-networks/" rel="nofollow noreferrer">here</a>.</p>
  
  <p>The same will be true for the deconvolutional layers. I didn't find it stated explicitly in the <a href="https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf" rel="nofollow noreferrer">original paper</a>, but here's <a href="https://arxiv.org/pdf/1511.00561.pdf" rel="nofollow noreferrer">another work</a> that applies fully-convolutional network for image segmentation. On the figure 2, you can see ReLu activations: they are placed after (simple) convolutional layers, in both upsampling and downsampling parts. They don't use dropout, but batchnorm instead, but if there were a dropout, just as before, it should be placed after the convolutional layer. The keep probability, though, should probably be higher - this relates to hyperparameters tuning and may be different depending on the problem and available data.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-11T04:58:13.150Z
Answers.LastActivityDate: 2017-10-11T04:58:13.150Z
Answers.UserDisplayName: Maxim
Body: <p>I am trying to replicate the <em>fully convolutional networks</em> (FCN) concept described <a href="https://www.youtube.com/watch?v=nDPWywWRIRo&amp;t=16m30s" rel="nofollow noreferrer">here</a> for semantic segmentation. It seems people have successfully trained such models by removing fully connected layers from the popular pre-trained models such as VGG and adding <em>upsampling</em> and <em>unpooling layers</em>.</p>
  
  <p>I understand that <em>transpose convolution</em> and <em>unpooling</em> in upsampling layers provide counterparts of <em>convolution</em> and <em>max (or average) pooling</em> in earlier downsampling layers respectively, but what are the counterparts for non-linearities such as ReLU? What about dropout? There seems to be no discussion of this in the video.</p>
  
Tags: <neural-networks><convolutional-neural-networks><activation-functions><dropout><fully-convolutional-networks>
Title: What are the counterparts of non-linearities and dropout in fully convolutional networks?
UserDisplayName: abhinavkulkarni
ThreadText: # What are the counterparts of non-linearities and dropout in fully convolutional networks?
  
  (Asked by: username_0 on 2017-10-11)
  
  I am trying to replicate the *fully convolutional networks* (FCN) concept described [here](https://www.youtube.com/watch?v=nDPWywWRIRo&t=16m30s) for semantic segmentation. It seems people have successfully trained such models by removing fully connected layers from the popular pre-trained models such as VGG and adding *upsampling* and *unpooling layers*.
  
  I understand that *transpose convolution* and *unpooling* in upsampling layers provide counterparts of *convolution* and *max (or average) pooling* in earlier downsampling layers respectively, but what are the counterparts for non-linearities such as ReLU? What about dropout? There seems to be no discussion of this in the video.
  
  **Tags:** neural-networks, convolutional-neural-networks, activation-functions, dropout, fully-convolutional-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-10-11. Score: 1
  
  All-convolutional neural network is a more general concept which can be (and is often) used without deconvolutional and unpolling layers, e.g. for an ordinary classification task. The idea is to replace the pooling and fully-connected layer with particular convolutional layers that do the same. Note that activation functions and dropout are not affected by this transformation. You can read more about it [here](http://cs231n.github.io/convolutional-networks/).
  
  The same will be true for the deconvolutional layers. I didn't find it stated explicitly in the [original paper](https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf), but here's [another work](https://arxiv.org/pdf/1511.00561.pdf) that applies fully-convolutional network for image segmentation. On the figure 2, you can see ReLu activations: they are placed after (simple) convolutional layers, in both upsampling and downsampling parts. They don't use dropout, but batchnorm instead, but if there were a dropout, just as before, it should be placed after the convolutional layer. The keep probability, though, should probably be higher - this relates to hyperparameters tuning and may be different depending on the problem and available data.
  
  
  ---
  
  