Body: <p>Why does <a href="https://arxiv.org/pdf/1311.2901.pdf" rel="nofollow noreferrer">DeconvNet (Zeiler, 2014)</a> use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply the ReLU again coming back to the input?</p>
  
  <p><strong>update:</strong>  I better explain my problem:</p>
  
  <p>given an input image <span class="math-container">$x$</span> and ConvLayer <span class="math-container">$CL$</span> composed of: </p>
  
  <ol>
  <li>a convolution</li>
  <li>an activation function ReLU</li>
  <li>a pool operation</li>
  </ol>
  
  <p><span class="math-container">$f$</span> is the output of ConvLayer given an input <span class="math-container">$x$</span>, i.e. <span class="math-container">$f=CL(x)$</span>.</p>
  
  <p>So, the Deconv target is to "reverse" the output <span class="math-container">$f$</span> (the feature map) to restore an approximate version of <span class="math-container">$x$</span>. To this aim, the authors define a function <span class="math-container">$CL^{-1}$</span> composed of 3 subfunctions: </p>
  
  <p>a. unpool </p>
  
  <p>b. activation function ReLU (useless in my opinion, because <span class="math-container">$f$</span> is already positive due to the application of the 2. step in <span class="math-container">$CL(f)$</span>) </p>
  
  <p>c. transposed convolution. </p>
  
  <p>In other words <span class="math-container">$x\simeq CL^{-1}(f)$</span> where <span class="math-container">$CL^{-1} (f) = transpconv(relu(unpool(f)))$</span>. But, if <span class="math-container">$f$</span> is the output computed as <span class="math-container">$f=CL(x)$</span>, it is already positive, so the b. step is useless.</p>
  
  <p>This is what I understood from the paper. Where I wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-03T10:15:29.767Z
Comments.Text: While computing error (training phase) its just like any other layer. Error can be negative, hence the ReLU, I guess.
Comments.UserDisplayName: Karan Shah
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-11T12:06:50.963Z
Comments.Text: but in the paper seems that $CL^{-1}$ is applied to $f$ (which is the feature map computed in the forward pass)
Comments.UserDisplayName: volperossa
Tags: <neural-networks><deep-learning><convolutional-neural-networks><backpropagation><relu>
Title: Why do DeconvNet use ReLU in the backward pass?
UserDisplayName: volperossa
ThreadText: # Why do DeconvNet use ReLU in the backward pass?
  
  (Asked by: username_0 on 2020-03-30)
  
  Why does [DeconvNet (Zeiler, 2014)](https://arxiv.org/pdf/1311.2901.pdf) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply the ReLU again coming back to the input?
  
  **update:** I better explain my problem:
  
  given an input image $x$ and ConvLayer $CL$ composed of:
  
  1. a convolution
  2. an activation function ReLU
  3. a pool operation
  
  $f$ is the output of ConvLayer given an input $x$, i.e. $f=CL(x)$.
  
  So, the Deconv target is to "reverse" the output $f$ (the feature map) to restore an approximate version of $x$. To this aim, the authors define a function $CL^{-1}$ composed of 3 subfunctions:
  
  a. unpool
  
  b. activation function ReLU (useless in my opinion, because $f$ is already positive due to the application of the 2. step in $CL(f)$)
  
  c. transposed convolution.
  
  In other words $x\simeq CL^{-1}(f)$ where $CL^{-1} (f) = transpconv(relu(unpool(f)))$. But, if $f$ is the output computed as $f=CL(x)$, it is already positive, so the b. step is useless.
  
  This is what I understood from the paper. Where I wrong?
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, backpropagation, relu
  > **Comments:**
  > **username_1**: While computing error (training phase) its just like any other layer. Error can be negative, hence the ReLU, I guess.  
  > **username_0**: but in the paper seems that $CL^{-1}$ is applied to $f$ (which is the feature map computed in the forward pass)  
  
  ---
  
  ## 0 answers
  
  