Answers.Body: <p><span class="math-container">$F_l$</span> is the activation of the filter. They state in the paper that they base their method on VGG-Network, which uses ReLU as its activation function. In fact, VGG uses it in all of its hidden layers. ReLU is defined as</p>
  <p><span class="math-container">$$f(x) = max(0,x)$$</span></p>
  <p>Since ReLU is 0 for all x's below 0, the equation above holds; When x is non-positive, all terms in the loss function are constants with respect to <span class="math-container">$F_{ij}^l$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T11:13:11.217Z
Answers.LastActivityDate: 2021-08-13T12:07:36.103Z
Answers.LastEditDate: 2021-08-13T12:07:36.103Z
Answers.UserDisplayName: Avatrin
Body: <p>I am currently in the process of reading and understanding the process of style transfer. I came across this equation in the research paper which went like -
  <a href="https://i.sstatic.net/gqaVU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gqaVU.png" alt="enter image description here" /></a></p>
  <p>For context, here is the paragraph -</p>
  <blockquote>
  <p>Generally each layer in the network defines a non-linear
  filter bank whose complexity increases with the position of
  the layer in the network. Hence a given input image is
  encoded in each layer of the Convolutional Neural Network
  by the filter responses to that image. A layer with <span class="math-container">$N_l$</span> distinct filters has <span class="math-container">$N$</span> feature maps each of size <span class="math-container">$M$</span> , where <span class="math-container">$M_l$</span>
  is the height times the width of the feature map. So the re-
  sponses in a layer l can be stored in a matrix <span class="math-container">$Fl ∈ R^{N_l×M_l}$</span>
  where F l is the activation of the ith filter at position j in ij
  layer l.
  To visualise the image information that is encoded at
  different layers of the hierarchy one can perform gradient descent on a white noise image to find another image that matches the feature responses of the original image (Fig 1, content reconstructions). Let <span class="math-container">$\vec p$</span> and <span class="math-container">$\vec x$</span> be the original image and the image that is generated, and <span class="math-container">$P^l$</span> and <span class="math-container">$F^l$</span> their respective feature representation in layer l. We then define the squared-error loss between the two feature representations
  <span class="math-container">$\mathcal{L_{content}(\vec p, \vec x, l)} = {1\over 2} \Sigma_{i,j} \big(F_{ij}^l - P_{ij}^l \big)$</span>. The derivative of this loss with respect to the activations in layer <span class="math-container">$l$</span> [<em><strong>the equation above</strong></em> <span class="math-container">$(2)$</span>].</p>
  </blockquote>
  <p>I just want to know why the partial derivative is <span class="math-container">$0$</span> when <span class="math-container">$F_{ij}^l &lt; 0$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-13T13:10:55.397Z
Comments.Text: Hello. Please, put your **specific** question in the title. "Math behind style transfer" is very vague/general and not a question. Thank you!
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-13T13:18:13.833Z
Comments.Text: K I actually tried to sum up my entire Q in that one line - wasnt able to in the beginning XD. I'll do it soon
Comments.UserDisplayName: HarshDarji
Tags: <convolutional-neural-networks><relu><calculus><vgg><style-transfer>
Title: Why the partial derivative is $0$ when $F_{ij}^l < 0$?. Math behind style transfer
UserDisplayName: HarshDarji
ThreadText: # Why the partial derivative is $0$ when $F_{ij}^l < 0$?. Math behind style transfer
  
  (Asked by: username_0 on 2021-08-13)
  
  I am currently in the process of reading and understanding the process of style transfer. I came across this equation in the research paper which went like -
  [![enter image description here](https://i.sstatic.net/gqaVU.png)](https://i.sstatic.net/gqaVU.png)
  
  For context, here is the paragraph -
  
  > Generally each layer in the network defines a non-linear
  > filter bank whose complexity increases with the position of
  > the layer in the network. Hence a given input image is
  > encoded in each layer of the Convolutional Neural Network
  > by the filter responses to that image. A layer with $N\_l$ distinct filters has $N$ feature maps each of size $M$ , where $M\_l$
  > is the height times the width of the feature map. So the re-
  > sponses in a layer l can be stored in a matrix $Fl ∈ R^{N\_l×M\_l}$
  > where F l is the activation of the ith filter at position j in ij
  > layer l.
  > To visualise the image information that is encoded at
  > different layers of the hierarchy one can perform gradient descent on a white noise image to find another image that matches the feature responses of the original image (Fig 1, content reconstructions). Let $\vec p$ and $\vec x$ be the original image and the image that is generated, and $P^l$ and $F^l$ their respective feature representation in layer l. We then define the squared-error loss between the two feature representations
  > $\mathcal{L\_{content}(\vec p, \vec x, l)} = {1\over 2} \Sigma\_{i,j} \big(F\_{ij}^l - P\_{ij}^l \big)$. The derivative of this loss with respect to the activations in layer $l$ [***the equation above*** $(2)$].
  
  I just want to know why the partial derivative is $0$ when $F\_{ij}^l < 0$.
  
  **Tags:** convolutional-neural-networks, relu, calculus, vgg, style-transfer
  
  > **Comments:**
  > **username_1**: Hello. Please, put your **specific** question in the title. "Math behind style transfer" is very vague/general and not a question. Thank you!  
  > **username_0**: K I actually tried to sum up my entire Q in that one line - wasnt able to in the beginning XD. I'll do it soon  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-08-13. Score: 2
  
  $F\_l$ is the activation of the filter. They state in the paper that they base their method on VGG-Network, which uses ReLU as its activation function. In fact, VGG uses it in all of its hidden layers. ReLU is defined as
  
  $$f(x) = max(0,x)$$
  
  Since ReLU is 0 for all x's below 0, the equation above holds; When x is non-positive, all terms in the loss function are constants with respect to $F\_{ij}^l$.
  
  
  ---
  
  