Answers.Body: <p>I have found that you should compute derivatives <span class="math-container">$\frac{\partial L}{\partial A}, \frac{\partial L}{\partial Z}$</span> in Flatten layer and then reshape Conv2D input shape.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-01T22:45:04.403Z
Answers.Text: Would you share an example of neural network model having Flatten --> Convolution?
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T07:48:36.027Z
Answers.LastActivityDate: 2020-05-06T10:47:26.017Z
Answers.LastEditDate: 2020-05-06T10:47:26.017Z
Answers.UserDisplayName: PentaHackedAll
Body: <p>I'm trying to create a convolutional neural network without frameworks (such as PyTorch, TensorFlow, Keras, and so on) with Python.</p>
  
  <p>Here's a description of CNN taken from <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network" rel="nofollow noreferrer">the Wikipedia article</a></p>
  
  <blockquote>
    <p>In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery. They are also known as shift invariant or space invariant artificial neural networks (SIANN), based on their shared-weights architecture and translation invariance characteristics. They have applications in image and video recognition, recommender systems, image classification, medical image analysis, natural language processing, and financial time series.</p>
  </blockquote>
  
  <p>A CNN has different types of layers, such as convolution, pooling (max or average), flatten and dense (or fully-connected) layers.</p>
  
  <p>I have a few questions.</p>
  
  <ol>
  <li><p>Should we compute gradients (such as <span class="math-container">$\frac{\partial L}{\partial A_i}$</span>,<span class="math-container">$\frac{\partial L}{\partial Z_i}$</span>,<span class="math-container">$\frac{\partial L}{\partial A_{i-1}}$</span> and so on) in flatten layer or not?</p></li>
  <li><p>If no, then how should I compute <span class="math-container">$\frac{\partial L}{\partial A_i}$</span> and <span class="math-container">$\frac{\partial L}{\partial Z_i}$</span> of first layer of convolutional layer? With <span class="math-container">$\frac{\partial L}{[\frac{\partial g(A_i)}{\partial x}]}$</span> or with 
  <span class="math-container">$\frac{\partial L}{\partial dA_{i+2}}$</span>(P.S. as you know iteration of BackPropagation is reverse, so I used i+n for denote the previous layer)? </p></li>
  <li><p>Or can I compute derivatives in Flatten layer with 
  <span class="math-container">$$\frac{\partial J}{\partial A} = W_{i+1}^T Z_{i+1}$$</span>(i+1 denotes prev.layer in BackProp)
  <span class="math-container">$$\frac{\partial L}{\partial Z} = \frac{\partial L}{\partial A} *\frac{\partial g(A_i)}{\partial x} $$</span>
  and then reshape of Conv2D shape? </p></li>
  </ol>
  
  <p>P.S. I found questions like mine (names are same), but there're not answer to my question as I asking about formula.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-25T12:03:48.553Z
Comments.Text: What are $A$ and $Z$ here?
Comments.UserDisplayName: nbro
Tags: <machine-learning><convolutional-neural-networks><backpropagation>
Title: Should I compute the gradients with respect to the flatten layer in a convolutional neural network?
UserDisplayName: PentaHackedAll
ThreadText: # Should I compute the gradients with respect to the flatten layer in a convolutional neural network?
  
  (Asked by: username_0 on 2020-04-30)
  
  I'm trying to create a convolutional neural network without frameworks (such as PyTorch, TensorFlow, Keras, and so on) with Python.
  
  Here's a description of CNN taken from [the Wikipedia article](https://en.wikipedia.org/wiki/Convolutional_neural_network)
  
  > In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of deep neural networks, most commonly applied to analyzing visual imagery. They are also known as shift invariant or space invariant artificial neural networks (SIANN), based on their shared-weights architecture and translation invariance characteristics. They have applications in image and video recognition, recommender systems, image classification, medical image analysis, natural language processing, and financial time series.
  
  A CNN has different types of layers, such as convolution, pooling (max or average), flatten and dense (or fully-connected) layers.
  
  I have a few questions.
  
  1. Should we compute gradients (such as $\frac{\partial L}{\partial A\_i}$,$\frac{\partial L}{\partial Z\_i}$,$\frac{\partial L}{\partial A\_{i-1}}$ and so on) in flatten layer or not?
  2. If no, then how should I compute $\frac{\partial L}{\partial A\_i}$ and $\frac{\partial L}{\partial Z\_i}$ of first layer of convolutional layer? With $\frac{\partial L}{[\frac{\partial g(A\_i)}{\partial x}]}$ or with
     $\frac{\partial L}{\partial dA\_{i+2}}$(P.S. as you know iteration of BackPropagation is reverse, so I used i+n for denote the previous layer)?
  3. Or can I compute derivatives in Flatten layer with
     $$\frac{\partial J}{\partial A} = W\_{i+1}^T Z\_{i+1}$$(i+1 denotes prev.layer in BackProp)
     $$\frac{\partial L}{\partial Z} = \frac{\partial L}{\partial A} \*\frac{\partial g(A\_i)}{\partial x} $$
     and then reshape of Conv2D shape?
  
  P.S. I found questions like mine (names are same), but there're not answer to my question as I asking about formula.
  
  **Tags:** machine-learning, convolutional-neural-networks, backpropagation
  > **Comments:**
  > **username_1**: What are $A$ and $Z$ here?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-05-01. Score: 1 | Accepted answer
  
  I have found that you should compute derivatives $\frac{\partial L}{\partial A}, \frac{\partial L}{\partial Z}$ in Flatten layer and then reshape Conv2D input shape.
  
  > **Comments:**
  > **username_2**: Would you share an example of neural network model having Flatten --> Convolution?  
  
  ---
  
  