Answers.Body: <p>Yes, a simple reshape would do the trick. A flattening layer is just a tool for reshaping data/activations to make them compatible with other layers/functions. The flattening layer doesn't change the activations themselves, so there is no special backpropagation handling needed other than changing back the shape.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-04T19:52:23.580Z
Answers.LastActivityDate: 2019-04-04T19:52:23.580Z
Answers.UserDisplayName: Philip Raeisghasem
Answers.Body: <p>The <code>Flatten</code> layer has no learnable parameters in itself (the operation it performs is fully defined by construction); still, it has to propagate the gradient to the previous layers.</p>
  <p>In general, the <code>Flatten</code> operation is well-posed, as whatever is the input shape you know what the output shape is.</p>
  <p>When you backpropagate, you are supposed to do an &quot;Unflatten&quot;, which maps a flattened tensor into a tensor of a given shape, and you know what that specific shape is from the forward pass, so it is also a well-posed operation.</p>
  <p>More formally</p>
  <p>Say you have <code>Img1</code> in input of your <code>Flatten</code> layer</p>
  <p><span class="math-container">$$
  \begin{pmatrix}
  f_{1,1}(x; w_{1,1}) &amp; f_{1,2}(x; w_{1,2}) \\ 
  f_{2,1}(x; w_{2,1}) &amp; f_{2,2}(x; w_{2,2})
  \end{pmatrix}
  $$</span></p>
  <p>So, in the output you have</p>
  <p><span class="math-container">$$
  \begin{pmatrix}
  f_{1,1}(x; w_{1,1}) &amp; f_{1,2}(x; w_{1,2}) &amp; f_{2,1}(x; w_{2,1}) &amp; f_{2,2}(x; w_{2,2})
  \end{pmatrix}
  $$</span></p>
  <p>When you compute the gradient you have</p>
  <p><span class="math-container">$$
  \frac{df_{i,j}(x; w_{i,j})}{dw_{i,j}}
  $$</span></p>
  <p>and everything in the same position as in the forward pass, so the unflatten maps from the <code>(1, 4)</code> tensor to the <code>(2, 2)</code> tensor.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-06T07:10:25.853Z
Answers.LastActivityDate: 2020-09-10T21:38:13.503Z
Answers.LastEditDate: 2020-09-10T21:38:13.503Z
Answers.UserDisplayName: Nicola Bernini
Body: <p>I am making a NN library without any other external NN library, so I am implementing all layers, including the flatten layer, and algorithms (forward and backward pass) from scratch. I know the forward implementation of the flatten layer, but is the backward just reshaping it or not? If yes, can I just call a simple NumPy's <code>reshape</code> function to reshape it?</p>
  
Tags: <neural-networks><convolutional-neural-networks><backpropagation><implementation>
Title: How should I implement the backward pass through a flatten layer of a CNN?
UserDisplayName: Clement
ThreadText: # How should I implement the backward pass through a flatten layer of a CNN?
  
  (Asked by: username_0 on 2019-04-04)
  
  I am making a NN library without any other external NN library, so I am implementing all layers, including the flatten layer, and algorithms (forward and backward pass) from scratch. I know the forward implementation of the flatten layer, but is the backward just reshaping it or not? If yes, can I just call a simple NumPy's `reshape` function to reshape it?
  
  **Tags:** neural-networks, convolutional-neural-networks, backpropagation, implementation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-04-04. Score: 1 | Accepted answer
  
  Yes, a simple reshape would do the trick. A flattening layer is just a tool for reshaping data/activations to make them compatible with other layers/functions. The flattening layer doesn't change the activations themselves, so there is no special backpropagation handling needed other than changing back the shape.
  
  
  ---
  
  ### Answer by username_2 on 2020-09-06. Score: 2
  
  The `Flatten` layer has no learnable parameters in itself (the operation it performs is fully defined by construction); still, it has to propagate the gradient to the previous layers.
  
  In general, the `Flatten` operation is well-posed, as whatever is the input shape you know what the output shape is.
  
  When you backpropagate, you are supposed to do an "Unflatten", which maps a flattened tensor into a tensor of a given shape, and you know what that specific shape is from the forward pass, so it is also a well-posed operation.
  
  More formally
  
  Say you have `Img1` in input of your `Flatten` layer
  
  $$
  \begin{pmatrix}
  f\_{1,1}(x; w\_{1,1}) & f\_{1,2}(x; w\_{1,2}) \\
  f\_{2,1}(x; w\_{2,1}) & f\_{2,2}(x; w\_{2,2})
  \end{pmatrix}
  $$
  
  So, in the output you have
  
  $$
  \begin{pmatrix}
  f\_{1,1}(x; w\_{1,1}) & f\_{1,2}(x; w\_{1,2}) & f\_{2,1}(x; w\_{2,1}) & f\_{2,2}(x; w\_{2,2})
  \end{pmatrix}
  $$
  
  When you compute the gradient you have
  
  $$
  \frac{df\_{i,j}(x; w\_{i,j})}{dw\_{i,j}}
  $$
  
  and everything in the same position as in the forward pass, so the unflatten maps from the `(1, 4)` tensor to the `(2, 2)` tensor.
  
  
  ---
  
  