Answers.Body: <p>When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward direction are also sums and <span class="math-container">$\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$</span>. </p>
  
  <p>That is the case already with gradients of kernels (which are sums over the image area), and is equally the case for overlapping aggregation, including maximums, minimums or averages.</p>
  
  <p>So in the 1d case, if you have a max pool over the input params <span class="math-container">$[a_0, a_1, a_2, a_3, a_4]$</span> a max function <span class="math-container">$m_0 = max(a_0, a_1, a_2)$</span>, <span class="math-container">$m_1 = max(a_2, a_3, a_4)$</span> which overlap at <span class="math-container">$a_2$</span>, and gradients <span class="math-container">$\nabla_{\mathbf{m}} J = [\frac{\partial J}{\partial m_0}, \frac{\partial J}{\partial m_1}]$</span>, then you would allocate those gradients to vector <span class="math-container">$\mathbf{a}$</span> according to which items in each group was the max of that group, adding them when they overlapped.</p>
  
  <p>Examples:</p>
  
  <p>If <span class="math-container">$\mathbf{a} = [3,0,1,2,0]$</span> and  <span class="math-container">$\nabla_{\mathbf{m}}J = [0.7, 0.9]$</span>, then <span class="math-container">$\nabla_{\mathbf{a}}J = [0.7, 0, 0, 0.9, 0]$</span></p>
  
  <p>If <span class="math-container">$\mathbf{a} = [3,0,4,2,0]$</span> and  <span class="math-container">$\nabla_{\mathbf{m}}J = [0.7, 0.9]$</span>, then <span class="math-container">$\nabla_{\mathbf{a}}J = [0, 0, 1.6, 0, 0]$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-15T13:42:11.013Z
Answers.LastActivityDate: 2019-12-16T07:14:02.650Z
Answers.LastEditDate: 2019-12-16T07:14:02.650Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Denote x(h,w) input to max-pooling, and y(h,w) - output. </p>
  
  <p>Then <span class="math-container">$\frac{dL}{dx}(h,w)= \sum \frac{dL}{dy}(h',w')$</span></p>
  
  <p>over all y(h',w') which have been obtained  from x(h.w) such that y(h',w') = x(h,w).</p>
  
  <p>related to <a href="http://courses.cs.tau.ac.il/Caffe_workshop/Bootcamp/pdf_lectures/Lecture%203%20CNN%20-%20backpropagation.pdf" rel="nofollow noreferrer">this p 11</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-16T07:20:14.590Z
Answers.Text: So this is the same answer as mine (add the gradients when they overlap), but for the 2D case.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-15T21:58:12.857Z
Answers.LastActivityDate: 2019-12-15T21:58:12.857Z
Answers.UserDisplayName: estamos
Body: <p>While studying backpropagation in CNNs, I can't understand how can we compute the gradient of max pooling with overlapping regions.</p>
  <p>That's also a question from <a href="http://courses.cs.tau.ac.il/Caffe_workshop/Bootcamp/pdf_lectures/Lecture%203%20CNN%20-%20backpropagation.pdf" rel="nofollow noreferrer">this</a> quiz and can be also found on <a href="https://books.google.gr/books?id=72UkDwAAQBAJ&amp;pg=PA128&amp;lpg=PA128&amp;dq=Compute+the+gradient+of+max+pooling+with+overlapping+regions.&amp;source=bl&amp;ots=RknIRCMsbW&amp;sig=ACfU3U1zStijp120L9lTh5jG-sLL3XxZtQ&amp;hl=el&amp;sa=X&amp;ved=2ahUKEwimjczOrbfmAhVBsaQKHVHsAhQQ6AEwCXoECAgQAQ#v=onepage&amp;q=Compute%20the%20gradient%20of%20max%20pooling%20with%20overlapping%20regions.&amp;f=false" rel="nofollow noreferrer">this</a> book.</p>
  
Tags: <convolutional-neural-networks><backpropagation><gradient><pooling><max-pooling>
Title: How can we compute the gradient of max pooling with overlapping regions?
UserDisplayName: estamos
ThreadText: # How can we compute the gradient of max pooling with overlapping regions?
  
  (Asked by: username_0 on 2019-12-15)
  
  While studying backpropagation in CNNs, I can't understand how can we compute the gradient of max pooling with overlapping regions.
  
  That's also a question from [this](http://courses.cs.tau.ac.il/Caffe_workshop/Bootcamp/pdf_lectures/Lecture%203%20CNN%20-%20backpropagation.pdf) quiz and can be also found on [this](https://books.google.gr/books?id=72UkDwAAQBAJ&pg=PA128&lpg=PA128&dq=Compute+the+gradient+of+max+pooling+with+overlapping+regions.&source=bl&ots=RknIRCMsbW&sig=ACfU3U1zStijp120L9lTh5jG-sLL3XxZtQ&hl=el&sa=X&ved=2ahUKEwimjczOrbfmAhVBsaQKHVHsAhQQ6AEwCXoECAgQAQ#v=onepage&q=Compute%20the%20gradient%20of%20max%20pooling%20with%20overlapping%20regions.&f=false) book.
  
  **Tags:** convolutional-neural-networks, backpropagation, gradient, pooling, max-pooling
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-12-15. Score: 2 | Accepted answer
  
  When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward direction are also sums and $\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$.
  
  That is the case already with gradients of kernels (which are sums over the image area), and is equally the case for overlapping aggregation, including maximums, minimums or averages.
  
  So in the 1d case, if you have a max pool over the input params $[a\_0, a\_1, a\_2, a\_3, a\_4]$ a max function $m\_0 = max(a\_0, a\_1, a\_2)$, $m\_1 = max(a\_2, a\_3, a\_4)$ which overlap at $a\_2$, and gradients $\nabla\_{\mathbf{m}} J = [\frac{\partial J}{\partial m\_0}, \frac{\partial J}{\partial m\_1}]$, then you would allocate those gradients to vector $\mathbf{a}$ according to which items in each group was the max of that group, adding them when they overlapped.
  
  Examples:
  
  If $\mathbf{a} = [3,0,1,2,0]$ and $\nabla\_{\mathbf{m}}J = [0.7, 0.9]$, then $\nabla\_{\mathbf{a}}J = [0.7, 0, 0, 0.9, 0]$
  
  If $\mathbf{a} = [3,0,4,2,0]$ and $\nabla\_{\mathbf{m}}J = [0.7, 0.9]$, then $\nabla\_{\mathbf{a}}J = [0, 0, 1.6, 0, 0]$
  
  
  ---
  
  ### Answer by username_0 on 2019-12-15. Score: 0
  
  Denote x(h,w) input to max-pooling, and y(h,w) - output.
  
  Then $\frac{dL}{dx}(h,w)= \sum \frac{dL}{dy}(h',w')$
  
  over all y(h',w') which have been obtained from x(h.w) such that y(h',w') = x(h,w).
  
  related to [this p 11](http://courses.cs.tau.ac.il/Caffe_workshop/Bootcamp/pdf_lectures/Lecture%203%20CNN%20-%20backpropagation.pdf)
  
  > **Comments:**
  > **username_1**: So this is the same answer as mine (add the gradients when they overlap), but for the 2D case.  
  
  ---
  
  