Answers.Body: <p>Let's first recapitulate why the function that calculates the maximum between two or more numbers, <span class="math-container">$z=\operatorname{max}(x_1, x_2)$</span>, is not a linear function.</p>
  
  <p>A linear function is defined as <span class="math-container">$y=f(x) = ax + b$</span>, so <span class="math-container">$y$</span> linearly increases with <span class="math-container">$x$</span>. Visually, <span class="math-container">$f$</span> corresponds to a straight line (or hyperplane, in the case of 2 or more input variables).</p>
  
  <p>If <span class="math-container">$z$</span> does not correspond to such a straight line (or hyperplane), then it cannot be a linear function (by definition). </p>
  
  <p>Let <span class="math-container">$x_1 = 1$</span> and let <span class="math-container">$x_2 \in [0, 2]$</span>. Then <span class="math-container">$z=\operatorname{max}(x_1, x_2) = x_1$</span> for all <span class="math-container">$x_2 \in [0, 1]$</span>. In other words, for the sub-range <span class="math-container">$x_2 \in [0, 1]$</span>, the maximum between <span class="math-container">$x_1$</span> and <span class="math-container">$x_2$</span> is a constant function (a horizontal line at <span class="math-container">$x_1=1$</span>). However, for the sub-range <span class="math-container">$x_2 \in [1, 2]$</span>, <span class="math-container">$z$</span> correspond to <span class="math-container">$x_2$</span>, that is, <span class="math-container">$z$</span> linearly increases with <span class="math-container">$x_2$</span>. Given that max is not a linear function in a special case, it can't also be a linear function in general.</p>
  
  <p>Here's a plot (computed with <a href="https://www.wolframalpha.com/input/?i=plot+max%28x%2C+y%29" rel="noreferrer">Wolfram Alpha</a>) of the maximum between two numbers (so it is clearly a function of two variables, hence the plot is 3D).</p>
  
  <p><a href="https://i.sstatic.net/RNRu6.png" rel="noreferrer"><img src="https://i.sstatic.net/RNRu6.png" alt="enter image description here"></a></p>
  
  <p>Note that, in this plot, both variables, <span class="math-container">$x$</span> and <span class="math-container">$y$</span>, can linearly increase, as opposed to having one of the variables fixed (which I used only to give you a simple and hopefully intuitive example that the maximum is not a linear function).</p>
  
  <p>In the case of convolution networks, although max-pooling is a non-linear operation, it is primarily used to <strong>reduce the dimensionality</strong> of the input, so that to reduce overfitting and computation. In any case, max-pooling doesn't non-linearly transform the input <strong>element-wise</strong>.</p>
  
  <p>The average function is a linear function because it linearly increases with the inputs. Here's a <a href="https://www.wolframalpha.com/input/?i=plot+%28x%2By%29%2F2" rel="noreferrer">plot of the average between two numbers</a>, which is clearly a hyperplane.</p>
  
  <p><a href="https://i.sstatic.net/a3oHU.png" rel="noreferrer"><img src="https://i.sstatic.net/a3oHU.png" alt="enter image description here"></a></p>
  
  <p>In the case of convolution networks, the average pooling is also used to reduce the dimensionality.</p>
  
  <p>To answer your question more directly, the non-linearity is usually applied element-wise, but neither max-pooling nor average pooling can do that (even if you downsample with a <span class="math-container">$1 \times 1$</span> window, i.e. you do not downsample at all). </p>
  
  <p>Nevertheless, you don't necessarily need a non-linear activation function after the convolution operation (if you use max-pooling), but the performance will be worse than if you use a non-linear activation, as reported in the paper <a href="https://arxiv.org/pdf/1606.02228.pdf" rel="noreferrer">Systematic evaluation of CNN advances on the
  ImageNet</a> (figure 2).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-10T00:10:29.677Z
Answers.Text: Is the same not true for ReLU? And even more so for LeakyReLU?
Answers.UserDisplayName: Kasia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-10T00:12:57.553Z
Answers.Text: @Kasia What exactly are you referring to?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-10T00:31:13.960Z
Answers.Text: ReLU is simply a max function. Couldn't it basically be considered a max-pool with additional constant dimension?
Answers.UserDisplayName: Kasia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-10T01:34:35.583Z
Answers.Text: @Kasia In max-pooling, conceptually, you slide a window. When you apply ReLU you do not slide any window. It's true that ReLU is a max between 0 and the input, but I fail to understand how you want to make it a pooling operation.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-10T19:20:33.150Z
Answers.Text: @DuttA So I spent a day thinking about it. In an intuitive approach, ReLU is something like an on/off for gradient flow. Either lets it go through freely, or turns it off completely. In the same way max-pool allows the gradient to flow freely through the maximum value, and blocks it for all the other values inside the sliding window. That' s the resemblance I see.
Answers.UserDisplayName: Kasia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-10T20:10:06.783Z
Answers.Text: @Kasia Yes, the [derivative of the ReLU](https://stats.stackexchange.com/q/333394/82135) is consistent with your description.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-09T23:03:19.290Z
Answers.LastActivityDate: 2020-02-09T23:44:37.177Z
Answers.LastEditDate: 2020-02-09T23:44:37.177Z
Answers.UserDisplayName: nbro
Body: <p>Is there any need to use a non-linear activation function (ReLU, LeakyReLU, Sigmoid, etc.) if the result of the convolution layer is passed through the sliding window max function, like max-pooling, which is non-linear itself? What about the average pooling?</p>
  
Tags: <deep-learning><convolutional-neural-networks><activation-functions><pooling><max-pooling>
Title: Is a non-linear activation function needed if we perform max-pooling after the convolution layer?
UserDisplayName: Kasia
ThreadText: # Is a non-linear activation function needed if we perform max-pooling after the convolution layer?
  
  (Asked by: username_0 on 2020-02-09)
  
  Is there any need to use a non-linear activation function (ReLU, LeakyReLU, Sigmoid, etc.) if the result of the convolution layer is passed through the sliding window max function, like max-pooling, which is non-linear itself? What about the average pooling?
  
  **Tags:** deep-learning, convolutional-neural-networks, activation-functions, pooling, max-pooling
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-09. Score: 6 | Accepted answer
  
  Let's first recapitulate why the function that calculates the maximum between two or more numbers, $z=\operatorname{max}(x\_1, x\_2)$, is not a linear function.
  
  A linear function is defined as $y=f(x) = ax + b$, so $y$ linearly increases with $x$. Visually, $f$ corresponds to a straight line (or hyperplane, in the case of 2 or more input variables).
  
  If $z$ does not correspond to such a straight line (or hyperplane), then it cannot be a linear function (by definition).
  
  Let $x\_1 = 1$ and let $x\_2 \in [0, 2]$. Then $z=\operatorname{max}(x\_1, x\_2) = x\_1$ for all $x\_2 \in [0, 1]$. In other words, for the sub-range $x\_2 \in [0, 1]$, the maximum between $x\_1$ and $x\_2$ is a constant function (a horizontal line at $x\_1=1$). However, for the sub-range $x\_2 \in [1, 2]$, $z$ correspond to $x\_2$, that is, $z$ linearly increases with $x\_2$. Given that max is not a linear function in a special case, it can't also be a linear function in general.
  
  Here's a plot (computed with [Wolfram Alpha](https://www.wolframalpha.com/input/?i=plot+max%28x%2C+y%29)) of the maximum between two numbers (so it is clearly a function of two variables, hence the plot is 3D).
  
  [![enter image description here](https://i.sstatic.net/RNRu6.png)](https://i.sstatic.net/RNRu6.png)
  
  Note that, in this plot, both variables, $x$ and $y$, can linearly increase, as opposed to having one of the variables fixed (which I used only to give you a simple and hopefully intuitive example that the maximum is not a linear function).
  
  In the case of convolution networks, although max-pooling is a non-linear operation, it is primarily used to **reduce the dimensionality** of the input, so that to reduce overfitting and computation. In any case, max-pooling doesn't non-linearly transform the input **element-wise**.
  
  The average function is a linear function because it linearly increases with the inputs. Here's a [plot of the average between two numbers](https://www.wolframalpha.com/input/?i=plot+%28x%2By%29%2F2), which is clearly a hyperplane.
  
  [![enter image description here](https://i.sstatic.net/a3oHU.png)](https://i.sstatic.net/a3oHU.png)
  
  In the case of convolution networks, the average pooling is also used to reduce the dimensionality.
  
  To answer your question more directly, the non-linearity is usually applied element-wise, but neither max-pooling nor average pooling can do that (even if you downsample with a $1 \times 1$ window, i.e. you do not downsample at all).
  
  Nevertheless, you don't necessarily need a non-linear activation function after the convolution operation (if you use max-pooling), but the performance will be worse than if you use a non-linear activation, as reported in the paper [Systematic evaluation of CNN advances on the
  ImageNet](https://arxiv.org/pdf/1606.02228.pdf) (figure 2).
  
  > **Comments:**
  > **username_0**: Is the same not true for ReLU? And even more so for LeakyReLU?  
  > **username_1**: @username_0 What exactly are you referring to?  
  > **username_0**: ReLU is simply a max function. Couldn't it basically be considered a max-pool with additional constant dimension?  
  > **username_1**: @username_0 In max-pooling, conceptually, you slide a window. When you apply ReLU you do not slide any window. It's true that ReLU is a max between 0 and the input, but I fail to understand how you want to make it a pooling operation.  
  > **username_0**: @DuttA So I spent a day thinking about it. In an intuitive approach, ReLU is something like an on/off for gradient flow. Either lets it go through freely, or turns it off completely. In the same way max-pool allows the gradient to flow freely through the maximum value, and blocks it for all the other values inside the sliding window. That' s the resemblance I see.  
  > **username_1**: @username_0 Yes, the [derivative of the ReLU](https://stats.stackexchange.com/q/333394/82135) is consistent with your description.  
  
  ---
  
  