Answers.Body: <p>Yes, when you have non-linearity it is not possible to combine your convolution steps.</p>
  <p>However, you can approximate the two layers network with one layer net according to the Universal Approximation Theorem. You will probably need to use something like a knowledge distillation technique to do it. Note that, the theorem doesn't say anything about the number of neurons required or if the learning techniques that we usually use will work well.</p>
  <p>Also, <span class="math-container">$ReLU(x)$</span> is a linear mapping when <span class="math-container">$x \geq 0$</span>, so if your input, weight, and biases are <span class="math-container">$\geq 0$</span>, the net can be exactly modeled with a single layer network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-01T01:19:29.623Z
Answers.Text: Thx! I will investigate about "Universal Approximation Theorem", I've never heard. I am trying to optimize cascade convolutions as much as possible since I am implementing convolutional neural networks in an electronic device.
Answers.UserDisplayName: Diego Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-01T01:13:05.083Z
Answers.LastActivityDate: 2020-07-01T01:13:05.083Z
Answers.UserDisplayName: saiRegrefree
Body: <p>The associative property of multidimensional discrete convolution says that:</p>
  <p><span class="math-container">$$Y=(x \circledast h_1) \circledast h_2=x\circledast(h_1\circledast h_2)$$</span></p>
  <p>where <span class="math-container">$h_1$</span> and <span class="math-container">$h_2$</span> are the filters and <span class="math-container">$x$</span> is the input.</p>
  <p>I was able to do exploit this property in Keras with <code>Conv2D</code>: first, I convolve <span class="math-container">$h_1$</span> and <span class="math-container">$h_2$</span>, then I convolve the result with <span class="math-container">$x$</span> (i.e. the rightmost part of the equation above).</p>
  <p>Up to this point, I don't have any problem, and I also understand that convolution is linear.</p>
  <p>The problem is when two Conv2D layers have a non-linear activation function after the convolution. For example, consider the following two operations</p>
  <p><span class="math-container">$$Y_1=\text{ReLU}(x \circledast h_1)$$</span>
  <span class="math-container">$$Y_2=\text{ReLU}(Y_1\circledast h_2)$$</span></p>
  <p>It is possible to apply the associative property if the first or both layers have a non-linear activation function (in the case above ReLU, but it could be any activation function)? I don't think so. Any idea or related paper or some kind of approach?</p>
  
Tags: <convolutional-neural-networks><activation-functions><convolution>
Title: Is it possible to apply the associative property of the convolution operation when it is followed by a non-linearity?
UserDisplayName: Diego Ruiz
ThreadText: # Is it possible to apply the associative property of the convolution operation when it is followed by a non-linearity?
  
  (Asked by: username_0 on 2020-06-30)
  
  The associative property of multidimensional discrete convolution says that:
  
  $$Y=(x \circledast h\_1) \circledast h\_2=x\circledast(h\_1\circledast h\_2)$$
  
  where $h\_1$ and $h\_2$ are the filters and $x$ is the input.
  
  I was able to do exploit this property in Keras with `Conv2D`: first, I convolve $h\_1$ and $h\_2$, then I convolve the result with $x$ (i.e. the rightmost part of the equation above).
  
  Up to this point, I don't have any problem, and I also understand that convolution is linear.
  
  The problem is when two Conv2D layers have a non-linear activation function after the convolution. For example, consider the following two operations
  
  $$Y\_1=\text{ReLU}(x \circledast h\_1)$$
  $$Y\_2=\text{ReLU}(Y\_1\circledast h\_2)$$
  
  It is possible to apply the associative property if the first or both layers have a non-linear activation function (in the case above ReLU, but it could be any activation function)? I don't think so. Any idea or related paper or some kind of approach?
  
  **Tags:** convolutional-neural-networks, activation-functions, convolution
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-01. Score: 1 | Accepted answer
  
  Yes, when you have non-linearity it is not possible to combine your convolution steps.
  
  However, you can approximate the two layers network with one layer net according to the Universal Approximation Theorem. You will probably need to use something like a knowledge distillation technique to do it. Note that, the theorem doesn't say anything about the number of neurons required or if the learning techniques that we usually use will work well.
  
  Also, $ReLU(x)$ is a linear mapping when $x \geq 0$, so if your input, weight, and biases are $\geq 0$, the net can be exactly modeled with a single layer network.
  
  > **Comments:**
  > **username_0**: Thx! I will investigate about "Universal Approximation Theorem", I've never heard. I am trying to optimize cascade convolutions as much as possible since I am implementing convolutional neural networks in an electronic device.  
  
  ---
  
  