Answers.Body: <p>It is definitley possible to make the links between neurons use more complex functions. Provided those functions are differentiable, backpropagation still works, and the resulting compound function might be able to learn something useful. The general name for such a thing is a <a href="https://www.deepideas.net/deep-learning-from-scratch-i-computational-graphs/" rel="nofollow noreferrer">computational graph</a> and the standardised structures used in most neural networks are a subset of all possible (and maybe useful) computational graphs.</p>
  
  <p>When adding complex and non-linear functions into a neural network, this is usually alternated with simpler linear layers using the weights. A generalised function of a single neuron as used in most neural networks looks like this:</p>
  
  <p><span class="math-container">$$a = f(\sum_i w_i x_i + b)$$</span></p>
  
  <p>Where <span class="math-container">$i$</span> indexes all inputs to the neuron, <span class="math-container">$x_i$</span> are the input values, <span class="math-container">$w_i$</span> the weights associated with each input, <span class="math-container">$b$</span> is a bias term and <span class="math-container">$f()$</span> is a differentiable non-linear <em>activation function</em>. Training process learns <span class="math-container">$w_i$</span> and <span class="math-container">$b$</span>. The output <span class="math-container">$a$</span> is the the neuron's activation value, that may be taken as output of the neural network, or fed in to some other neuron as one of the next neuron's <span class="math-container">$x_i$</span>.</p>
  
  <p>A simple feed-forward network using this basic neuron function, with at least one hidden layer which has a non-linear activation function can already learn approximations to any given function - a result proved in the <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">universal approximation theorem</a>. </p>
  
  <p>The practical result of the universal approximation theorem is that you need motivation other than increasing diversity in order to make neural network functions more complex. If you were considering altering one of the <span class="math-container">$w_i x_i$</span> multiplications, and replacing with a more complex learnable function, you can effectively achieve the same thing by adding another neuron whose output <span class="math-container">$a$</span> is used as <span class="math-container">$x_i$</span> - or simply adding a layer in most neural network libraries.</p>
  
  <p>In some situations there may be good reasons to make lower-level changes:</p>
  
  <ul>
  <li><p>If you know the function you are learning relates to a theoretical model with a specific mathematical form, you can deliberately set up functions that mirror that with learnable parameters. Typcially that is done as transforms on inputs, but could also be part of a more complex computational graph if necessary.</p></li>
  <li><p>In neural network architectures, you can consider things such as gate combinations in <a href="https://en.wikipedia.org/wiki/Long_short-term_memory" rel="nofollow noreferrer">LSTM cells</a>, or <a href="https://en.wikipedia.org/wiki/Residual_neural_network" rel="nofollow noreferrer">skip connections in residual networks</a> as examples where the functions have deliberately be made more complex to achieve a specific goal - in both those cases in order to increase effectiveness of backpropagation in deep structures.</p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-23T10:01:56.240Z
Answers.LastActivityDate: 2020-05-23T10:01:56.240Z
Answers.UserDisplayName: Neil Slater
Body: <p>In a neural network, a neuron typically computes a linear function <span class="math-container">$f(x) = w*x$</span>, where <span class="math-container">$w$</span> is the weight and <span class="math-container">$x$</span> is the input.</p>
  
  <p>Why not replacing the linear function with more complex functions, such as <span class="math-container">$f(x,w,a,b,c) = w*(x + b)^a + c$</span>? </p>
  
  <p>It will provide much more diversity into neural networks. </p>
  
  <p>Does this have a name? Has this been used?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-23T11:02:23.273Z
Comments.Text: In your previous version of this post, you were using the word "weight" as if it was a function, but it's not. The weight is a number. The neuron computes some function. I think it's important you differentiate the two! So, it's the neuron that can compute more complex functions and "not the weight that can compute more complex functions".
Comments.UserDisplayName: nbro
Tags: <neural-networks><activation-functions>
Title: Why not replacing the simple linear functions that neurons compute with more complex functions?
UserDisplayName: Henry
ThreadText: # Why not replacing the simple linear functions that neurons compute with more complex functions?
  
  (Asked by: username_0 on 2020-05-23)
  
  In a neural network, a neuron typically computes a linear function $f(x) = w\*x$, where $w$ is the weight and $x$ is the input.
  
  Why not replacing the linear function with more complex functions, such as $f(x,w,a,b,c) = w\*(x + b)^a + c$?
  
  It will provide much more diversity into neural networks.
  
  Does this have a name? Has this been used?
  
  **Tags:** neural-networks, activation-functions
  > **Comments:**
  > **username_1**: In your previous version of this post, you were using the word "weight" as if it was a function, but it's not. The weight is a number. The neuron computes some function. I think it's important you differentiate the two! So, it's the neuron that can compute more complex functions and "not the weight that can compute more complex functions".  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-05-23. Score: 4 | Accepted answer
  
  It is definitley possible to make the links between neurons use more complex functions. Provided those functions are differentiable, backpropagation still works, and the resulting compound function might be able to learn something useful. The general name for such a thing is a [computational graph](https://www.deepideas.net/deep-learning-from-scratch-i-computational-graphs/) and the standardised structures used in most neural networks are a subset of all possible (and maybe useful) computational graphs.
  
  When adding complex and non-linear functions into a neural network, this is usually alternated with simpler linear layers using the weights. A generalised function of a single neuron as used in most neural networks looks like this:
  
  $$a = f(\sum\_i w\_i x\_i + b)$$
  
  Where $i$ indexes all inputs to the neuron, $x\_i$ are the input values, $w\_i$ the weights associated with each input, $b$ is a bias term and $f()$ is a differentiable non-linear *activation function*. Training process learns $w\_i$ and $b$. The output $a$ is the the neuron's activation value, that may be taken as output of the neural network, or fed in to some other neuron as one of the next neuron's $x\_i$.
  
  A simple feed-forward network using this basic neuron function, with at least one hidden layer which has a non-linear activation function can already learn approximations to any given function - a result proved in the [universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem).
  
  The practical result of the universal approximation theorem is that you need motivation other than increasing diversity in order to make neural network functions more complex. If you were considering altering one of the $w\_i x\_i$ multiplications, and replacing with a more complex learnable function, you can effectively achieve the same thing by adding another neuron whose output $a$ is used as $x\_i$ - or simply adding a layer in most neural network libraries.
  
  In some situations there may be good reasons to make lower-level changes:
  
  * If you know the function you are learning relates to a theoretical model with a specific mathematical form, you can deliberately set up functions that mirror that with learnable parameters. Typcially that is done as transforms on inputs, but could also be part of a more complex computational graph if necessary.
  * In neural network architectures, you can consider things such as gate combinations in [LSTM cells](https://en.wikipedia.org/wiki/Long_short-term_memory), or [skip connections in residual networks](https://en.wikipedia.org/wiki/Residual_neural_network) as examples where the functions have deliberately be made more complex to achieve a specific goal - in both those cases in order to increase effectiveness of backpropagation in deep structures.
  
  
  ---
  
  