Answers.Body: <p>Doing this naively will give a huge performance hit as it isn't as easily parallizable on the gpu.</p>
  <p>If you are wondering &quot;Kolmogorov–Arnold Networks&quot; do (something similar to) what you describe. <br>
  <a href="https://arxiv.org/pdf/2404.19756" rel="noreferrer">KAN: Kolmogorov–Arnold Networks</a></p>
  <blockquote>
  <p>While MLPs have fixed activation functions on nodes (“neurons”), KANs have learnable
  activation functions on edges (“weights”).</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-16T12:40:22.510Z
Answers.Text: Came here to write this answer. Another important difference is that KANs don't have linear weights at all -- the weights are replaced by spines.
Answers.UserDisplayName: Sycorax
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-15T14:42:19.063Z
Answers.LastActivityDate: 2024-10-15T14:42:19.063Z
Answers.UserDisplayName: hal9000
Answers.Body: <p>Because you still have to decide the set of activations to use, and ideally, you will have a set of <span class="math-container">$a,b,c,d,...$</span> for each neuron, therefore you have a lot of parameters to add, with probably marginal performance increase</p>
  <p>Furthermore, consider that a linear transformation (the weighted sum you are doing) composed with a affine transformation (the linear layer that would come after it) is still linear, so not sure that is necessary</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-15T12:09:20.880Z
Answers.LastActivityDate: 2024-10-15T12:09:20.880Z
Answers.UserDisplayName: Alberto
Body: <p>For each layer, we could define a vector of activation functions like:
  <span class="math-container">$$\begin{bmatrix} f(x) \\ f(x) \\ f(x) \\ \vdots \end{bmatrix}$$</span></p>
  <p>and then have that be applied on the neurons of each layer.
  If we define the activation function as something like:
  <span class="math-container">$$ f(x) = a.x + b.ReLU(x) + c.\sigma(x) + d.\tanh(x) + \cdots $$</span>
  In which the coefficients <span class="math-container">$a$</span>, <span class="math-container">$b$</span>, <span class="math-container">$c$</span>, <span class="math-container">$\cdots$</span> are adjustable parameters, then we can  calculate the derivative of the loss function with respect to each coefficient and use gradient descent on them to train the activation function for each neuron, shouldn't that make the network more flexible and maybe powerful?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-15T20:57:54.540Z
Comments.Text: A side-drawback of doing this is that you lose the advantages of a lot of the component functions. The output is no longer bounded like for $\sigma$ and tanh, and it's no longer as fast as the linear or RELU. (It's also redundant, since you can express $\sigma$ and tanh in terms of each other with the right coefficients and bias)
Comments.UserDisplayName: Ray
Tags: <neural-networks><activation-functions>
Title: Why aren't activation functions variable as well instead of being fixed?
UserDisplayName: hansoko
ThreadText: # Why aren't activation functions variable as well instead of being fixed?
  
  (Asked by: username_0 on 2024-10-15)
  
  For each layer, we could define a vector of activation functions like:
  $$\begin{bmatrix} f(x) \\ f(x) \\ f(x) \\ \vdots \end{bmatrix}$$
  
  and then have that be applied on the neurons of each layer.
  If we define the activation function as something like:
  $$ f(x) = a.x + b.ReLU(x) + c.\sigma(x) + d.\tanh(x) + \cdots $$
  In which the coefficients $a$, $b$, $c$, $\cdots$ are adjustable parameters, then we can calculate the derivative of the loss function with respect to each coefficient and use gradient descent on them to train the activation function for each neuron, shouldn't that make the network more flexible and maybe powerful?
  
  **Tags:** neural-networks, activation-functions
  
  > **Comments:**
  > **username_1**: A side-drawback of doing this is that you lose the advantages of a lot of the component functions. The output is no longer bounded like for $\sigma$ and tanh, and it's no longer as fast as the linear or RELU. (It's also redundant, since you can express $\sigma$ and tanh in terms of each other with the right coefficients and bias)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-10-15. Score: 5
  
  Doing this naively will give a huge performance hit as it isn't as easily parallizable on the gpu.
  
  If you are wondering "Kolmogorov–Arnold Networks" do (something similar to) what you describe.   
  [KAN: Kolmogorov–Arnold Networks](https://arxiv.org/pdf/2404.19756)
  
  > While MLPs have fixed activation functions on nodes (“neurons”), KANs have learnable
  > activation functions on edges (“weights”).
  
  > **Comments:**
  > **username_3**: Came here to write this answer. Another important difference is that KANs don't have linear weights at all -- the weights are replaced by spines.  
  
  ---
  
  ### Answer by username_4 on 2024-10-15. Score: 4
  
  Because you still have to decide the set of activations to use, and ideally, you will have a set of $a,b,c,d,...$ for each neuron, therefore you have a lot of parameters to add, with probably marginal performance increase
  
  Furthermore, consider that a linear transformation (the weighted sum you are doing) composed with a affine transformation (the linear layer that would come after it) is still linear, so not sure that is necessary
  
  
  ---
  
  