Answers.Body: <p>In a neural network, each layer is being applied separately. in my notation I use lower case showing vectors and captial cases showing matrices
  eg <span class="math-container">$x_1$</span> is the input (arbitrary size) y1 is the output of the first layer, W1 is the matrix of weights and f1 is the non-linearity such as sigmoid, which is applied elementwise.</p>
  <p><span class="math-container">$y1 = f1( W_1.x_1 + b1)$</span></p>
  <p>in the second layer, the output of the previous layer will be fed in (e.g y1). then we have again the similar operations (possibly with different dimesions of vectors and matrice W). here f2 can be another elementwise function such as relu.</p>
  <p><span class="math-container">$y2 = f2( W_2.y_1 + b2)$</span></p>
  <p>I hope my explanation makes it clear. otherwise feel free to ask follow up questions</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T14:22:11.263Z
Answers.LastActivityDate: 2024-07-09T14:22:11.263Z
Answers.UserDisplayName: saleh
Answers.Body: <p>First of all, it looks like you are under impression that a neural network is structured like this (example for 4 inputs and outputs):</p>
  <p><span class="math-container">$$ \begin{array}{rcl} 
  y_1 &amp; = &amp; \text{sigmoid}(w_{11}x_1 + w_{12}x_2+w_{13}x_3+w_{14}x_4+b_1)\\
  y_2 &amp; = &amp; \text{sigmoid}(w_{21}x_1 + w_{22}x_2+w_{23}x_3+w_{24}x_4+b_2)\\
  y_3 &amp; = &amp; \text{softmax}(w_{31}x_1 + w_{32}x_2+w_{33}x_3+w_{34}x_4+b_3)\\
  y_4 &amp; = &amp; \text{softmax}(w_{41}x_1 + w_{42}x_2+w_{43}x_3+w_{44}x_4+b_4)\\
  \end{array}
  $$</span></p>
  <p>If that is how you understand it, then you've missed the point of &quot;deep&quot; neural networks. (Sequential) deep neural network are structured in several layers:</p>
  <p><span class="math-container">$$
   l_i^1 = \sigma^1(\sum_jw_{ij}^1x_j+b^1_i) \\
   l_i^2 = \sigma^2(\sum_jw_{ij}^2l^1_j+b^2_i) \\
   \ddots\\
   l_i^\alpha = \sigma^\alpha(\sum_jw_{ij}^\alpha l^{\alpha-1}_j+b^\alpha_i) \\
   y_i = \sigma^{out}(\sum_jw_{ij}^{out} l^\alpha_j+b^{out}_i) \\
  $$</span></p>
  <p>The activation function <span class="math-container">$\sigma^l$</span> of a single layer stays the same.</p>
  <blockquote>
  <p>To me, it's like saying a function is approximated using sine functions with N different k values and then also randomly a few Hermite polynomials are thrown in as well.</p>
  </blockquote>
  <p>To follow this analogy - you are mistaken by thinking that we are <em>simultaneously</em> trying to decompose over <span class="math-container">$\sin$</span> functions and hermitian polynomials. What actually happens is that we <em>first</em> decompose over <span class="math-container">$\sin$</span> functions and <em>then</em> decompose over hermitian polynomials. This actually can makes sense from practical point of view.</p>
  <blockquote>
  <p>Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).</p>
  </blockquote>
  <p>They are not, but sigmoids (or ReLUs) are not orthogonal to each other either. Orthogonality has noting to do with any of it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-15T18:16:23.023Z
Answers.Text: I agree about the linear combination of affine functions thrown in an activation function is what constitutes a neuron at layer $l$. Are you saying that we form something of a "separate" function approximation at each layer, using a different or not different activation function? What otherwise do you mean by "we *first* decomponse over $sin$ functions and *then* decompose over hermitian polynomials? I read that as, "we could decompose with $sin$ for one layer, then decompose with hermitians the next". If that's true, why do we do that? It sounds redundant. Surely I still have it wrong.
Answers.UserDisplayName: sangstar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-15T18:21:35.853Z
Answers.Text: From my understanding, neural networks ultimately aim to approximate a function. You can approximate any function with a combination of certain functions like the hermitian polynomials we mentioned. Each neuron can constitute a different sigmoid, and all of those sigmoids in a layer are used to calculate the activation function in the next layer for each neuron. To be honest, I come from a fairly not-super-high-level in applied mathematics, and I seem to get a sense of a lot of overcomplexity in function approximation with all these layers, but I know this is incredibly naive to say.
Answers.UserDisplayName: sangstar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-15T18:24:45.100Z
Answers.Text: My background is basically saying that you can approximate $y = mx + b$ perfectly well with a sum of cosines and sines, or hermite polynomials, or what have you (if they form an orthogonal basis / have an inner product of 0), but in a neural network, by analogy, it sounds like the function $y = mx + b$ is first decomposed (for example -- not actually in practice) with sines and cosines, and *then* with hermitian polynomials. Why do we.. decompose again? What good does it do us? I know this is a painfully fundamental question, so I want to be clear my misunderstandings are made clear.
Answers.UserDisplayName: sangstar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-15T17:28:55.717Z
Answers.LastActivityDate: 2021-04-15T17:28:55.717Z
Answers.UserDisplayName: Kostya
Body: <p>From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and weights as their constants, like Fourier sums and other orthogonal basis functions.</p>
  <p>How, then, can one neural network layer use activation function, like a sigmoid, and another one like the output using softmax? How do we know a linear combination of sigmoids and <em>something else</em> can still build that function no matter what? To me, it's like saying a function is approximated using sine functions with <span class="math-container">$N$</span> different <span class="math-container">$k$</span> values and then also randomly a few Hermite polynomials are thrown in as well. In this case, Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).</p>
  <p>This question highlights some misconceptions I have about activation functions, perhaps, and I'd like to know where I'm going wrong here.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-15T14:11:19.770Z
Comments.Text: have a look at this question [Neural Network composed of multiple activation functions](https://stackoverflow.com/questions/37947558/neural-network-composed-of-multiple-activation-functions#:~:text=2%20Answers&text=A%20neural%20network%20is%20just,for%20solving%20a%20specific%20function.) ... this might help
Comments.UserDisplayName: Faizy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-20T03:05:38.693Z
Comments.Text: This approximation of $x^3 + x^2 - x - 1$ might help you understand a bit better: https://www.desmos.com/calculator/cfvtjusqmq. Play around with the added functions, and try combining more than one activation function and you'll find you'll still be able to approximate the function without issue. This is what a neural network is doing, just instead of 2 dimensional space it's $n$-dimensional
Comments.UserDisplayName: Recessive
Tags: <math><activation-functions><function-approximation><universal-approximation-theorems>
Title: Why can a neural network use more than one activation function?
UserDisplayName: sangstar
ThreadText: # Why can a neural network use more than one activation function?
  
  (Asked by: username_0 on 2021-04-15)
  
  From trying to understand neural networks better, I've come upon a tentative notion that an activation function aims to build a function it's approximating via linear combinations with biases and weights as their constants, like Fourier sums and other orthogonal basis functions.
  
  How, then, can one neural network layer use activation function, like a sigmoid, and another one like the output using softmax? How do we know a linear combination of sigmoids and *something else* can still build that function no matter what? To me, it's like saying a function is approximated using sine functions with $N$ different $k$ values and then also randomly a few Hermite polynomials are thrown in as well. In this case, Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).
  
  This question highlights some misconceptions I have about activation functions, perhaps, and I'd like to know where I'm going wrong here.
  
  **Tags:** math, activation-functions, function-approximation, universal-approximation-theorems
  
  > **Comments:**
  > **username_1**: have a look at this question [Neural Network composed of multiple activation functions](https://stackoverflow.com/questions/37947558/neural-network-composed-of-multiple-activation-functions#:~:text=2%20Answers&text=A%20neural%20network%20is%20just,for%20solving%20a%20specific%20function.) ... this might help  
  > **username_2**: This approximation of $x^3 + x^2 - x - 1$ might help you understand a bit better: https://www.desmos.com/calculator/cfvtjusqmq. Play around with the added functions, and try combining more than one activation function and you'll find you'll still be able to approximate the function without issue. This is what a neural network is doing, just instead of 2 dimensional space it's $n$-dimensional  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-07-09. Score: 0
  
  In a neural network, each layer is being applied separately. in my notation I use lower case showing vectors and captial cases showing matrices
  eg $x\_1$ is the input (arbitrary size) y1 is the output of the first layer, W1 is the matrix of weights and f1 is the non-linearity such as sigmoid, which is applied elementwise.
  
  $y1 = f1( W\_1.x\_1 + b1)$
  
  in the second layer, the output of the previous layer will be fed in (e.g y1). then we have again the similar operations (possibly with different dimesions of vectors and matrice W). here f2 can be another elementwise function such as relu.
  
  $y2 = f2( W\_2.y\_1 + b2)$
  
  I hope my explanation makes it clear. otherwise feel free to ask follow up questions
  
  
  ---
  
  ### Answer by username_4 on 2021-04-15. Score: 0
  
  First of all, it looks like you are under impression that a neural network is structured like this (example for 4 inputs and outputs):
  
  $$ \begin{array}{rcl}
  y\_1 & = & \text{sigmoid}(w\_{11}x\_1 + w\_{12}x\_2+w\_{13}x\_3+w\_{14}x\_4+b\_1)\\
  y\_2 & = & \text{sigmoid}(w\_{21}x\_1 + w\_{22}x\_2+w\_{23}x\_3+w\_{24}x\_4+b\_2)\\
  y\_3 & = & \text{softmax}(w\_{31}x\_1 + w\_{32}x\_2+w\_{33}x\_3+w\_{34}x\_4+b\_3)\\
  y\_4 & = & \text{softmax}(w\_{41}x\_1 + w\_{42}x\_2+w\_{43}x\_3+w\_{44}x\_4+b\_4)\\
  \end{array}
  $$
  
  If that is how you understand it, then you've missed the point of "deep" neural networks. (Sequential) deep neural network are structured in several layers:
  
  $$
  l\_i^1 = \sigma^1(\sum\_jw\_{ij}^1x\_j+b^1\_i) \\
  l\_i^2 = \sigma^2(\sum\_jw\_{ij}^2l^1\_j+b^2\_i) \\
  \ddots\\
  l\_i^\alpha = \sigma^\alpha(\sum\_jw\_{ij}^\alpha l^{\alpha-1}\_j+b^\alpha\_i) \\
  y\_i = \sigma^{out}(\sum\_jw\_{ij}^{out} l^\alpha\_j+b^{out}\_i) \\
  $$
  
  The activation function $\sigma^l$ of a single layer stays the same.
  
  > To me, it's like saying a function is approximated using sine functions with N different k values and then also randomly a few Hermite polynomials are thrown in as well.
  
  To follow this analogy - you are mistaken by thinking that we are *simultaneously* trying to decompose over $\sin$ functions and hermitian polynomials. What actually happens is that we *first* decompose over $\sin$ functions and *then* decompose over hermitian polynomials. This actually can makes sense from practical point of view.
  
  > Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).
  
  They are not, but sigmoids (or ReLUs) are not orthogonal to each other either. Orthogonality has noting to do with any of it.
  
  > **Comments:**
  > **username_0**: I agree about the linear combination of affine functions thrown in an activation function is what constitutes a neuron at layer $l$. Are you saying that we form something of a "separate" function approximation at each layer, using a different or not different activation function? What otherwise do you mean by "we *first* decomponse over $sin$ functions and *then* decompose over hermitian polynomials? I read that as, "we could decompose with $sin$ for one layer, then decompose with hermitians the next". If that's true, why do we do that? It sounds redundant. Surely I still have it wrong.  
  > **username_0**: From my understanding, neural networks ultimately aim to approximate a function. You can approximate any function with a combination of certain functions like the hermitian polynomials we mentioned. Each neuron can constitute a different sigmoid, and all of those sigmoids in a layer are used to calculate the activation function in the next layer for each neuron. To be honest, I come from a fairly not-super-high-level in applied mathematics, and I seem to get a sense of a lot of overcomplexity in function approximation with all these layers, but I know this is incredibly naive to say.  
  > **username_0**: My background is basically saying that you can approximate $y = mx + b$ perfectly well with a sum of cosines and sines, or hermite polynomials, or what have you (if they form an orthogonal basis / have an inner product of 0), but in a neural network, by analogy, it sounds like the function $y = mx + b$ is first decomposed (for example -- not actually in practice) with sines and cosines, and *then* with hermitian polynomials. Why do we.. decompose again? What good does it do us? I know this is a painfully fundamental question, so I want to be clear my misunderstandings are made clear.  
  
  ---
  
  