Answers.Body: <p>Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your <code>tanh</code> network had to start from zero).</p>
  <p>Generally I would say it's impossible to have a good approximation of sinus with just 3 neurons, but if you want to consider one period of sinus, then you can do something. for clarity look at this picture:</p>
  <p><a href="https://i.sstatic.net/nSAJK.png" rel="noreferrer"><img src="https://i.sstatic.net/nSAJK.png" alt="enter image description here" /></a></p>
  <p>I've write the code for this task in colab and you can find it <a href="https://colab.research.google.com/drive/1X4wdJ4v_uq2yD-ixjpLeErVPKC_RgYiI?usp=sharing" rel="noreferrer">here</a>, and you can play with it if you want.</p>
  <p>If you run the network several times you may get different results (because of different initializations) and you can see some of them at the <strong>Results</strong> section of the link above. What you showed us in the images above are just two possibilities. But it's interesting that you can get better results with <code>tanh</code> rather than <code>sigmoid</code> and if you want to know why, I highly recommend you to look at <a href="https://www.youtube.com/watch?v=wEoyxE0GP2M&amp;list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&amp;index=6" rel="noreferrer">this</a> lecture of CS231n. In summary it's because <code>tanh</code> has the negative part and the network can learn better with it.</p>
  <p>But actually their power of approximation are almost similar because <code>2*sigmoid(1.5*x) - 1</code> almost looks the same as <code>tanh(x)</code> and you can find it by looking the picture below:
  <a href="https://i.sstatic.net/iJ5Ex.png" rel="noreferrer"><img src="https://i.sstatic.net/iJ5Ex.png" alt="enter image description here" /></a></p>
  <p>So why you can't get the same results as <code>tanh</code>? that's because <code>tanh</code> suits the problem better and if the network wants to get the same result as <code>tanh</code> with <code>sigmoid</code> it should learn their transformation parameters and learning these parameters makes the learning task harder. So It's not impossible to get the same result with <code>sigmoid</code> but it's harder. And to show you that its possible, I have set the parameters of the network using <code>sigmoid</code> manually and got the result below (you can get better results if you have more time):
  <a href="https://i.sstatic.net/6njbk.png" rel="noreferrer"><img src="https://i.sstatic.net/6njbk.png" alt="enter image description here" /></a></p>
  <p>At last if you want to know why you can't get the same result with 2 neurons instead of 3 neurons, it's better to understand what does the network do with 3 neurons.<br />
  If you look at the output of the first layer, you may see something like this (which are outputs of two neurons it has):</p>
  <p><a href="https://i.sstatic.net/rcsNq.png" rel="noreferrer"><img src="https://i.sstatic.net/rcsNq.png" alt="enter image description here" /></a></p>
  <p>Then the next layer gets the difference between the output of these two neurons (which is like sinus) and applies <code>sigmoid</code> or <code>tanh</code> to it, and that's how you get a good result. But when you have just one neuron in the first layer, you can't imagine some scenario like this and approximating one period of sinus is out of it's ability (underfitting).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-23T09:06:44.383Z
Answers.Text: Thank you very much for your detailed answer. One question: How did you come up with the manually set weights for the sigmoid approximation of the sinus? Did you figure them out just by trial and error/ educated guesses or did you use any system?
Answers.UserDisplayName: JavAlex
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-23T09:38:38.253Z
Answers.Text: Happy if it helped. It was combination of both. The ratio of the weights was trial, but I knew what the network should do and what the output should look like in each layer (as I described in the answer). So it's not hard to find the weights. Feel free to uncomment the related parts in the colab link and play with the weights and biases to see their affect on output of each layer.
Answers.UserDisplayName: amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-20T06:34:46.397Z
Answers.LastActivityDate: 2020-10-20T06:34:46.397Z
Answers.UserDisplayName: amin
Body: <p>The main goal is: <em>Find the smallest possible neural network to approximate the <span class="math-container">$sin$</span> function.</em></p>
  <p>Moreover, I want to find a qualitative reason why this network is the smallest possible network.</p>
  <p>I have created 8000 random <span class="math-container">$x$</span> values with corresponding target values <span class="math-container">$sin(x)$</span>. The network, which am currently considering, consists of 1 input neuron, 3 neurons in two hidden layers, and 1 output neuron:</p>
  <p><strong>Network architecture:</strong></p>
  <p><a href="https://i.sstatic.net/MBElA.png" rel="noreferrer"><img src="https://i.sstatic.net/MBElA.png" alt="Network architecture" /></a></p>
  <p>The neural network can be written as function
  <span class="math-container">$$y = sig(w_3 \cdot sig(w_1 \cdot x) + w_4 \cdot sig(w_2 \cdot x)),$$</span>
  where <span class="math-container">$\text{sig}$</span> is the sigmoid activation function.</p>
  <p><strong><span class="math-container">$tanh$</span> activation function:</strong><br />
  When I use <span class="math-container">$tanh$</span> as an activation function, the network is able to hit the 2 extrema of the <span class="math-container">$sin$</span> function:</p>
  <p><a href="https://i.sstatic.net/tvpYv.png" rel="noreferrer"><img src="https://i.sstatic.net/tvpYv.png" alt="*<span class="math-container">$tanh$</span> activation function" /></a></p>
  <p><strong>Sigmoid activation function:</strong><br />
  However, when I use the sigmoid activation function <span class="math-container">$\text{sig}$</span>, only the first extremum is hit. The network output is not a periodic function but converges:</p>
  <p><a href="https://i.sstatic.net/wVnzq.png" rel="noreferrer"><img src="https://i.sstatic.net/wVnzq.png" alt="Sigmoid activation function" /></a></p>
  <p>My questions are now:</p>
  <ul>
  <li>Why does one get a better approximation with the <span class="math-container">$tanh$</span> activation function? What is a qualitative argument for that?</li>
  <li>Why does one need at least 3 hidden neurons? What is the reason that the approximation with <span class="math-container">$tanh$</span> does not work anymore, if one uses only 2 hidden neurons?</li>
  </ul>
  <p>I really appreciate all your ideas on this problem!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-19T01:18:28.873Z
Comments.Text: Is the smallest network that can "approximate" the sin function not just $f(x) = \sigma(w\cdot x + b)$ where $w = 1$, $b=0$, and $\sigma(\cdot) = \sin(\cdot)$?
Comments.UserDisplayName: Cameron Chandler
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-19T19:11:55.730Z
Comments.Text: thank you for your comment. But I am looking for the smallest network with a tanh or sigmoid activation function
Comments.UserDisplayName: JavAlex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-19T22:32:44.490Z
Comments.Text: Is x restricted to some interval ?
Comments.UserDisplayName: pasaba por aqui
Tags: <neural-networks><activation-functions><function-approximation><universal-approximation-theorems>
Title: Smallest possible network to approximate the $sin$ function
UserDisplayName: JavAlex
ThreadText: # Smallest possible network to approximate the $sin$ function
  
  (Asked by: username_0 on 2020-10-17)
  
  The main goal is: *Find the smallest possible neural network to approximate the $sin$ function.*
  
  Moreover, I want to find a qualitative reason why this network is the smallest possible network.
  
  I have created 8000 random $x$ values with corresponding target values $sin(x)$. The network, which am currently considering, consists of 1 input neuron, 3 neurons in two hidden layers, and 1 output neuron:
  
  **Network architecture:**
  
  [![Network architecture](https://i.sstatic.net/MBElA.png)](https://i.sstatic.net/MBElA.png)
  
  The neural network can be written as function
  $$y = sig(w\_3 \cdot sig(w\_1 \cdot x) + w\_4 \cdot sig(w\_2 \cdot x)),$$
  where $\text{sig}$ is the sigmoid activation function.
  
  **$tanh$ activation function:**  
  When I use $tanh$ as an activation function, the network is able to hit the 2 extrema of the $sin$ function:
  
  [![*<span class=](https://i.sstatic.net/tvpYv.png)$tanh$ activation function" />](https://i.sstatic.net/tvpYv.png)
  
  **Sigmoid activation function:**  
  However, when I use the sigmoid activation function $\text{sig}$, only the first extremum is hit. The network output is not a periodic function but converges:
  
  [![Sigmoid activation function](https://i.sstatic.net/wVnzq.png)](https://i.sstatic.net/wVnzq.png)
  
  My questions are now:
  
  * Why does one get a better approximation with the $tanh$ activation function? What is a qualitative argument for that?
  * Why does one need at least 3 hidden neurons? What is the reason that the approximation with $tanh$ does not work anymore, if one uses only 2 hidden neurons?
  
  I really appreciate all your ideas on this problem!
  
  **Tags:** neural-networks, activation-functions, function-approximation, universal-approximation-theorems
  > **Comments:**
  > **username_1**: Is the smallest network that can "approximate" the sin function not just $f(x) = \sigma(w\cdot x + b)$ where $w = 1$, $b=0$, and $\sigma(\cdot) = \sin(\cdot)$?  
  > **username_0**: thank you for your comment. But I am looking for the smallest network with a tanh or sigmoid activation function  
  > **username_2**: Is x restricted to some interval ?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-10-20. Score: 6 | Accepted answer
  
  Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your `tanh` network had to start from zero).
  
  Generally I would say it's impossible to have a good approximation of sinus with just 3 neurons, but if you want to consider one period of sinus, then you can do something. for clarity look at this picture:
  
  [![enter image description here](https://i.sstatic.net/nSAJK.png)](https://i.sstatic.net/nSAJK.png)
  
  I've write the code for this task in colab and you can find it [here](https://colab.research.google.com/drive/1X4wdJ4v_uq2yD-ixjpLeErVPKC_RgYiI?usp=sharing), and you can play with it if you want.
  
  If you run the network several times you may get different results (because of different initializations) and you can see some of them at the **Results** section of the link above. What you showed us in the images above are just two possibilities. But it's interesting that you can get better results with `tanh` rather than `sigmoid` and if you want to know why, I highly recommend you to look at [this](https://www.youtube.com/watch?v=wEoyxE0GP2M&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&index=6) lecture of CS231n. In summary it's because `tanh` has the negative part and the network can learn better with it.
  
  But actually their power of approximation are almost similar because `2*sigmoid(1.5*x) - 1` almost looks the same as `tanh(x)` and you can find it by looking the picture below:
  [![enter image description here](https://i.sstatic.net/iJ5Ex.png)](https://i.sstatic.net/iJ5Ex.png)
  
  So why you can't get the same results as `tanh`? that's because `tanh` suits the problem better and if the network wants to get the same result as `tanh` with `sigmoid` it should learn their transformation parameters and learning these parameters makes the learning task harder. So It's not impossible to get the same result with `sigmoid` but it's harder. And to show you that its possible, I have set the parameters of the network using `sigmoid` manually and got the result below (you can get better results if you have more time):
  [![enter image description here](https://i.sstatic.net/6njbk.png)](https://i.sstatic.net/6njbk.png)
  
  At last if you want to know why you can't get the same result with 2 neurons instead of 3 neurons, it's better to understand what does the network do with 3 neurons.  
  If you look at the output of the first layer, you may see something like this (which are outputs of two neurons it has):
  
  [![enter image description here](https://i.sstatic.net/rcsNq.png)](https://i.sstatic.net/rcsNq.png)
  
  Then the next layer gets the difference between the output of these two neurons (which is like sinus) and applies `sigmoid` or `tanh` to it, and that's how you get a good result. But when you have just one neuron in the first layer, you can't imagine some scenario like this and approximating one period of sinus is out of it's ability (underfitting).
  
  > **Comments:**
  > **username_0**: Thank you very much for your detailed answer. One question: How did you come up with the manually set weights for the sigmoid approximation of the sinus? Did you figure them out just by trial and error/ educated guesses or did you use any system?  
  > **username_3**: Happy if it helped. It was combination of both. The ratio of the weights was trial, but I knew what the network should do and what the output should look like in each layer (as I described in the answer). So it's not hard to find the weights. Feel free to uncomment the related parts in the colab link and play with the weights and biases to see their affect on output of each layer.  
  
  ---
  
  