Answers.Body: <p>Two things are happening here:</p>
  <ul>
  <li><p>As you increase the number of <span class="math-container">$sin$</span> cycles that the neural network needs to approximate, the problem becomes harder for it to learn. The network - being 5 layers with 64 neurons each - actually has plenty of capacity to learn up to the approx 10 cycles you are trying to teach it. However, training it becomes progressively harder.</p>
  </li>
  <li><p>scikit-learn's <code>MLPRegressor</code> has a lot of hyperparameters, which are mostly set with sensible defaults, but it looks like* it is not set to try hard when training begins to stall and become more stochastic. The main issue with that is how quickly it will give up when cost function is not improving epoch-by-epoch, and it is common for measurable improvements to take multiple epochs, especially with a small data set (you only have a few hundred samples) when progress epoch-to-epoch can become quite noisy.</p>
  </li>
  </ul>
  <p>I managed to fix this and make training more robust by . . .</p>
  <p><strong>Increasing training set size slightly</strong></p>
  <pre><code>  x = np.arange(0, sineLength, 0.05)
  </code></pre>
  <p>You could probably do a lot more here, the larger the set, the less noisy each improvement per epoch will be. But I wanted to use other hyperparameters to show what you might need to do if training is tougher and you cannot afford to easily increase the training set size.</p>
  <p><strong>Adding hyperparameters that made the fit routine try harder</strong></p>
  <pre><code>    reg = MLPRegressor(hidden_layer_sizes=tuple([64]*5), activation=&quot;relu&quot;, 
        random_state=1, max_iter=2000, early_stopping=True, 
        validation_fraction=0.2, n_iter_no_change=200, 
        tol=1e-5).fit(X_trainscaled, y_train)
  </code></pre>
  <p>Notes:</p>
  <ul>
  <li>I added using validation for early stopping, to avoid over-fitting. You should always use a validation set (in any non-toy exercise), although you could use other regularisation approaches. I did not check whether any of your bad scores were due to overfitting, but I don't think it was likely. This consumes more of the training data, which is another reason why I increased the size of the training set.</li>
  <li>I increased <code>n_iter_no_change</code> significantly, to overcome the noise in the improvements per epoch. Small data sets may need many epochs in order to reveal the real trend in the training process.</li>
  <li>I decreased <code>tol</code>, to allow for almost any improvement to be considered worthwhile.</li>
  <li>I reduced <code>max_iter</code> to prevent training taking too long on this toy problem, and because I probably over-compensated with <code>n_iter_no_change</code> and <code>tol</code> changes. That meant I did get quite a few warnings of <code>Maximum iterations (2000) reached and the optimization hasn't converged yet</code> but the scores were nearly always very good after 2000 epochs, so I did not feel any need to increase that. Probably your original code was giving up after 100 or even less epochs because loss was not improving, and that was what gave you the bad r2 scores.</li>
  </ul>
  <p>The above changes resulted in this graph for me:
  <a href="https://i.sstatic.net/PqmDp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PqmDp.png" alt="Graph of r2score against input range of function" /></a></p>
  <p>. . . almost all r2score values were above 0.99, just one scored badly at 0.22, likely due to random chance. I could probably tune the hyperparameters further to make it perfect, or maybe double the size of the training set, but I will leave that as an exercise for you.</p>
  <hr />
  <p>* This is the first time I have used <code>MLPRegressor</code>, hence &quot;looks like&quot;. I am not certain what all the consequences are for how they have set defaults.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-02T23:48:24.430Z
Answers.LastActivityDate: 2022-03-02T23:48:24.430Z
Answers.UserDisplayName: Neil Slater
Body: <h2>TL;DR;</h2>
  <p>How to build a neural network that properly approximates the sine function with different ranges?</p>
  <h2>Context and Question:</h2>
  <p>From <a href="https://stackoverflow.com/questions/13897316/approximating-the-sine-function-with-a-neural-network">this question</a> I decided to use the <a href="https://stackoverflow.com/a/67386162/3210864">Sergey's answer</a>, however I used a different range of values for <span class="math-container">$x$</span> which yielded a very poor estimation of the sine function. I noticed that for large values of the <span class="math-container">$x$</span> range it will provide incorrect results.</p>
  <p>I plotted the r<span class="math-container">$^2$</span>-score for different values of the <span class="math-container">$x$</span> range obtaining the following:
  <a href="https://i.sstatic.net/xhMsh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xhMsh.png" alt="R2 Scores" /></a></p>
  <p>Am I missing someting? Why is this happening?, how can one build a neural network for approximating the sine function?</p>
  <h2>The Code:</h2>
  <p>Here is the code I used to create the plot in case you want to reproduce it</p>
  <pre><code>from sklearn.neural_network import MLPRegressor
  from sklearn.model_selection import train_test_split
  from sklearn.preprocessing import StandardScaler
  from sklearn.metrics import r2_score
  import numpy as np
  import matplotlib.pyplot as plt
  
  sineLengths = []
  scores = []
  
  for sineLength in np.arange(40, 60, 0.1):
      sineLengths.append(sineLength)
      X = []
      x = np.arange(0, sineLength, 0.1)
      y = np.sin(x)
  
      for i in range(len(x)):
          X.append([x[i]])
  
      X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=1, test_size=0.2)
      sc_X = StandardScaler()
      X_trainscaled=sc_X.fit_transform(X_train)
      X_testscaled=sc_X.transform(X_test)
  
      reg = MLPRegressor(hidden_layer_sizes=tuple([64]*5), activation=&quot;relu&quot; ,random_state=1, max_iter=20000).fit(X_trainscaled, y_train)
      y_pred=reg.predict(X_testscaled)
      r2score = r2_score(y_pred, y_test)
      print(&quot;The Score with &quot;, r2score)
      scores.append(r2score)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-02T18:20:03.937Z
Comments.Text: If I am reading this correctly, your feature for `X` is five different x input values in a strict sequence. Why? It would mean you are not approximating $sin(x)$ but some other related function $f(x,y,z,a,b)$ but constraining the valid inputs to $f$ whilst expecting $f(\cdot,\cdot,\cdot,\cdot,b) = sin(b)$
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-02T22:01:19.670Z
Comments.Text: @NeilSlater, Thanks for your comment. I'm working in predicting a signal value taking into account the immediate previous historic values, I figured out that I could expect $f(x,y,z,a,b) = f(x) + f(y) + f(z) + f(a) + f(b)$, then, in this specific example $f(x) = f(y) = f(z) = f(a)=0$ and $f(b) = sin(b)$. However I took your comment into account and updated the code in the post as well as the scores and the range. I also scaled the $X$ with `fit_transform` and `transform`, and obtained the samples with the same step across ranges. It still doesn't work. Am I forgetting something?
Comments.UserDisplayName: Hans
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-02T22:03:05.687Z
Comments.Text: @NeilSlater, I also updated the plot, btw :)
Comments.UserDisplayName: Hans
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-02T22:18:36.617Z
Comments.Text: In the previous code, you were not providing the 5 previous values of $y$ as a sequence to predict the next value of $y$, but the 4 previous values of $x$ plus the current value of $x$, which is very different from predicting the continuation of a sequence. Either way, thanks for making the edit and including code that matches the question better.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-02T22:38:04.533Z
Comments.Text: @NeilSlater, haha, you're right, my bad. Thx for pointing that out, and I agree that now the edition reflects the question more clearly
Comments.UserDisplayName: Hans
Tags: <neural-networks><prediction><regression><multilayer-perceptrons><scikit-learn>
Title: How to make a proper approximation of Sine function with Neural Networks?
UserDisplayName: Hans
ThreadText: # How to make a proper approximation of Sine function with Neural Networks?
  
  (Asked by: username_0 on 2022-03-02)
  
  ## TL;DR;
  
  How to build a neural network that properly approximates the sine function with different ranges?
  
  ## Context and Question:
  
  From [this question](https://stackoverflow.com/questions/13897316/approximating-the-sine-function-with-a-neural-network) I decided to use the [Sergey's answer](https://stackoverflow.com/a/67386162/3210864), however I used a different range of values for $x$ which yielded a very poor estimation of the sine function. I noticed that for large values of the $x$ range it will provide incorrect results.
  
  I plotted the r$^2$-score for different values of the $x$ range obtaining the following:
  [![R2 Scores](https://i.sstatic.net/xhMsh.png)](https://i.sstatic.net/xhMsh.png)
  
  Am I missing someting? Why is this happening?, how can one build a neural network for approximating the sine function?
  
  ## The Code:
  
  Here is the code I used to create the plot in case you want to reproduce it
  
  ```
  from sklearn.neural_network import MLPRegressor
  from sklearn.model_selection import train_test_split
  from sklearn.preprocessing import StandardScaler
  from sklearn.metrics import r2_score
  import numpy as np
  import matplotlib.pyplot as plt
  
  sineLengths = []
  scores = []
  
  for sineLength in np.arange(40, 60, 0.1):
      sineLengths.append(sineLength)
      X = []
      x = np.arange(0, sineLength, 0.1)
      y = np.sin(x)
  
      for i in range(len(x)):
          X.append([x[i]])
  
      X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=1, test_size=0.2)
      sc_X = StandardScaler()
      X_trainscaled=sc_X.fit_transform(X_train)
      X_testscaled=sc_X.transform(X_test)
  
      reg = MLPRegressor(hidden_layer_sizes=tuple([64]*5), activation="relu" ,random_state=1, max_iter=20000).fit(X_trainscaled, y_train)
      y_pred=reg.predict(X_testscaled)
      r2score = r2_score(y_pred, y_test)
      print("The Score with ", r2score)
      scores.append(r2score)
  
  ```
  
  **Tags:** neural-networks, prediction, regression, multilayer-perceptrons, scikit-learn
  
  > **Comments:**
  > **username_1**: If I am reading this correctly, your feature for `X` is five different x input values in a strict sequence. Why? It would mean you are not approximating $sin(x)$ but some other related function $f(x,y,z,a,b)$ but constraining the valid inputs to $f$ whilst expecting $f(\cdot,\cdot,\cdot,\cdot,b) = sin(b)$  
  > **username_0**: @username_1, Thanks for your comment. I'm working in predicting a signal value taking into account the immediate previous historic values, I figured out that I could expect $f(x,y,z,a,b) = f(x) + f(y) + f(z) + f(a) + f(b)$, then, in this specific example $f(x) = f(y) = f(z) = f(a)=0$ and $f(b) = sin(b)$. However I took your comment into account and updated the code in the post as well as the scores and the range. I also scaled the $X$ with `fit_transform` and `transform`, and obtained the samples with the same step across ranges. It still doesn't work. Am I forgetting something?  
  > **username_0**: @username_1, I also updated the plot, btw :)  
  > **username_1**: In the previous code, you were not providing the 5 previous values of $y$ as a sequence to predict the next value of $y$, but the 4 previous values of $x$ plus the current value of $x$, which is very different from predicting the continuation of a sequence. Either way, thanks for making the edit and including code that matches the question better.  
  > **username_0**: @username_1, haha, you're right, my bad. Thx for pointing that out, and I agree that now the edition reflects the question more clearly  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-02. Score: 1 | Accepted answer
  
  Two things are happening here:
  
  * As you increase the number of $sin$ cycles that the neural network needs to approximate, the problem becomes harder for it to learn. The network - being 5 layers with 64 neurons each - actually has plenty of capacity to learn up to the approx 10 cycles you are trying to teach it. However, training it becomes progressively harder.
  * scikit-learn's `MLPRegressor` has a lot of hyperparameters, which are mostly set with sensible defaults, but it looks like\* it is not set to try hard when training begins to stall and become more stochastic. The main issue with that is how quickly it will give up when cost function is not improving epoch-by-epoch, and it is common for measurable improvements to take multiple epochs, especially with a small data set (you only have a few hundred samples) when progress epoch-to-epoch can become quite noisy.
  
  I managed to fix this and make training more robust by . . .
  
  **Increasing training set size slightly**
  
  ```
    x = np.arange(0, sineLength, 0.05)
  
  ```
  
  You could probably do a lot more here, the larger the set, the less noisy each improvement per epoch will be. But I wanted to use other hyperparameters to show what you might need to do if training is tougher and you cannot afford to easily increase the training set size.
  
  **Adding hyperparameters that made the fit routine try harder**
  
  ```
      reg = MLPRegressor(hidden_layer_sizes=tuple([64]*5), activation="relu", 
        random_state=1, max_iter=2000, early_stopping=True, 
        validation_fraction=0.2, n_iter_no_change=200, 
        tol=1e-5).fit(X_trainscaled, y_train)
  
  ```
  
  Notes:
  
  * I added using validation for early stopping, to avoid over-fitting. You should always use a validation set (in any non-toy exercise), although you could use other regularisation approaches. I did not check whether any of your bad scores were due to overfitting, but I don't think it was likely. This consumes more of the training data, which is another reason why I increased the size of the training set.
  * I increased `n_iter_no_change` significantly, to overcome the noise in the improvements per epoch. Small data sets may need many epochs in order to reveal the real trend in the training process.
  * I decreased `tol`, to allow for almost any improvement to be considered worthwhile.
  * I reduced `max_iter` to prevent training taking too long on this toy problem, and because I probably over-compensated with `n_iter_no_change` and `tol` changes. That meant I did get quite a few warnings of `Maximum iterations (2000) reached and the optimization hasn't converged yet` but the scores were nearly always very good after 2000 epochs, so I did not feel any need to increase that. Probably your original code was giving up after 100 or even less epochs because loss was not improving, and that was what gave you the bad r2 scores.
  
  The above changes resulted in this graph for me:
  [![Graph of r2score against input range of function](https://i.sstatic.net/PqmDp.png)](https://i.sstatic.net/PqmDp.png)
  
  . . . almost all r2score values were above 0.99, just one scored badly at 0.22, likely due to random chance. I could probably tune the hyperparameters further to make it perfect, or maybe double the size of the training set, but I will leave that as an exercise for you.
  
  ---
  
  \* This is the first time I have used `MLPRegressor`, hence "looks like". I am not certain what all the consequences are for how they have set defaults.
  
  
  ---
  
  