Answers.Body: <p>You're trying to learn a cubic function that explodes in values and your issue is <strong>scaling</strong>. I have been able to learn a better approximation by scaling data and using <strong>tanh</strong> as activation function.</p>
  <p>Code and result are as below:</p>
  <p><a href="https://i.sstatic.net/pHqhO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pHqhO.png" alt="enter image description here" /></a></p>
  <p>Convergence around X=100 happens because of tanh activation. Relu will not work better because of negative values that is the result of scaling. You can try playing with <strong>Leaky Relu</strong> activation and various <strong>alpha</strong> values.</p>
  <pre><code>import pandas as pd
  import numpy as np
  import matplotlib.pyplot as plt
  
  from tensorflow import keras
  from sklearn.preprocessing import StandardScaler
  from sklearn.model_selection import train_test_split
  
  def mass_of_sphere(R):
      return (4/3) * np.pi * (R**3)
  
  X = np.linspace(1, 120, 500000)
  y = [mass_of_sphere(x) for x in X]
  
  X = np.array(X).reshape(-1, 1)
  y = np.array(y)
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
  
  scaler_X = StandardScaler()
  X_train = scaler_X.fit_transform(X_train)
  X_test = scaler_X.transform(X_test)
  
  scaler_y = StandardScaler()
  y_train = scaler_y.fit_transform(y_train.reshape(-1, 1))
  y_test = scaler_y.transform(y_test.reshape(-1, 1)).reshape(-1)
  
  model = keras.Sequential([keras.layers.Dense(1),
                            keras.layers.Dense(128, activation = &quot;tanh&quot;),
                            keras.layers.Dense(1, activation = &quot;tanh&quot;)])
  
  early_stopping = keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 5)
  model.compile(optimizer = 'rmsprop', loss = 'mse')
  history = model.fit(X_train, y_train, epochs = 100, callbacks=[early_stopping], 
                      batch_size = 2048, validation_data=(X_test, y_test))
  
  y_hat = scaler_y.inverse_transform(model.predict(X_test)).reshape(-1)
  y_test = scaler_y.inverse_transform(y_test).reshape(-1)
  
  f, ax = plt.subplots(figsize = (12, 4))
  ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_test), color = 'blue', label = 'Real')
  ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_hat), color = 'orange', label = 'DL')
  ax.legend()
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-26T13:51:37.123Z
Answers.Text: Excellent. Rather surprising for me because I have read that relu should (almost) always work. For the moment I do not undersand the divergence with values bigger than 100. I will study your answer and try all your suggestions. I hope I shall be able to fit it beyond 100 :). I supposed that it was not neccesary to do any sort of "feature engineering", as is very frequently asserted. –
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-26T14:07:16.913Z
Answers.Text: For better understanding, inspect tanh figure, it maps $(-\infty, \infty)$ to $(-1, 1)$. Having clarified that, inspect the scaled value of the mass of $R = 100$. 
  
  scaler_y.transform(np.array([[mass_of_sphere(100)]]))
  
  You can simply bypass this limitation by either changing activation function or using a higher range of training values, going for perhaps 500 instead of 120.
  
  I just tested with values between (1, 500) and convergence occured at 400. Not an elegant solution but good to have.
Answers.UserDisplayName: meliksahturker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T12:36:43.980Z
Answers.Text: I've got a very good fit using "elu" as an activation function. But I was surprised because comparing results (inside the range used) obtained from mass_of_sphere() and model.predict I found enourmous differences. For the moment, I do not identify the error
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T20:24:50.483Z
Answers.Text: OK maybe a better approach: Scale your data using MinMaxScaler() to range(-1, 1) instead of StandardScaler(). Train your network with tanh activation in last layer again.
  Don't forget to scale back your prediction.
Answers.UserDisplayName: meliksahturker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-26T13:32:28.957Z
Answers.LastActivityDate: 2021-07-26T13:41:10.390Z
Answers.LastEditDate: 2021-07-26T13:41:10.390Z
Answers.UserDisplayName: meliksahturker
Answers.Body: <p>A simpler model seems to be the best option</p>
  <pre><code>model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])
  model.compile(optimizer='sgd', loss='mean_squared_error')
  xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)
  ys2 = np.array([4/3*r*np.pi**3 for r in xs])    
  model.fit(xs, ys2, epochs=500,validation_split=0.2)
  
  def masa_circulo(x):
      return 4/3*x*np.pi**3
  </code></pre>
  <p>Testing it graphically</p>
  <pre><code>x = [x for x in range(1,int(1e6),int(1e3))]
  y_masa_circulo = [ masa_circulo(m) for m in x]
  y_masa_predicha= [model.predict([m])[0] for m in x]
  
  import matplotlib.pyplot as plt
  fig,axes = plt.subplots(1,2)
  axes[0].plot(y_masa_circulo);
  axes[0].set_title(&quot;y_masa_circulo&quot;)
  axes[0].set_ylabel(&quot;y_masa_circulo&quot;)
  axes[0].set_xlabel(&quot;Radio&quot;)
  
  axes[1].plot(y_masa_predicha);
  plt.title(&quot;y_masa_circulo&quot;)
  plt.title(&quot;y_masa_predicha&quot;);
  axes[1].set_ylabel(&quot;y_masa_predicha&quot;);
  axes[1].set_xlabel(&quot;Radio&quot;);
  </code></pre>
  <p><a href="https://i.sstatic.net/3Scb1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3Scb1.png" alt="Graphical comparison function_results vs nn_results" /></a></p>
  <p>No need of scaling the data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-27T18:35:38.753Z
Answers.LastActivityDate: 2021-08-27T19:53:13.033Z
Answers.LastEditDate: 2021-08-27T19:53:13.033Z
Answers.UserDisplayName: Hermes Morales
Body: <p>I have been trying to adjust a neural network to a simple function: the mass of an sphere.
  I have tried with different architectures, for example, a single hidden layer and two hidden layers, always with 128 neurons each, and training them for 5000 epochs.
  The code is the usual one. Just in case, I publish one of them</p>
  <pre><code>model = keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])
                          ,keras.layers.Dense(128, activation=&quot;relu&quot;)
                          ,keras.layers.Dense(1, activation=&quot;relu&quot;)])
  model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])
  history = model.fit(x, y, validation_split=0.2, epochs=5000)
  </code></pre>
  <p>The results are shown in the graphs.</p>
  <p><a href="https://i.sstatic.net/q6Nad.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/q6Nad.png" alt="two hidden layers, 128 neurons each" /></a></p>
  <p><a href="https://i.sstatic.net/jYAuV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jYAuV.png" alt="one hidden layer, 128 neurons" /></a></p>
  <p>I suspect that I am making an error somewhere, because I have seen that deep learning is able to match complex functions with much less epochs. I shall appreciate any hint to fix this problem and obtain a good fit with the deep learning function.</p>
  <p>In order to make it clear I post the graph's code.</p>
  <pre><code>rs =[x for x in range(20)]
  def masas_circulo(x):
      masas_circulos =[]
      rs =[r for r in range(x)]
      for r in rs:
          masas_circulos.append(model.predict([r])[0][0])
  
     return masas_circulos
  
   masas_circulos = masas_circulo(20) 
   masas_circulos
   esferas = [4/3*np.pi*r**3 for r in range(20)]
   import matplotlib.pyplot as plt
   plt.plot(rs,masas_circulos,label=&quot;DL&quot;)
   plt.plot(rs,esferas,label=&quot;Real&quot;);
   plt.title(&quot;Mass of an sphere.\nDL (1hl,128 n,5000 e) vs ground_truth&quot;)
   plt.xlabel(&quot;Radius&quot;)
   plt.ylabel(&quot;Sphere&quot;)
   plt.legend();
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:06:11.833Z
Comments.Text: What are you trying to predict?: is it $V$ given $r$? for the formula $V=4/3\pi r^3$? if so, is it the output of your network, the single scalar, expected to be $V$? For me that seems like a plotting error: check how you are drawing the output of your network for different $r$
Comments.UserDisplayName: JVGD
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:15:51.567Z
Comments.Text: Ies, I am trying to predict V  given r for the formula  V=4/3πr^3. I have added the graphs' code.
Comments.UserDisplayName: Hermes Morales
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T14:21:02.377Z
Comments.Text: Sphere's density=1 
Comments.UserDisplayName: Hermes Morales
Tags: <deep-learning>
Title: Not able to find a good fit for a simple function with neural networks
UserDisplayName: Hermes Morales
ThreadText: # Not able to find a good fit for a simple function with neural networks
  
  (Asked by: username_0 on 2021-07-26)
  
  I have been trying to adjust a neural network to a simple function: the mass of an sphere.
  I have tried with different architectures, for example, a single hidden layer and two hidden layers, always with 128 neurons each, and training them for 5000 epochs.
  The code is the usual one. Just in case, I publish one of them
  
  ```
  model = keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])
                          ,keras.layers.Dense(128, activation="relu")
                          ,keras.layers.Dense(1, activation="relu")])
  model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])
  history = model.fit(x, y, validation_split=0.2, epochs=5000)
  
  ```
  
  The results are shown in the graphs.
  
  [![two hidden layers, 128 neurons each](https://i.sstatic.net/q6Nad.png)](https://i.sstatic.net/q6Nad.png)
  
  [![one hidden layer, 128 neurons](https://i.sstatic.net/jYAuV.png)](https://i.sstatic.net/jYAuV.png)
  
  I suspect that I am making an error somewhere, because I have seen that deep learning is able to match complex functions with much less epochs. I shall appreciate any hint to fix this problem and obtain a good fit with the deep learning function.
  
  In order to make it clear I post the graph's code.
  
  ```
  rs =[x for x in range(20)]
  def masas_circulo(x):
      masas_circulos =[]
      rs =[r for r in range(x)]
      for r in rs:
          masas_circulos.append(model.predict([r])[0][0])
  
     return masas_circulos
  
   masas_circulos = masas_circulo(20) 
   masas_circulos
   esferas = [4/3*np.pi*r**3 for r in range(20)]
   import matplotlib.pyplot as plt
   plt.plot(rs,masas_circulos,label="DL")
   plt.plot(rs,esferas,label="Real");
   plt.title("Mass of an sphere.\nDL (1hl,128 n,5000 e) vs ground_truth")
   plt.xlabel("Radius")
   plt.ylabel("Sphere")
   plt.legend();
  
  ```
  
  **Tags:** deep-learning
  
  > **Comments:**
  > **username_1**: What are you trying to predict?: is it $V$ given $r$? for the formula $V=4/3\pi r^3$? if so, is it the output of your network, the single scalar, expected to be $V$? For me that seems like a plotting error: check how you are drawing the output of your network for different $r$  
  > **username_0**: Ies, I am trying to predict V  given r for the formula  V=4/3πr^3. I have added the graphs' code.  
  > **username_0**: Sphere's density=1   
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-07-26. Score: 2 | Accepted answer
  
  You're trying to learn a cubic function that explodes in values and your issue is **scaling**. I have been able to learn a better approximation by scaling data and using **tanh** as activation function.
  
  Code and result are as below:
  
  [![enter image description here](https://i.sstatic.net/pHqhO.png)](https://i.sstatic.net/pHqhO.png)
  
  Convergence around X=100 happens because of tanh activation. Relu will not work better because of negative values that is the result of scaling. You can try playing with **Leaky Relu** activation and various **alpha** values.
  
  ```
  import pandas as pd
  import numpy as np
  import matplotlib.pyplot as plt
  
  from tensorflow import keras
  from sklearn.preprocessing import StandardScaler
  from sklearn.model_selection import train_test_split
  
  def mass_of_sphere(R):
      return (4/3) * np.pi * (R**3)
  
  X = np.linspace(1, 120, 500000)
  y = [mass_of_sphere(x) for x in X]
  
  X = np.array(X).reshape(-1, 1)
  y = np.array(y)
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
  
  scaler_X = StandardScaler()
  X_train = scaler_X.fit_transform(X_train)
  X_test = scaler_X.transform(X_test)
  
  scaler_y = StandardScaler()
  y_train = scaler_y.fit_transform(y_train.reshape(-1, 1))
  y_test = scaler_y.transform(y_test.reshape(-1, 1)).reshape(-1)
  
  model = keras.Sequential([keras.layers.Dense(1),
                            keras.layers.Dense(128, activation = "tanh"),
                            keras.layers.Dense(1, activation = "tanh")])
  
  early_stopping = keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 5)
  model.compile(optimizer = 'rmsprop', loss = 'mse')
  history = model.fit(X_train, y_train, epochs = 100, callbacks=[early_stopping], 
                      batch_size = 2048, validation_data=(X_test, y_test))
  
  y_hat = scaler_y.inverse_transform(model.predict(X_test)).reshape(-1)
  y_test = scaler_y.inverse_transform(y_test).reshape(-1)
  
  f, ax = plt.subplots(figsize = (12, 4))
  ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_test), color = 'blue', label = 'Real')
  ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_hat), color = 'orange', label = 'DL')
  ax.legend()
  
  ```
  
  > **Comments:**
  > **username_0**: Excellent. Rather surprising for me because I have read that relu should (almost) always work. For the moment I do not undersand the divergence with values bigger than 100. I will study your answer and try all your suggestions. I hope I shall be able to fit it beyond 100 :). I supposed that it was not neccesary to do any sort of "feature engineering", as is very frequently asserted. –  
  > **username_2**: For better understanding, inspect tanh figure, it maps $(-\infty, \infty)$ to $(-1, 1)$. Having clarified that, inspect the scaled value of the mass of $R = 100$.   scaler_y.transform(np.array([[mass_of_sphere(100)]]))  You can simply bypass this limitation by either changing activation function or using a higher range of training values, going for perhaps 500 instead of 120.  I just tested with values between (1, 500) and convergence occured at 400. Not an elegant solution but good to have.  
  > **username_0**: I've got a very good fit using "elu" as an activation function. But I was surprised because comparing results (inside the range used) obtained from mass_of_sphere() and model.predict I found enourmous differences. For the moment, I do not identify the error  
  > **username_2**: OK maybe a better approach: Scale your data using MinMaxScaler() to range(-1, 1) instead of StandardScaler(). Train your network with tanh activation in last layer again. Don't forget to scale back your prediction.  
  
  ---
  
  ### Answer by username_0 on 2021-08-27. Score: 0
  
  A simpler model seems to be the best option
  
  ```
  model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])
  model.compile(optimizer='sgd', loss='mean_squared_error')
  xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)
  ys2 = np.array([4/3*r*np.pi**3 for r in xs])    
  model.fit(xs, ys2, epochs=500,validation_split=0.2)
  
  def masa_circulo(x):
      return 4/3*x*np.pi**3
  
  ```
  
  Testing it graphically
  
  ```
  x = [x for x in range(1,int(1e6),int(1e3))]
  y_masa_circulo = [ masa_circulo(m) for m in x]
  y_masa_predicha= [model.predict([m])[0] for m in x]
  
  import matplotlib.pyplot as plt
  fig,axes = plt.subplots(1,2)
  axes[0].plot(y_masa_circulo);
  axes[0].set_title("y_masa_circulo")
  axes[0].set_ylabel("y_masa_circulo")
  axes[0].set_xlabel("Radio")
  
  axes[1].plot(y_masa_predicha);
  plt.title("y_masa_circulo")
  plt.title("y_masa_predicha");
  axes[1].set_ylabel("y_masa_predicha");
  axes[1].set_xlabel("Radio");
  
  ```
  
  [![Graphical comparison function_results vs nn_results](https://i.sstatic.net/3Scb1.png)](https://i.sstatic.net/3Scb1.png)
  
  No need of scaling the data.
  
  
  ---
  
  