Answers.Body: <p>You're trying to learn a cubic function that explodes in values and your issue is <strong>scaling</strong>. I have been able to learn a better approximation by scaling data and using <strong>tanh</strong> as activation function.</p>
  <p>Code and result are as below:</p>
  <p><a href="https://i.sstatic.net/pHqhO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pHqhO.png" alt="enter image description here" /></a></p>
  <p>Convergence around X=100 happens because of tanh activation. Relu will not work better because of negative values that is the result of scaling. You can try playing with <strong>Leaky Relu</strong> activation and various <strong>alpha</strong> values.</p>
  <pre><code>import pandas as pd
  import numpy as np
  import matplotlib.pyplot as plt
  
  from tensorflow import keras
  from sklearn.preprocessing import StandardScaler
  from sklearn.model_selection import train_test_split
  
  def mass_of_sphere(R):
      return (4/3) * np.pi * (R**3)
  
  X = np.linspace(1, 120, 500000)
  y = [mass_of_sphere(x) for x in X]
  
  X = np.array(X).reshape(-1, 1)
  y = np.array(y)
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
  
  scaler_X = StandardScaler()
  X_train = scaler_X.fit_transform(X_train)
  X_test = scaler_X.transform(X_test)
  
  scaler_y = StandardScaler()
  y_train = scaler_y.fit_transform(y_train.reshape(-1, 1))
  y_test = scaler_y.transform(y_test.reshape(-1, 1)).reshape(-1)
  
  model = keras.Sequential([keras.layers.Dense(1),
                            keras.layers.Dense(128, activation = &quot;tanh&quot;),
                            keras.layers.Dense(1, activation = &quot;tanh&quot;)])
  
  early_stopping = keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 5)
  model.compile(optimizer = 'rmsprop', loss = 'mse')
  history = model.fit(X_train, y_train, epochs = 100, callbacks=[early_stopping], 
                      batch_size = 2048, validation_data=(X_test, y_test))
  
  y_hat = scaler_y.inverse_transform(model.predict(X_test)).reshape(-1)
  y_test = scaler_y.inverse_transform(y_test).reshape(-1)
  
  f, ax = plt.subplots(figsize = (12, 4))
  ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_test), color = 'blue', label = 'Real')
  ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_hat), color = 'orange', label = 'DL')
  ax.legend()
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-26T13:51:37.123Z
Answers.Text: Excellent. Rather surprising for me because I have read that relu should (almost) always work. For the moment I do not undersand the divergence with values bigger than 100. I will study your answer and try all your suggestions. I hope I shall be able to fit it beyond 100 :). I supposed that it was not neccesary to do any sort of "feature engineering", as is very frequently asserted. –
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-26T14:07:16.913Z
Answers.Text: For better understanding, inspect tanh figure, it maps $(-\infty, \infty)$ to $(-1, 1)$. Having clarified that, inspect the scaled value of the mass of $R = 100$. 
  
  scaler_y.transform(np.array([[mass_of_sphere(100)]]))
  
  You can simply bypass this limitation by either changing activation function or using a higher range of training values, going for perhaps 500 instead of 120.
  
  I just tested with values between (1, 500) and convergence occured at 400. Not an elegant solution but good to have.
Answers.UserDisplayName: meliksahturker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T12:36:43.980Z
Answers.Text: I've got a very good fit using "elu" as an activation function. But I was surprised because comparing results (inside the range used) obtained from mass_of_sphere() and model.predict I found enourmous differences. For the moment, I do not identify the error
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T20:24:50.483Z
Answers.Text: OK maybe a better approach: Scale your data using MinMaxScaler() to range(-1, 1) instead of StandardScaler(). Train your network with tanh activation in last layer again.
  Don't forget to scale back your prediction.
Answers.UserDisplayName: meliksahturker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-26T13:32:28.957Z
Answers.LastActivityDate: 2021-07-26T13:41:10.390Z
Answers.LastEditDate: 2021-07-26T13:41:10.390Z
Answers.UserDisplayName: meliksahturker
Answers.Body: <p>A simpler model seems to be the best option</p>
  <pre><code>model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])
  model.compile(optimizer='sgd', loss='mean_squared_error')
  xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)
  ys2 = np.array([4/3*r*np.pi**3 for r in xs])    
  model.fit(xs, ys2, epochs=500,validation_split=0.2)
  
  def masa_circulo(x):
      return 4/3*x*np.pi**3
  </code></pre>
  <p>Testing it graphically</p>
  <pre><code>x = [x for x in range(1,int(1e6),int(1e3))]
  y_masa_circulo = [ masa_circulo(m) for m in x]
  y_masa_predicha= [model.predict([m])[0] for m in x]
  
  import matplotlib.pyplot as plt
  fig,axes = plt.subplots(1,2)
  axes[0].plot(y_masa_circulo);
  axes[0].set_title(&quot;y_masa_circulo&quot;)
  axes[0].set_ylabel(&quot;y_masa_circulo&quot;)
  axes[0].set_xlabel(&quot;Radio&quot;)
  
  axes[1].plot(y_masa_predicha);
  plt.title(&quot;y_masa_circulo&quot;)
  plt.title(&quot;y_masa_predicha&quot;);
  axes[1].set_ylabel(&quot;y_masa_predicha&quot;);
  axes[1].set_xlabel(&quot;Radio&quot;);
  </code></pre>
  <p><a href="https://i.sstatic.net/3Scb1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3Scb1.png" alt="Graphical comparison function_results vs nn_results" /></a></p>
  <p>No need of scaling the data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-27T18:35:38.753Z
Answers.LastActivityDate: 2021-08-27T19:53:13.033Z
Answers.LastEditDate: 2021-08-27T19:53:13.033Z
Answers.UserDisplayName: Hermes Morales
Body: <p>I have been trying to adjust a neural network to a simple function: the mass of an sphere.
  I have tried with different architectures, for example, a single hidden layer and two hidden layers, always with 128 neurons each, and training them for 5000 epochs.
  The code is the usual one. Just in case, I publish one of them</p>
  <pre><code>model = keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])
                          ,keras.layers.Dense(128, activation=&quot;relu&quot;)
                          ,keras.layers.Dense(1, activation=&quot;relu&quot;)])
  model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])
  history = model.fit(x, y, validation_split=0.2, epochs=5000)
  </code></pre>
  <p>The results are shown in the graphs.</p>
  <p><a href="https://i.sstatic.net/q6Nad.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/q6Nad.png" alt="two hidden layers, 128 neurons each" /></a></p>
  <p><a href="https://i.sstatic.net/jYAuV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jYAuV.png" alt="one hidden layer, 128 neurons" /></a></p>
  <p>I suspect that I am making an error somewhere, because I have seen that deep learning is able to match complex functions with much less epochs. I shall appreciate any hint to fix this problem and obtain a good fit with the deep learning function.</p>
  <p>In order to make it clear I post the graph's code.</p>
  <pre><code>rs =[x for x in range(20)]
  def masas_circulo(x):
      masas_circulos =[]
      rs =[r for r in range(x)]
      for r in rs:
          masas_circulos.append(model.predict([r])[0][0])
  
     return masas_circulos
  
   masas_circulos = masas_circulo(20) 
   masas_circulos
   esferas = [4/3*np.pi*r**3 for r in range(20)]
   import matplotlib.pyplot as plt
   plt.plot(rs,masas_circulos,label=&quot;DL&quot;)
   plt.plot(rs,esferas,label=&quot;Real&quot;);
   plt.title(&quot;Mass of an sphere.\nDL (1hl,128 n,5000 e) vs ground_truth&quot;)
   plt.xlabel(&quot;Radius&quot;)
   plt.ylabel(&quot;Sphere&quot;)
   plt.legend();
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:06:11.833Z
Comments.Text: What are you trying to predict?: is it $V$ given $r$? for the formula $V=4/3\pi r^3$? if so, is it the output of your network, the single scalar, expected to be $V$? For me that seems like a plotting error: check how you are drawing the output of your network for different $r$
Comments.UserDisplayName: JVGD
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:15:51.567Z
Comments.Text: Ies, I am trying to predict V  given r for the formula  V=4/3πr^3. I have added the graphs' code.
Comments.UserDisplayName: Hermes Morales
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T14:21:02.377Z
Comments.Text: Sphere's density=1 
Comments.UserDisplayName: Hermes Morales
Tags: <deep-learning>
Title: Not able to find a good fit for a simple function with neural networks
UserDisplayName: Hermes Morales
ThreadText: # Not able to find a good fit for a simple function with neural networks
  
  (Asked by: username_0 on 2021-07-26)
  
  I have been trying to adjust a neural network to a simple function: the mass of an sphere.
  I have tried with different architectures, for example, a single hidden layer and two hidden layers, always with 128 neurons each, and training them for 5000 epochs.
  The code is the usual one. Just in case, I publish one of them
  model = keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])
                          ,keras.layers.Dense(128, activation="relu")
                          ,keras.layers.Dense(1, activation="relu")])
  model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])
  history = model.fit(x, y, validation_split=0.2, epochs=5000)
  
  The results are shown in the graphs.
  
  
  I suspect that I am making an error somewhere, because I have seen that deep learning is able to match complex functions with much less epochs. I shall appreciate any hint to fix this problem and obtain a good fit with the deep learning function.
  In order to make it clear I post the graph's code.
  rs =[x for x in range(20)]
  def masas_circulo(x):
      masas_circulos =[]
      rs =[r for r in range(x)]
      for r in rs:
          masas_circulos.append(model.predict([r])[0][0])
  
     return masas_circulos
  
   masas_circulos = masas_circulo(20) 
   masas_circulos
   esferas = [4/3*np.pi*r**3 for r in range(20)]
   import matplotlib.pyplot as plt
   plt.plot(rs,masas_circulos,label="DL")
   plt.plot(rs,esferas,label="Real");
   plt.title("Mass of an sphere.\nDL (1hl,128 n,5000 e) vs ground_truth")
   plt.xlabel("Radius")
   plt.ylabel("Sphere")
   plt.legend();
  
  **Tags:** deep-learning
  > **Comments:**
  > **username_1**: What are you trying to predict?: is it $V$ given $r$? for the formula $V=4/3\pi r^3$? if so, is it the output of your network, the single scalar, expected to be $V$? For me that seems like a plotting error: check how you are drawing the output of your network for different $r$  
  > **username_0**: Ies, I am trying to predict V  given r for the formula  V=4/3πr^3. I have added the graphs' code.  
  > **username_0**: Sphere's density=1   
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-07-26. Score: 2 | Accepted answer
  
  You're trying to learn a cubic function that explodes in values and your issue is scaling. I have been able to learn a better approximation by scaling data and using tanh as activation function.
  Code and result are as below:
  
  Convergence around X=100 happens because of tanh activation. Relu will not work better because of negative values that is the result of scaling. You can try playing with Leaky Relu activation and various alpha values.
  import pandas as pd
  import numpy as np
  import matplotlib.pyplot as plt
  
  from tensorflow import keras
  from sklearn.preprocessing import StandardScaler
  from sklearn.model_selection import train_test_split
  
  def mass_of_sphere(R):
      return (4/3) * np.pi * (R**3)
  
  X = np.linspace(1, 120, 500000)
  y = [mass_of_sphere(x) for x in X]
  
  X = np.array(X).reshape(-1, 1)
  y = np.array(y)
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
  
  scaler_X = StandardScaler()
  X_train = scaler_X.fit_transform(X_train)
  X_test = scaler_X.transform(X_test)
  
  scaler_y = StandardScaler()
  y_train = scaler_y.fit_transform(y_train.reshape(-1, 1))
  y_test = scaler_y.transform(y_test.reshape(-1, 1)).reshape(-1)
  
  model = keras.Sequential([keras.layers.Dense(1),
                            keras.layers.Dense(128, activation = "tanh"),
                            keras.layers.Dense(1, activation = "tanh")])
  
  early_stopping = keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 5)
  model.compile(optimizer = 'rmsprop', loss = 'mse')
  history = model.fit(X_train, y_train, epochs = 100, callbacks=[early_stopping], 
                      batch_size = 2048, validation_data=(X_test, y_test))
  
  y_hat = scaler_y.inverse_transform(model.predict(X_test)).reshape(-1)
  y_test = scaler_y.inverse_transform(y_test).reshape(-1)
  
  f, ax = plt.subplots(figsize = (12, 4))
  ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_test), color = 'blue', label = 'Real')
  ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_hat), color = 'orange', label = 'DL')
  ax.legend()
  
  > **Comments:**
  > **username_0**: Excellent. Rather surprising for me because I have read that relu should (almost) always work. For the moment I do not undersand the divergence with values bigger than 100. I will study your answer and try all your suggestions. I hope I shall be able to fit it beyond 100 :). I supposed that it was not neccesary to do any sort of "feature engineering", as is very frequently asserted. –  
  > **username_2**: For better understanding, inspect tanh figure, it maps $(-\infty, \infty)$ to $(-1, 1)$. Having clarified that, inspect the scaled value of the mass of $R = 100$.   scaler_y.transform(np.array([[mass_of_sphere(100)]]))  You can simply bypass this limitation by either changing activation function or using a higher range of training values, going for perhaps 500 instead of 120.  I just tested with values between (1, 500) and convergence occured at 400. Not an elegant solution but good to have.  
  > **username_0**: I've got a very good fit using "elu" as an activation function. But I was surprised because comparing results (inside the range used) obtained from mass_of_sphere() and model.predict I found enourmous differences. For the moment, I do not identify the error  
  > **username_2**: OK maybe a better approach: Scale your data using MinMaxScaler() to range(-1, 1) instead of StandardScaler(). Train your network with tanh activation in last layer again. Don't forget to scale back your prediction.  
  
  ---
  
  ### Answer by username_0 on 2021-08-27. Score: 0
  
  A simpler model seems to be the best option
  model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])
  model.compile(optimizer='sgd', loss='mean_squared_error')
  xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)
  ys2 = np.array([4/3*r*np.pi**3 for r in xs])    
  model.fit(xs, ys2, epochs=500,validation_split=0.2)
  
  def masa_circulo(x):
      return 4/3*x*np.pi**3
  
  Testing it graphically
  x = [x for x in range(1,int(1e6),int(1e3))]
  y_masa_circulo = [ masa_circulo(m) for m in x]
  y_masa_predicha= [model.predict([m])[0] for m in x]
  
  import matplotlib.pyplot as plt
  fig,axes = plt.subplots(1,2)
  axes[0].plot(y_masa_circulo);
  axes[0].set_title("y_masa_circulo")
  axes[0].set_ylabel("y_masa_circulo")
  axes[0].set_xlabel("Radio")
  
  axes[1].plot(y_masa_predicha);
  plt.title("y_masa_circulo")
  plt.title("y_masa_predicha");
  axes[1].set_ylabel("y_masa_predicha");
  axes[1].set_xlabel("Radio");
  
  
  No need of scaling the data.
  
  
  ---
  
  