Answers.Body: <p><span class="math-container">$f(x) = x^2 + b$</span> is a polynomial (more precisely, a parabola) so it is continuous, thus, a neural network (with at least one hidden layer) should be able to approximate that function (given the universal approximation theorem).</p>
  
  <p>After a very quick look at your code, I noticed you aren't using an activation function for your dense layers (i.e. your activation function is <code>None</code>). Try to use e.g. ReLU.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T06:48:16.393Z
Answers.Text: Thank you @nbro for your answer. I will test more, with different activation functions, including ReLU. +1 for pointing out the universal approximation theorem, of which I was not aware yet. That is really helpful to know.
Answers.UserDisplayName: Mike de Klerk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T03:12:08.223Z
Answers.LastActivityDate: 2020-02-20T03:12:08.223Z
Answers.UserDisplayName: nbro
Answers.Body: <ol>
  <li><p>if you know in advance that your data is polynomial and you want to fit the best coefficient, then this is an easier (classical machine learning) problem .
  you would just assume a maximum degree e.g <span class="math-container">$y_{pred} = a_0+a_1.x+a_2x^2+...+a_n.x^n  $</span><br />
  you can just initialize <span class="math-container">$a_i$</span>s and consider MSE loss <span class="math-container">$(y_{groundtruth}-y_{pred})^2$</span>.<br />
  then derivative with respect to <span class="math-container">$a_i$</span>s is <span class="math-container">$2(y_{groundtruth}-y_{pred}).(- d(y_{pred})/d(a_i)) = 2(y_{groundtruth}-y_{pred}).(- x^i)$</span><br />
  so having access to the gradinet, you can easily do gradient descent which is trivial.</p>
  </li>
  <li><p>So I assume this (the general form of the data generator) is not known.
  in this case as you did, we assume a general form and let neural network learn it. it is important to note that you should limit the support (range) of your data by normalization as you did by normalization because it is impossible to find an approximation which holds true every where. but in a small range of [0,1] it is much easier to approximate the desired function.</p>
  </li>
  </ol>
  <p>in general it is possible to approximate this function but on a limited range e.g on [0,1], unless we have the correct general form (in this case polynomial) and then just fit the correct parameters using gradient descent or other methods.</p>
  <p>I hope these two cases shed some lights on your question</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T14:12:07.920Z
Answers.LastActivityDate: 2024-07-09T14:12:07.920Z
Answers.UserDisplayName: saleh
Body: <p>I am new to ANN. I am trying out several 'simple' algorithms to see what ANN can (or cannot) be used for and how. I played around with Conv2d once and had it recognize images successfully. Now I am looking into trend line analyses. I have succeeded in training a network where it solved for linear equations. Now I am trying to see if it can be trained to solve for <span class="math-container">$y$</span> in the formula <span class="math-container">$y = b + x^2$</span>.</p>
  <p>No matter what parameters I change, or the number of dense layers, I get high values for loss and validation loss, and the predictions are incorrect.</p>
  <p>Is it possible to solve this equation, and with what network? If it is not possible, why not? I am not looking to solve a practical problem, rather build up understanding and intuition about ANNs.</p>
  <p>See the code I tried with below</p>
  <pre><code>#region Imports
  from __future__ import absolute_import, division, print_function, unicode_literals
  import math 
  import numpy as np 
  import tensorflow as tf
  from tensorflow.keras import models, optimizers
  from tensorflow.keras.models import Sequential
  from tensorflow.keras.layers import Activation, Dropout, Flatten, Dense, Lambda
  import tensorflow.keras.backend as K
  #endregion
  
  #region Constants
  learningRate = 0.01
  epochs: int = 1000
  batch_size = None
  trainingValidationFactor = 0.75
  nrOfSamples = 100
  activation = None
  #endregion
  
  #region Function definitions
  def CreateNetwork(inputDimension):
    model = Sequential()
    model.add(Dense(2, input_dim=2, activation=activation))
    model.add(Dense(64, use_bias=True, activation=activation))
    model.add(Dense(32, use_bias=True, activation=activation))
    model.add(Dense(1))
    adam = optimizers.Adam(learning_rate=learningRate)
    # sgd = optimizers.SGD(lr=learningRate, decay=1e-6, momentum=0.9, nesterov=True)
    # adamax = optimizers.Adamax(learning_rate=learningRate)
    model.compile(loss='mse', optimizer=adam)
    return model
  
  def SplitDataForValidation(factor, data, labels):
    upperBoundary = int(len(data) * factor)
  
    trainingData = data[:upperBoundary]
    trainingLabels = labels[:upperBoundary]
  
    validationData = data[upperBoundary:]
    validationLabels = labels[upperBoundary:]
    return ((trainingData, trainingLabels), (validationData, validationLabels))
  
  def Train(network, training, validation):
    trainingData, trainingLabels = training
    history = network.fit(
      trainingData
      ,trainingLabels
      ,validation_data=validation
      ,epochs=epochs 
      ,batch_size=batch_size
    )
  
    return history
  
  def subtractMean(data):
    mean = np.mean(data)
    data -= mean
    return mean
  
  def rescale(data):
    max = np.amax(data)
    factor = 1 / max
    data *= factor
    return factor
  
  def Normalize(data, labels):
    dataScaleFactor = rescale(data)
    dataMean = subtractMean(data)
  
    labels *= dataScaleFactor
    labelsMean = np.mean(labels)
    labels -= labelsMean
  
  def Randomize(data, labels):
    rng_state = np.random.get_state()
    np.random.shuffle(data)
    np.random.set_state(rng_state)
    np.random.shuffle(labels)
  
  def CreateTestData(nrOfSamples):
    data = np.zeros(shape=(nrOfSamples,2))
    labels = np.zeros(nrOfSamples)
  
    for i in range(nrOfSamples):
      for j in range(2):
        randomInt = np.random.randint(1, 5)
        data[i, j] = (randomInt * i) + 10
      labels[i] = data[i, 0] + math.pow(data[i, 1], 2)
    
    Randomize(data, labels)
    return (data, labels)
  #endregion
  
  allData, allLabels = CreateTestData(nrOfSamples)
  Normalize(allData, allLabels)
  training, validation = SplitDataForValidation(trainingValidationFactor, allData, allLabels)
  
  inputDimension = np.size(allData, 1)
  network = CreateNetwork(inputDimension)
  
  history = Train(network, training, validation)
  
  prediction = network.predict([
    [2, 2], # Should be 2 + 2 * 2 = 6
    [4, 7], # Should be 4 + 7 * 7 = 53
    [23, 56], # Should be 23 + 56 * 56 = 3159
    [128,256] # Should be 128 + 256 * 256 = 65664
  ])
  print(str(prediction))
  </code></pre>
  
Tags: <neural-networks><keras><function-approximation><model-request>
Title: Which neural network can approximate the function $y = x^2 + b$?
UserDisplayName: Mike de Klerk
ThreadText: # Which neural network can approximate the function $y = x^2 + b$?
  
  (Asked by: username_0 on 2020-02-19)
  
  I am new to ANN. I am trying out several 'simple' algorithms to see what ANN can (or cannot) be used for and how. I played around with Conv2d once and had it recognize images successfully. Now I am looking into trend line analyses. I have succeeded in training a network where it solved for linear equations. Now I am trying to see if it can be trained to solve for $y$ in the formula $y = b + x^2$.
  No matter what parameters I change, or the number of dense layers, I get high values for loss and validation loss, and the predictions are incorrect.
  Is it possible to solve this equation, and with what network? If it is not possible, why not? I am not looking to solve a practical problem, rather build up understanding and intuition about ANNs.
  See the code I tried with below
  #region Imports
  from __future__ import absolute_import, division, print_function, unicode_literals
  import math 
  import numpy as np 
  import tensorflow as tf
  from tensorflow.keras import models, optimizers
  from tensorflow.keras.models import Sequential
  from tensorflow.keras.layers import Activation, Dropout, Flatten, Dense, Lambda
  import tensorflow.keras.backend as K
  #endregion
  
  #region Constants
  learningRate = 0.01
  epochs: int = 1000
  batch_size = None
  trainingValidationFactor = 0.75
  nrOfSamples = 100
  activation = None
  #endregion
  
  #region Function definitions
  def CreateNetwork(inputDimension):
    model = Sequential()
    model.add(Dense(2, input_dim=2, activation=activation))
    model.add(Dense(64, use_bias=True, activation=activation))
    model.add(Dense(32, use_bias=True, activation=activation))
    model.add(Dense(1))
    adam = optimizers.Adam(learning_rate=learningRate)
    # sgd = optimizers.SGD(lr=learningRate, decay=1e-6, momentum=0.9, nesterov=True)
    # adamax = optimizers.Adamax(learning_rate=learningRate)
    model.compile(loss='mse', optimizer=adam)
    return model
  
  def SplitDataForValidation(factor, data, labels):
    upperBoundary = int(len(data) * factor)
  
    trainingData = data[:upperBoundary]
    trainingLabels = labels[:upperBoundary]
  
    validationData = data[upperBoundary:]
    validationLabels = labels[upperBoundary:]
    return ((trainingData, trainingLabels), (validationData, validationLabels))
  
  def Train(network, training, validation):
    trainingData, trainingLabels = training
    history = network.fit(
      trainingData
      ,trainingLabels
      ,validation_data=validation
      ,epochs=epochs 
      ,batch_size=batch_size
    )
  
    return history
  
  def subtractMean(data):
    mean = np.mean(data)
    data -= mean
    return mean
  
  def rescale(data):
    max = np.amax(data)
    factor = 1 / max
    data *= factor
    return factor
  
  def Normalize(data, labels):
    dataScaleFactor = rescale(data)
    dataMean = subtractMean(data)
  
    labels *= dataScaleFactor
    labelsMean = np.mean(labels)
    labels -= labelsMean
  
  def Randomize(data, labels):
    rng_state = np.random.get_state()
    np.random.shuffle(data)
    np.random.set_state(rng_state)
    np.random.shuffle(labels)
  
  def CreateTestData(nrOfSamples):
    data = np.zeros(shape=(nrOfSamples,2))
    labels = np.zeros(nrOfSamples)
  
    for i in range(nrOfSamples):
      for j in range(2):
        randomInt = np.random.randint(1, 5)
        data[i, j] = (randomInt * i) + 10
      labels[i] = data[i, 0] + math.pow(data[i, 1], 2)
    
    Randomize(data, labels)
    return (data, labels)
  #endregion
  
  allData, allLabels = CreateTestData(nrOfSamples)
  Normalize(allData, allLabels)
  training, validation = SplitDataForValidation(trainingValidationFactor, allData, allLabels)
  
  inputDimension = np.size(allData, 1)
  network = CreateNetwork(inputDimension)
  
  history = Train(network, training, validation)
  
  prediction = network.predict([
    [2, 2], # Should be 2 + 2 * 2 = 6
    [4, 7], # Should be 4 + 7 * 7 = 53
    [23, 56], # Should be 23 + 56 * 56 = 3159
    [128,256] # Should be 128 + 256 * 256 = 65664
  ])
  print(str(prediction))
  
  **Tags:** neural-networks, keras, function-approximation, model-request
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-02-20. Score: 2
  
  $f(x) = x^2 + b$ is a polynomial (more precisely, a parabola) so it is continuous, thus, a neural network (with at least one hidden layer) should be able to approximate that function (given the universal approximation theorem).
  
  After a very quick look at your code, I noticed you aren't using an activation function for your dense layers (i.e. your activation function is `None`). Try to use e.g. ReLU.
  
  > **Comments:**
  > **username_0**: Thank you @nbro for your answer. I will test more, with different activation functions, including ReLU. +1 for pointing out the universal approximation theorem, of which I was not aware yet. That is really helpful to know.  
  
  ---
  
  ### Answer by username_2 on 2024-07-09. Score: 0
  
  1. if you know in advance that your data is polynomial and you want to fit the best coefficient, then this is an easier (classical machine learning) problem .
     you would just assume a maximum degree e.g $y\_{pred} = a\_0+a\_1.x+a\_2x^2+...+a\_n.x^n $  
     you can just initialize $a\_i$s and consider MSE loss $(y\_{groundtruth}-y\_{pred})^2$.  
     then derivative with respect to $a\_i$s is $2(y\_{groundtruth}-y\_{pred}).(- d(y\_{pred})/d(a\_i)) = 2(y\_{groundtruth}-y\_{pred}).(- x^i)$  
     so having access to the gradinet, you can easily do gradient descent which is trivial.
  2. So I assume this (the general form of the data generator) is not known.
     in this case as you did, we assume a general form and let neural network learn it. it is important to note that you should limit the support (range) of your data by normalization as you did by normalization because it is impossible to find an approximation which holds true every where. but in a small range of [0,1] it is much easier to approximate the desired function.
  
  in general it is possible to approximate this function but on a limited range e.g on [0,1], unless we have the correct general form (in this case polynomial) and then just fit the correct parameters using gradient descent or other methods.
  
  I hope these two cases shed some lights on your question
  
  
  ---
  
  