Answers.Body: <p>I found a solution to this some time back. I have been studying function approximation (within linear regression) for some time. 
  Here's how I did it:</p>
  
  <p><a href="http://neuralnetworksanddeeplearning.com/chap4.html" rel="nofollow noreferrer">Neural Networks have been proved to be universal function approximators</a>. So, even a single hidden layer would be sufficient to approximate a function simple as <strong>addition</strong> (Even somewhat complex functions like the <strong><a href="https://www.codeproject.com/Articles/1237026/Simple-MLP-Backpropagation-Artificial-Neural-Netwo" rel="nofollow noreferrer">Sine</a></strong> and <strong>any random CONTINUOUS <a href="https://colab.research.google.com/drive/1K9agqd9Pn8llY1uwkFD68YqFgPeSTBCQ" rel="nofollow noreferrer">wiggly function</a></strong> have been approximated) </p>
  
  <p>First, I used a high level API like TensorFlow and Keras and implemented it <a href="https://colab.research.google.com/drive/1s8VYf5tYGj83SxOtJQbiKs7AJd_eNQX5" rel="nofollow noreferrer">here</a></p>
  
  <p>The model was trained on the data (input-output pairs)</p>
  
  <pre><code>R    = np.array([-4, -10,  -2,  8, 5, 22,  3],  dtype=float)
  B    = np.array([4, -10,  0,  0, 15, 5,  1],  dtype=float)
  G    = np.array([0, 10,  5,  8, 1, 2,  38],  dtype=float)
  
  Y    = np.array([0, -10, 3, 16, 21, 29, 42],  dtype=float)
  </code></pre>
  
  <p>And trained as follows:</p>
  
  <pre><code>#Create a hidden layer with 2 neurons
  hidden = tf.keras.layers.Dense(units=2, input_shape=[3])
  
  #Create the output (final) layer which symbolises value of **Y**
  output = tf.keras.layers.Dense(units=1)
  
  #Combine layers to form the neural network and compile it
  model = tf.keras.Sequential([hidden, output])
  model.compile(loss='mean_squared_error', optimizer=tf.keras.optimizers.Adam(0.1))
  history = model.fit(RBG,Y, epochs=500, verbose=False)
  </code></pre>
  
  <p>The model converges in about 50 epochs </p>
  
  <p><a href="https://i.sstatic.net/97HZy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/97HZy.png" alt="enter image description here"></a></p>
  
  <p>Also, I have implemented the same using only C/C++ and used GNU plot to visualize the results. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-19T13:47:33.570Z
Answers.LastActivityDate: 2020-01-19T13:47:33.570Z
Answers.UserDisplayName: Pe Dro
Answers.Body: <p>There are always a large number of possible functions that can produce a given set of input-output values. The challenge is to find a <em>simplest</em> function (according to whatever criteria you choose) to produce those values.</p>
  
  <p>One approach is to write a general function of the input variables, comprising terms of all order in R, G, and B, with a coefficient for each term, then search for values of the coefficients that A) reproduce the known input-output values accurately and B) leave the largest number of coefficients equal to zero.  </p>
  
  <p>Several different algorithms can be used to do the search efficiently.  My choice would be a genetic algorithm to seek the minimum of the RMS difference between the produced and known I-O values, summed with the product of a gradually increasing parameter and the number of nonzero coefficients.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-03T15:17:55.967Z
Answers.Text: I have brushed over Genetic Algorithm and found it to be really novel for me and very useful. But, I need to deploy the trained model onto a MCU and hence it will be too bulky for the device.
Answers.UserDisplayName: Pe Dro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T14:54:26.127Z
Answers.LastActivityDate: 2019-09-02T14:54:26.127Z
Answers.UserDisplayName: S. McGrew
Answers.Body: <p>As I see your explanation, what you're looking for can be stated via another way saying</p>
  
  <pre><code>Y = aX + b
  </code></pre>
  
  <p>where Y is output vector, X input vector and a &amp; b are the coefficients you want to find.</p>
  
  <p><strong>Why so? And what is happening?</strong></p>
  
  <p>First thing first: I recommend to look the video [4] about how matrices and vectors work together and form after multiplication very familiar equations:</p>
  
  <pre><code>Y = a1 * x1 + a2 * x2 + a3 * x3
  </code></pre>
  
  <p>Now you see, you do not only get R + G + B but also some constants supplied for each of the variables.</p>
  
  <p>About polynomial solutions I found [2] and [3] but reading through [3] you'll soon notice it is about completely different approach:</p>
  
  <pre><code>Y = a1 * x + a2 * x^2 + a3 * x^3
  </code></pre>
  
  <p>which you don't want.</p>
  
  <p>So, you find something called linear regression and a so called Deep Neural Network for example to solve it [1].</p>
  
  <p>I would summarize the source [1] in these steps:</p>
  
  <ul>
  <li><p>You have to find some training data. That is examples of correct Y values, when X values are known.</p></li>
  <li><p>Then you build in Python note book some code that has: a Neural Network with hidden layer(s), Activation Function, Back Propagation and Objective Function.</p></li>
  <li><p>Many many iterations with the data, called Training.</p></li>
  </ul>
  
  <p>There are plenty of samples and courses online about these in detail, but with correct tools and tutorials some dozen lines and no more are needed.</p>
  
  <p>Process is ended with validation phase with some more known data items and results. It can tell you how well the estimated model works.</p>
  
  <hr>
  
  <p><strong>Final Notes:</strong></p>
  
  <p><em>As you may observe, the solution includes quite a many funny terms you have to learn somehow before mastering the task. For example Udemy has great online courses on this topic, also free tutorials are available on another sites. Your plans sound quite ambitious compared to the knowledge you have so far, so I really do recommend you learn little bit more to be able to fine tune the already given examples online. For example tutorial [5] includes one. It is at first quite complicated code, you'd need quite a lot practice to master it line by line.</em></p>
  
  <hr>
  
  <p><strong>In short:</strong></p>
  
  <p>Find your favorite tutorial, study neural networks a little bit (basics) and pick code sample to start experimenting. It is a long way but it is worth it.</p>
  
  <hr>
  
  <p><strong>Source:</strong></p>
  
  <p>[1] <a href="https://lightsapplications.wordpress.com/linear-regression-and-deep-learning/" rel="nofollow noreferrer">https://lightsapplications.wordpress.com/linear-regression-and-deep-learning/</a></p>
  
  <p>[2] <a href="https://www.ritchieng.com/machine-learning-polynomial-regression/" rel="nofollow noreferrer">https://www.ritchieng.com/machine-learning-polynomial-regression/</a></p>
  
  <p>[3] <a href="https://arachnoid.com/polysolve/" rel="nofollow noreferrer">https://arachnoid.com/polysolve/</a></p>
  
  <p>[4] <a href="https://www.youtube.com/watch?v=F2lJ7oSwcyY" rel="nofollow noreferrer">https://www.youtube.com/watch?v=F2lJ7oSwcyY</a></p>
  
  <p>[5] <a href="https://missinglink.ai/guides/neural-network-concepts/backpropagation-neural-networks-process-examples-code-minus-math/" rel="nofollow noreferrer">https://missinglink.ai/guides/neural-network-concepts/backpropagation-neural-networks-process-examples-code-minus-math/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-03T15:17:58.960Z
Answers.Text: Thank Your for the detailed reply. I have begun learning Neural Networks and have also found some papers related to my application. The links provided by you are awesome.
Answers.UserDisplayName: Pe Dro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T18:40:12.723Z
Answers.LastActivityDate: 2019-09-02T18:40:12.723Z
Answers.UserDisplayName: mico
Body: <p>I would like to take in some input values for <span class="math-container">$n$</span> variables, say <span class="math-container">$R$</span>, <span class="math-container">$B$</span>, and <span class="math-container">$G$</span>. Let <span class="math-container">$Y$</span> denote the response variable of these <span class="math-container">$n$</span> inputs (in this example, we have <span class="math-container">$3$</span> inputs). Other than these, I would like to use a reference/target value to compare the results. </p>
  
  <p>Now, suppose the relation between the inputs (<span class="math-container">$R$</span>, <span class="math-container">$B$</span> and <span class="math-container">$G$</span>) with the output <span class="math-container">$Y$</span> is (let's say):</p>
  
  <p><span class="math-container">$$Y = R + B + G$$</span></p>
  
  <p>But the system/machine has no knowledge of this relation. It can only read its inputs, <span class="math-container">$R$</span>, <span class="math-container">$B$</span> and <span class="math-container">$G$</span>, and the output, <span class="math-container">$Y$</span>. Also, the system is provided with the reference value, say, <span class="math-container">$\text{REF} = 30$</span> (suppose). </p>
  
  <p>The aim of the machine is to find this relation between its inputs and output(s). For this, I have come across some quite useful material online like <a href="https://ai.stackexchange.com/q/7434/2444">this forum query</a> and <a href="https://pdfs.semanticscholar.org/05ce/b32839c26c8d2cb38d5529cf7720a68c3fab.pdf" rel="nofollow noreferrer">Approximation by Superpositions of a Sigmoidal Function</a> by <strong>G. Cybenko</strong> and felt that it were possible. Also, I doubt that <strong>Polynomial Regression</strong> may be helpful as suggested <a href="https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/1467-9868.00192" rel="nofollow noreferrer">Here</a>.</p>
  
  <p>One vague approach that comes to my mind is to use a truth table like approach to somehow deduce the effect of the inputs on the output and hence, get a function for it. But neither am I sure how to proceed with it, nor do I trust its credibility. </p>
  
  <p>Is there any alternative/already existing method to accomplish this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-31T18:11:08.310Z
Comments.Text: I would like a direction for that. I know it is related to machine learning and artificial intelligence. Had I had the liberty to start from the very basic of ML, I would love it. But time bounds me.
Comments.UserDisplayName: Pe Dro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-04T13:44:46.140Z
Comments.Text: @DuttaA Actually this isn't a standard ML problem. If $Y$ can be any arbitrary function of the inputs, ML is usually used to find a black box model to predict $Y$ and not care about the question "what is the actual function?" The OP chose an unfortunate example where the question "what is the function?" can be answered exactly using linear regression. But what if it was something like $Y = \sin(x)/x$? A neural net can approximate it well within a certain range. But how can we actually *learn* that $Y = \sin(x)/x$? This is not a standard ML task.
Comments.UserDisplayName: Bridgeburners
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-04T14:22:47.060Z
Comments.Text: @Bridgeburners As you say that the problem quoted here can be solved using linear regression, you would be surprised to know that my immediate need is exactly  solving this exact equation.i.e. R,G,B and Y. But inevitably, I will need to solve more complicated equations like the Sine function etc.
Comments.UserDisplayName: Pe Dro
Tags: <machine-learning><math>
Title: How can I determine the mathematical relation between the input and output variables?
UserDisplayName: Pe Dro
ThreadText: # How can I determine the mathematical relation between the input and output variables?
  
  (Asked by: username_0 on 2019-08-30)
  
  I would like to take in some input values for $n$ variables, say $R$, $B$, and $G$. Let $Y$ denote the response variable of these $n$ inputs (in this example, we have $3$ inputs). Other than these, I would like to use a reference/target value to compare the results.
  
  Now, suppose the relation between the inputs ($R$, $B$ and $G$) with the output $Y$ is (let's say):
  
  $$Y = R + B + G$$
  
  But the system/machine has no knowledge of this relation. It can only read its inputs, $R$, $B$ and $G$, and the output, $Y$. Also, the system is provided with the reference value, say, $\text{REF} = 30$ (suppose).
  
  The aim of the machine is to find this relation between its inputs and output(s). For this, I have come across some quite useful material online like [this forum query](https://ai.stackexchange.com/q/7434/2444) and [Approximation by Superpositions of a Sigmoidal Function](https://pdfs.semanticscholar.org/05ce/b32839c26c8d2cb38d5529cf7720a68c3fab.pdf) by **G. Cybenko** and felt that it were possible. Also, I doubt that **Polynomial Regression** may be helpful as suggested [Here](https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/1467-9868.00192).
  
  One vague approach that comes to my mind is to use a truth table like approach to somehow deduce the effect of the inputs on the output and hence, get a function for it. But neither am I sure how to proceed with it, nor do I trust its credibility.
  
  Is there any alternative/already existing method to accomplish this?
  
  **Tags:** machine-learning, math
  
  > **Comments:**
  > **username_0**: I would like a direction for that. I know it is related to machine learning and artificial intelligence. Had I had the liberty to start from the very basic of ML, I would love it. But time bounds me.  
  > **username_1**: @DuttaA Actually this isn't a standard ML problem. If $Y$ can be any arbitrary function of the inputs, ML is usually used to find a black box model to predict $Y$ and not care about the question "what is the actual function?" The OP chose an unfortunate example where the question "what is the function?" can be answered exactly using linear regression. But what if it was something like $Y = \sin(x)/x$? A neural net can approximate it well within a certain range. But how can we actually *learn* that $Y = \sin(x)/x$? This is not a standard ML task.  
  > **username_0**: @username_1 As you say that the problem quoted here can be solved using linear regression, you would be surprised to know that my immediate need is exactly  solving this exact equation.i.e. R,G,B and Y. But inevitably, I will need to solve more complicated equations like the Sine function etc.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2020-01-19. Score: 0 | Accepted answer
  
  I found a solution to this some time back. I have been studying function approximation (within linear regression) for some time.
  Here's how I did it:
  
  [Neural Networks have been proved to be universal function approximators](http://neuralnetworksanddeeplearning.com/chap4.html). So, even a single hidden layer would be sufficient to approximate a function simple as **addition** (Even somewhat complex functions like the **[Sine](https://www.codeproject.com/Articles/1237026/Simple-MLP-Backpropagation-Artificial-Neural-Netwo)** and **any random CONTINUOUS [wiggly function](https://colab.research.google.com/drive/1K9agqd9Pn8llY1uwkFD68YqFgPeSTBCQ)** have been approximated)
  
  First, I used a high level API like TensorFlow and Keras and implemented it [here](https://colab.research.google.com/drive/1s8VYf5tYGj83SxOtJQbiKs7AJd_eNQX5)
  
  The model was trained on the data (input-output pairs)
  
  ```
  R    = np.array([-4, -10,  -2,  8, 5, 22,  3],  dtype=float)
  B    = np.array([4, -10,  0,  0, 15, 5,  1],  dtype=float)
  G    = np.array([0, 10,  5,  8, 1, 2,  38],  dtype=float)
  
  Y    = np.array([0, -10, 3, 16, 21, 29, 42],  dtype=float)
  
  ```
  
  And trained as follows:
  
  ```
  #Create a hidden layer with 2 neurons
  hidden = tf.keras.layers.Dense(units=2, input_shape=[3])
  
  #Create the output (final) layer which symbolises value of **Y**
  output = tf.keras.layers.Dense(units=1)
  
  #Combine layers to form the neural network and compile it
  model = tf.keras.Sequential([hidden, output])
  model.compile(loss='mean_squared_error', optimizer=tf.keras.optimizers.Adam(0.1))
  history = model.fit(RBG,Y, epochs=500, verbose=False)
  
  ```
  
  The model converges in about 50 epochs
  
  [![enter image description here](https://i.sstatic.net/97HZy.png)](https://i.sstatic.net/97HZy.png)
  
  Also, I have implemented the same using only C/C++ and used GNU plot to visualize the results.
  
  
  ---
  
  ### Answer by username_2 on 2019-09-02. Score: 1
  
  There are always a large number of possible functions that can produce a given set of input-output values. The challenge is to find a *simplest* function (according to whatever criteria you choose) to produce those values.
  
  One approach is to write a general function of the input variables, comprising terms of all order in R, G, and B, with a coefficient for each term, then search for values of the coefficients that A) reproduce the known input-output values accurately and B) leave the largest number of coefficients equal to zero.
  
  Several different algorithms can be used to do the search efficiently. My choice would be a genetic algorithm to seek the minimum of the RMS difference between the produced and known I-O values, summed with the product of a gradually increasing parameter and the number of nonzero coefficients.
  
  > **Comments:**
  > **username_0**: I have brushed over Genetic Algorithm and found it to be really novel for me and very useful. But, I need to deploy the trained model onto a MCU and hence it will be too bulky for the device.  
  
  ---
  
  ### Answer by username_3 on 2019-09-02. Score: 0
  
  As I see your explanation, what you're looking for can be stated via another way saying
  
  ```
  Y = aX + b
  
  ```
  
  where Y is output vector, X input vector and a & b are the coefficients you want to find.
  
  **Why so? And what is happening?**
  
  First thing first: I recommend to look the video [4] about how matrices and vectors work together and form after multiplication very familiar equations:
  
  ```
  Y = a1 * x1 + a2 * x2 + a3 * x3
  
  ```
  
  Now you see, you do not only get R + G + B but also some constants supplied for each of the variables.
  
  About polynomial solutions I found [2] and [3] but reading through [3] you'll soon notice it is about completely different approach:
  
  ```
  Y = a1 * x + a2 * x^2 + a3 * x^3
  
  ```
  
  which you don't want.
  
  So, you find something called linear regression and a so called Deep Neural Network for example to solve it [1].
  
  I would summarize the source [1] in these steps:
  
  * You have to find some training data. That is examples of correct Y values, when X values are known.
  * Then you build in Python note book some code that has: a Neural Network with hidden layer(s), Activation Function, Back Propagation and Objective Function.
  * Many many iterations with the data, called Training.
  
  There are plenty of samples and courses online about these in detail, but with correct tools and tutorials some dozen lines and no more are needed.
  
  Process is ended with validation phase with some more known data items and results. It can tell you how well the estimated model works.
  
  ---
  
  **Final Notes:**
  
  *As you may observe, the solution includes quite a many funny terms you have to learn somehow before mastering the task. For example Udemy has great online courses on this topic, also free tutorials are available on another sites. Your plans sound quite ambitious compared to the knowledge you have so far, so I really do recommend you learn little bit more to be able to fine tune the already given examples online. For example tutorial [5] includes one. It is at first quite complicated code, you'd need quite a lot practice to master it line by line.*
  
  ---
  
  **In short:**
  
  Find your favorite tutorial, study neural networks a little bit (basics) and pick code sample to start experimenting. It is a long way but it is worth it.
  
  ---
  
  **Source:**
  
  [1] <https://lightsapplications.wordpress.com/linear-regression-and-deep-learning/>
  
  [2] <https://www.ritchieng.com/machine-learning-polynomial-regression/>
  
  [3] <https://arachnoid.com/polysolve/>
  
  [4] <https://www.youtube.com/watch?v=F2lJ7oSwcyY>
  
  [5] <https://missinglink.ai/guides/neural-network-concepts/backpropagation-neural-networks-process-examples-code-minus-math/>
  
  > **Comments:**
  > **username_0**: Thank Your for the detailed reply. I have begun learning Neural Networks and have also found some papers related to my application. The links provided by you are awesome.  
  
  ---
  
  