Answers.Body: <p>If the concept class specified is</p>
  
  <p><span class="math-container">$$f(x, y) = k \, \sin(2 \pi f_x x) \, sin(2 \pi f_y y) \\ \land 0 &lt; x &lt; 1 \\ \land 0 &lt; y &lt; 1 \; \text{,}$$</span></p>
  
  <p>and the optimum fit to example data is expected occur when <span class="math-container">$k \approx 1 \land f_x \approx 1 \land f_y \approx 1$</span>, then it is not an AI problem. It is a problem that can be solved with a least squares convergence, probably in conjunction with a Fourier transform.</p>
  
  <p>If nothing is known about <span class="math-container">$f(x, y)$</span> except continuity and that it is single valued with respect to <span class="math-container">$(x, y)$</span>, then few conclusions can be drawn about best approach. In such a case, the domain of <span class="math-container">$x$</span> and <span class="math-container">$y$</span> are irrelevant because they can be normalized. Furthermore, the tree of operations, such as <span class="math-container">$\sin()$</span> and multiplication, are irrelevant too, because the function could just as easily be</p>
  
  <p><span class="math-container">$$f(x, y) = \ln(x) + \Gamma(y) - k \, \text{.}$$</span></p>
  
  <p>The question indicates the design involves CNN and RNN components for analyzing images and sequential data. It is not clear whether the CNN is for the discovery of objects or waves (given the <span class="math-container">$\sin()$</span> in the function mentioned) and whether those objects move between frames so that the RNN must detect motion.</p>
  
  <p>Nothing is given about the pool of example data available or planned to be available or the expected outputs of the system. If data is sequential, where is <span class="math-container">$t$</span> in the function? What is the objective of image analysis?</p>
  
  <p>Although a deep MLP (multilayer perceptron) with SGD can learn an arbitrary function, it is hardly an architecture, the mention of images, CNN, RNN, and sequential data, MLP with SGD does not seem to match.</p>
  
  <p>Regarding activation functions, the inner layer functions would depend on the higher level design requirements. The activation functions of the last layer of a single artificial network is usually chosen to match the data type and range of desired output for each output channel (dimension).</p>
  
  <p>If the objective of this question is to take images and sequential data and produce something useful without <em>a priori</em> defining what useful means, then it is an unsolved AI problem thus far and no known topology comprised of artificial networks and other AI building blocks provide a solution. The autonomous development of internal concepts of usefulness would need to be developed mathematically and algorithmically and become practically speed optimized in hardware and software first.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-11T08:53:52.680Z
Answers.LastActivityDate: 2018-12-11T08:53:52.680Z
Answers.UserDisplayName: Douglas Daseeco
Answers.Body: <p>For example, suppose I want to approximate the function
  mashine learning is used to approximate an UNKNOWN function. If you do want to do this with NN, you will have a regression task with small full connected network with like:</p>
  
  <ul>
  <li>Input is 2 values of range 0 ..1</li>
  <li>a few hidden layers, just start trying with one, of size like 3-6 neurons , activation  sigmoid or tanh</li>
  <li>need a non-linear function here to approx non linear sin * sin, whatever </li>
  <li>last layer with no activation function just making a difference from... known value f(x,y)</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-12T12:54:54.827Z
Answers.LastActivityDate: 2019-06-12T12:54:54.827Z
Answers.UserDisplayName: user8426627
Body: <p>We have convolutional neural networks and recurrent neural networks for analyzing, respectively, images and sequential data.</p>
  <p>Now, suppose I want to approximate the unknown function <span class="math-container">$f(x,y) = \sin(2\pi x)\sin(2\pi y)$</span>, with domain <span class="math-container">$\Omega = [0,1]\times [0,1]$</span>, that is, <span class="math-container">$x$</span> and <span class="math-container">$y$</span> can be between <span class="math-container">$0$</span> and <span class="math-container">$1$</span> (inclusive).</p>
  <p>How do I determine which neural network architecture is more appropriate to approximate this function? Which kind of activation functions would be better suited for this?</p>
  <p>Note that, generally, I don't know a priori which function the neural network has to learn. I am just asking for this specific <span class="math-container">$f(x, y)$</span>, as it could be a solution for a differential equation. And <span class="math-container">$\Omega$</span> is the domain, i.e., I don't care about the output of the neural network outside <span class="math-container">$\Omega$</span>.</p>
  
Tags: <neural-networks><activation-functions><function-approximation><model-request><network-design>
Title: Which neural network should I use to approximate a specific but unknown function?
UserDisplayName: timudk
ThreadText: # Which neural network should I use to approximate a specific but unknown function?
  
  (Asked by: username_0 on 2018-12-03)
  
  We have convolutional neural networks and recurrent neural networks for analyzing, respectively, images and sequential data.
  
  Now, suppose I want to approximate the unknown function $f(x,y) = \sin(2\pi x)\sin(2\pi y)$, with domain $\Omega = [0,1]\times [0,1]$, that is, $x$ and $y$ can be between $0$ and $1$ (inclusive).
  
  How do I determine which neural network architecture is more appropriate to approximate this function? Which kind of activation functions would be better suited for this?
  
  Note that, generally, I don't know a priori which function the neural network has to learn. I am just asking for this specific $f(x, y)$, as it could be a solution for a differential equation. And $\Omega$ is the domain, i.e., I don't care about the output of the neural network outside $\Omega$.
  
  **Tags:** neural-networks, activation-functions, function-approximation, model-request, network-design
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-12-11. Score: 1
  
  If the concept class specified is
  
  $$f(x, y) = k \, \sin(2 \pi f\_x x) \, sin(2 \pi f\_y y) \\ \land 0 < x < 1 \\ \land 0 < y < 1 \; \text{,}$$
  
  and the optimum fit to example data is expected occur when $k \approx 1 \land f\_x \approx 1 \land f\_y \approx 1$, then it is not an AI problem. It is a problem that can be solved with a least squares convergence, probably in conjunction with a Fourier transform.
  
  If nothing is known about $f(x, y)$ except continuity and that it is single valued with respect to $(x, y)$, then few conclusions can be drawn about best approach. In such a case, the domain of $x$ and $y$ are irrelevant because they can be normalized. Furthermore, the tree of operations, such as $\sin()$ and multiplication, are irrelevant too, because the function could just as easily be
  
  $$f(x, y) = \ln(x) + \Gamma(y) - k \, \text{.}$$
  
  The question indicates the design involves CNN and RNN components for analyzing images and sequential data. It is not clear whether the CNN is for the discovery of objects or waves (given the $\sin()$ in the function mentioned) and whether those objects move between frames so that the RNN must detect motion.
  
  Nothing is given about the pool of example data available or planned to be available or the expected outputs of the system. If data is sequential, where is $t$ in the function? What is the objective of image analysis?
  
  Although a deep MLP (multilayer perceptron) with SGD can learn an arbitrary function, it is hardly an architecture, the mention of images, CNN, RNN, and sequential data, MLP with SGD does not seem to match.
  
  Regarding activation functions, the inner layer functions would depend on the higher level design requirements. The activation functions of the last layer of a single artificial network is usually chosen to match the data type and range of desired output for each output channel (dimension).
  
  If the objective of this question is to take images and sequential data and produce something useful without *a priori* defining what useful means, then it is an unsolved AI problem thus far and no known topology comprised of artificial networks and other AI building blocks provide a solution. The autonomous development of internal concepts of usefulness would need to be developed mathematically and algorithmically and become practically speed optimized in hardware and software first.
  
  
  ---
  
  ### Answer by username_2 on 2019-06-12. Score: -1
  
  For example, suppose I want to approximate the function
  mashine learning is used to approximate an UNKNOWN function. If you do want to do this with NN, you will have a regression task with small full connected network with like:
  
  * Input is 2 values of range 0 ..1
  * a few hidden layers, just start trying with one, of size like 3-6 neurons , activation sigmoid or tanh
  * need a non-linear function here to approx non linear sin \* sin, whatever
  * last layer with no activation function just making a difference from... known value f(x,y)
  
  
  ---
  
  