Answers.Body: <p><a href="https://www.wikiwand.com/en/Universal_approximation_theorem" rel="nofollow noreferrer">The universal approximation theorem</a> states that, theoretically, any function (trivially including sine waves) can be approximated using a neural network of large enough size. Neural networks include RNNs (i.e. vanillar, GRU, LSTM etc).</p>
  <p>Practically, RNNs can definitely approximate a sine function. Research on GANs has had various papers reproducing sine waves (of different frequencies and phases) given random input (<a href="https://arxiv.org/abs/1706.02633" rel="nofollow noreferrer">1</a>, <a href="https://proceedings.neurips.cc/paper_files/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf" rel="nofollow noreferrer">2</a>).</p>
  <p>If you would only want to generate a sine wave with one phase and one frequency, then simply train the discriminator of the GANs to do so, and you easily have a generator RNN which generates approximately perfect sine waves. In such a case (without any desired variation in the outputs), it is enough to simply provide 1s as input to all timesteps, giving you the 'no input' criterion.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-08T14:14:57.430Z
Answers.Text: Thanks. It took me some time to read the two links. I learnt a lot, but do not fully understand the relevance (or my question was not clear). Why is it a GAN project? I do not want to make something "like" a sinus curve, but I do want the system to learn a particular, given sinus curve. With other words, I want a generator RNN to give me an output that predicts my known curve. For every datapoint I can do a backpropagation to get a better generator, but I am lost with the cost function of it (instead of a prediction, I get a moving average, i.e. a sinus, just different).
Answers.UserDisplayName: jollytall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-08T16:03:21.013Z
Answers.Text: GANs are one type of generative model. Other types of architectures can also be trained but i just thought of these papers which show the sine wave generation and hence answered your original question. However for getting advice on a project, you might need to open a new question because the comments are not for this kind of discussions.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-08T16:26:54.150Z
Answers.Text: Sorry if my original question was not clear, but it always was about to predict a - given, existing - sinus curve and not to generate one that looks like. So even opening a new question would be the same, but anyway thanks.
Answers.UserDisplayName: jollytall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-08T17:39:25.207Z
Answers.Text: You could do that with the given architectures. Simply detach the discriminator, and generate the first t steps and check the latest step. You could probably do it better simply using a supervised rnn. But the papers were simply to show that it was possible and done in literature. If you want to simply generate a given/existing sine wave from an integer t, just do it in a supervised learning fashion.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-07T12:25:12.350Z
Answers.LastActivityDate: 2023-05-07T12:25:12.350Z
Answers.UserDisplayName: Robin van Hoorn
Answers.Body: <p>Consider that RNNs have an <em>internal state</em>, <span class="math-container">$h$</span>, apart from the input sequence (and in particular, LSTMs have two of them), that got updated at each timestep of the input sequence during back-propagation through-time.</p>
  <blockquote>
  <p>I am wondering if THEORETICALLY and practically, an RNN can be constructed that takes no input (or if technically needed one x = 1) and still predicts a sinus curve?</p>
  </blockquote>
  <p>Well, an input is always required and you can set it to a constant value, each time. Anyway, you could leverage the hidden state at time <span class="math-container">$t$</span> to predict the next sine value at time <span class="math-container">$t+1$</span>: you feed it as an input to the RNN. This may work but I guess only in a sequential manner, I mean you can't predict values at arbitrary points of the curve but only the next one in time: assuming <span class="math-container">$f$</span> is the RNN, you have <span class="math-container">$h_{t+1}, \hat y_{t+1} = f(h_t)$</span> i.e. the next internal state <span class="math-container">$h_{t+1}$</span>, and the estimated sine value <span class="math-container">$\hat y_{t+1}$</span> also at the next timestep <span class="math-container">$t+1$</span>.</p>
  <blockquote>
  <p>Extra version: Let there be one input parameter, that is the time passed since the last point. The expected prediction is again, the sinus value after that passed time.</p>
  </blockquote>
  <p>This is more reasonable. You can implement it by either let the timestep by the only input, or from the previous idea also use the last hidden state i.e. <span class="math-container">$h_{t+1}, \hat y_{t+t'} = f(t'; h_t)$</span> where <span class="math-container">$t'$</span> is the time passed since last point.</p>
  <p>Anyway, in both cases you should use a mean squared error, and appropriately set the target sine values according to the point in time you want to predict.</p>
  <p>(Regarding the universal approximation theorem: RNNs are <em>universal computers</em>, meaning that they can approximate/learn whatever computable program, at least in principle. There should be at least one paper about this, see the works of Juergen Schmidhuber.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-12T05:25:50.203Z
Answers.Text: How can the second be more reasonable (i.e. easier - I guess)? The first is a special case of the second, when the only input (i.e. time passed) is always the same.   
  I actually tried the same as said, i.e. use LSTM (1000 h and cs values), a one layer DNN (1000:1 no AF) to get from cs the $\hat{y}_{t+1}$ value and a error square as the CF for backpropagation. What I got was moving average, hence the question.
Answers.UserDisplayName: jollytall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-12T05:29:03.553Z
Answers.Text: On the approximation theorem: It is clear that any continuous function can be approximated by a NN. The question, if with the given model (i.e. in every step no input, or max the time passed; no previous values fed as input), the model can theoretically learn it.
Answers.UserDisplayName: jollytall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-11T17:55:57.300Z
Answers.LastActivityDate: 2023-05-11T17:55:57.300Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I read a number of tutorials on how to make an RNN (simple, LSTM, etc.) that predicts a sinus curve. They all use as an input (x) in every step a set of past sinus values.</p>
  <p>I am wondering if THEORETICALLY and if yes, practically as well, can an RNN be constructed that takes no input (or if technically needed one x = 1) and still predicts a sinus curve?</p>
  <p>A bit of details:</p>
  <p>Let's assume that the &quot;inputs&quot; (but as said the input is none) are coming on a time scale equidistantly and the expected output is a sinus curve. The timestep is significantly smaller than 2PI and the ratio to 2PI is non-rational (i.e. the same value is never repeated). E.g. assume that the timestep is 0.1 (so more than 62 points for one cycle). The feedback for backpropagation can be anything, like the sinus value that was expected, or the error square, or anything else. The number of cell states is also freely choosable.
  When I tried to construct such a model, instead of a predicted sinus curve, I got a moving average of past values.</p>
  <p>Extra version (if the above simple could work): Let there be one input parameter, that is the time passed since the last point. The expected prediction is again, the sinus value after that passed time. Here again, we can assume that the time steps (not equidistant in this case) are still significantly smaller than the cycle length of sinus.</p>
  
Tags: <recurrent-neural-networks><function-approximation>
Title: Can an RNN predict a sinus curve with no input?
UserDisplayName: jollytall
ThreadText: # Can an RNN predict a sinus curve with no input?
  
  (Asked by: username_0 on 2023-05-06)
  
  I read a number of tutorials on how to make an RNN (simple, LSTM, etc.) that predicts a sinus curve. They all use as an input (x) in every step a set of past sinus values.
  
  I am wondering if THEORETICALLY and if yes, practically as well, can an RNN be constructed that takes no input (or if technically needed one x = 1) and still predicts a sinus curve?
  
  A bit of details:
  
  Let's assume that the "inputs" (but as said the input is none) are coming on a time scale equidistantly and the expected output is a sinus curve. The timestep is significantly smaller than 2PI and the ratio to 2PI is non-rational (i.e. the same value is never repeated). E.g. assume that the timestep is 0.1 (so more than 62 points for one cycle). The feedback for backpropagation can be anything, like the sinus value that was expected, or the error square, or anything else. The number of cell states is also freely choosable.
  When I tried to construct such a model, instead of a predicted sinus curve, I got a moving average of past values.
  
  Extra version (if the above simple could work): Let there be one input parameter, that is the time passed since the last point. The expected prediction is again, the sinus value after that passed time. Here again, we can assume that the time steps (not equidistant in this case) are still significantly smaller than the cycle length of sinus.
  
  **Tags:** recurrent-neural-networks, function-approximation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-05-07. Score: 1
  
  [The universal approximation theorem](https://www.wikiwand.com/en/Universal_approximation_theorem) states that, theoretically, any function (trivially including sine waves) can be approximated using a neural network of large enough size. Neural networks include RNNs (i.e. vanillar, GRU, LSTM etc).
  
  Practically, RNNs can definitely approximate a sine function. Research on GANs has had various papers reproducing sine waves (of different frequencies and phases) given random input ([1](https://arxiv.org/abs/1706.02633), [2](https://proceedings.neurips.cc/paper_files/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf)).
  
  If you would only want to generate a sine wave with one phase and one frequency, then simply train the discriminator of the GANs to do so, and you easily have a generator RNN which generates approximately perfect sine waves. In such a case (without any desired variation in the outputs), it is enough to simply provide 1s as input to all timesteps, giving you the 'no input' criterion.
  
  > **Comments:**
  > **username_0**: Thanks. It took me some time to read the two links. I learnt a lot, but do not fully understand the relevance (or my question was not clear). Why is it a GAN project? I do not want to make something "like" a sinus curve, but I do want the system to learn a particular, given sinus curve. With other words, I want a generator RNN to give me an output that predicts my known curve. For every datapoint I can do a backpropagation to get a better generator, but I am lost with the cost function of it (instead of a prediction, I get a moving average, i.e. a sinus, just different).  
  > **username_1**: GANs are one type of generative model. Other types of architectures can also be trained but i just thought of these papers which show the sine wave generation and hence answered your original question. However for getting advice on a project, you might need to open a new question because the comments are not for this kind of discussions.  
  > **username_0**: Sorry if my original question was not clear, but it always was about to predict a - given, existing - sinus curve and not to generate one that looks like. So even opening a new question would be the same, but anyway thanks.  
  > **username_1**: You could do that with the given architectures. Simply detach the discriminator, and generate the first t steps and check the latest step. You could probably do it better simply using a supervised rnn. But the papers were simply to show that it was possible and done in literature. If you want to simply generate a given/existing sine wave from an integer t, just do it in a supervised learning fashion.  
  
  ---
  
  ### Answer by username_2 on 2023-05-11. Score: 1
  
  Consider that RNNs have an *internal state*, $h$, apart from the input sequence (and in particular, LSTMs have two of them), that got updated at each timestep of the input sequence during back-propagation through-time.
  
  > I am wondering if THEORETICALLY and practically, an RNN can be constructed that takes no input (or if technically needed one x = 1) and still predicts a sinus curve?
  
  Well, an input is always required and you can set it to a constant value, each time. Anyway, you could leverage the hidden state at time $t$ to predict the next sine value at time $t+1$: you feed it as an input to the RNN. This may work but I guess only in a sequential manner, I mean you can't predict values at arbitrary points of the curve but only the next one in time: assuming $f$ is the RNN, you have $h\_{t+1}, \hat y\_{t+1} = f(h\_t)$ i.e. the next internal state $h\_{t+1}$, and the estimated sine value $\hat y\_{t+1}$ also at the next timestep $t+1$.
  
  > Extra version: Let there be one input parameter, that is the time passed since the last point. The expected prediction is again, the sinus value after that passed time.
  
  This is more reasonable. You can implement it by either let the timestep by the only input, or from the previous idea also use the last hidden state i.e. $h\_{t+1}, \hat y\_{t+t'} = f(t'; h\_t)$ where $t'$ is the time passed since last point.
  
  Anyway, in both cases you should use a mean squared error, and appropriately set the target sine values according to the point in time you want to predict.
  
  (Regarding the universal approximation theorem: RNNs are *universal computers*, meaning that they can approximate/learn whatever computable program, at least in principle. There should be at least one paper about this, see the works of Juergen Schmidhuber.)
  
  > **Comments:**
  > **username_0**: How can the second be more reasonable (i.e. easier - I guess)? The first is a special case of the second, when the only input (i.e. time passed) is always the same.    I actually tried the same as said, i.e. use LSTM (1000 h and cs values), a one layer DNN (1000:1 no AF) to get from cs the $\hat{y}_{t+1}$ value and a error square as the CF for backpropagation. What I got was moving average, hence the question.  
  > **username_0**: On the approximation theorem: It is clear that any continuous function can be approximated by a NN. The question, if with the given model (i.e. in every step no input, or max the time passed; no previous values fed as input), the model can theoretically learn it.  
  
  ---
  
  