Answers.Body: <p>I think there are two problems with your network.
  The first one, always having very similar outputs, is the rather simple one. As it seems, your network suffers from the so-called, very common <a href="https://developers.google.com/machine-learning/gan/problems" rel="nofollow noreferrer">Mode Collapse</a> problem. The attached link provides both an explanation and some potential remedy to that problem.</p>
  <p>The second problem is more fundamental. You say that you want your network to produce random numbers. Or numbers that at least appear as such. However, once training is finished, your model is going to be a static function which will not change any further. Given the same input x, it will always produce the same output y. Consequently, unless the inputs to your network contain some randomness already or are, at least, always slightly dissimilar, you will not end up having a random generator. So, whether that is going to be useful will depend on your usecase. But if you make sure that the true random variable (like date&amp;time) serves as input to the RNN and the RNN just translates this into some different format, that might work again. Just keep in mind that randomness can never arise out of a trained model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T05:15:20.407Z
Answers.Text: Thanks, I'll have a look at your link. Regarding randomness -- I agree, but the point of my demonstration is to partially illustrate the fact that trained models, [like human brains](https://www.reddit.com/r/programming/comments/1p3ti1/you_are_bad_at_entropy/), are poor sources of entropy. Nonetheless, I would expect that they can be as good as human brains are, and at least produce sequences that aren't completely *trivial* to predict.
Answers.UserDisplayName: Abhimanyu Pallavi Sudhir
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-23T19:53:16.337Z
Answers.Text: A RNN could theoretically emulate a simple PRNG, and I think that would serve OP's need. PRNGs are also deterministic and work with a starting seed value. The OP wants to generate sequences that *look* random, they have not asked for "true random", so I think a PRNG would be OK. However, I don't know how to create an RNN that could behave like a PRNG - what architecture and training would be likely to work.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-30T12:57:53.137Z
Answers.Text: The discriminator is going to have all the same data the generator does, so it should be able to predict all the numbers if it just copies the generator. But one hope is that perhaps it won't be able to converge towards the same weights just by seeing the output.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-09T10:19:12.717Z
Answers.Text: I suspect a fundamental problem here is how you determine what the correct outputs should be so that you can measure loss.  It's a very subjective problem.  Could the next number be equal to the last?  Certainly.  Could it be much higher?  Sure.  Much lower? Absolutely.  It sounds like OP is trying to capture the impression someone would have looking at the data.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-23T17:38:07.900Z
Answers.LastActivityDate: 2020-06-23T17:38:07.900Z
Answers.UserDisplayName: Daniel B.
Body: <p>I'm new to relatively RNNs, and I'm trying to train <em>generative</em> and <em>guessing</em> neural networks to produce sequences of real numbers that look random. My architecture looks like this (each &quot;circle&quot; in the output is the adverserial network's guess for the generated circle vertically below it -- having seen only the terms before it):</p>
  <p><img src="https://i.sstatic.net/fWSu9.png" alt="" /></p>
  <p>Note that the adverserial network is rewarded for predicting outputs <em>close</em> to the true values, i.e. the loss function looks like <code>tf.math.reduce_max((sequence - predictions) ** 2)</code> (I have also tried <code>reduce_mean</code>).</p>
  <p>I don't know if there's something obviously wrong with my architecture, but when I try to train this network (and I've added a reasonable number of layers), it doesn't really work very well.</p>
  <ul>
  <li><a href="https://colab.research.google.com/drive/1Zd3l1tW64r9OizML_Vz_J3cYpPlZzrap?usp=sharing" rel="nofollow noreferrer">Colab Notebook</a></li>
  </ul>
  <p>If you look at the result of the last code block, you'll see that my generative neural network produces things like</p>
  <ul>
  <li><code>[0.9907787, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827]</code></li>
  </ul>
  <p>But it could <em>easily</em> improve itself by simply training to jump around more, since you'll observe that the adverserial network also predicts numbers very close to the given number (even when the sequence it is given to predict is one that jumps around a lot!).</p>
  <p>What am I doing wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-09T10:17:03.257Z
Comments.Text: If you are trying for purely random numbers, how do you determine what a "true" value would be to measure loss?
Comments.UserDisplayName: David Hoelzer
Tags: <neural-networks><machine-learning><deep-learning><recurrent-neural-networks><generative-adversarial-networks>
Title: Why does my "entropy generation" RNN do so badly?
UserDisplayName: Abhimanyu Pallavi Sudhir
ThreadText: # Why does my "entropy generation" RNN do so badly?
  
  (Asked by: username_0 on 2020-06-23)
  
  I'm new to relatively RNNs, and I'm trying to train *generative* and *guessing* neural networks to produce sequences of real numbers that look random. My architecture looks like this (each "circle" in the output is the adverserial network's guess for the generated circle vertically below it -- having seen only the terms before it):
  
  ![](https://i.sstatic.net/fWSu9.png)
  
  Note that the adverserial network is rewarded for predicting outputs *close* to the true values, i.e. the loss function looks like `tf.math.reduce_max((sequence - predictions) ** 2)` (I have also tried `reduce_mean`).
  
  I don't know if there's something obviously wrong with my architecture, but when I try to train this network (and I've added a reasonable number of layers), it doesn't really work very well.
  
  * [Colab Notebook](https://colab.research.google.com/drive/1Zd3l1tW64r9OizML_Vz_J3cYpPlZzrap?usp=sharing)
  
  If you look at the result of the last code block, you'll see that my generative neural network produces things like
  
  * `[0.9907787, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827, 0.9907827]`
  
  But it could *easily* improve itself by simply training to jump around more, since you'll observe that the adverserial network also predicts numbers very close to the given number (even when the sequence it is given to predict is one that jumps around a lot!).
  
  What am I doing wrong?
  
  **Tags:** neural-networks, machine-learning, deep-learning, recurrent-neural-networks, generative-adversarial-networks
  > **Comments:**
  > **username_1**: If you are trying for purely random numbers, how do you determine what a "true" value would be to measure loss?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-06-23. Score: 1
  
  I think there are two problems with your network.
  The first one, always having very similar outputs, is the rather simple one. As it seems, your network suffers from the so-called, very common [Mode Collapse](https://developers.google.com/machine-learning/gan/problems) problem. The attached link provides both an explanation and some potential remedy to that problem.
  
  The second problem is more fundamental. You say that you want your network to produce random numbers. Or numbers that at least appear as such. However, once training is finished, your model is going to be a static function which will not change any further. Given the same input x, it will always produce the same output y. Consequently, unless the inputs to your network contain some randomness already or are, at least, always slightly dissimilar, you will not end up having a random generator. So, whether that is going to be useful will depend on your usecase. But if you make sure that the true random variable (like date&time) serves as input to the RNN and the RNN just translates this into some different format, that might work again. Just keep in mind that randomness can never arise out of a trained model.
  
  > **Comments:**
  > **username_0**: Thanks, I'll have a look at your link. Regarding randomness -- I agree, but the point of my demonstration is to partially illustrate the fact that trained models, [like human brains](https://www.reddit.com/r/programming/comments/1p3ti1/you_are_bad_at_entropy/), are poor sources of entropy. Nonetheless, I would expect that they can be as good as human brains are, and at least produce sequences that aren't completely *trivial* to predict.  
  > **username_3**: A RNN could theoretically emulate a simple PRNG, and I think that would serve OP's need. PRNGs are also deterministic and work with a starting seed value. The OP wants to generate sequences that *look* random, they have not asked for "true random", so I think a PRNG would be OK. However, I don't know how to create an RNN that could behave like a PRNG - what architecture and training would be likely to work.  
  > **username_4**: The discriminator is going to have all the same data the generator does, so it should be able to predict all the numbers if it just copies the generator. But one hope is that perhaps it won't be able to converge towards the same weights just by seeing the output.  
  > **username_1**: I suspect a fundamental problem here is how you determine what the correct outputs should be so that you can measure loss.  It's a very subjective problem.  Could the next number be equal to the last?  Certainly.  Could it be much higher?  Sure.  Much lower? Absolutely.  It sounds like OP is trying to capture the impression someone would have looking at the data.  
  
  ---
  
  