Answers.Body: <p>The starting point is that for a fair dice thrown fairly the p(n) is 1/n where n is the number of sides.</p>
  <p>You said both</p>
  <p>and</p>
  <blockquote>
  <p>there are too many variables (up 40 dimensions with value range 1-100) in input, I don't know how these properties relate and an empirical approach would require too much data.</p>
  </blockquote>
  <p>It seems that this problem has 2 solutions:</p>
  <ol>
  <li>Don't use a neural net and create a 'std' statistical model.
  It may be possible since you said:</li>
  </ol>
  <blockquote>
  <p>I know there is some underlying rule that simplify a lot the problem (ie. reduce the actual number of dimension of the input)</p>
  </blockquote>
  <ol>
  <li>Use a neural network (with softmax at the end) - for a fair dice; with enough training data the classifier should arrive as 1/n as the approximating function for a fair dice. The other 40 dimensions/settings your mentioned are the inputs. I think a 'basic' neural network with dense layers only could work for your task.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-03T21:52:45.590Z
Answers.LastActivityDate: 2020-11-16T05:21:27.817Z
Answers.LastEditDate: 2020-11-16T05:21:27.817Z
Answers.UserDisplayName: tmaj
Answers.Body: <p>Neural network isnt what you want here. You have a limited number of events and draws from some unknown distribution that you want to recover.  </p>
  
  <p>In that case, just use the empirical probabilities <span class="math-container">$p(event_i) = \frac{\# event_i}{total\ events}$</span> which given enough draws will converge to the true probability. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-03T19:25:53.173Z
Answers.Text: I'm not sure I can do that because there are too many variables (up 40 dimensions with value range 1-100) in input, I don't know how these properties relate and an empirical approach would require too much data. But it could be a valid approach if I make some assumptions on the model to simplify it.
Answers.UserDisplayName: Steve Marion
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-03T20:05:25.360Z
Answers.Text: If you cannot take an empirical approach how will you generate training data for your ML algorithm?
Answers.UserDisplayName: DrMcCleod
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-03T17:48:38.393Z
Answers.LastActivityDate: 2019-07-03T17:48:38.393Z
Answers.UserDisplayName: mshlis
Answers.Body: <p>This approach:</p>
  
  <blockquote>
    <p>train it as a classification problem which output confidence, and hope that the confidence will reflect the actual probability. Sometimes the network would output the correct probability and fail the test, but on average it would tend to the correct probability.</p>
  </blockquote>
  
  <p>will work with some limitations. If you use a classifier with softmax activation and multiclass log loss</p>
  
  <p><span class="math-container">$$\mathcal{L}(\mathbf{\hat{y}},\mathbf{y}) = -\mathbf{y} \cdot  \text{log}(\mathbf{\hat{y}})$$</span></p>
  
  <p>where <span class="math-container">$\mathbf{\hat{y}}$</span> is the network output as a vector, and <span class="math-container">$\mathbf{y}$</span> is the actual output from an individual sample. Your input should be the settings of the die. </p>
  
  <p>Optimising this loss will converge on approximated probabilities for each discrete output. You can demonstrate this with some simple examples - for instance if you train a network with a single input - a one-hot-encoded die type from the classic D&amp;D dice sets, plus deliberately chosen examples of different results in the right frequencies, you will end up with a classifier that predicts roughly <span class="math-container">$p=0.25$</span> for results of 1,2,3,4 for a d4 and <span class="math-container">$p=0.125$</span> for results of 1,2,3,4,5,6,7,8 for a d8</p>
  
  <p>So it works mathematically. Whether it works for your situation depends on details. You need enough data samples to cover both the distribution of results under each setting, and any complexities of how that distribution varies with the settings. In the limit of wanting very accurate predictions of probability within a complex space you will need a huge number of samples. You should be able to find a compromise between accuracy and generalisation by trying different levels of regularisation - this will be necessary as over-fitting to input/output sample pairs as seen is going to be a serious problem for a neural network trained on this data. </p>
  
  <p>One thing you can do to help a classifier learn probabilities is always take some number of samples with the same settings - e.g. 10 or 100 or 1000 each with the same settings - that should guarantee that the network cannot simply converge to predict high <span class="math-container">$p$</span> values for single outputs as seen, as it will have counter-examples to work with.</p>
  
  <p>You mention that you have 40 dimensions of settings. Whether this is an issue will depend on how the probability distribution varies based on those settings. However, at minimum you should be thinking in terms of millions of samples for training, or possibly a fast on-demand generator that can generate 1000s of new samples per second to train with.</p>
  
  <p>You can test accuracy by building histograms using some fixed (and as yet unseen) setting and comparing to NN predictions of probabilities of that setting. Even getting accurate test results is likely to require 1000s of samples.</p>
  
  <blockquote>
    <p>However it may require a lot of training and data.</p>
  </blockquote>
  
  <p>If you cannot obtain a very large training set here, then a purely statistical "black box" approach is probably not feasible, regardless of whether you use neural networks, or more raw analysis. What neural networks add is smooth interpolation between different settings values, as a form of approximation. This seems desirable for your problem, as you will never fully explore 40 dimensions of 100 values in the lifetime of the universe - but you need some confidence that minor changes in settings equate to minor changes in probability distributions in most cases. </p>
  
  <p>It's OK to have one or two major shifts across the input space, but if the distribution depends in some cryptographic primitive or similar complex high frequency (over space) and high amplitude on the input variables, there is no way to obtain approximations using statistics.</p>
  
  <p>The alternative to statistical approaches is to find some way to break open this black box through analysis. No AI system can do that in general at the moment, so you would rely on human ingenuity.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-04T13:03:30.473Z
Answers.Text: while I have a lot of input variables, I know there is some underlying rule that simplify a lot the problem (ie. reduce the actual number of dimension of the input). if I apply a statistical analysis I would need to make assumption on the rules to simplify beforehand. I would rather the NN to infer it from its training. I can generate 1-2 tests/s. what about the 2nd approach of evaluating the loss on a group of test cases ?
Answers.UserDisplayName: Steve Marion
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-04T13:07:42.990Z
Answers.Text: @SteveMarion: In that case it may be tractable. You will still need a lot of training data for the network to learn about this embedded lower-dimension problem.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-04T13:11:18.980Z
Answers.Text: @SteveMarion: A NN **is** a statistical method. It  "infers" data that is available statistically. You don't need to pre-analyse anything before feeding into a NN, and I don't suggest you need to in this answer. With 1-2 tests/second it should take you about 2 weeks to generate a dataset of a million records. Assuming that is automatic and not a manual process, that seems quite a reasonable time investment for data gathering. Of course that number is a guess and there are ways to check whether more data would be more useful before making the effort
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-04T08:12:46.533Z
Answers.LastActivityDate: 2019-07-04T08:53:17.230Z
Answers.LastEditDate: 2019-07-04T08:53:17.230Z
Answers.UserDisplayName: Neil Slater
Body: <p>Let's say I have an adjustable loaded die, and I want to train a neural network to give me the probability of each face, depending on the settings of the loaded die.</p>
  
  <p>I can't mesure its performance on individual die roll, since it does not give me a probability.</p>
  
  <p>I could batch a lot of roll to calculate a probability and use this batch as an individual test case, but my problem does not allow this (let's say the settings are complex and randomized between each roll).</p>
  
  <p>I have 2 ideas:</p>
  
  <ol>
  <li>train it as a classification problem which output confidence, and hope that the confidence will reflect the actual probability. Sometimes the network would output the correct probability and fail the test, but on average it would tend to the correct probability. However it may require a lot of training and data.</li>
  <li>batch random rolls together and compare the mean/median/standard deviation of the measured result vs the predictions. It could work but I don't know the good batch size.</li>
  </ol>
  
  <p>Thank you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-03T19:33:14.077Z
Comments.Text: @DuttaA while a function cannot predict a random ouput, It could theorically predict probabilities, which are not random. either I need to built test cases by grouping individual entries into a statistical distribution, or find another clever method.
Comments.UserDisplayName: Steve Marion
Tags: <neural-networks><machine-learning>
Title: How can I train a neural network to give probability of a random event?
UserDisplayName: Steve Marion
ThreadText: # How can I train a neural network to give probability of a random event?
  
  (Asked by: username_0 on 2019-07-03)
  
  Let's say I have an adjustable loaded die, and I want to train a neural network to give me the probability of each face, depending on the settings of the loaded die.
  
  I can't mesure its performance on individual die roll, since it does not give me a probability.
  
  I could batch a lot of roll to calculate a probability and use this batch as an individual test case, but my problem does not allow this (let's say the settings are complex and randomized between each roll).
  
  I have 2 ideas:
  
  1. train it as a classification problem which output confidence, and hope that the confidence will reflect the actual probability. Sometimes the network would output the correct probability and fail the test, but on average it would tend to the correct probability. However it may require a lot of training and data.
  2. batch random rolls together and compare the mean/median/standard deviation of the measured result vs the predictions. It could work but I don't know the good batch size.
  
  Thank you.
  
  **Tags:** neural-networks, machine-learning
  > **Comments:**
  > **username_0**: @DuttaA while a function cannot predict a random ouput, It could theorically predict probabilities, which are not random. either I need to built test cases by grouping individual entries into a statistical distribution, or find another clever method.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-08-03. Score: 1
  
  The starting point is that for a fair dice thrown fairly the p(n) is 1/n where n is the number of sides.
  
  You said both
  
  and
  
  > there are too many variables (up 40 dimensions with value range 1-100) in input, I don't know how these properties relate and an empirical approach would require too much data.
  
  It seems that this problem has 2 solutions:
  
  1. Don't use a neural net and create a 'std' statistical model.
     It may be possible since you said:
  
  > I know there is some underlying rule that simplify a lot the problem (ie. reduce the actual number of dimension of the input)
  
  1. Use a neural network (with softmax at the end) - for a fair dice; with enough training data the classifier should arrive as 1/n as the approximating function for a fair dice. The other 40 dimensions/settings your mentioned are the inputs. I think a 'basic' neural network with dense layers only could work for your task.
  
  
  ---
  
  ### Answer by username_2 on 2019-07-03. Score: 0
  
  Neural network isnt what you want here. You have a limited number of events and draws from some unknown distribution that you want to recover.
  
  In that case, just use the empirical probabilities $p(event\_i) = \frac{\# event\_i}{total\ events}$ which given enough draws will converge to the true probability.
  
  > **Comments:**
  > **username_0**: I'm not sure I can do that because there are too many variables (up 40 dimensions with value range 1-100) in input, I don't know how these properties relate and an empirical approach would require too much data. But it could be a valid approach if I make some assumptions on the model to simplify it.  
  > **username_3**: If you cannot take an empirical approach how will you generate training data for your ML algorithm?  
  
  ---
  
  ### Answer by username_4 on 2019-07-04. Score: 0
  
  This approach:
  
  > train it as a classification problem which output confidence, and hope that the confidence will reflect the actual probability. Sometimes the network would output the correct probability and fail the test, but on average it would tend to the correct probability.
  
  will work with some limitations. If you use a classifier with softmax activation and multiclass log loss
  
  $$\mathcal{L}(\mathbf{\hat{y}},\mathbf{y}) = -\mathbf{y} \cdot \text{log}(\mathbf{\hat{y}})$$
  
  where $\mathbf{\hat{y}}$ is the network output as a vector, and $\mathbf{y}$ is the actual output from an individual sample. Your input should be the settings of the die.
  
  Optimising this loss will converge on approximated probabilities for each discrete output. You can demonstrate this with some simple examples - for instance if you train a network with a single input - a one-hot-encoded die type from the classic D&D dice sets, plus deliberately chosen examples of different results in the right frequencies, you will end up with a classifier that predicts roughly $p=0.25$ for results of 1,2,3,4 for a d4 and $p=0.125$ for results of 1,2,3,4,5,6,7,8 for a d8
  
  So it works mathematically. Whether it works for your situation depends on details. You need enough data samples to cover both the distribution of results under each setting, and any complexities of how that distribution varies with the settings. In the limit of wanting very accurate predictions of probability within a complex space you will need a huge number of samples. You should be able to find a compromise between accuracy and generalisation by trying different levels of regularisation - this will be necessary as over-fitting to input/output sample pairs as seen is going to be a serious problem for a neural network trained on this data.
  
  One thing you can do to help a classifier learn probabilities is always take some number of samples with the same settings - e.g. 10 or 100 or 1000 each with the same settings - that should guarantee that the network cannot simply converge to predict high $p$ values for single outputs as seen, as it will have counter-examples to work with.
  
  You mention that you have 40 dimensions of settings. Whether this is an issue will depend on how the probability distribution varies based on those settings. However, at minimum you should be thinking in terms of millions of samples for training, or possibly a fast on-demand generator that can generate 1000s of new samples per second to train with.
  
  You can test accuracy by building histograms using some fixed (and as yet unseen) setting and comparing to NN predictions of probabilities of that setting. Even getting accurate test results is likely to require 1000s of samples.
  
  > However it may require a lot of training and data.
  
  If you cannot obtain a very large training set here, then a purely statistical "black box" approach is probably not feasible, regardless of whether you use neural networks, or more raw analysis. What neural networks add is smooth interpolation between different settings values, as a form of approximation. This seems desirable for your problem, as you will never fully explore 40 dimensions of 100 values in the lifetime of the universe - but you need some confidence that minor changes in settings equate to minor changes in probability distributions in most cases.
  
  It's OK to have one or two major shifts across the input space, but if the distribution depends in some cryptographic primitive or similar complex high frequency (over space) and high amplitude on the input variables, there is no way to obtain approximations using statistics.
  
  The alternative to statistical approaches is to find some way to break open this black box through analysis. No AI system can do that in general at the moment, so you would rely on human ingenuity.
  
  > **Comments:**
  > **username_0**: while I have a lot of input variables, I know there is some underlying rule that simplify a lot the problem (ie. reduce the actual number of dimension of the input). if I apply a statistical analysis I would need to make assumption on the rules to simplify beforehand. I would rather the NN to infer it from its training. I can generate 1-2 tests/s. what about the 2nd approach of evaluating the loss on a group of test cases ?  
  > **username_4**: @SteveMarion: In that case it may be tractable. You will still need a lot of training data for the network to learn about this embedded lower-dimension problem.  
  > **username_4**: @SteveMarion: A NN **is** a statistical method. It  "infers" data that is available statistically. You don't need to pre-analyse anything before feeding into a NN, and I don't suggest you need to in this answer. With 1-2 tests/second it should take you about 2 weeks to generate a dataset of a million records. Assuming that is automatic and not a manual process, that seems quite a reasonable time investment for data gathering. Of course that number is a guess and there are ways to check whether more data would be more useful before making the effort  
  
  ---
  
  