Answers.Body: <p>A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect relationships to be highly non-linear then additional layers might be required, but from your description of the problem, I would be surprised if you needed more than few layers, and a relatively small network.</p>
  <p>However, I note that:</p>
  <blockquote>
  <p>The input variables are numerous but not that many, maybe 7-10.</p>
  </blockquote>
  <p>This gives you <span class="math-container">$3^{10} = 59049$</span> possible inputs. That's not much in terms of amount of data needed for ML statistical models. Assuming that even the best predictions are still probabilistic, then you may only need a million or so examples to create a reasonably accurate lookup table, not needing a neural network at all.</p>
  <p>The strength of a neural network is to be able to generalise well from less examples than that. Of course, this is not perfect, but it would be able to do things such as notice if inputs 1,2 and 3 agree then that is always the most likely answer. If that turns out to be true (and not an accident of having low numbers of samples), then the NN could learn that useful pattern using far less data than a table-based approach.</p>
  <blockquote>
  <p>I have read that feedforward nns have limitations regarding control flow and loops, so I'm not sure if that structure will be appropriate.</p>
  </blockquote>
  <p>This is true, but does not impact your situation, because there is no control flow or loops involved. You have described a simple function. Whilst you or I might inspect the data and look backwards and forwards across it before coming to a decision, a neural network approximating a function does not need to do that, and in simple cases there is usually no benefit to doing so - a statistical summary of the correct mapping from input to output is more than sufficient and likely the best that can be done.</p>
  <blockquote>
  <p>I'm not sure if recurrent nn will be appropriate either as I don't care what the previous inputs were.</p>
  </blockquote>
  <p>As all your inputs represent the same kind of thing, you could implement as a RNN with a single input, -1, 0 or +1, always feeding in the predictions by type in the same order. It might resemble how you are thinking about the problem as a human (at least a better analogy than the direct statistical match in a feed-forward network), especially if you implemented a variant of attention. However, I don't think there would be any benefit to that in improved accuracy, and it would be a significant challenge to build that if you are new to AI.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T10:40:26.463Z
Answers.Text: Thanks for the answer. What if the inputs were fuzzy? Say instead of -1,0,1 it predicts more like -0.9, 0.1, 0.8 ? Does this change anything? What if the input values were higher than 1, would the nn normalize that automatically? Should I use relu?
Answers.UserDisplayName: minusatwelfth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T13:09:36.633Z
Answers.Text: @minusatwelfth If the inputs were fuzzy then you could not use a tabular approach as suggested in this answer, because there would be too many input variations. The NN would still work just fine though, and would treat e.g. -0.8 as similar to -0.7 (you probably want that). Neural networks do not automatically normalize inputs. An input range between -2 and +2 would be safe enough, up to you whether you want to clean them up before using. If the values could be really high you would want to pre-process before feeding to the NN, because NNs cope badly with large magnitude inputs in general.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T13:51:36.097Z
Answers.Text: Thanks, that's pretty much everyt-... I read somewhere that I should try the weights to be low numbers first (-1 to 1). Is this true? With the platform im working with, i cant afford to optimize for every single weight value as it would take too long to compute.
Answers.UserDisplayName: minusatwelfth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T13:54:09.993Z
Answers.Text: @minusatwelfth: The best numbers depend on size of layers. Most neural network libraries will use sane weight initialisation amounts. If you are implementing your own NN routines (using e.g. NumPy) then you will need to research what to use for initialisation, alongside how to do backpropagation etc. Otherwise if you are using a modern framework like Keras or PyTorch, the defaults will work fine for you
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T14:20:05.473Z
Answers.Text: Ah I see. Sadly I'm not using any of those softwares you mentioned, I have to initialise the weights and the steps manually
Answers.UserDisplayName: minusatwelfth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T15:54:35.970Z
Answers.Text: @minusatwelfth: It is not possible to give you rules for initialisation in the space of a comment. I suggest look it up. Keywords to search "Glorot initialisation" or "Xavier initialisation". It's not too complicated. Easier than understanding backprop in any case
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T10:21:22.037Z
Answers.LastActivityDate: 2020-09-03T10:21:22.037Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am very new to the field of AI so please bear with me.
  Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The input variables are numerous but not that many, maybe 7-10.</p>
  <p>These input variables are certain formulae that involve calculations to do with wind, time, angle, momentum etc, and each formula returns which side it thinks the dice will like roll. Let's say that intuitively, by looking at these variables, I can make a very good guess at which side the dice lands on. If for example 6 out of 7 input variables say it likely that the dice will land on 1 but the 7th input suggests that it will land on 0, I would guess it lands on 1. As a human, I'm essentially consulting these inputs as a kind of &quot;brains trust&quot;, and I act as a judge to make the final decision based on the brains trust. Of course in that example, my logic as a judge was simply majority rules, but what if some other more complicated non-linear method of judging was needed?</p>
  <p>I essentially want my neural network to take this role as a judge. I have read that feedforward nns have limitations regarding control flow and loops, so I'm not sure if that structure will be appropriate. I'm not sure if recurrent nn will be appropriate either as I don't care what the previous inputs were.</p>
  <p>Thanks</p>
  
Tags: <neural-networks><reinforcement-learning><convolutional-neural-networks><recurrent-neural-networks><feedforward-neural-networks>
Title: What's a good neural network for this problem?
UserDisplayName: minusatwelfth
ThreadText: # What's a good neural network for this problem?
  
  (Asked by: username_0 on 2020-09-03)
  
  I am very new to the field of AI so please bear with me.
  Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The input variables are numerous but not that many, maybe 7-10.
  
  These input variables are certain formulae that involve calculations to do with wind, time, angle, momentum etc, and each formula returns which side it thinks the dice will like roll. Let's say that intuitively, by looking at these variables, I can make a very good guess at which side the dice lands on. If for example 6 out of 7 input variables say it likely that the dice will land on 1 but the 7th input suggests that it will land on 0, I would guess it lands on 1. As a human, I'm essentially consulting these inputs as a kind of "brains trust", and I act as a judge to make the final decision based on the brains trust. Of course in that example, my logic as a judge was simply majority rules, but what if some other more complicated non-linear method of judging was needed?
  
  I essentially want my neural network to take this role as a judge. I have read that feedforward nns have limitations regarding control flow and loops, so I'm not sure if that structure will be appropriate. I'm not sure if recurrent nn will be appropriate either as I don't care what the previous inputs were.
  
  Thanks
  
  **Tags:** neural-networks, reinforcement-learning, convolutional-neural-networks, recurrent-neural-networks, feedforward-neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-03. Score: 4 | Accepted answer
  
  A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect relationships to be highly non-linear then additional layers might be required, but from your description of the problem, I would be surprised if you needed more than few layers, and a relatively small network.
  
  However, I note that:
  
  > The input variables are numerous but not that many, maybe 7-10.
  
  This gives you $3^{10} = 59049$ possible inputs. That's not much in terms of amount of data needed for ML statistical models. Assuming that even the best predictions are still probabilistic, then you may only need a million or so examples to create a reasonably accurate lookup table, not needing a neural network at all.
  
  The strength of a neural network is to be able to generalise well from less examples than that. Of course, this is not perfect, but it would be able to do things such as notice if inputs 1,2 and 3 agree then that is always the most likely answer. If that turns out to be true (and not an accident of having low numbers of samples), then the NN could learn that useful pattern using far less data than a table-based approach.
  
  > I have read that feedforward nns have limitations regarding control flow and loops, so I'm not sure if that structure will be appropriate.
  
  This is true, but does not impact your situation, because there is no control flow or loops involved. You have described a simple function. Whilst you or I might inspect the data and look backwards and forwards across it before coming to a decision, a neural network approximating a function does not need to do that, and in simple cases there is usually no benefit to doing so - a statistical summary of the correct mapping from input to output is more than sufficient and likely the best that can be done.
  
  > I'm not sure if recurrent nn will be appropriate either as I don't care what the previous inputs were.
  
  As all your inputs represent the same kind of thing, you could implement as a RNN with a single input, -1, 0 or +1, always feeding in the predictions by type in the same order. It might resemble how you are thinking about the problem as a human (at least a better analogy than the direct statistical match in a feed-forward network), especially if you implemented a variant of attention. However, I don't think there would be any benefit to that in improved accuracy, and it would be a significant challenge to build that if you are new to AI.
  
  > **Comments:**
  > **username_0**: Thanks for the answer. What if the inputs were fuzzy? Say instead of -1,0,1 it predicts more like -0.9, 0.1, 0.8 ? Does this change anything? What if the input values were higher than 1, would the nn normalize that automatically? Should I use relu?  
  > **username_1**: @username_0 If the inputs were fuzzy then you could not use a tabular approach as suggested in this answer, because there would be too many input variations. The NN would still work just fine though, and would treat e.g. -0.8 as similar to -0.7 (you probably want that). Neural networks do not automatically normalize inputs. An input range between -2 and +2 would be safe enough, up to you whether you want to clean them up before using. If the values could be really high you would want to pre-process before feeding to the NN, because NNs cope badly with large magnitude inputs in general.  
  > **username_0**: Thanks, that's pretty much everyt-... I read somewhere that I should try the weights to be low numbers first (-1 to 1). Is this true? With the platform im working with, i cant afford to optimize for every single weight value as it would take too long to compute.  
  > **username_1**: @username_0: The best numbers depend on size of layers. Most neural network libraries will use sane weight initialisation amounts. If you are implementing your own NN routines (using e.g. NumPy) then you will need to research what to use for initialisation, alongside how to do backpropagation etc. Otherwise if you are using a modern framework like Keras or PyTorch, the defaults will work fine for you  
  > **username_0**: Ah I see. Sadly I'm not using any of those softwares you mentioned, I have to initialise the weights and the steps manually  
  > **username_1**: @username_0: It is not possible to give you rules for initialisation in the space of a comment. I suggest look it up. Keywords to search "Glorot initialisation" or "Xavier initialisation". It's not too complicated. Easier than understanding backprop in any case  
  
  ---
  
  