Body: <p>I am trying to ascertain if my neural network is able to generalize or if it’s simply using memory/overfitting to solve a task. I would like my model to generalise.</p>
  
  <p>Currently, I train the neural network on a randomly generated 3x3 frozen lake environment - with no holes. (The network simply chooses an action for each state it is presented.)</p>
  
  <p>Then, I test the model on a much larger frozen lake environment. Still no holes. Still randomly generated. The test environment size is assigned by a random value of 5-15 for each axis (height/width), randomly generated.</p>
  
  <p>Then I determine the "degree of generalization" by how many large environments the network is able to solve. At present, it solves 100/100 on the 3x3, and about 83/100 on the larger test environments. </p>
  
  <p>When I track the solutions it generates, I can see that the network always takes the shortest route available, which is great.</p>
  
  <p>Do you guys have any ideas, inputs or criticism on the method I use to determine the degree of generalization?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-12T08:28:28.353Z
Comments.Text: Do you have any working definition of "abstract thinking" that you want answers to apply? It is a hard concept to define at the level of maths and statistics where you are measuring it. Wiithout a working definition from you, answers will tend to be more philosophical than practical
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-12T10:11:18.953Z
Comments.Text: Hi Neil! No i don’t. I just want to avoid overfitting/memorisation, and want the network to be able to generalise and apply to similar environments as described. Does this make sense?
Comments.UserDisplayName: skywalkerdk
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-12T10:28:12.770Z
Comments.Text: Yes. I was thinking that really you are asking about generalisation and not about "abstract thought" at all. Measuring generalisation is a very practical thing that you can do, and some discussion can be had about limits of this. "Abstract thought" - in the sense that it is used in natural English - is arguably not happening at all here, your agent has learned a kind of reflex behaviour a bit like riding a bicycle or running away from danger.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-12T10:31:15.710Z
Comments.Text: Thanks for your input Neil. English is not my native language, so i will edit the post to reflect the correct meaning. - Do you happen to have any input on the issue at hand?
Comments.UserDisplayName: skywalkerdk
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-12T10:36:28.573Z
Comments.Text: Yes, I do have some thoughts. I may get time to post an answer later today
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><machine-learning><deep-learning><reinforcement-learning><generalization>
Title: How do I determine the generalisation ability of a neural network?
UserDisplayName: skywalkerdk
ThreadText: # How do I determine the generalisation ability of a neural network?
  
  (Asked by: username_0 on 2019-07-12)
  
  I am trying to ascertain if my neural network is able to generalize or if it’s simply using memory/overfitting to solve a task. I would like my model to generalise.
  
  Currently, I train the neural network on a randomly generated 3x3 frozen lake environment - with no holes. (The network simply chooses an action for each state it is presented.)
  
  Then, I test the model on a much larger frozen lake environment. Still no holes. Still randomly generated. The test environment size is assigned by a random value of 5-15 for each axis (height/width), randomly generated.
  
  Then I determine the "degree of generalization" by how many large environments the network is able to solve. At present, it solves 100/100 on the 3x3, and about 83/100 on the larger test environments.
  
  When I track the solutions it generates, I can see that the network always takes the shortest route available, which is great.
  
  Do you guys have any ideas, inputs or criticism on the method I use to determine the degree of generalization?
  
  **Tags:** neural-networks, machine-learning, deep-learning, reinforcement-learning, generalization
  
  > **Comments:**
  > **username_1**: Do you have any working definition of "abstract thinking" that you want answers to apply? It is a hard concept to define at the level of maths and statistics where you are measuring it. Wiithout a working definition from you, answers will tend to be more philosophical than practical  
  > **username_0**: Hi username_1! No i don’t. I just want to avoid overfitting/memorisation, and want the network to be able to generalise and apply to similar environments as described. Does this make sense?  
  > **username_1**: Yes. I was thinking that really you are asking about generalisation and not about "abstract thought" at all. Measuring generalisation is a very practical thing that you can do, and some discussion can be had about limits of this. "Abstract thought" - in the sense that it is used in natural English - is arguably not happening at all here, your agent has learned a kind of reflex behaviour a bit like riding a bicycle or running away from danger.  
  > **username_0**: Thanks for your input username_1. English is not my native language, so i will edit the post to reflect the correct meaning. - Do you happen to have any input on the issue at hand?  
  > **username_1**: Yes, I do have some thoughts. I may get time to post an answer later today  
  
  ---
  
  ## 0 answers
  
  