Answers.Body: <p>You have identified an optimization in your problem space and desire to bake this into your neural net. I suggest preprocessing: Compose your optimization with a neural net that does a subset of what you want.</p>
  
  <p>In other words, normalize your input by manually coding a rotation algorithm that rotates inputs to capture the equivalence highlighted in your post. Then feed the output of this transformation to your neural net, for training and all other uses. This means you are training the neural net to tackle the sub-problem you identified - rotations are redundant.</p>
  
  <p>Test your normalizer by generating random input, rotating it to all four potential transformations, run the normalizer on each one, then check that they are all equivalent.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-08T13:19:13.927Z
Answers.LastActivityDate: 2017-08-08T13:19:13.927Z
Answers.UserDisplayName: Harrichael
Answers.Body: <p>To be purist about it, begin with considering the input differently, as a circular array of size four, each item containing a pair of bits, and additionally a center bit:</p>
  <blockquote>
  <p>... 01, 01, 11, 10 ...</p>
  <p>0</p>
  </blockquote>
  <p>Throughout the design of the network, continue this circular structure and center point paradigm.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-09T09:40:36.573Z
Answers.LastActivityDate: 2017-08-09T09:40:36.573Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Douglas Daseeco
Answers.Body: <p>If I understand well your single output node will be the next status of the square in the middle. You don't need to worry about the number of nodes in the hidden layers while you have sufficient resources to train the model. This problem is very easy to learn for a neural network so no size concerns.</p>
  
  <p>You need to do supervised training that means you need to feed in input data and the matching expected output. You need to be sure that in your training data all 4 rotations are assigned to the same output. This way your network should learn to treat all of these the same way.  </p>
  
  <p>You made me curious so I tried myself. My solution could learn 100% correct in about 20 epochs running within a few seconds on my old laptop. I only slightly changed the output to be categorical either [0,1] or [1,0] but that gives the same result that you are looking for. Just for reference here is the code written in python:  </p>
  
  <pre><code>from keras.models import Sequential
  from keras.layers import Input, Dense
  from keras.models import Model
  from keras import optimizers
  from keras.utils.np_utils import to_categorical
  import helper
  
  x_,y_ = helper.fnn_csv_toXY("conway.csv","output",False)
  y_binary = to_categorical(y_)
  
  model = Sequential()
  model.add(Dense(100, activation='relu', kernel_initializer='glorot_uniform',input_shape =(9,)))
  model.add(Dense(20, activation='relu', kernel_initializer='glorot_uniform'))
  model.add(Dense(2, activation='softmax'))
  adam=optimizers.Adam()
  model.compile(optimizer=adam,
                loss='categorical_crossentropy',
                metrics=['acc'])
  model.fit(x_, y_binary, epochs=100)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-08T12:02:34.793Z
Answers.Text: The network I will be using will be simulated by a class written by myself, hence the memory concerns.
Answers.UserDisplayName: Aric
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-08T12:27:35.533Z
Answers.Text: If you want reduce memory usage by the network then reducing the number of possible inputs (by rotating) will help to have a smaller network. The easier the learning task is the smaller network needed. In this case pre-processing would be better. Nevertheless NN is for concept learning and to learn the concept of Game of Life you should feed in all patterns. If your aim is to absolutely minimize memory footprint solve the problem on a linear way.
Answers.UserDisplayName: Manngo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-08T12:31:43.780Z
Answers.Text: If memory is not a concern, I would rather the network performs this operation for the same reasons you stated. Pre-processing removes some of the task from the network, simplifying it.
Answers.UserDisplayName: Aric
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-08T12:56:53.243Z
Answers.Text: Correct, then go for the pre-processing. I think with this the question is answered. You may chose to implement it if you run into issues with memory. Tip: use float for the weights that takes only 32 bits instead of double that takes 64. This will use less memory.
Answers.UserDisplayName: Manngo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-08T11:56:20.673Z
Answers.LastActivityDate: 2017-08-08T11:56:20.673Z
Answers.UserDisplayName: Manngo
Body: <p>I'm attempting to program my own system to run a neural network. To reduce the number of nodes needed, it was suggested to make it treat rotations of the input equally.</p>
  
  <p>My network aims to learn and predict Conway's Game of Life by looking at every square and its surrounding squares in a grid, and giving the output for that square. Its input is a string of 9 bits:</p>
  
  <p><a href="https://i.sstatic.net/p9UwK.png" rel="noreferrer"><img src="https://i.sstatic.net/p9UwK.png" alt="Glider"></a></p>
  
  <p>The above is represented as 010 001 111.</p>
  
  <p>There are three other rotations of this shape however, and all of them produce the same output:</p>
  
  <p><a href="https://i.sstatic.net/eyDkl.png" rel="noreferrer"><img src="https://i.sstatic.net/eyDkl.png" alt="Glider rotations"></a></p>
  
  <p>My network topology is 9 input nodes and 1 output node for the next state of the centre square in the input. How can I construct the hidden layer(s) so that they take each of these rotations as the same, cutting the number of possible inputs down to a quarter of the original?</p>
  
  <p><strong>Edit:</strong></p>
  
  <p>There is also a flip of each rotation which produces an identical result. Incorporating these will cut my inputs by 1/8th. With the glider, my aim is for all of these inputs to be treated exactly the same. Will this have to be done with pre-processing, or can I incorporate it into the network?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-07T18:25:37.320Z
Comments.Text: Outstanding question!  I have a similar, impeding issue with my own project and will be very interested to learn about the most efficient techniques to reduce for symmetry.
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-07T18:38:14.717Z
Comments.Text: @DukeZhou I get the impression that it will take a while to get an answer however. I'm prepared to use up my meagre supply of rep to set up a bounty if needed...
Comments.UserDisplayName: Aric
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-11T08:01:01.220Z
Comments.Text: Another solution is to pre-process the input, such that all 4 rotations are converted into the same image before being fed to the network.
Comments.UserDisplayName: BlueMoon93
Tags: <neural-networks>
Title: How can I make my network treat rotations of the input equally?
UserDisplayName: Aric
ThreadText: # How can I make my network treat rotations of the input equally?
  
  (Asked by: username_0 on 2017-08-07)
  
  I'm attempting to program my own system to run a neural network. To reduce the number of nodes needed, it was suggested to make it treat rotations of the input equally.
  
  My network aims to learn and predict Conway's Game of Life by looking at every square and its surrounding squares in a grid, and giving the output for that square. Its input is a string of 9 bits:
  
  [![Glider](https://i.sstatic.net/p9UwK.png)](https://i.sstatic.net/p9UwK.png)
  
  The above is represented as 010 001 111.
  
  There are three other rotations of this shape however, and all of them produce the same output:
  
  [![Glider rotations](https://i.sstatic.net/eyDkl.png)](https://i.sstatic.net/eyDkl.png)
  
  My network topology is 9 input nodes and 1 output node for the next state of the centre square in the input. How can I construct the hidden layer(s) so that they take each of these rotations as the same, cutting the number of possible inputs down to a quarter of the original?
  
  **Edit:**
  
  There is also a flip of each rotation which produces an identical result. Incorporating these will cut my inputs by 1/8th. With the glider, my aim is for all of these inputs to be treated exactly the same. Will this have to be done with pre-processing, or can I incorporate it into the network?
  
  **Tags:** neural-networks
  > **Comments:**
  > **username_1**: Outstanding question!  I have a similar, impeding issue with my own project and will be very interested to learn about the most efficient techniques to reduce for symmetry.  
  > **username_0**: @DukeZhou I get the impression that it will take a while to get an answer however. I'm prepared to use up my meagre supply of rep to set up a bounty if needed...  
  > **username_2**: Another solution is to pre-process the input, such that all 4 rotations are converted into the same image before being fed to the network.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2017-08-08. Score: 4 | Accepted answer
  
  If I understand well your single output node will be the next status of the square in the middle. You don't need to worry about the number of nodes in the hidden layers while you have sufficient resources to train the model. This problem is very easy to learn for a neural network so no size concerns.
  You need to do supervised training that means you need to feed in input data and the matching expected output. You need to be sure that in your training data all 4 rotations are assigned to the same output. This way your network should learn to treat all of these the same way.  
  You made me curious so I tried myself. My solution could learn 100% correct in about 20 epochs running within a few seconds on my old laptop. I only slightly changed the output to be categorical either [0,1] or [1,0] but that gives the same result that you are looking for. Just for reference here is the code written in python:  
  from keras.models import Sequential
  from keras.layers import Input, Dense
  from keras.models import Model
  from keras import optimizers
  from keras.utils.np_utils import to_categorical
  import helper
  
  x_,y_ = helper.fnn_csv_toXY("conway.csv","output",False)
  y_binary = to_categorical(y_)
  
  model = Sequential()
  model.add(Dense(100, activation='relu', kernel_initializer='glorot_uniform',input_shape =(9,)))
  model.add(Dense(20, activation='relu', kernel_initializer='glorot_uniform'))
  model.add(Dense(2, activation='softmax'))
  adam=optimizers.Adam()
  model.compile(optimizer=adam,
                loss='categorical_crossentropy',
                metrics=['acc'])
  model.fit(x_, y_binary, epochs=100)
  
  > **Comments:**
  > **username_0**: The network I will be using will be simulated by a class written by myself, hence the memory concerns.  
  > **username_5**: If you want reduce memory usage by the network then reducing the number of possible inputs (by rotating) will help to have a smaller network. The easier the learning task is the smaller network needed. In this case pre-processing would be better. Nevertheless NN is for concept learning and to learn the concept of Game of Life you should feed in all patterns. If your aim is to absolutely minimize memory footprint solve the problem on a linear way.  
  > **username_0**: If memory is not a concern, I would rather the network performs this operation for the same reasons you stated. Pre-processing removes some of the task from the network, simplifying it.  
  > **username_5**: Correct, then go for the pre-processing. I think with this the question is answered. You may chose to implement it if you run into issues with memory. Tip: use float for the weights that takes only 32 bits instead of double that takes 64. This will use less memory.  
  
  ---
  
  ### Answer by username_3 on 2017-08-08. Score: 4
  
  You have identified an optimization in your problem space and desire to bake this into your neural net. I suggest preprocessing: Compose your optimization with a neural net that does a subset of what you want.
  
  In other words, normalize your input by manually coding a rotation algorithm that rotates inputs to capture the equivalence highlighted in your post. Then feed the output of this transformation to your neural net, for training and all other uses. This means you are training the neural net to tackle the sub-problem you identified - rotations are redundant.
  
  Test your normalizer by generating random input, rotating it to all four potential transformations, run the normalizer on each one, then check that they are all equivalent.
  
  
  ---
  
  ### Answer by username_4 on 2017-08-09. Score: 1
  
  To be purist about it, begin with considering the input differently, as a circular array of size four, each item containing a pair of bits, and additionally a center bit:
  
  > ... 01, 01, 11, 10 ...
  >
  > 0
  
  Throughout the design of the network, continue this circular structure and center point paradigm.
  
  
  ---
  
  