Answers.Body: <p>The way you describe with one hot encoding is correct.</p>
  <p>Note that how the state is encoded is a separate question from the neural network, so I'm not sure what convolutional neural networks have to do with the question. In the famous atari game example, the input is a sequence of RGB images; a cnn is used to process the images. In your example you probably just want to use a regular Dense network, as your input is just the one hot encoding and not images.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T02:28:33.960Z
Answers.Text: Ahh, i've gone with a Dense network now. Do you think I should use activation functions? at the moment my network is outputting values between -1 and 1. I'm guessing it'd be better to use the probabilities as my reward function is a sum of percentages
Answers.UserDisplayName: Arlo Rostirolla
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-22T16:53:02.127Z
Answers.Text: Q tables/functions are attempting to predict the rewards for a given action assuming all of the perfect decisions are made from there forward until the end of the episode.  It is not only the present reward but a discounted value of all future rewards.  So no matter the value of the rewards, an output neuron for the Q-Value of an action should be able to represent (-inf, inf), this is typically linear activation.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T07:38:58.343Z
Answers.LastActivityDate: 2021-09-04T07:38:58.343Z
Answers.UserDisplayName: Taw
Answers.Body: <p>If there can be multiple items on a square something more complex than one-hot is needed.  As the name implies, only one category can be hot (set to true) in  one-hot.  You need multi-hot.  Further, if multiple instances of items can be on a grid square (2 food, 1 wall) it might be beneficial to encode it proportionality-wise (.66 food, .33 wall).</p>
  <p>CNN's inductive bias is locality.  It's looking for items near each other in the grid space with certain values.  It identifies groups of items that are similar to some pattern (ie looking for the ball or paddle in pong).  I'll leave it to you to decide if this is interesting in your environment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-22T16:49:00.357Z
Answers.LastActivityDate: 2024-01-22T17:07:41.957Z
Answers.LastEditDate: 2024-01-22T17:07:41.957Z
Answers.UserDisplayName: foreverska
Body: <p>I've read most of the posts on here regarding this subject, however most of them deal with gameboards where there are two different categories of single pieces on a board without walls etc.</p>
  <p>My game board has walls, and multiple instances of food. There are 8 different categories,
  Walls, enemy food, my food, enemy powerup, my powerup, attackable enemies, threatening enemies, and current teammate.</p>
  <p>I have one hot encoded all of this data into a tensor of size (8, 16, 32) where (16, 32) are the sizes of the game grid. However I'm not sure whether this is appropriate since many of the categories have multiple occurrences of each category in a single  (walls, food). Is it appropriate to use one hot encoding to represent categories in spatial data, where multiple one's may be present?</p>
  <p>The alternative I was considering was to use a CNN, however many posts have said it is inappropriate for one hot data. My reasoning was that since the data is a abstract Boolean grid representing the RGB frames, it might be appropriate.</p>
  <p>Does anyone have any suggestions as to the best way to represent a spatial Boolean grid representing multiple categories for input to a network?</p>
  
Tags: <convolutional-neural-networks><deep-rl><game-ai><double-q-learning><one-hot-encoding>
Title: How to embed game grid state with walls as an input to neural network
UserDisplayName: Arlo Rostirolla
ThreadText: # How to embed game grid state with walls as an input to neural network
  
  (Asked by: username_0 on 2021-09-04)
  
  I've read most of the posts on here regarding this subject, however most of them deal with gameboards where there are two different categories of single pieces on a board without walls etc.
  
  My game board has walls, and multiple instances of food. There are 8 different categories,
  Walls, enemy food, my food, enemy powerup, my powerup, attackable enemies, threatening enemies, and current teammate.
  
  I have one hot encoded all of this data into a tensor of size (8, 16, 32) where (16, 32) are the sizes of the game grid. However I'm not sure whether this is appropriate since many of the categories have multiple occurrences of each category in a single (walls, food). Is it appropriate to use one hot encoding to represent categories in spatial data, where multiple one's may be present?
  
  The alternative I was considering was to use a CNN, however many posts have said it is inappropriate for one hot data. My reasoning was that since the data is a abstract Boolean grid representing the RGB frames, it might be appropriate.
  
  Does anyone have any suggestions as to the best way to represent a spatial Boolean grid representing multiple categories for input to a network?
  
  **Tags:** convolutional-neural-networks, deep-rl, game-ai, double-q-learning, one-hot-encoding
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-09-04. Score: 1
  
  The way you describe with one hot encoding is correct.
  
  Note that how the state is encoded is a separate question from the neural network, so I'm not sure what convolutional neural networks have to do with the question. In the famous atari game example, the input is a sequence of RGB images; a cnn is used to process the images. In your example you probably just want to use a regular Dense network, as your input is just the one hot encoding and not images.
  
  > **Comments:**
  > **username_0**: Ahh, i've gone with a Dense network now. Do you think I should use activation functions? at the moment my network is outputting values between -1 and 1. I'm guessing it'd be better to use the probabilities as my reward function is a sum of percentages  
  > **username_2**: Q tables/functions are attempting to predict the rewards for a given action assuming all of the perfect decisions are made from there forward until the end of the episode.  It is not only the present reward but a discounted value of all future rewards.  So no matter the value of the rewards, an output neuron for the Q-Value of an action should be able to represent (-inf, inf), this is typically linear activation.  
  
  ---
  
  ### Answer by username_2 on 2024-01-22. Score: 1
  
  If there can be multiple items on a square something more complex than one-hot is needed. As the name implies, only one category can be hot (set to true) in one-hot. You need multi-hot. Further, if multiple instances of items can be on a grid square (2 food, 1 wall) it might be beneficial to encode it proportionality-wise (.66 food, .33 wall).
  
  CNN's inductive bias is locality. It's looking for items near each other in the grid space with certain values. It identifies groups of items that are similar to some pattern (ie looking for the ball or paddle in pong). I'll leave it to you to decide if this is interesting in your environment.
  
  
  ---
  
  