Body: <p>I am making a school project where I should develop any kind of game where I can have one reactive agent and one agent based on machine learning competing with each other.</p>
  
  <p>My game consists of a salesmen problem. Basically, I have 3 types of entities, consumers, salesmen, and hotspots. </p>
  
  <p>The consumers are represented by the person with a green background. There are 8 of them. They basically move around the whole game using random walks and they tend o aggregate on the HotSpots (the orange icon with the router in it).</p>
  
  <p>The salesmen are represented by the person with the dark grey background. One of them is controlled by a reactive agent that has in it some rules that I programmed and the other one is controlled by my DQN model.</p>
  
  <p>The salesmen have 5 available actions, move <strong>up</strong>, <strong>right</strong>, <strong>down</strong>, <strong>left</strong> or <strong>sell</strong>.
  When they choose to sell the simulation will try to sell to the closest consumer in a predetermined max range. If no consumers exist in that range or if the consumer rejects to buy then the sell fails.</p>
  
  <p>I started training a Deep Q Network that I built using TensorFlow. As input features, I am giving the agent current position, the position of each consumer and a boolean saying if the consumer was recently asked to buy or not (consumers that were asked to buy something will reject future offers for a determined amount of time with 100% probability).
  For the output layer, I have 5 nodes, one for each available action.</p>
  
  <p>Here is a screenshot of the game:
  The red number on the right-bottom corner of each agent represents their total utility.</p>
  
  <p><a href="https://i.sstatic.net/xQ16I.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xQ16I.png" alt="enter image description here"></a></p>
  
  <p><strong>I decided to give the agents the following rewards:</strong></p>
  
  <ul>
  <li><strong>SELL_SUCCESSED_REWARD = 3</strong> - The agents receives 3 points for each success sell. </li>
  <li><strong>SELL_FAILED_REWARD = -0.010</strong> - The agent loses 0.01 points for each failed sell</li>
  <li><strong>MOVING_REWARD = -0.001</strong> - The agent loses 0.001 points for each move</li>
  <li><strong>NOT_MOVING_REWARD = -0.0125</strong> - The agent loses 0.0125 points for standing in the same position (ie. not moving or trying to move against a wall)</li>
  </ul>
  
  <p>I started training my agent but I seems to do not learn anything! I left it training for around 3 hours and I could not see any improvement. I tried different activation functions, batch sizes, exploration rates etc but no noticiable effect.</p>
  
  <p>My question is: Can a DQN learn in this type of enviroment where there are a lot of random walks?</p>
  
  <p>If yes what could be my problem? Not enought training time? Bad input features? Bad implementation?</p>
  
  <p>Here are the files with my implementation of DQN:</p>
  
  <p>Agent: <a href="https://github.com/daniel3303/aasma-project/blob/master/src/Agent/DeepLearningAgent/DeepLearningAgent.py" rel="nofollow noreferrer">https://github.com/daniel3303/aasma-project/blob/master/src/Agent/DeepLearningAgent/DeepLearningAgent.py</a></p>
  
  <p>Training: <a href="https://github.com/daniel3303/aasma-project/blob/master/train_dqn.py" rel="nofollow noreferrer">https://github.com/daniel3303/aasma-project/blob/master/train_dqn.py</a></p>
  
  <p>Thanks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-09T07:42:02.203Z
Comments.Text: First thing that seems weird to me is your reward function. You give negative reward for moving and negative reward for not moving ? So how is the agent supposed to know what to do ? Reward function should reflect your final goal, that is to sell a lot of items, so you should give positive reward when item is sold and negative one when you failed. This may or may not help but I think it's worth to try at least
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-09T09:10:25.850Z
Comments.Text: I don't think there is anything wrong with the reward scheme here. Instead I would suspect an implementation issue. Could you clarify that the grid of houses is the same throughout (as you don't represent positions that the agents cannot occupy in the state)? It may help if you give a more thorough description of your inputs/features with one or two examples, as that could be key, and I don't really want to invest in reading your code to find that out
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><machine-learning><deep-learning><reinforcement-learning><tensorflow>
Title: DQN not able to learn in a game where other agents perform random walks
UserDisplayName: Daniel Oliveira
ThreadText: # DQN not able to learn in a game where other agents perform random walks
  
  (Asked by: username_0 on 2019-05-08)
  
  I am making a school project where I should develop any kind of game where I can have one reactive agent and one agent based on machine learning competing with each other.
  
  My game consists of a salesmen problem. Basically, I have 3 types of entities, consumers, salesmen, and hotspots.
  
  The consumers are represented by the person with a green background. There are 8 of them. They basically move around the whole game using random walks and they tend o aggregate on the HotSpots (the orange icon with the router in it).
  
  The salesmen are represented by the person with the dark grey background. One of them is controlled by a reactive agent that has in it some rules that I programmed and the other one is controlled by my DQN model.
  
  The salesmen have 5 available actions, move **up**, **right**, **down**, **left** or **sell**.
  When they choose to sell the simulation will try to sell to the closest consumer in a predetermined max range. If no consumers exist in that range or if the consumer rejects to buy then the sell fails.
  
  I started training a Deep Q Network that I built using TensorFlow. As input features, I am giving the agent current position, the position of each consumer and a boolean saying if the consumer was recently asked to buy or not (consumers that were asked to buy something will reject future offers for a determined amount of time with 100% probability).
  For the output layer, I have 5 nodes, one for each available action.
  
  Here is a screenshot of the game:
  The red number on the right-bottom corner of each agent represents their total utility.
  
  [![enter image description here](https://i.sstatic.net/xQ16I.png)](https://i.sstatic.net/xQ16I.png)
  
  **I decided to give the agents the following rewards:**
  
  * **SELL\_SUCCESSED\_REWARD = 3** - The agents receives 3 points for each success sell.
  * **SELL\_FAILED\_REWARD = -0.010** - The agent loses 0.01 points for each failed sell
  * **MOVING\_REWARD = -0.001** - The agent loses 0.001 points for each move
  * **NOT\_MOVING\_REWARD = -0.0125** - The agent loses 0.0125 points for standing in the same position (ie. not moving or trying to move against a wall)
  
  I started training my agent but I seems to do not learn anything! I left it training for around 3 hours and I could not see any improvement. I tried different activation functions, batch sizes, exploration rates etc but no noticiable effect.
  
  My question is: Can a DQN learn in this type of enviroment where there are a lot of random walks?
  
  If yes what could be my problem? Not enought training time? Bad input features? Bad implementation?
  
  Here are the files with my implementation of DQN:
  
  Agent: <https://github.com/daniel3303/aasma-project/blob/master/src/Agent/DeepLearningAgent/DeepLearningAgent.py>
  
  Training: <https://github.com/daniel3303/aasma-project/blob/master/train_dqn.py>
  
  Thanks.
  
  **Tags:** neural-networks, machine-learning, deep-learning, reinforcement-learning, tensorflow
  
  > **Comments:**
  > **username_1**: First thing that seems weird to me is your reward function. You give negative reward for moving and negative reward for not moving ? So how is the agent supposed to know what to do ? Reward function should reflect your final goal, that is to sell a lot of items, so you should give positive reward when item is sold and negative one when you failed. This may or may not help but I think it's worth to try at least  
  > **username_2**: I don't think there is anything wrong with the reward scheme here. Instead I would suspect an implementation issue. Could you clarify that the grid of houses is the same throughout (as you don't represent positions that the agents cannot occupy in the state)? It may help if you give a more thorough description of your inputs/features with one or two examples, as that could be key, and I don't really want to invest in reading your code to find that out  
  
  ---
  
  ## 0 answers
  
  