Body: <p>I have devised an gridworld-like environment where a RL agent is tasked to cover all the blank squares by passing through them. Possible actions are up, down, left, right. The reward scheme is the following: +1 for covering a blank cell, and -1 per step. So, if the cell was colored after a step, the summed reward is (+1) + (-1) = 0, otherwise it is (0) + (-1) = -1. The environment is a tensor whose layers encode the positions to be covered and the position of the agent.</p>
  <p>Under this reward scheme, DQN fails to find a solution (implementation: <a href="https://github.com/DLR-RM/stable-baselines3" rel="nofollow noreferrer">stable_baselines3</a>). However, when the rewards are reduced by a factor of 10 to +0.1/-0.1, then the algorithm learns an optimal path.</p>
  <p>I wonder why that happens. I have tried reducing learning rate and gradient clipping (by norm) for the first case to see whether it will improve the learning, but it does not.</p>
  <p>The activation function used is ReLU</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-24T12:49:34.323Z
Comments.Text: Likely to be a bug or problem with your implementation. First thought on what might be relevant: How is your DQN network structured, and what transform function is being applied to the output (is it linear, ReLU, sigmoid, tanh for example)? Please add that detail using [edit]. Also consider breaking up into shorter paragraphs, maybe show a picture of the environment etc.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-24T12:52:51.893Z
Comments.Text: Another relevant point might be your state representation. When the history of actions is important it is also important to capture that somehow in the state. However, as you say scaling the rewards allows correct learning, it seems less likely to be that issue to me.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-24T18:00:33.817Z
Comments.Text: @NeilSlater, I just unit tested my environment ensure that the rewards are assigned as described: everything is ok. Extended the description of the problem as well. I doubt there is an error in the implementation because I am using stable_baselines3. 
  Re state representation and history, it is important, yes, but not relevant for this issue...
Comments.UserDisplayName: d56
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-24T18:30:31.907Z
Comments.Text: Are you using ReLU in the output layer? How many steps are there, and what are the expected returns from random policy and optimal policy (when the rewards are simply +1/-1)? Are you using any regularisation?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-26T13:44:18.703Z
Comments.Text: Well, the output layer does not have ReLUs because I am using DQN and it should predict the state-action value. I am using 1-step DQN in the environment that has 51 cells to cover on a 100 cell grid, the maximum number of moves (steps) is capped at 200. No regularization used. Trying to reprocude stable-baselines3 behavior with my own code atm
Comments.UserDisplayName: d56
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-26T14:53:44.540Z
Comments.Text: Ok, those seem like sensible choices. I cannot spot what you have wrong.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-26T16:00:01.643Z
Comments.Text: btw, may I contact you in private (which platform if yes)?
Comments.UserDisplayName: d56
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-26T18:55:37.120Z
Comments.Text: No, sorry. But if you want to discuss something AI related briefly, then you could @ me on chat here. I am on holiday for a couple of days so probably won't reply immediately
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-02T10:16:30.527Z
Comments.Text: yes, sure! I was a bit busy too these days :)
Comments.UserDisplayName: d56
Tags: <neural-networks><reinforcement-learning><dqn><deep-rl>
Title: Why scaling reward drastically affects performance?
UserDisplayName: d56
ThreadText: # Why scaling reward drastically affects performance?
  
  (Asked by: username_0 on 2020-08-24)
  
  I have devised an gridworld-like environment where a RL agent is tasked to cover all the blank squares by passing through them. Possible actions are up, down, left, right. The reward scheme is the following: +1 for covering a blank cell, and -1 per step. So, if the cell was colored after a step, the summed reward is (+1) + (-1) = 0, otherwise it is (0) + (-1) = -1. The environment is a tensor whose layers encode the positions to be covered and the position of the agent.
  
  Under this reward scheme, DQN fails to find a solution (implementation: [stable\_baselines3](https://github.com/DLR-RM/stable-baselines3)). However, when the rewards are reduced by a factor of 10 to +0.1/-0.1, then the algorithm learns an optimal path.
  
  I wonder why that happens. I have tried reducing learning rate and gradient clipping (by norm) for the first case to see whether it will improve the learning, but it does not.
  
  The activation function used is ReLU
  
  **Tags:** neural-networks, reinforcement-learning, dqn, deep-rl
  
  > **Comments:**
  > **username_1**: Likely to be a bug or problem with your implementation. First thought on what might be relevant: How is your DQN network structured, and what transform function is being applied to the output (is it linear, ReLU, sigmoid, tanh for example)? Please add that detail using [edit]. Also consider breaking up into shorter paragraphs, maybe show a picture of the environment etc.  
  > **username_1**: Another relevant point might be your state representation. When the history of actions is important it is also important to capture that somehow in the state. However, as you say scaling the rewards allows correct learning, it seems less likely to be that issue to me.  
  > **username_0**: @username_1, I just unit tested my environment ensure that the rewards are assigned as described: everything is ok. Extended the description of the problem as well. I doubt there is an error in the implementation because I am using stable_baselines3.  Re state representation and history, it is important, yes, but not relevant for this issue...  
  > **username_1**: Are you using ReLU in the output layer? How many steps are there, and what are the expected returns from random policy and optimal policy (when the rewards are simply +1/-1)? Are you using any regularisation?  
  > **username_0**: Well, the output layer does not have ReLUs because I am using DQN and it should predict the state-action value. I am using 1-step DQN in the environment that has 51 cells to cover on a 100 cell grid, the maximum number of moves (steps) is capped at 200. No regularization used. Trying to reprocude stable-baselines3 behavior with my own code atm  
  > **username_1**: Ok, those seem like sensible choices. I cannot spot what you have wrong.  
  > **username_0**: btw, may I contact you in private (which platform if yes)?  
  > **username_1**: No, sorry. But if you want to discuss something AI related briefly, then you could @ me on chat here. I am on holiday for a couple of days so probably won't reply immediately  
  > **username_0**: yes, sure! I was a bit busy too these days :)  
  
  ---
  
  ## 0 answers
  
  