Answers.Body: <p>Thanks for the responses - after taking into account the 'risk' the network experiences when working with unstable financial data I <em>modified the activation layers from RelU to Sigmoid</em> - this has lead to a considerable improvement to both results (profit!) and additionally the rate at which actions are carried out. </p>
  
  <p>I will also edit the above question to provide further details.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-24T15:04:46.740Z
Answers.LastActivityDate: 2019-04-24T15:04:46.740Z
Answers.UserDisplayName: Ryan Hodgson
Body: <p>I understand that gamma is an important factor in determining the rewards for a deep Q agent, however during testing of my network I am noticing that the agent is outputting more actions to "do nothing" as it learns more about it's given data set (financial stock data).</p>
  
  <p>I have tried tweaking the gamma at different levels ranging from 0 - 1 and everywhere in between, however as the agent continues to learn, the times between actions is getting longer, and longer. This behaviour is undesirable, and it is preferable that the agent be making more often, short-term actions even if they result in reduced reward.</p>
  
  <p>Does anyone have any tips on how to achieve this? Would a minus gamma have adverse effects on the network?</p>
  
  <p>TLDR:
  Time between actions becoming increasingly long over time, would prefer an agent that makes many actions over long-term ones.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-21T19:11:36.960Z
Comments.Text: The value of gamma would determine *which* actions an agent would take It wouldn't have a significant impact on the time between taking actions. It sounds like something else in your implementation is eating up time.
Comments.UserDisplayName: Philip Raeisghasem
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-21T19:25:35.863Z
Comments.Text: Agreed with Philip. There is nothing inherently part of Q learning, DQN, A2C etc that makes them slow down with more experience. Are you perhaps storing experience and using it directly to search for the probable best action?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-21T22:01:58.270Z
Comments.Text: Thanks for the response - i am using experience replay with a memory length of 1000 with an action space of 3 (either 0,1 or 2) an action of 0 would mean "doing nothing" and my observations show that as the length of training time increases, the amount of time between getting a 1 or 2 action increases. I initially thought this was caused by my epsilon function, however the slowdown continues after epsilon has fully decayed
Comments.UserDisplayName: Ryan Hodgson
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-22T12:05:47.423Z
Comments.Text: So, to clarify, the amount of time between decisions is not wall clock time (which would imply inefficient code), but time step intervals? The agent is in fact making action choices at normal speed, but choosing a specific "do nothing" action more frequently that you would like? I think in order to help with this, you need to explain more about your environment and reward scheme.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-22T12:08:03.537Z
Comments.Text: You should note that it may be entirely realistic for the agent to make few "active" decisions if you have put it into a high risk environment, such as gambling or financial markets, where it has determined that the expected reward from acting is negative in most scenarios (including short-term ones)
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-23T01:22:19.473Z
Comments.Text: Yes I should have clarified - this is with my network working through a dataset of financial (price) data. Thanks for the answers I believe what you mentioned regarding the risk of the environment may be impacting this.
Comments.UserDisplayName: Ryan Hodgson
Tags: <deep-learning><q-learning><dqn><rewards>
Title: Encourage Deep Q to seek short-term reward
UserDisplayName: Ryan Hodgson
ThreadText: # Encourage Deep Q to seek short-term reward
  
  (Asked by: username_0 on 2019-04-21)
  
  I understand that gamma is an important factor in determining the rewards for a deep Q agent, however during testing of my network I am noticing that the agent is outputting more actions to "do nothing" as it learns more about it's given data set (financial stock data).
  
  I have tried tweaking the gamma at different levels ranging from 0 - 1 and everywhere in between, however as the agent continues to learn, the times between actions is getting longer, and longer. This behaviour is undesirable, and it is preferable that the agent be making more often, short-term actions even if they result in reduced reward.
  
  Does anyone have any tips on how to achieve this? Would a minus gamma have adverse effects on the network?
  
  TLDR:
  Time between actions becoming increasingly long over time, would prefer an agent that makes many actions over long-term ones.
  
  **Tags:** deep-learning, q-learning, dqn, rewards
  
  > **Comments:**
  > **username_1**: The value of gamma would determine *which* actions an agent would take It wouldn't have a significant impact on the time between taking actions. It sounds like something else in your implementation is eating up time.  
  > **username_2**: Agreed with username_1. There is nothing inherently part of Q learning, DQN, A2C etc that makes them slow down with more experience. Are you perhaps storing experience and using it directly to search for the probable best action?  
  > **username_0**: Thanks for the response - i am using experience replay with a memory length of 1000 with an action space of 3 (either 0,1 or 2) an action of 0 would mean "doing nothing" and my observations show that as the length of training time increases, the amount of time between getting a 1 or 2 action increases. I initially thought this was caused by my epsilon function, however the slowdown continues after epsilon has fully decayed  
  > **username_2**: So, to clarify, the amount of time between decisions is not wall clock time (which would imply inefficient code), but time step intervals? The agent is in fact making action choices at normal speed, but choosing a specific "do nothing" action more frequently that you would like? I think in order to help with this, you need to explain more about your environment and reward scheme.  
  > **username_2**: You should note that it may be entirely realistic for the agent to make few "active" decisions if you have put it into a high risk environment, such as gambling or financial markets, where it has determined that the expected reward from acting is negative in most scenarios (including short-term ones)  
  > **username_0**: Yes I should have clarified - this is with my network working through a dataset of financial (price) data. Thanks for the answers I believe what you mentioned regarding the risk of the environment may be impacting this.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-04-24. Score: 0 | Accepted answer
  
  Thanks for the responses - after taking into account the 'risk' the network experiences when working with unstable financial data I *modified the activation layers from RelU to Sigmoid* - this has lead to a considerable improvement to both results (profit!) and additionally the rate at which actions are carried out.
  
  I will also edit the above question to provide further details.
  
  
  ---
  
  