Answers.Body: <p>Given that your state space is continuous, then I would recommend using <a href="https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf" rel="nofollow noreferrer">Deep Q-Learning</a>. As you say, running several episodes will definitely be beneficial so that the agent is able to explore the space more thoroughly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-18T18:48:52.900Z
Answers.LastActivityDate: 2020-06-18T18:48:52.900Z
Answers.UserDisplayName: David
Body: <p>Let's say an RL trading system places trades based on pricing data.</p>
  <p>Each episode represents 1 hour of trading, and there are 24 hours of data available. The Q table represents for a given state, what is the most action with the highest utility.</p>
  <p>The state is a sequence of prices, and the action is either buy, hold, sell.</p>
  <p>Instead of &quot;Loop for each episode&quot; as per the Sarsa algorithm :</p>
  <p><a href="https://i.sstatic.net/tG9O1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tG9O1.png" alt="enter image description here" /></a></p>
  <p>I add an additional outer loop. Now instead of just looping for each episode we have:</p>
  <pre><code>for 1 to N:     
     &quot;Loop for each episode&quot;
  </code></pre>
  <p>Manually set N or exit out of the loop on convergence.</p>
  <p>Is this the correct approach? Iterating multiple times over the episodes will produce more valuable state-action pairs in the Q table, because e greedy is not deterministic and for each iteration may exploit an action to greater reward than other episode epochs.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T15:14:23.087Z
Comments.Text: do you have a fixed data set or are you able to generate data from the environment? if it is the latter, then yes your approach is correct
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T15:29:15.930Z
Comments.Text: @David Ireland the dataset is fixed as for a given time interval the price values do not change. I'm confused by your comment 'generate data' as the model utilises data from the environment, the source of the environment pricing data is a trading exchange. So I can generate data by reading the pricing data but I'm not sure if I'm interpreting correctly 'generate data from the environment' .
Comments.UserDisplayName: blue-sky
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T16:58:01.330Z
Comments.Text: The way I understood your post was that you have an episode that is 24 steps long. Each step you get a series of prices and you have to decide whether to hold/sell/buy. If so, then are you able to interact with an environment arbitrarily, e.g. can you run simulations of this? or is your data set fixed?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T17:45:11.563Z
Comments.Text: @DavidIreland yes can run simulations where the reward of choosing buy/hold/sell for each step is observed. The dataset is fixed in that the number of prices for the environment is immutable. The episode length is mutable, for example If I set the episode length from 1 to two hours then there are 12 episodes available and the number of price attributes for each state will double.
Comments.UserDisplayName: blue-sky
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T18:32:30.247Z
Comments.Text: Okay, then it seems like it should work by following the table. However, is your state space continuous? If so, then you may want to use Deep Q-Learning.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T18:43:31.400Z
Comments.Text: @DavidIreland the state space is continuous. I'm considering Deep Q-Learning but I'm also considering quantizing the data or using an auto encoder to try and ensure states are revisited during training. Do you want to provide an answer to this question or I'll collate your comments and provide an answer.
Comments.UserDisplayName: blue-sky
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T18:46:50.310Z
Comments.Text: I will write an answer for DQN.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><algorithmic-trading>
Title: Looping over Sarsa algorithm for better Q values
UserDisplayName: blue-sky
ThreadText: # Looping over Sarsa algorithm for better Q values
  
  (Asked by: username_0 on 2020-06-18)
  
  Let's say an RL trading system places trades based on pricing data.
  Each episode represents 1 hour of trading, and there are 24 hours of data available. The Q table represents for a given state, what is the most action with the highest utility.
  The state is a sequence of prices, and the action is either buy, hold, sell.
  Instead of "Loop for each episode" as per the Sarsa algorithm :
  
  I add an additional outer loop. Now instead of just looping for each episode we have:
  for 1 to N:     
     "Loop for each episode"
  
  Manually set N or exit out of the loop on convergence.
  Is this the correct approach? Iterating multiple times over the episodes will produce more valuable state-action pairs in the Q table, because e greedy is not deterministic and for each iteration may exploit an action to greater reward than other episode epochs.
  
  **Tags:** reinforcement-learning, algorithmic-trading
  > **Comments:**
  > **username_1**: do you have a fixed data set or are you able to generate data from the environment? if it is the latter, then yes your approach is correct  
  > **username_0**: @David Ireland the dataset is fixed as for a given time interval the price values do not change. I'm confused by your comment 'generate data' as the model utilises data from the environment, the source of the environment pricing data is a trading exchange. So I can generate data by reading the pricing data but I'm not sure if I'm interpreting correctly 'generate data from the environment' .  
  > **username_1**: The way I understood your post was that you have an episode that is 24 steps long. Each step you get a series of prices and you have to decide whether to hold/sell/buy. If so, then are you able to interact with an environment arbitrarily, e.g. can you run simulations of this? or is your data set fixed?  
  > **username_0**: @DavidIreland yes can run simulations where the reward of choosing buy/hold/sell for each step is observed. The dataset is fixed in that the number of prices for the environment is immutable. The episode length is mutable, for example If I set the episode length from 1 to two hours then there are 12 episodes available and the number of price attributes for each state will double.  
  > **username_1**: Okay, then it seems like it should work by following the table. However, is your state space continuous? If so, then you may want to use Deep Q-Learning.  
  > **username_0**: @DavidIreland the state space is continuous. I'm considering Deep Q-Learning but I'm also considering quantizing the data or using an auto encoder to try and ensure states are revisited during training. Do you want to provide an answer to this question or I'll collate your comments and provide an answer.  
  > **username_1**: I will write an answer for DQN.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-18. Score: 1 | Accepted answer
  
  Given that your state space is continuous, then I would recommend using [Deep Q-Learning](https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf). As you say, running several episodes will definitely be beneficial so that the agent is able to explore the space more thoroughly.
  
  
  ---
  
  