<p>Given that your state space is continuous, then I would recommend using <a href="https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf" rel="nofollow noreferrer">Deep Q-Learning</a>. As you say, running several episodes will definitely be beneficial so that the agent is able to explore the space more thoroughly.</p>

CC BY-SA 4.0
2020-06-18T18:48:52.900Z
2020-06-18T18:48:52.900Z
David
<p>Let's say an RL trading system places trades based on pricing data.</p>
<p>Each episode represents 1 hour of trading, and there are 24 hours of data available. The Q table represents for a given state, what is the most action with the highest utility.</p>
<p>The state is a sequence of prices, and the action is either buy, hold, sell.</p>
<p>Instead of &quot;Loop for each episode&quot; as per the Sarsa algorithm :</p>
<p><a href="https://i.sstatic.net/tG9O1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tG9O1.png" alt="enter image description here" /></a></p>
<p>I add an additional outer loop. Now instead of just looping for each episode we have:</p>
<pre><code>for 1 to N:     
   &quot;Loop for each episode&quot;
</code></pre>
<p>Manually set N or exit out of the loop on convergence.</p>
<p>Is this the correct approach? Iterating multiple times over the episodes will produce more valuable state-action pairs in the Q table, because e greedy is not deterministic and for each iteration may exploit an action to greater reward than other episode epochs.</p>

CC BY-SA 4.0
2020-06-18T15:14:23.087Z
do you have a fixed data set or are you able to generate data from the environment? if it is the latter, then yes your approach is correct
David
CC BY-SA 4.0
2020-06-18T15:29:15.930Z
@David Ireland the dataset is fixed as for a given time interval the price values do not change. I'm confused by your comment 'generate data' as the model utilises data from the environment, the source of the environment pricing data is a trading exchange. So I can generate data by reading the pricing data but I'm not sure if I'm interpreting correctly 'generate data from the environment' .
blue-sky
CC BY-SA 4.0
2020-06-18T16:58:01.330Z
The way I understood your post was that you have an episode that is 24 steps long. Each step you get a series of prices and you have to decide whether to hold/sell/buy. If so, then are you able to interact with an environment arbitrarily, e.g. can you run simulations of this? or is your data set fixed?
David
CC BY-SA 4.0
2020-06-18T17:45:11.563Z
@DavidIreland yes can run simulations where the reward of choosing buy/hold/sell for each step is observed. The dataset is fixed in that the number of prices for the environment is immutable. The episode length is mutable, for example If I set the episode length from 1 to two hours then there are 12 episodes available and the number of price attributes for each state will double.
blue-sky
CC BY-SA 4.0
2020-06-18T18:32:30.247Z
Okay, then it seems like it should work by following the table. However, is your state space continuous? If so, then you may want to use Deep Q-Learning.
David
CC BY-SA 4.0
2020-06-18T18:43:31.400Z
@DavidIreland the state space is continuous. I'm considering Deep Q-Learning but I'm also considering quantizing the data or using an auto encoder to try and ensure states are revisited during training. Do you want to provide an answer to this question or I'll collate your comments and provide an answer.
blue-sky
CC BY-SA 4.0
2020-06-18T18:46:50.310Z
I will write an answer for DQN.
David
<reinforcement-learning><algorithmic-trading>
Looping over Sarsa algorithm for better Q values
blue-sky
# Looping over Sarsa algorithm for better Q values

(Asked by: username_0 on 2020-06-18)

Let's say an RL trading system places trades based on pricing data.
Each episode represents 1 hour of trading, and there are 24 hours of data available. The Q table represents for a given state, what is the most action with the highest utility.
The state is a sequence of prices, and the action is either buy, hold, sell.
Instead of "Loop for each episode" as per the Sarsa algorithm :

I add an additional outer loop. Now instead of just looping for each episode we have:
for 1 to N:     
   "Loop for each episode"

Manually set N or exit out of the loop on convergence.
Is this the correct approach? Iterating multiple times over the episodes will produce more valuable state-action pairs in the Q table, because e greedy is not deterministic and for each iteration may exploit an action to greater reward than other episode epochs.

**Tags:** reinforcement-learning, algorithmic-trading

> **Comments:**
> **username_1**: do you have a fixed data set or are you able to generate data from the environment? if it is the latter, then yes your approach is correct  
> **username_0**: @username_1 Ireland the dataset is fixed as for a given time interval the price values do not change. I'm confused by your comment 'generate data' as the model utilises data from the environment, the source of the environment pricing data is a trading exchange. So I can generate data by reading the pricing data but I'm not sure if I'm interpreting correctly 'generate data from the environment' .  
> **username_1**: The way I understood your post was that you have an episode that is 24 steps long. Each step you get a series of prices and you have to decide whether to hold/sell/buy. If so, then are you able to interact with an environment arbitrarily, e.g. can you run simulations of this? or is your data set fixed?  
> **username_0**: @username_1Ireland yes can run simulations where the reward of choosing buy/hold/sell for each step is observed. The dataset is fixed in that the number of prices for the environment is immutable. The episode length is mutable, for example If I set the episode length from 1 to two hours then there are 12 episodes available and the number of price attributes for each state will double.  
> **username_1**: Okay, then it seems like it should work by following the table. However, is your state space continuous? If so, then you may want to use Deep Q-Learning.  
> **username_0**: @username_1Ireland the state space is continuous. I'm considering Deep Q-Learning but I'm also considering quantizing the data or using an auto encoder to try and ensure states are revisited during training. Do you want to provide an answer to this question or I'll collate your comments and provide an answer.  
> **username_1**: I will write an answer for DQN.  

---

## 1 answers

### Answer by username_1 on 2020-06-18. Score: 1 | Accepted answer

Given that your state space is continuous, then I would recommend using [Deep Q-Learning](https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf). As you say, running several episodes will definitely be beneficial so that the agent is able to explore the space more thoroughly.


---

