Body: <p>I am playing around with a stock trading agent trained via (deep) reinforcement learning, including memory replay. The agent is trained for 1000 episodes, where each episode consists of 180 timesteps (e.g. daily stock prices).</p>
  <p>My question is concerning the sampling of episodes for training.</p>
  <p>Assuming I've got daily stock prices going back 3 years, that's about 750 trading days/prices.</p>
  <p>How should I sample this data set to get enough episodes for training?</p>
  <p>With an episode length of 180 and an episode count of 1000, I'd need 180k &quot;days&quot; to choose from, if I wouldn't want any duplication.</p>
  <p>Do I even need to sample 1000 non-overlapping windows from my dataset or can I sample my episodes using a sliding window approach? Could I even just randomly sample the dataset for episodes? For example, calculate a random date and build the episode from the 180 days following that random starting date?</p>
  <p>The reward for each action is calculated as follows, <code>p</code> are the prices and <code>t</code> is the current timestep of the episode.</p>
  <ul>
  <li>CASH: <code>0</code></li>
  <li>BUY:  <code>p(t+1) - p(t) - fee</code></li>
  <li>HOLD: <code>p(t+1) - p(t)</code></li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T13:33:28.160Z
Comments.Text: Are you want to predict the stock price using RL?
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T14:16:04.933Z
Comments.Text: No, the model predicts one out of three actions (open/close/hold).
Comments.UserDisplayName: Scarysize
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T14:23:35.300Z
Comments.Text: Why don't use the sliding window and iterative the whole data over and over. You can also go to investing.com and find more training data.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T14:59:48.810Z
Comments.Text: That is the question, whether it is "ok" to use a sliding window or not. I can't get more data for stocks which have only been traded for a few years ;)
Comments.UserDisplayName: Scarysize
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T15:32:03.267Z
Comments.Text: I may be wrong about your problem, but I think you are trying to make action (open/clsoe/hold) based on previous days stock price. If this is the case the next state is independent of our current action if you take sliding window aproach. How you formulated the MDP? I think it's a contextual bandit problem for the sliding window approach. Same goes to other methods.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T16:02:06.300Z
Comments.Text: Yes the states should be independent of the action taken & from each other. I've edited the question to clarify how the reward for each action is calculated.
Comments.UserDisplayName: Scarysize
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T16:11:59.400Z
Comments.Text: what is the fee?
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T16:13:59.083Z
Comments.Text: Make the state as follows $s_{t} = (p_{t},p_{t-1},,...,p_{t-n}, cash \ in \ hand)$, then it will be a complete RL problem.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T16:14:46.727Z
Comments.Text: The fee models the transaction cost of buying a stock, e.g. 0.5% of the trade price.
Comments.UserDisplayName: Scarysize
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-10T16:23:10.437Z
Comments.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/111627/discussion-between-scarysize-and-swakshar-deb).
Comments.UserDisplayName: Scarysize
Tags: <training><deep-rl><reward-functions><algorithmic-trading>
Title: Given the daily stock prices of the last 3 years, how should I sample the training data for episodic RL?
UserDisplayName: Scarysize
ThreadText: # Given the daily stock prices of the last 3 years, how should I sample the training data for episodic RL?
  
  (Asked by: username_0 on 2020-08-10)
  
  I am playing around with a stock trading agent trained via (deep) reinforcement learning, including memory replay. The agent is trained for 1000 episodes, where each episode consists of 180 timesteps (e.g. daily stock prices).
  
  My question is concerning the sampling of episodes for training.
  
  Assuming I've got daily stock prices going back 3 years, that's about 750 trading days/prices.
  
  How should I sample this data set to get enough episodes for training?
  
  With an episode length of 180 and an episode count of 1000, I'd need 180k "days" to choose from, if I wouldn't want any duplication.
  
  Do I even need to sample 1000 non-overlapping windows from my dataset or can I sample my episodes using a sliding window approach? Could I even just randomly sample the dataset for episodes? For example, calculate a random date and build the episode from the 180 days following that random starting date?
  
  The reward for each action is calculated as follows, `p` are the prices and `t` is the current timestep of the episode.
  
  * CASH: `0`
  * BUY: `p(t+1) - p(t) - fee`
  * HOLD: `p(t+1) - p(t)`
  
  **Tags:** training, deep-rl, reward-functions, algorithmic-trading
  
  > **Comments:**
  > **username_1**: Are you want to predict the stock price using RL?  
  > **username_0**: No, the model predicts one out of three actions (open/close/hold).  
  > **username_1**: Why don't use the sliding window and iterative the whole data over and over. You can also go to investing.com and find more training data.  
  > **username_0**: That is the question, whether it is "ok" to use a sliding window or not. I can't get more data for stocks which have only been traded for a few years ;)  
  > **username_1**: I may be wrong about your problem, but I think you are trying to make action (open/clsoe/hold) based on previous days stock price. If this is the case the next state is independent of our current action if you take sliding window aproach. How you formulated the MDP? I think it's a contextual bandit problem for the sliding window approach. Same goes to other methods.  
  > **username_0**: Yes the states should be independent of the action taken & from each other. I've edited the question to clarify how the reward for each action is calculated.  
  > **username_1**: what is the fee?  
  > **username_1**: Make the state as follows $s_{t} = (p_{t},p_{t-1},,...,p_{t-n}, cash \ in \ hand)$, then it will be a complete RL problem.  
  > **username_0**: The fee models the transaction cost of buying a stock, e.g. 0.5% of the trade price.  
  > **username_0**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/111627/discussion-between-username_0-and-username_1-deb).  
  
  ---
  
  ## 0 answers
  
  