Body: <p>I've been playing around building a reinforcement learned-based trading bot using the <strong>stable-baselines3</strong> library. I've come up with an environment that seems to be able to learn how to make profitable trades on the dataset that I feed it. If I keep training long enough it is seemingly making good money.</p>
  <p>Now, I have a background in supervised machine learning, and I was already quite sure that my agent was &quot;overfitting&quot; to the dataset that I was feeding it. This was confirmed by my experiment: when feeding my trained model a new validation dataset the performance was on-par with a random agent, and it was no longer making any money. In other words, my agent manages to exploit relations in the training data very well, but when evaluating the out-of-sample performance on unseen test dataset it appears that my agent just learned noise. In the context of my trading bot it seems that the agent memorises the financial time-series, and it not actually useful for the purpose of trading as the patterns it learned are not generalisable.</p>
  <p>Now my question: how do we overcome this issue?</p>
  <p>In supervised learning, there are various ways to help prevent that our model will overfit. We can use a validation set during training so that we can apply early stopping (stop training when validation accuracy no longer improves). We can use regularization techniques such as drop-out layers. But how do we deal with the concept of overfitting in reinforcement learning? Some things I had in mind that could be useful:</p>
  <ul>
  <li>Does it make sense to apply regularization to my neural network that learns the value function and policy?</li>
  <li>Is there something equivalent to early stopping?</li>
  <li>Will adding more data or simply decreasing the number of timesteps help?</li>
  <li>Could it possibly help to generate training batches by randomly selecting certain periods of my full financial time-series, rather than going through the same time-series over and over again? My assumption is that in that way the network no longer relies on the same sequence, and the patterns it learns could be more robust...</li>
  </ul>
  <p>Looking forward to hearing your view on this!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-17T05:19:15.133Z
Comments.Text: Can you comment on what you're doing a little further? I'm guessing you're representing the state space as price volume data/financial data/economic data/news surrounding a given security X. Your choice of action (buy/sell in x amount) supposing that you are not large enough to move the market does not affect the next state s' - in fact, modelling s' is the main task it seems. Furthermore, the reward function r(s,a) does depend on your action, but in a non-deterministic way - does your algorithm handle this? A little more information is needed to answer your question.
Comments.UserDisplayName: Snowball
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-17T05:23:07.200Z
Comments.Text: But to point you in a general direction, you're dealing with batch RL (as opposed to online RL). The difference is that in the batch RL setting, your dataset is fixed, and you do not get to interact with the environment to observe rewards and next states. For any bootstrapped target, in a continuous batch RL setting, you may encounter overestimation bias, which can lead your agent to behave poorly. This is somewhat analogous to overfitting in the supervised learning setting. I suggest reading this: https://arxiv.org/pdf/2005.01643.pdf
Comments.UserDisplayName: Snowball
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-17T05:34:45.487Z
Comments.Text: Finally, just to make a comment about the U.S. stock market: for most liquid stocks, in the short run, the price movements will be incredibly noisy (near-random), mostly governed by news (which, by definition is unknown information prior to it coming to light). The noise to signal ratio is incredibly large, so this problem is probably harder than any commonly available RL benchmark you can find. This may be why it seems like your agent is behaving randomly. Furthermore, it doesn't seem to me transition dynamics are stationary in financial markets - you may want to also consider this obstacle.
Comments.UserDisplayName: Snowball
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-17T19:00:38.727Z
Comments.Text: Thanks for your comments! Actually I am using data from cryptocurrencies, but the idea is indeed like you mention. I feed historical data related to price/volume to the agent as the state, and the action space is [hold, buy, sell]. The reward function is triggered when the agent buys or sells. The agent gets rewarded if it sold on a profitable trade (i.e., sell price is higher than buy price) or if it has bought low/avoided an unprofitable trade (i.e., if a buy is on a lower price than previous sell). I do am aware of the high signal-to-noise ratio, so have realistic expectations.
Comments.UserDisplayName: dubvice
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-17T19:02:33.057Z
Comments.Text: Additionally, the train and test dataset should be reasonable equal, as they both describe the price of the same asset (bitcoin) and have similar dynamics, in the sense that there are "bull markets", drawdowns, volatile periods, etc. So my hope is that is can extrapolate some the things learned from the train data to actually outperform a random agent on the test data.
Comments.UserDisplayName: dubvice
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-18T04:33:23.190Z
Comments.Text: Does the state space also contain your current portfolio position?
Comments.UserDisplayName: Snowball
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-18T04:34:09.047Z
Comments.Text: Prior to throwing it real data, I would try to generate some autocorrelated data with obvious momentum trends just to see if your setup is working on clearly inefficient markets.
Comments.UserDisplayName: Snowball
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-18T09:20:53.937Z
Comments.Text: Yes, the state space does include the current position [cash, bitcoin] and if it is holding bitcoin, the difference between the entry and current price. Moreover, I am using maskable PPO: https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html
  
  
  This way, I am preventing that the bot tries to buy bitcoin while it is already holding a position, or selling while it is holding cash.
Comments.UserDisplayName: dubvice
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-18T09:22:39.167Z
Comments.Text: Running a simulated timeseries through the bot surely sounds like a good idea to start with. Also the term "overestimation bias" seems to perfectly fit my case. I was curious to what extent general techniques in reinforcement learning exist to overcome this issue of "overestimation bias". My intuition says that stopping training at some point could help (i.e., using "early stopping"). But not sure if that's a thing in RL...
Comments.UserDisplayName: dubvice
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-19T23:40:34.263Z
Comments.Text: It's not exactly like overfitting in the supervised learning sense. What happens in a continuous action space is that you are using function approximation to estimate Q(s,a). For actions a_hat that are out of distribution (i.e. not drawn from your dataset), you have no idea what Q(s,a_hat) should be, and are forming an estimate based on your dataset. These estimates turn out to be very important, because TD learning relies on next state value estimates. If Q(s, a_hat) is erroneously high, due to overestimation, then you're going to overestimate the value of actions which you have not observed.
Comments.UserDisplayName: Snowball
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-19T23:41:18.257Z
Comments.Text: Essentially you mistakenly exploit areas of the Q-function which are erroneously high, and base your actions on this.
Comments.UserDisplayName: Snowball
Tags: <reinforcement-learning><deep-rl><time-series><stable-baselines>
Title: RL-based trading bot: how to deal with overfitting
UserDisplayName: dubvice
ThreadText: # RL-based trading bot: how to deal with overfitting
  
  (Asked by: username_0 on 2022-06-10)
  
  I've been playing around building a reinforcement learned-based trading bot using the **stable-baselines3** library. I've come up with an environment that seems to be able to learn how to make profitable trades on the dataset that I feed it. If I keep training long enough it is seemingly making good money.
  
  Now, I have a background in supervised machine learning, and I was already quite sure that my agent was "overfitting" to the dataset that I was feeding it. This was confirmed by my experiment: when feeding my trained model a new validation dataset the performance was on-par with a random agent, and it was no longer making any money. In other words, my agent manages to exploit relations in the training data very well, but when evaluating the out-of-sample performance on unseen test dataset it appears that my agent just learned noise. In the context of my trading bot it seems that the agent memorises the financial time-series, and it not actually useful for the purpose of trading as the patterns it learned are not generalisable.
  
  Now my question: how do we overcome this issue?
  
  In supervised learning, there are various ways to help prevent that our model will overfit. We can use a validation set during training so that we can apply early stopping (stop training when validation accuracy no longer improves). We can use regularization techniques such as drop-out layers. But how do we deal with the concept of overfitting in reinforcement learning? Some things I had in mind that could be useful:
  
  * Does it make sense to apply regularization to my neural network that learns the value function and policy?
  * Is there something equivalent to early stopping?
  * Will adding more data or simply decreasing the number of timesteps help?
  * Could it possibly help to generate training batches by randomly selecting certain periods of my full financial time-series, rather than going through the same time-series over and over again? My assumption is that in that way the network no longer relies on the same sequence, and the patterns it learns could be more robust...
  
  Looking forward to hearing your view on this!
  
  **Tags:** reinforcement-learning, deep-rl, time-series, stable-baselines
  > **Comments:**
  > **username_1**: Can you comment on what you're doing a little further? I'm guessing you're representing the state space as price volume data/financial data/economic data/news surrounding a given security X. Your choice of action (buy/sell in x amount) supposing that you are not large enough to move the market does not affect the next state s' - in fact, modelling s' is the main task it seems. Furthermore, the reward function r(s,a) does depend on your action, but in a non-deterministic way - does your algorithm handle this? A little more information is needed to answer your question.  
  > **username_1**: But to point you in a general direction, you're dealing with batch RL (as opposed to online RL). The difference is that in the batch RL setting, your dataset is fixed, and you do not get to interact with the environment to observe rewards and next states. For any bootstrapped target, in a continuous batch RL setting, you may encounter overestimation bias, which can lead your agent to behave poorly. This is somewhat analogous to overfitting in the supervised learning setting. I suggest reading this: https://arxiv.org/pdf/2005.01643.pdf  
  > **username_1**: Finally, just to make a comment about the U.S. stock market: for most liquid stocks, in the short run, the price movements will be incredibly noisy (near-random), mostly governed by news (which, by definition is unknown information prior to it coming to light). The noise to signal ratio is incredibly large, so this problem is probably harder than any commonly available RL benchmark you can find. This may be why it seems like your agent is behaving randomly. Furthermore, it doesn't seem to me transition dynamics are stationary in financial markets - you may want to also consider this obstacle.  
  > **username_0**: Thanks for your comments! Actually I am using data from cryptocurrencies, but the idea is indeed like you mention. I feed historical data related to price/volume to the agent as the state, and the action space is [hold, buy, sell]. The reward function is triggered when the agent buys or sells. The agent gets rewarded if it sold on a profitable trade (i.e., sell price is higher than buy price) or if it has bought low/avoided an unprofitable trade (i.e., if a buy is on a lower price than previous sell). I do am aware of the high signal-to-noise ratio, so have realistic expectations.  
  > **username_0**: Additionally, the train and test dataset should be reasonable equal, as they both describe the price of the same asset (bitcoin) and have similar dynamics, in the sense that there are "bull markets", drawdowns, volatile periods, etc. So my hope is that is can extrapolate some the things learned from the train data to actually outperform a random agent on the test data.  
  > **username_1**: Does the state space also contain your current portfolio position?  
  > **username_1**: Prior to throwing it real data, I would try to generate some autocorrelated data with obvious momentum trends just to see if your setup is working on clearly inefficient markets.  
  > **username_0**: Yes, the state space does include the current position [cash, bitcoin] and if it is holding bitcoin, the difference between the entry and current price. Moreover, I am using maskable PPO: https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html   This way, I am preventing that the bot tries to buy bitcoin while it is already holding a position, or selling while it is holding cash.  
  > **username_0**: Running a simulated timeseries through the bot surely sounds like a good idea to start with. Also the term "overestimation bias" seems to perfectly fit my case. I was curious to what extent general techniques in reinforcement learning exist to overcome this issue of "overestimation bias". My intuition says that stopping training at some point could help (i.e., using "early stopping"). But not sure if that's a thing in RL...  
  > **username_1**: It's not exactly like overfitting in the supervised learning sense. What happens in a continuous action space is that you are using function approximation to estimate Q(s,a). For actions a_hat that are out of distribution (i.e. not drawn from your dataset), you have no idea what Q(s,a_hat) should be, and are forming an estimate based on your dataset. These estimates turn out to be very important, because TD learning relies on next state value estimates. If Q(s, a_hat) is erroneously high, due to overestimation, then you're going to overestimate the value of actions which you have not observed.  
  > **username_1**: Essentially you mistakenly exploit areas of the Q-function which are erroneously high, and base your actions on this.  
  
  ---
  
  ## 0 answers
  
  