Answers.Body: <p>Sounds a little like your algorithm is giving expert advice already.  You either buy and HODL or don't buy at all.</p>
  <p>I'm not sure you will find literature on designing your environment.  Trading bots are relatively closely held secrets.  Another mechanism to consider would be a punishment for holding a stock, especially with a ramping intensity (-0.1, -0.3, -0.6, etc).  This type of thing often is used to encourage RL to find shortest paths to a goal in mazes.  But it's hard to blame an algorithm for deciding not to participate in such an environment given the choice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T15:36:31.010Z
Answers.LastActivityDate: 2023-10-14T15:49:52.627Z
Answers.LastEditDate: 2023-10-14T15:49:52.627Z
Answers.UserDisplayName: foreverska
Body: <p>I am playing around with RL to develop a trading bot (using DQN). (Disclaimer: I know, that short term stock movements are near-random and having a bot that is actually useful not likely to happen. But since I'm currently only evaluating on training data, I should be able to get a bot to overfit on the training data and perform seemingly well).</p>
  <p>The environment is such that each state corresponds to a point in time, actions are [buy, hold, sell], the bot knows whether currently bought in (and since how many time steps) and how the current price relates to the buy price, as well as information about price movements. The reward for sell actions is the gain made from the whole buy-sell combination and I have experimented with and without using reward distribution from the sell action across preceeding hold and buy actions.</p>
  <p>Since I want the bot to learn to sell fairly quickly (instead of holding forever), I set up the simulation to auto-sell if the bot hasn't sold after three weeks. (Remember, the number of time slots since the buy is a feature of the training data). The problem is that the network always converges to either extreme of this time period. The bot either keeps holding, forcing auto-sell in each simulation run or - if I make the reward of the auto-sell actions more negative - the bot starts to always sell on the first time slot right after buying.</p>
  <p>Is there literature on rewarding/punishing automated actions such as the auto-sell? When thinking about it, presuming we have an already well-performing network, then on average the automated sell will yield a lower reward (the expected value over the long run is around zero, minus the trading fees) than the rewards yielded by sells from the well-performing bot. Hence, the behavior should converge away from the auto-sell instances even without specific punishment, no?</p>
  
Tags: <reinforcement-learning><deep-rl><dqn><algorithmic-trading>
Title: Trading bot with RL, automated actions, nonconvergence
UserDisplayName: Andy
ThreadText: # Trading bot with RL, automated actions, nonconvergence
  
  (Asked by: username_0 on 2023-10-14)
  
  I am playing around with RL to develop a trading bot (using DQN). (Disclaimer: I know, that short term stock movements are near-random and having a bot that is actually useful not likely to happen. But since I'm currently only evaluating on training data, I should be able to get a bot to overfit on the training data and perform seemingly well).
  
  The environment is such that each state corresponds to a point in time, actions are [buy, hold, sell], the bot knows whether currently bought in (and since how many time steps) and how the current price relates to the buy price, as well as information about price movements. The reward for sell actions is the gain made from the whole buy-sell combination and I have experimented with and without using reward distribution from the sell action across preceeding hold and buy actions.
  
  Since I want the bot to learn to sell fairly quickly (instead of holding forever), I set up the simulation to auto-sell if the bot hasn't sold after three weeks. (Remember, the number of time slots since the buy is a feature of the training data). The problem is that the network always converges to either extreme of this time period. The bot either keeps holding, forcing auto-sell in each simulation run or - if I make the reward of the auto-sell actions more negative - the bot starts to always sell on the first time slot right after buying.
  
  Is there literature on rewarding/punishing automated actions such as the auto-sell? When thinking about it, presuming we have an already well-performing network, then on average the automated sell will yield a lower reward (the expected value over the long run is around zero, minus the trading fees) than the rewards yielded by sells from the well-performing bot. Hence, the behavior should converge away from the auto-sell instances even without specific punishment, no?
  
  **Tags:** reinforcement-learning, deep-rl, dqn, algorithmic-trading
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-14. Score: 1
  
  Sounds a little like your algorithm is giving expert advice already. You either buy and HODL or don't buy at all.
  
  I'm not sure you will find literature on designing your environment. Trading bots are relatively closely held secrets. Another mechanism to consider would be a punishment for holding a stock, especially with a ramping intensity (-0.1, -0.3, -0.6, etc). This type of thing often is used to encourage RL to find shortest paths to a goal in mazes. But it's hard to blame an algorithm for deciding not to participate in such an environment given the choice.
  
  
  ---
  
  