Answers.Body: <p>Your agent's actions will (probably) not have much impact on the observed financial time series. However, they will make a large difference to other things - namely what stock your agent is holding and the account balance.</p>
  <p>If you are happy to ignore the agent's current portfolio and balance as not part of your problem, effectively treating these items as infinite sinks, then yes a multi-armed bandit might be a reasonable solution. But then so might any other sequence-predicting algorithm, if what you are searching for is some kind of financial prediction of good times to buy or sell.</p>
  <p>If the portfolio and cash balance are an important part of your problem, you should add them to the state and use reinforcement learning. You might do this if your goal is to model a single investor playing the markets.</p>
  <p>Note that although it may be possible to use machine learning techniques to analyse markets, and base investments on advice of a trained AI agent, it is a very risky venture. There are lots of ways you can fool yourself into believing too strongly in your solution and you stand to lose significant amounts of money.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-04T17:47:16.020Z
Answers.Text: Thank you @NeilSlater! While playing with on-policy Monte Carlo algo I start to think that the same results could be achieved simply by taking the average return for every possible action(long/short) in every possible state. I mean we don't even need to explore/exploit... just find those values, perform walk-forward backtesting or s.th. like that and done. All that with RL starts to not make sense for me in terms of generating buy and sell trading signals.
Answers.UserDisplayName: kobo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-03T17:01:59.547Z
Answers.LastActivityDate: 2023-01-03T17:01:59.547Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>The question is whether the rewards are i.i.d. In such time-series problems, the rewards are not i.i.d; the reward at timestep t+1 depends on the reward at timestep t. Therefore, even though the actions/states may be somewhat independent, the rewards should i.i.d in order to apply MA bandits. Otherwise, the problem is about RL.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-03T18:49:39.617Z
Answers.LastActivityDate: 2023-01-03T18:49:39.617Z
Answers.UserDisplayName: ddaedalus
Body: <p>If we take simple financial timeseries data(stock/commodity/currency prices), State(t+1) does not depend on the action that we choose to take at State(t) as in Maze or Chess problem.</p>
  <p>Simple example: as states we can have the sum of the daily returns of 5 different ETFs. Based on that, we want to take action - either buy(go long) or sell(go short) in another ETF. No matter what we choose however, our action will not determine what the next state would be (we do not have any control of what the returns of those 5 ETFs will be tomorrow).</p>
  <p>In that case of simple financial time series data, would <strong>multi-armed-bandit</strong> approach be more suitable?</p>
  
Tags: <reinforcement-learning><comparison><time-series><multi-armed-bandits>
Title: Should I use multi-armed-bandits or RL for a financial time-series problem?
UserDisplayName: kobo
ThreadText: # Should I use multi-armed-bandits or RL for a financial time-series problem?
  
  (Asked by: username_0 on 2023-01-03)
  
  If we take simple financial timeseries data(stock/commodity/currency prices), State(t+1) does not depend on the action that we choose to take at State(t) as in Maze or Chess problem.
  
  Simple example: as states we can have the sum of the daily returns of 5 different ETFs. Based on that, we want to take action - either buy(go long) or sell(go short) in another ETF. No matter what we choose however, our action will not determine what the next state would be (we do not have any control of what the returns of those 5 ETFs will be tomorrow).
  
  In that case of simple financial time series data, would **multi-armed-bandit** approach be more suitable?
  
  **Tags:** reinforcement-learning, comparison, time-series, multi-armed-bandits
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-01-03. Score: 1 | Accepted answer
  
  Your agent's actions will (probably) not have much impact on the observed financial time series. However, they will make a large difference to other things - namely what stock your agent is holding and the account balance.
  
  If you are happy to ignore the agent's current portfolio and balance as not part of your problem, effectively treating these items as infinite sinks, then yes a multi-armed bandit might be a reasonable solution. But then so might any other sequence-predicting algorithm, if what you are searching for is some kind of financial prediction of good times to buy or sell.
  
  If the portfolio and cash balance are an important part of your problem, you should add them to the state and use reinforcement learning. You might do this if your goal is to model a single investor playing the markets.
  
  Note that although it may be possible to use machine learning techniques to analyse markets, and base investments on advice of a trained AI agent, it is a very risky venture. There are lots of ways you can fool yourself into believing too strongly in your solution and you stand to lose significant amounts of money.
  
  > **Comments:**
  > **username_0**: Thank you @username_1! While playing with on-policy Monte Carlo algo I start to think that the same results could be achieved simply by taking the average return for every possible action(long/short) in every possible state. I mean we don't even need to explore/exploit... just find those values, perform walk-forward backtesting or s.th. like that and done. All that with RL starts to not make sense for me in terms of generating buy and sell trading signals.  
  
  ---
  
  ### Answer by username_2 on 2023-01-03. Score: 0
  
  The question is whether the rewards are i.i.d. In such time-series problems, the rewards are not i.i.d; the reward at timestep t+1 depends on the reward at timestep t. Therefore, even though the actions/states may be somewhat independent, the rewards should i.i.d in order to apply MA bandits. Otherwise, the problem is about RL.
  
  
  ---
  
  