Answers.Body: <p>Generally researchers (<a href="https://cs.adelaide.edu.au/%7Ezbyszek/Papers/trading.pdf" rel="nofollow noreferrer">Ghandar</a> et al, <a href="https://cs.adelaide.edu.au/%7Ezbyszek/papers.html" rel="nofollow noreferrer">Michalewicz</a>, <a href="https://www.researchgate.net/publication/3903854_A_genetic_fuzzy_expert_system_for_stock_market_timing" rel="nofollow noreferrer">Lam</a>) have used the profit or return on investment (ROI) as a reward (fitness) function.</p>
  <p><span class="math-container">$ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) \right] - \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }{ \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }$</span></p>
  <p>where <span class="math-container">$I_b(t)$</span> and <span class="math-container">$I_s(t)$</span> are equal to one if a rule signals a buy and sell, respectively, and zero otherwise; <span class="math-container">$sc$</span> represents the selling cost and <span class="math-container">$bc$</span> the buying cost. ROI is the difference between final bank balance and starting bank balance after trading.</p>
  <p>You are correct, that the machine learning algorithm will then be influenced by spikes just before a sell.</p>
  <p><a href="http://www.aiecon.org/conference/2008/CIEF/Evaluation%20of%20Fitness%20Functions%20for%20Evolved%20Stock%20Market%20Forecasting/s98028571.pdf" rel="nofollow noreferrer">Nicholls et al</a> showed that using the average profit or area under the trade resulted in better performing trading rules. This approach was used by <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.61.6892&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">Schoreels et al</a>. This approach focuses on being in the market to capitalize on profit. It does not penalize the trading rule when it is in the market and the market is going down. The accumulated asset value (AAV) is defined as:</p>
  <p><span class="math-container">$AAV = \frac{\sum_{i=1}^N [(Price_s - sc) - (Price_b + bc)]}{N}$</span></p>
  <p>where <span class="math-container">$i$</span> is a buy and sell trading event, <span class="math-container">$N$</span> is the number of buy and sell events, <span class="math-container">$s$</span> the day the sale took place, and <span class="math-container">$b$</span> is the day the purchase took place.</p>
  <p><a href="http://hdl.handle.net/2263/68477" rel="nofollow noreferrer">Nicholls MSc thesis</a> [available April 2019] showed that the fitness function used by <a href="https://www.cs.montana.edu/courses/spring2007/536/materials/Lopez/genetic.pdf" rel="nofollow noreferrer">Allen and Karjalainen</a> is the preferred fitness function when evolving trading rules for the JSE using evolutionary programs.</p>
  <p>Allen and Karjalainen used a fitness function based on the compounded excess returns over the buy-and-hold (buy the first day, sell the last day) strategy. The excess return is given by:</p>
  <p><span class="math-container">$\Delta r = r - r_{bh}$</span></p>
  <p>where the continuously compounded return of the trading rule is computed as</p>
  <p><span class="math-container">$r = \sum_{t=1}^T r_i I_b(t) + \sum_{t=1}^T r_f I_s(t) + n\log\left(\frac{1-c}{1+c'}\right)$</span></p>
  <p>and the return for the buy-and-hold strategy is calculated as</p>
  <p><span class="math-container">$r_{bh} = \sum_{t=1}^T r_t + \log\left(\frac{1-c}{1+c'}\right)$</span></p>
  <p>In the above,</p>
  <p><span class="math-container">$r_i = \log P_t - \log P_{t-1}$</span></p>
  <p>and <span class="math-container">$P$</span> is the daily close price for a given day <span class="math-container">$t$</span>, <span class="math-container">$c$</span> denotes the one-way transaction cost; <span class="math-container">$r_f$</span> is the risk free cost when the trader is not trading, <span class="math-container">$I_b(t)$</span> and <span class="math-container">$I_s(t)$</span> are equal to one if a rule signals buy and sell, respectively, and zero otherwise; <span class="math-container">$n$</span> denotes the number of trades and <span class="math-container">$r_{bh}$</span> represents the returns of a buy-and-hold, while <span class="math-container">$r$</span> represents the returns of the trader.</p>
  <p>A fixed trading cost of <span class="math-container">$c = 0.25\%$</span> of the transaction was defined but this could be anything like a STATE fee + Brocker fee + Tax, and might even be 2 different values, one for buying and one for selling. Which was the approach used by Nicholls. The continuously compounded return function rewards an individual when the share value is dropping and the individual is out of the market. The continuously compounded return function penalises the individual when the market is rising and the individual is out of the market.</p>
  <p>I would recommend that you use the compounded excess return over the buy and hold strategy as your reward function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-31T16:05:05.567Z
Answers.Text: Thanks for sharing the detailed answer. Would you mind explaining what $c'$ is in the denominator of the $\log$, as well as what $r_f$, the "risk free cost when the trader is not trading" is?
Answers.UserDisplayName: PyRsquared
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-31T16:51:55.067Z
Answers.Text: @PyRsquared $c′$ is the trading cost when selling, while $c$ is the trading cost when buying.  Allen and Karjalainen used a fixed value for both $c$ and $c'$ of $0.25\%$ of the trading price, while Nicholls et al used two different values one for buying and one for selling. $r_{f}$ is the risk free rate of return, this is the interest one would get if the money stayed in the bank eg $10\%$ would be $1.01$. Nicholls et al used a value of $1$ which results in the day's price
Answers.UserDisplayName: Jason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-31T16:58:42.017Z
Answers.Text: @PyRsquared  rf is the risk free rate of return per day so the 10% in my previous comment is crazy high, it should be divided by 365, and of course that is then compounded, to keep it simple, I kept it at 1.
Answers.UserDisplayName: Jason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-31T20:39:32.787Z
Answers.Text: in any case, don't the log terms cancel out once you subtract $r_{bh}$ from $r$?
Answers.UserDisplayName: PyRsquared
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-27T11:47:12.553Z
Answers.LastActivityDate: 2020-07-06T05:03:32.030Z
Answers.LastEditDate: 2020-07-06T05:03:32.030Z
Answers.UserDisplayName: Jason
Body: <p>I am working to build a deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are <code>{&quot;Buy&quot;: 0 , &quot;Do Nothing&quot;: 1, &quot;Sell&quot;: 2}</code>.</p>
  <p>Suppose that all the features are well suited for this task. I wanted to use just the standard &quot;Profit &amp; Loss&quot; as a reward, but I hardly thought to get something similar to the above image. The standard P&amp;L will simply place the pair (limit buy order, limit sell order) on every up movement. I don't want that because very often it won't cover the commission and it is not a good indicator to trade manually. I would be interested that the agent can maximize the profit and give me a minimum profit of $100 on every pair (limit buy order, limit sell order).</p>
  <p>I would be interested in something similar to the picture below.</p>
  <p><a href="https://i.sstatic.net/1PlBP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1PlBP.png" alt="enter image description here" /></a></p>
  <p>Is there a reward function that could allow me to get such a result? If so, what is it?</p>
  <p><strong>UPDATE</strong></p>
  <p>Is the following utility function can work with the purpose of that question?</p>
  <p><span class="math-container">$$
  U(x) = \max(\\\$100, x)
  $$</span></p>
  <p>That seems correct, but I don't know how the agent will be penalized if it covers a wrong transaction, i.e. the pair (limit buy order, limit sell order) creates a loss of money.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-21T08:02:40.420Z
Comments.Text: Today we have a relevant xkcd: https://xkcd.com/2101/ - I have re-read your question and I think that the big issue for you is "Suppose that all the features are well suited for this task." I am certain that no amount of RL will help you unless you solve the prediction problem *first*. Looking at reward functions is not going to help you. RL does not magically make the prediction problem go away, it just obscures it. However, if you could create a predictive model that does better than random luck, then you could use RL to discover an investment strategy.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-21T15:01:11.460Z
Comments.Text: Use the prediction, for multiple time steps ahead, as state features that help the RL to learn a strategy. For this to work, you need your prediction model to have some accuracy better than random guessing at the time scale and profit amounts that you care about.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-21T15:03:08.273Z
Comments.Text: Please take care on this project. Applying ML to market data is not a new thing, and professional traders with strong math and computing knowledge have already taken  the low-hanging fruit years ago. Your best outcome is likely learning enough to appreciate how hard the problem is. Your worst outcome is thinking you've solved it then losing all your savings when it turns out that was just a random test result.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-21T20:36:10.710Z
Comments.Text: @NeilSlater I need someone to adjust the code so that we can have predict more than one step ahead. I can surely pay you for your job
Comments.UserDisplayName: fgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-21T20:41:07.357Z
Comments.Text: Not interested, sorry, and I think you will find that the further you predict ahead, the worse the predictions will become (like weather forecasting), quickly becoming useless for your purpose. However, that does seem like the kind of question you could ask on DataScience stack exchange - explain the architecture of your predictive model, including the rough nature of inputs, and ask how it could be adapted from single-step to multi-step predictions.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-25T11:00:34.353Z
Comments.Text: While the ML approach to this kind of problems could be interesting, I feel like this is a quantitative finance question, more precisely a question of taking into account market friction in portfolio optimisation. I suggest you read the latest reasearch on the subject to undestand the formalism and the limitations of the standard approaches (combinatorial explosion, lack of predictive value). The bottom line is that this problem won't probably be magically solved by ML. If the features were well suited for this task the problem would have been solved a long time ago.
Comments.UserDisplayName: Lucas Morin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-27T13:59:47.620Z
Comments.Text: @fgauth there are better alternatives to a neural network for trading rule generation. Have you looked at Genetic Programs?
Comments.UserDisplayName: Jason
Tags: <reinforcement-learning><deep-rl><reward-functions><reward-design><algorithmic-trading>
Title: Suitable reward function for trading buy and sell orders
UserDisplayName: fgauth
ThreadText: # Suitable reward function for trading buy and sell orders
  
  (Asked by: username_0 on 2019-01-20)
  
  I am working to build a deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are `{"Buy": 0 , "Do Nothing": 1, "Sell": 2}`.
  
  Suppose that all the features are well suited for this task. I wanted to use just the standard "Profit & Loss" as a reward, but I hardly thought to get something similar to the above image. The standard P&L will simply place the pair (limit buy order, limit sell order) on every up movement. I don't want that because very often it won't cover the commission and it is not a good indicator to trade manually. I would be interested that the agent can maximize the profit and give me a minimum profit of $100 on every pair (limit buy order, limit sell order).
  
  I would be interested in something similar to the picture below.
  
  [![enter image description here](https://i.sstatic.net/1PlBP.png)](https://i.sstatic.net/1PlBP.png)
  
  Is there a reward function that could allow me to get such a result? If so, what is it?
  
  **UPDATE**
  
  Is the following utility function can work with the purpose of that question?
  
  $$
  U(x) = \max(\\\$100, x)
  $$
  
  That seems correct, but I don't know how the agent will be penalized if it covers a wrong transaction, i.e. the pair (limit buy order, limit sell order) creates a loss of money.
  
  **Tags:** reinforcement-learning, deep-rl, reward-functions, reward-design, algorithmic-trading
  
  > **Comments:**
  > **username_1**: Today we have a relevant xkcd: https://xkcd.com/2101/ - I have re-read your question and I think that the big issue for you is "Suppose that all the features are well suited for this task." I am certain that no amount of RL will help you unless you solve the prediction problem *first*. Looking at reward functions is not going to help you. RL does not magically make the prediction problem go away, it just obscures it. However, if you could create a predictive model that does better than random luck, then you could use RL to discover an investment strategy.  
  > **username_1**: Use the prediction, for multiple time steps ahead, as state features that help the RL to learn a strategy. For this to work, you need your prediction model to have some accuracy better than random guessing at the time scale and profit amounts that you care about.  
  > **username_1**: Please take care on this project. Applying ML to market data is not a new thing, and professional traders with strong math and computing knowledge have already taken  the low-hanging fruit years ago. Your best outcome is likely learning enough to appreciate how hard the problem is. Your worst outcome is thinking you've solved it then losing all your savings when it turns out that was just a random test result.  
  > **username_0**: @username_1 I need someone to adjust the code so that we can have predict more than one step ahead. I can surely pay you for your job  
  > **username_1**: Not interested, sorry, and I think you will find that the further you predict ahead, the worse the predictions will become (like weather forecasting), quickly becoming useless for your purpose. However, that does seem like the kind of question you could ask on DataScience stack exchange - explain the architecture of your predictive model, including the rough nature of inputs, and ask how it could be adapted from single-step to multi-step predictions.  
  > **username_2**: While the ML approach to this kind of problems could be interesting, I feel like this is a quantitative finance question, more precisely a question of taking into account market friction in portfolio optimisation. I suggest you read the latest reasearch on the subject to undestand the formalism and the limitations of the standard approaches (combinatorial explosion, lack of predictive value). The bottom line is that this problem won't probably be magically solved by ML. If the features were well suited for this task the problem would have been solved a long time ago.  
  > **username_3**: @username_0 there are better alternatives to a neural network for trading rule generation. Have you looked at Genetic Programs?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-02-27. Score: 9
  
  Generally researchers ([Ghandar](https://cs.adelaide.edu.au/%7Ezbyszek/Papers/trading.pdf) et al, [Michalewicz](https://cs.adelaide.edu.au/%7Ezbyszek/papers.html), [Lam](https://www.researchgate.net/publication/3903854_A_genetic_fuzzy_expert_system_for_stock_market_timing)) have used the profit or return on investment (ROI) as a reward (fitness) function.
  
  $ROI = \frac{ \left[\sum\_{t=1}^T (Price\_t - sc) \times I\_s(t) \right] - \left[ \sum\_{t=1}^T (Price\_t + bc) \times I\_b(t) \right] }{ \left[ \sum\_{t=1}^T (Price\_t + bc) \times I\_b(t) \right] }$
  
  where $I\_b(t)$ and $I\_s(t)$ are equal to one if a rule signals a buy and sell, respectively, and zero otherwise; $sc$ represents the selling cost and $bc$ the buying cost. ROI is the difference between final bank balance and starting bank balance after trading.
  
  You are correct, that the machine learning algorithm will then be influenced by spikes just before a sell.
  
  [Nicholls et al](http://www.aiecon.org/conference/2008/CIEF/Evaluation%20of%20Fitness%20Functions%20for%20Evolved%20Stock%20Market%20Forecasting/s98028571.pdf) showed that using the average profit or area under the trade resulted in better performing trading rules. This approach was used by [Schoreels et al](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.61.6892&rep=rep1&type=pdf). This approach focuses on being in the market to capitalize on profit. It does not penalize the trading rule when it is in the market and the market is going down. The accumulated asset value (AAV) is defined as:
  
  $AAV = \frac{\sum\_{i=1}^N [(Price\_s - sc) - (Price\_b + bc)]}{N}$
  
  where $i$ is a buy and sell trading event, $N$ is the number of buy and sell events, $s$ the day the sale took place, and $b$ is the day the purchase took place.
  
  [Nicholls MSc thesis](http://hdl.handle.net/2263/68477) [available April 2019] showed that the fitness function used by [Allen and Karjalainen](https://www.cs.montana.edu/courses/spring2007/536/materials/Lopez/genetic.pdf) is the preferred fitness function when evolving trading rules for the JSE using evolutionary programs.
  
  Allen and Karjalainen used a fitness function based on the compounded excess returns over the buy-and-hold (buy the first day, sell the last day) strategy. The excess return is given by:
  
  $\Delta r = r - r\_{bh}$
  
  where the continuously compounded return of the trading rule is computed as
  
  $r = \sum\_{t=1}^T r\_i I\_b(t) + \sum\_{t=1}^T r\_f I\_s(t) + n\log\left(\frac{1-c}{1+c'}\right)$
  
  and the return for the buy-and-hold strategy is calculated as
  
  $r\_{bh} = \sum\_{t=1}^T r\_t + \log\left(\frac{1-c}{1+c'}\right)$
  
  In the above,
  
  $r\_i = \log P\_t - \log P\_{t-1}$
  
  and $P$ is the daily close price for a given day $t$, $c$ denotes the one-way transaction cost; $r\_f$ is the risk free cost when the trader is not trading, $I\_b(t)$ and $I\_s(t)$ are equal to one if a rule signals buy and sell, respectively, and zero otherwise; $n$ denotes the number of trades and $r\_{bh}$ represents the returns of a buy-and-hold, while $r$ represents the returns of the trader.
  
  A fixed trading cost of $c = 0.25\%$ of the transaction was defined but this could be anything like a STATE fee + Brocker fee + Tax, and might even be 2 different values, one for buying and one for selling. Which was the approach used by Nicholls. The continuously compounded return function rewards an individual when the share value is dropping and the individual is out of the market. The continuously compounded return function penalises the individual when the market is rising and the individual is out of the market.
  
  I would recommend that you use the compounded excess return over the buy and hold strategy as your reward function.
  
  > **Comments:**
  > **username_4**: Thanks for sharing the detailed answer. Would you mind explaining what $c'$ is in the denominator of the $\log$, as well as what $r_f$, the "risk free cost when the trader is not trading" is?  
  > **username_3**: @username_4 $c′$ is the trading cost when selling, while $c$ is the trading cost when buying.  Allen and Karjalainen used a fixed value for both $c$ and $c'$ of $0.25\%$ of the trading price, while Nicholls et al used two different values one for buying and one for selling. $r_{f}$ is the risk free rate of return, this is the interest one would get if the money stayed in the bank eg $10\%$ would be $1.01$. Nicholls et al used a value of $1$ which results in the day's price  
  > **username_3**: @username_4  rf is the risk free rate of return per day so the 10% in my previous comment is crazy high, it should be divided by 365, and of course that is then compounded, to keep it simple, I kept it at 1.  
  > **username_4**: in any case, don't the log terms cancel out once you subtract $r_{bh}$ from $r$?  
  
  ---
  
  