title: Vectorized Backtest -- Short returns not correct?
body: <h2>Problem Statement</h2>
  <p>I have a dataframe of <code>position_signal</code> and <code>close</code> (closing prices) for a given asset. You can think of each row being a specific time step (minute, hour, etc.). I want to run a vectorized backtest to calculate my aggregated portfolio return at each point in time whether i'm long, short or not holding. In which case the <code>position_signal</code> column is either <code>1</code>, <code>-1</code>, <code>0</code>, respectively.</p>
  <h2>What I've Tried</h2>
  <p>I've tried a lot... seems like an easy problem. There are <a href="https://www.oreilly.com/library/view/python-for-algorithmic/9781492053347/ch04.html" rel="nofollow noreferrer">a lot</a> of <a href="https://medium.com/@craigmariani47/back-test-your-trading-strategies-with-vectorization-in-python-c58209ff3b1b" rel="nofollow noreferrer">tutorials</a> online that <a href="https://hudsonthames.org/correct-backtest-methodology-pairs-trading/" rel="nofollow noreferrer">claim to do it</a>. However the problem with all of these is that they either: assume cumulative returns <em>during trading</em> (so each day's returns are multiplied by the next, which isn't right if you're just holding a position) or they don't calculate short returns correctly.</p>
  <p>The code for most of them boils down to this:</p>
  <pre><code>df['log_returns'] = np.log(df['close']) - np.log(df['close'].shift(1))
  df['strategy_returns'] = df['position_signal'] * df['log_returns']
  df['cumulative_ret'] = df['strategy_returns'].cumsum().apply(np.exp) 
  </code></pre>
  <p>Now, the math is fairly annoying so I don't want to do it here... but essentially this works fine with longs, but breaks down when trying to calculate the returns for shorts. <strong>The reason is because it thinks that our returns on our shorts is just the negative of our loss on an equivalent long position. <em>Which is wrong</em>.</strong></p>
  <h2>Simple Example</h2>
  <p>Let's say I have a dataframe that represents two trades taken: one that is <code>long</code> from price = <code>$1</code> --&gt; <code>$2</code> and the other is short from <code>$2</code> -&gt; <code>$1</code>. Now, if we were to calculate the returns for this would simply be: 100% on the first trade, and 50% on the second.</p>
  <p>If we held a position of $100 at the start of the session, our first trade (+100%) would turn our $100 to $200, then our second trade (+50%) would turn our $200 to $300. So our final position is $300 (or 300%).</p>
  <p>The problem is that the code given in these tutorials does the long right but then thinks our returns for the short is 100% (because our returns are the negative of a -100% returns on a long).</p>
  <h2>Code for Proof</h2>
  <p>Here is a code snippet that demonstrates this. A minor thing to note, we enter at the price <em>above</em> the first row where <code>position_signal = 1/-1</code>, this basically means our <code>position_signal</code> is shifted by 1 to account for look-ahead bias. It's a semantics issue and not really an issue, you can do it however you'd like as long as returns are the same</p>
  <pre><code>df = pd.DataFrame([[0.0, 1.0],
                     [0.0, 1.2],
                     [0.0, 1.50],
                     [0.0, 1],
                     [1.0, 1.2], # We entered a LONG at the open of this timestep, which is the same as the close of the previous ($1.0)
                     [1.0, 1.3],
                     [1.0, 2.0], # We exit at close of this timestep, so $2
                     [0.0, 1.7],
                     [0.0, 2],
                     [-1.0, 1.798], # We entered a SHORT at the open of this timestep, which is the same as the close of the previous ($2.0) 
                     [-1.0, 0.50],
                     [-1.0, 1.3],
                     [-1.0, 1], # We exit at close of this timestep, so $1.0
                     [0.0, 1.5]],
                     columns=['position_signal','close'])
  
  df['log_returns'] = np.log(df.close) - np.log(df.close.shift(1))
  df['strategy_returns'] = df.log_returns * df.position_signal
  df['cumulative_returns'] = df.strategy_returns.cumsum().apply(np.exp)
  print(df)
  </code></pre>
  <h2>Code Output:</h2>
  <pre><code>    position_signal  close  log_returns  strategy_returns  cumulative_returns
  0               0.0  1.000          NaN               NaN                 NaN
  1               0.0  1.200     0.182322          0.000000            1.000000
  2               0.0  1.500     0.223144          0.000000            1.000000
  3               0.0  1.000    -0.405465         -0.000000            1.000000
  4               1.0  1.200     0.182322          0.182322            1.200000
  5               1.0  1.300     0.080043          0.080043            1.300000
  6               1.0  2.000     0.430783          0.430783            2.000000
  7               0.0  1.700    -0.162519         -0.000000            2.000000
  8               0.0  2.000     0.162519          0.000000            2.000000
  9              -1.0  1.798    -0.106472          0.106472            2.224694
  10             -1.0  0.500    -1.279822          1.279822            8.000000
  11             -1.0  1.300     0.955511         -0.955511            3.076923
  12             -1.0  1.000    -0.262364          0.262364            4.000000
  13              0.0  1.500     0.405465          0.000000            4.000000
  </code></pre>
  <p>As you can see, by the end it assumes our returns are 4.0 (400%), which is not what we want. It should be 3.0 (300%)</p>
  <hr />
  <hr />
  <h3>Other Variations:</h3>
  <p>1 -  There's a way to do this if we ignore the intermediate price ticks. If we only reduce this df to four rows: enter trade 1, exit trade 1, enter trade 2, exit trade 2, then we can just mask on position signal, and calculate returns on each trade as a single row and then run cumprod on that. I don't want that, I need to know my portfolio value in the intermediate. Which is why I added those rows for the example above.</p>
  <p>2 -  There's also a way to do this by dividing the pnl due to shorts and pnl due to longs. But this isn't good because if I'm down 50% due to a bad long, it affects my purchasing power for the next trade (i.e. being up 50% for the next trade doesn't make me break-even)</p>
  <p>3 - There's also the non-vectorized route, but that is too slow for me.</p>