Hi Ernie. Nice strategy. Just wanted a clarification. The return from yest low to today open could be flat, negative or positive. So by lowest 100 you are implying we rank the 500 stocks by % chng, remove those whose absolute value is greater than the 1SD value, and then choose the bottom 100. A lot of these will have negative returns frm yest low.. so thats acceptable ?
Also, are the stocks then sold by end of day ?
No, by lowest 100 I mean first apply the criteria that their returns must be < -1*std, and then pick the lowest 100 among the candidates. Of course, there may not be 100 candidates, in which case we will buy fewer stocks.
Yes, the portfolio is liquidated at the close.
So in essence you are picking the weakest stocks.. and expecting mean reversion to happen for the basket as a whole.. or for the basket to outperform the overall index per se..
That's correct. It is a mean-reverting strategy.
To be honest I do not see how one can use such a strategy. In my opinion this is simply going to boil down to luck. I think an improvement on this may to group n stocks in a basket and buy the stock that performed the least in the prior period. I believe ensembling things like correlation with some more advanced machine learning models would be the way to go. Note I have had success with that type of strategy in medium frequency trading (holding time a few minutes).
This strategy exploits a particular inefficiency in the opening auction price of equities. Over a period of about 2 years of live trading, it has a consistently high winning ratio, and numerous traders have traded this strategy. However, this inefficiency has disappeared, or at least greatly diminished, since the financial crisis. In fact, maybe it disappeared because too many traders learn about it.
The strategy you suggested with a few minutes holding period are in a completely different category. There is no reason why we cannot trade both types of strategies successfully.
In contrast to you, I have never had any success with machine learning based trading models.
"Swiftly" is somewhat subjective, depending on whether you think the drawdown is due to a specific event that won't repeat. But if there is no discernible reason for the decline, and the drawdown is longer than the the max drawdown duration in the backtest, then it is time for drastic cuts in leverage.
hi Ernie; what is the rationale for being long only? why not trade it on both sides?
I think it's important to always be creating new strategies, even if your returns are doing well with current ones.
May I suggest a conspiracy theory that since the financial crisis there is greater government involvement/manipulation in the markets. Somone supports yen, someone supports swiss frank, someone supports interest rates. It can be that someone supports stocks as well (or does it indirectly). And while supporting stocks they can really go for an easy target: buying those stocks that on openings actually trade higher, not those that fall deeper than the others. The capital concentrates in selected instruments that are being propelled beyond reasonable (a fruit, anyone?), ignoring broad market.
Hi Ernie -- first off, loved the book, which then clued me into your blog! Also, I greatly enjoy the content in addition to the community comments -- great intelligent and often challenging stuff which makes me research more! The email notice of a new post is always a welcome arrival!
On this topic, irrespective of a particular strategy, I agree many strategies can become unusable once they are "known". There is, though, an interesting judgement between: is my strategy just not working in the short term (temporary drawdown), or is it now overused and unusable. That may be difficult to detect in the medium term, but hopefully backtests illustrating prior drawdown periods may be a metric of comparison. We can't abandon each strategy quickly when we just "think" it's dead.
William -- funny example and spot on, that's the lesson here! And is also why I wouldn't expect to read of great (completed ready-to-trade) strategies in the literature. As long as we are all doing something a bit different, the markets work.
It turns out that the short side of this strategy does not perform very well. This seems to be a general characteristic of equities trading, which is not surprising considering the positive expected return of the market.
Interesting thought. Zerohedge recently mentioned (http://www.zerohedge.com/news/why-market-slowly-dying) that retail trading has decreased ever since 2008. So this inefficiency may have died with retail trading.
Thanks for your kind words on my book. I agree that it is difficult to tell if a strategy is just suffering from short-term drawdown. But I believe it is always better to be safe than sorry once the drawdown duration approaches the max duration in backtest.
This probably means that this strategy was followed by some big funds then which may have abandon it for some reason or actually started to fade it when they realized retail follows it. The strategy is too simplistic too work for a long time. Simple = no edge as anyone can figure it out.
Your explanation for the demise of this strategy may be true, though I personally stick to my explanation above: that there are too few retail traders nowadays who used to drive down the prices of stocks in a panic at the open to allow for this opportunity.
I disagree that it loses profitability because it is too simple. You can make it more complicated with all sorts of hedges, risk management measures, and mathematical analysis. It won't change the basic fact that as long as there are no retail participation, professional traders cannot hope to be profitable, because they will just be competing among themselves.
I understand your argument but this makes no sense to me: why would retail traders push prices down on the open when in a bull market most indicators they use would signal a buy instead of a sell?
IMO this had to do with the proliferation of pairs trading funds during that period. Most of those funds went bust during the bear market although they continued the strategy for as long as they could and as a result the effect you saw of it being profitable during the market drop. When those funds ran out of money the strategy stopped working.
Wondering what your and everyone else's thoughts are on whether such a strategy is feasible for retail traders (100K). I mean is it really cost effective to trade 100 names intraday?
Your comment is very thought-provoking, even though I disagree with all your points!
First, by retail traders, I am thinking of those who do not trade using technical indicators or otherwise algorithmically. My strategy bought stocks at the open when the market is down, sometimes very down. So obviously most traders are selling, and we can hypothesize that a sizeable portion of these sellers were retail traders. If, as you said, in a bull market every body would be buying, then why would the market be down at all at the open? I believe these retail selliing exaggerated the down move at the open. But nowadays, they no longer participate in the market, so we no longer have exaggerated down moves.
Second, far from the "bankruptcy" of pair traders that you suggested, mean-reverting stat arb strategies actually worked very well in 2008, as you can find out by backtesting the Andrew Lo toy strategy referenced in my article http://epchan.blogspot.ca/2012/01/what-worked-in-2011.html. I believe it is because of this great success that many more institutional traders have adopted this strategy, so they all buy at the open when the market is indicated down, again alleviating the exaggerated down move and wiping out everybody's profits.
As I suggested in my post, this is just the simplest version of the strategy. You can modify it so that it is suitable for a $100K account. In fact, I personally traded a version of it in an $100K account (with 4x leverage) initially, and with good success.
"My strategy bought stocks at the open when the market is down, sometimes very down. So obviously most traders are selling, and we can hypothesize that a sizeable portion of these sellers were retail traders. If, as you said, in a bull market every body would be buying, then why would the market be down at all at the open? "
I think the other anon tried to argue that the market is lower at the open because of arbitrage by funds where the sell outperforming stocks and buy underperforming ones. I think this is a better explanation than yours.
Actually, I think you are right that this gapping down may have to do with stat arb funds selling stocks that outperform in recent days, because if I impose that condition, the Sharpe ratios improve across the board, even though it is still weaker in the second part of the data. So I think my point about increasing participation of stat arb funds due to their superior performance during the financial crisis is also right.
Thanks for the debate -- this is a great example of how thinking about the root cause of a strategy's decline can inspire ways to rejuvenate it!
Generally-speaking about life & death of strategies, why not to apply some form of long term momentum to theorical equities-line of your strategies, choosing the best or simply turnig off when performances are negative?
Thanks Lorenzo for this good article !
Do you have any others articles on data snooping/anomalies/model selection ?
Good insight here regarding regression & trading. I've been thinking about logit/probit type regressions in trading but don't have experience with these types of models. I have googled around but there doesn't seem to be a lot of research in this area.
If anyone has any papers or comments on the application of these types of regressions in trading I would be interested. Please feel free to share here or you can mail me at algotr8der@gmail.com. Thanks.
It's a great article, thanks. I have one question: in the strategy above, u used previous low to open gap to decide what stocks to buy at open. But how do you buy at open after you already know the open price?
As with any strategy, there will be slippage costs involved. In this case, you can use pre-market prices to estimate the open, or you can trade just after the open.
Great article, thanks for all the hard work. I'm curious, if a strategy's performance begins to languish and you take stop trading it, do you continue to monitor its performance through simulation and consider trading it again if conditions become favorable or do you completely do away with it in favor of developing a newer strategy? I guess the middle ground here to developing an entirely new strategy would be to adjust parameters of the older strategy instead.
In my experience, adjusting the parameters of an old strategy that stopped working won't make it work again in the short term. I would backtest the strategy in 6-12 months and see if performance has returned, and if not, tweak the trading rules.
I don't think 2009 is the only regime change we have seen. I believe the Flash Crash in 2010 is another.
I developed a strategy in college as part of my economics thesis, which used an event study to identify a set of recommendations (upgrades, downgrades) that had statistical and economic significance on the underlying stock in an intraday basis. This led to a strategy of creating a portfolio which buys/sells stocks which received new analyst recommendations at the open, and closing all positions at the close. The backtest from 2001 to 2009 worked very well. Live trading started in 2009, results matched backtested results. Results began to weaken in late 2009, and all of a sudden, the strategy stopped working after the flash crash. I believe the market structure has been altered since May 2010 and the flash crash, although I do not know why.
We traded a similar strategy based on companies' guidance. The results also weakened in 2009, and we kill that strategy long before flash crash.
I was testing similar strategies and I found without stop loss they cannot be profitable. Did you apply any stop or simply closed positions at close? Did you find any correlation with the market capitalization of the stocks?
If you are referring to the earnings recommendations or guidance strategies, then yes, we have tested stop losses. But stop loss invited "stop-hunting" strategies, and it was not successful.
There was some difference in performance between large vs small cap stocks, though I don't remember which one is better, and I don't know if this difference persists.
I am referring to stops on the long positions opened at open according to this gap strategy.
Without stops it is not profitable, unless by trading 100 stocks it is ( i tested with less).
How can they hunt your stops?
Are your stops visible in the book? They should not be?
If you are referring to the gap strategy, then we have never found stops to be useful. It was profitable when we traded it without stops nor profit caps.
Stop-hunting can hurt you even without others knowing exactly where your stops are. For details, you can read the Trading & Exchanges book on my Recommended Book list on the right sidebar, section 11.4.
Have you tried to trade the same strategy but using the price in the middle of the day as the open?
This would basically be an intraday reversal. I suppose it would not work as well because traders had more time to react to such a price configuration, since it developed during the past half of the day, while at open there is less time for reaction, so fast (automatic) traders can have an edge.
To your question: yes, I have. It is equally profitable.
Thanks for one of the best blogs about quantitative trading - thought provoking articles and the best comments in the blog sphere around!
I agree too many traders knowing a strategy/edge will kill it naturally because you can only trade a limited volume on it.
But why 'it seldom comes back to health'. If the edge isn't working anymore people should stop trading it.
Maybe the mob (90% of traders loose) is still keeping the worn out strategy?
For me it is one of the biggest task to decide when to leave a trading system. I use a combination of DD-Duration and DD-depth with a grain of guts feeling.
Sometimes strategies die because of competition. Other times they die because of structural change in the market (e.g. decimalization, change in short-sale rules, rise of dark pools, etc.) Structural changes may do irreversible damage to strategies.
I also rely on drawdowns and gut-feel (some sense of the fundamental reasons) to decide when to drop a strategy.
I love your blog, especially the community that you've built up as reflected by the many thought provoking comments.
I have been modeling the variance of asset returns in FTSE 100 sectors trying to separate it into two components, an environmental component(ie macroeconomic news) and a truly random random component.
It is interesting that in the period 2000 to June 2008 the model says that the noise is virtually always 100% environmental, however come July only 65% is of the noise environmental. There is a definite transition in market microstructure from June 2008 on wards.
Btw October 2011 only 30% of the noise is environmental, the other 70% is random, possibly caused by HFT trading price action rather than fundamentals.
Once i've written a paper / blog post on it in more detail I'll post it in the comments.
Those are very interesting results -- thanks for sharing!
It turns out that after every stock market crash, the market switched from momentum to severe mean regression for an extended period of time. Increased "random noise" like you suggested would lead to increased mean reversion, since "environmental noise" do not reverse easily.
Perhaps the changes we observed is a consequence of this fundamental change, or maybe there are other factors such as heightened risk aversion, decreased volume/liquidity, and higher frequency trading.
- Calculate pessimistic stats; such as drawdown, annual return, sharpe; using VaR monte carlo simulation.
- Stop trading a strategy as soon as the pessimistics levels are reached.
Your suggestion seems sensible. I suppose Monte Carlo simulations can tell you when the distribution of prices or returns has really changed if the actual returns ended up on the tail end of the distributions, indicating a regime shift.
i know a lot of strategies which you may be interested. just let me know.
I hope you don't mind this comment.
From what i've implemented it looks like this strategy still works well. One slight difference that I think might exist is the standard deviation test, I check that the stock is less than or equal to the mean return + 1 standard deviation.
Another slight difference is that i'm using the 2011 S&P constituents (so have survival bias).
What sort of values in the sharpe ratio/win ratio/max drawdown performance metrics do you look for before taking a strategy live?
Yes, the strategy apparently worked well in 2012, but I don't know about 2009-2010.
I look for a Sharpe ratio > 1 before trading live. Max drawdown is less important since it depends on leverage. If we have only 1 strategy, then Sharpe ratio needs to be even higher, at least 1.5 in backtest.
In one of your comment, you said "It is equally profitable" for buying in the mid of the day. Were you using low from previous day? And ignoring the price data in the morning session?
Thanks. I should have rephrased my question to "are you using the previous day low and open of the noon bar?" I don't have access to intraday data and can't really run the test with any other time except daily open.
Also, what value did you used for slippage in your backtest (daily open entry)? When I looked at the 5min charts, the open is so chaotic. In your opinion, what level of slippage should I expected?
I also don't use historical intraday data except high and low. However, that doesn't mean you cannot enter intraday.
Due to the uncertainty of the data at open, 5bps for large cap, 10 bps for small cap tcost will be prudent.
Thanks for all the replies. But I think I don't quite understand what you mean. Because if you are using just daily open-hi-low-close, how can you back test using noon open price?
Does that mean when you trade intraday, you rank your stock using daily open and daily low(-1)?. But trade at noon on a different price, even though the ranking might be different at noon?
Or do you redo the ranking at noon, using the noon "open" price with previous day low? Then trade at noon. I do feel more comfortable with that because the instance I observed, its quiet after 11:00AM.
Sorry, I am a total newbie at this kind of topic (therefore such explicit questions).
What is tcost mean in your post? Commission only, commission & slippage? Is the bps for 1 way or round trip?
If your strategy says "Buy if the price is lower than the previous day's low", and your backtest shows that today's low is indeed lower than previous day's low, then you know the execution price is the previous day's low. You don't need intraday prices to do this kind of backtesting.
However, you won't be able to test a strategy that can only execute at 1pm, of course, without intraday historical data.
My tcost is one-way, include slippage and commission.
"Buy if the price is lower than the previous day's low, and your backtest shows that today's low is indeed lower than previous day's low, then you know the execution price is the previous day's low. You don't need intraday prices to do this kind of backtesting.".
But Andrew Lo's strategy is "buy the open"?
I believe you have lost track of the thread of conversation.
The thread started with the question "Have you tried to trade the same strategy but using the price in the middle of the day as the open?
This would basically be an intraday reversal. "
implying this is an intraday variation of Lo's strategy.
3) rank using before_noon_low(0) and noon_open(0), but noon_open(0) >= before_noon_low(0), since there is no reason to believe the market will gap at noon. However, we only care about its relative rank?
But after reading your replies, I don't think I understand how this variation work. Can you elaborate it a little bit?
Actually, intraday variations may not involve ranking. But I will leave it to your imagination to find out the exact strategy that is possible and profitable, lest I trample on someone else' intellectual property!
You can use LOO orders.
There is inevitably some slippage, since you have to use the 9:28 prices to generate these orders, and they will be filled at the auction price at the open.
If you want to backtest this accurately, you really have to use bid-ask quotes at 1 minute intervals.
Thanks Ernie for your quick reply. Since LOO orders were used for entry, there might be instances on many days when for example only 40 or 70 or 60 stocks out of 100 stocks got filled due to the usage of LOO orders. How did you model your strategy for this behavior? (Not being able to trade all 100 buy signals). And did you execute this strategy in F&O or in the Cash Market?
There are many versions of the buy on gap strategy. But as long as one uses limit orders, no versions would guarantee that you all your orders are filled. Why would that be a problem? You can backtest this quite accurately: just see if your limit price is higher or equal to the open auction price.
We can only trade in the cash market for stock strategies.
Thanks Ernie for your valuable comments. I will further backtest this strategy and try it live.
I think you misunderstood how one backtests strategies that use LOO.
Now I get it. But I trade on NSE India. NSE does not have LOO's. I use daily stock price data from google finance to backtest, and the price at 9:08 am during the NSE pre-market session is reflected as the open price for the day (when the market starts at 9:15 am) in google data. Hence it not possible for me to figure out what was the actual trade price of the first trade when the market opened at 9:15 am.
When I implemented it Live, with limit orders I am not able to executes all buy signals, if I opt for markets orders, the slippage is unpredictable and makes the strategy unprofitable.
If Goolge Finance doesn't give you the actual open price of the market, then you have to find another source data. You may even need to find intraday data. If you use IB, you can download free intraday data through their API, as long as you are subscribed to their live data for the Indian market.