Backtesting means simulating a strategy on past data to see what it would have produced. An indispensable tool — and a formidably effective machine for deceiving yourself.
Overfitting: enemy number one
With enough parameters and enough attempts, you can ALWAYS find a rule that would have worked beautifully on the past. That proves nothing: you have simply memorised the noise in the historical data rather than captured a real mechanism. An overfitted strategy collapses the moment it meets new data.
The biases that distort everything
- Survivorship bias: testing on companies still listed today mechanically excludes every one that went bankrupt. The result is artificially flattering.
- Look-ahead bias: using, at a given date, information that was not yet public (earnings published later).
- Data snooping: testing 500 variants and keeping only the best — by construction, it is good by chance.
- Forgetting costs: ignoring fees, spreads and taxes turns a losing strategy into a winner on paper.
The discipline that saves you
Split your data: build the strategy on one period, and test it on a period you have NEVER looked at (out-of-sample validation). If performance collapses on that second period, you had overfitted. That is unpleasant, and infinitely cheaper than discovering it with real money.
In Earnnest
The backtest engine lets you test a strategy against history. Earnnest's PRE programme was in fact rebuilt specifically to eliminate survivorship bias — which shows how seriously the subject is taken.
À retenir
- ✓A brilliant backtest proves nothing: it may have memorised nothing but noise.
- ✓Survivorship, look-ahead, data snooping, forgotten costs: the four killers.
- ✓Always validate on an out-of-sample period, never looked at during construction.