Every quantitative researcher has seen a beautiful equity curve. It rises smoothly, survives a drawdown or two, and reports statistics that would justify a sizeable allocation. Sometimes the strategy is real. Often the curve is a description of the data, not of a repeatable opportunity.
After years of constructing and reviewing strategies at Meridian Crest and Atlas Vale, I have come to treat a backtest as the beginning of a conversation rather than the end of an argument. The purpose of validation is not to confirm the result. It is to discover what the result requires in order to be true.
1. What is the economic mechanism?
A strategy should have an explanation that does not depend on its own past performance. Who is on the other side of the trade, and why would they persist in providing that opportunity? Risk transfer, liquidity provision, behavioral bias, structural constraints and information diffusion are plausible mechanisms. A pattern discovered by searching thousands of variants is not.
2. How many decisions were actually made?
A daily strategy over ten years contains roughly 2,500 observations, but the number of independent bets may be far smaller. If positions persist for months, the effective sample is small, and a high t-statistic computed as if every day were independent overstates confidence. Serial correlation and overlapping positions must be accounted for.
3. What did the strategy cost to run?
Transaction costs are not a constant percentage. They vary with liquidity, volatility, order size and urgency. A strategy that trades frequently in less liquid assets can look attractive before costs and ordinary after them. The correct test is not a fixed commission assumption but a cost model that responds to market conditions.
4. Which environment produced the result?
Many strategies are, in effect, long a particular regime: falling rates, stable volatility, cheap liquidity or a persistent trend. Performance should be decomposed by environment, not reported only in aggregate. If all of the excess return comes from one period, the strategy may be a historical episode rather than a method.
5. How sensitive is it to small changes?
Reasonable variations in lookback windows, thresholds, universes and rebalancing rules should not destroy the result. Sensitivity analysis is not a formality; it is a test of whether the signal is structural or an artifact of one specification. A strategy that only works at 20 days and fails at 19 and 21 is telling you something important.
6. What would make us stop trading it?
Before deployment, the team should write down the conditions under which the strategy would be considered broken: a specified drawdown, a collapse in the underlying mechanism, a structural change in liquidity, or a persistent divergence between expected and realized behavior. Without pre-committed criteria, every loss becomes arguable and discipline erodes.
Validation is the practice of trying to falsify your own strategy before the market does it for you.
From validation to monitoring
The work does not end at deployment. A live strategy should be monitored against the assumptions that justified it. Are execution costs in line with the model? Is the signal decaying? Has the composition of market participants changed? Is the strategy’s correlation to other holdings rising? These are not reasons to intervene constantly; they are evidence for periodic review.
At VEA, this discipline is reflected in the ORION framework’s emphasis on model version tracking, assumption verification and outcome review. The aim is to make the distance between expectation and reality visible early, while there is still time to learn from it.
A lower, more durable standard
None of these questions guarantees a good strategy. They do, however, make it harder to deploy a bad one with confidence. The standard I prefer is not “the backtest is impressive” but “we understand why this should work, we know what would make it fail, and we have agreed in advance how we would recognize failure.”
That is a lower standard than certainty. It is also more durable.
This note is educational and does not constitute investment advice.