Every backtest is an argument that the past, properly compressed, predicts the future. Artificial intelligence has made these arguments faster, smoother, and far more numerous — a model can now be walked through history while the coffee is still hot. Speed is precisely the danger. Ten thousand simulated pasts produce one magnificent portfolio history at the end; no simulator warns you that it was chosen by the survivor’s own luck.

Whatever the engine, the seams where backtests go wrong are old, few, and worth memorizing. What follows is the desk’s seam-inspection list, reduced to four.

Point-in-time: respecting the fog of then

The discipline is called point-in-time: every value used on a date must be only what was knowable on that date. It sounds obvious; it is bookkeeping hell. Economic series get revised for months after release. A company is renamed and the old news is re-linked. A data vendor improves its cleaning process and generously rewrites history to match. Each courtesy turns the historical record into a tidy,hindsight-flavored version of the mess that live trading actually faced — and a model trained or tested on the tidy version is being graded on answers it could not have known.

Look-ahead: knowing the future by accident

The classic leak is a feature that includes the answer: today’s stock return hiding inside today’s “sentiment,” a market cap computed at period-end and used at period-start. The leaks became subtler with AI. A text model trained on the whole archive has, in some sense, already read the earnings reports that follow every headline in your test set. A universe built with tomorrow’s liquidity screens honors securities only after they succeed. The question to ask of any impressive result is not is this too good but what, exactly, prevented the future from leaking in — and if the answer is a shrug, treat the number as fiction with footnotes.

Survivorship: the graveyard that skipped the photo

Markets edit their rosters without announcements: delistings, take-privates, reorganizations. A backtest universe assembled from today’s membership is a reunion photo of the survivors only, and it flatters every strategy — many a “consistent” result is simply the absence of the departed. Ask for the graveyard: how the universe has handled securities that disappeared, and whether the test ever held names that later failed. A vendor with no answer has described a world in which nobody ever dies, which is a different and much kinder market than the one that pays returns.

Revisions: the past, edited

Some histories are corrected — that is a virtue of statistics, not a sin. The sin is using the corrected past as though it had been the announced one. A consumption signal tested on revised numbers claims credit for prescience about data that, in real time, arrived blurry. The honest backtest separates as-first-announced from as-eventually-corrected and shows the signal against the blurred original, because the blur is what a live model would have had to survive.

The inspection list, pinned to the wall

Is every input restatable as “the value available on the decision date”?
Are training and testing materials strictly separated, including text?
Does the universe contain the names that later died, vanished or merged away?
Are results shown against first-release values wherever revisions exist?
If the answer to any of the above is unknown, is that unknown printed on the page?

That last question is what separates research from marketing. A result whose uncertainties are published can survive contact; one whose uncertainties are cosmetic cannot.

An endnote, in plain terms

This publication writes about method because method is what outlives any particular dataset: while this desk examines how evidence is built, it is not in the business of telling anyone what to own, and none of its pages should be read as a recommendation to act anywhere. If you are wrestling with a backtest whose seams you cannot inspect, describe the situation to the desk by form — methodological disagreements are welcome reading material.

The questions that keep backtests honest