How we test, before we tell you
whether it worked
Every result on this site follows the same three rules. Not because they guarantee a strategy works — most of what we test does not — but because they are what stop us from quietly reframing a loss as a win after the fact.
Rule 1: freeze the rule before you see the result
Before any holdout test runs, the exact rule being tested — entry gate, stop sizing, exit logic, every constant — is committed to a version-control history with a timestamp. That commit is the pre-registration. It exists whether the result that follows is good, bad, or a dead flat null, because it is written before the result exists.
This is not a promise, it's a git hash. Anyone can check that a given commit predates the holdout run: the file with the frozen rule has no result attached at that commit, and a later commit adds the result without touching the rule. Three examples, all inherited from the same research lineage this site publishes from:
| Generation | What was frozen | Commit |
|---|---|---|
| Gen-1 | Fixed-point entry thresholds | d3990f6 |
| Gen-2 | ATR-relative entry thresholds | 6b41c436 |
| Gen-3 | Exit law (stop-only + flatten) | 481eced0 |
Rule 2: the holdout is spent once
Each generation declares a success bar in the same commit that freezes the rule — typically a paired t-statistic threshold and a minimum expectancy in points per trade — and a holdout window it has not yet been run against. The holdout script itself refuses a second run once the result is recorded:
This closes off the second-most common way a backtest quietly cheats: running the holdout, not liking the number, adjusting something small, and running it again. Each of the three generations on this site got exactly one holdout run. Two failed to clear their own pre-declared bar. That is published as a fail, not re-run until it isn't.
Rule 3: benchmark against a coin flip, not a naive baseline
A profitable holdout result is not, by itself, evidence that a signal has predictive value. A profitable exit rule — cut losses early, let winners run — can be profitable on its own, independent of whether the entry direction is informed by anything at all. So every real trade is paired against a "coin-flip twin": a trade on the same day, at the same entry time, run through the identical stop and flatten logic, whose only difference is that its direction is fixed in advance by a hash instead of the strategy's signal.
Fixed in advance, not re-rolled per attempt, and never seen by whatever logic generates the real signal. Both legs run through the same code, the same day, the same exit. The only thing being tested is whether the entry direction carries information the coin flip doesn't have.
What we don't do
- We don't run a holdout twice and report the better number.
- We don't move the success bar after seeing the result.
- We don't compare a strategy's return to zero, or to "buy and hold," when the honest comparison is a coin flip running the same exit — a weaker baseline makes a null look like a win.
- We don't currently apply a formal multiple-comparisons correction (e.g. a Deflated Sharpe Ratio) across the three generations tested so far. Each generation's bar is pre-declared and single-shot on its own holdout, which limits but does not eliminate the risk of a false positive surviving three independent attempts by chance. We're naming that limitation here rather than after someone else finds it.