Rolling-Origin Validation

Every gate returns one of three states. They are not degrees of the same thing, and reading the middle one as failure is wrong about exactly the arms that perform best.

accepted

The gate passed at that horizon. The evidence was sufficient and the model met the threshold.

insufficient_evidence

The panel cannot resolve the gate. This says nothing about the model: a perfectly calibrated generator and a badly miscalibrated one return the same state when there is not enough independent data to tell them apart. It is a statement about the measurement.

rejected

The model failed the gate. The evidence was sufficient and the threshold was not met.

At one day the study has 5,900 independent observations and full power, which is why the h1 verdicts discriminate: the retired default is rejected there on calibration alone. At one year it has 269, and the tail gates need thousands.

This does not improve by waiting. The number of independent observations at a given horizon is the calendar span divided by that horizon, so a one-year gate needs roughly 5,000 years of history to resolve its 1% expected-shortfall floor. Collecting another decade moves 269 to about 279. Any long-horizon number on this platform is therefore reported as conditional and always will be, and no future run will change that by itself.

That is why an arm can carry zero rejecting gates and still not be accepted, and why absence of rejection is never read as acceptance anywhere in this study.

The harness answers one question at four horizons, 1, 21, 63 and 252 trading sessions: standing at a historical date and knowing only what was knowable then, how well does each generator's forecast distribution describe what actually happened next?

• It freezes and digest-verifies an approved research panel of Yahoo adjusted closes, and records complete data provenance so a replay is reproducible rather than merely repeatable.
• It builds causal expanding-window folds, so every fit sees only data from before its origin.
• It routes candidate paths through the production simulation and planning ledgers, not a research reimplementation. A generator is scored as the product would run it.

The developer study contains 628 folds over 161 origins, 3,140 scored model-horizon forecasts with zero failures, and all 3,725 expected planning replay rows with zero failures. The deterministic artifacts, a JSON evidence file and a 104-page PDF, are digest-pinned.

A backtest that lets any future information reach a fit produces optimistic results that nothing downstream can undo. Two mechanisms prevent it, and they address different failures.

Every score is strictly proper: it is minimized in expectation only by the true forecast distribution, so a generator cannot improve its score by hedging or by widening its intervals.

The continuous ranked probability score is the workhorse. For a forecast distribution with CDF and an observed outcome :

Estimated from a finite ensemble it is biased, and the bias shrinks with ensemble size, so a candidate run with fewer paths would score artificially well. The harness therefore uses the fair(unbiased) form throughout:

QLIKE is retained as a reported diagnostic rather than a confirmatory score. Its deviance form , with , diverges as the realized proxy approaches zero. At a one-session horizon the realized proxy is a single squared return, which is a chi-square with one degree of freedom, so the statistic is unbounded below in a way that no amount of resampling fixes.

Rolling origins at a 21-session stride with a 252-session horizon produce heavily overlapping targets. Treating those rows as independent would understate every standard error by a large factor, so the whole inference layer is built around not doing that.

• Origin-row dependence floors. A gate cannot be evaluated below a declared minimum number of effective non-overlapping blocks, whatever the raw row count says.
• Corrected automatic Politis-White block selection for the paired stationary bootstrap, with an audited structural overlap fallback that is published whenever its guard fires rather than silently substituted.
• Intersection-union non-inferiority tests. A candidate must be non-inferior on every component to pass, so the composite claim inherits the level without an extra correction. This is the conservative direction by construction.
• Authoritative Holm adjustment across the family of comparisons, controlling the family-wise error rate.
• Romano-Wolf sensitivity is explicitly pending until its finite-sample size behaviour on this design is validated. It is not quietly used and not quietly dropped.

Scoring well on average is not enough. A generator that wins on average but is never selected by a train-only rule is not usable, because in production you must choose before you see the outcome.

Four completed studies used the same harness to test specific hypotheses. Every arm in them is validation-only and stays outside the production generator enum, so none of them changed what a user can select.