Probability of Backtest Overfitting (PBO)

A researcher runs 200 momentum strategies on a Hyperliquid universe. The champion posts a Sharpe of 2.5 and a 60% in-sample win rate. The runner-up is at 2.3 and the bottom decile is around 0.4. Looks like a clear signal. The researcher reserves the last 20% of history as out-of-sample, applies the champion's parameters, and gets a Sharpe of 1.4 — still positive, still publishable, still about to lose money in production.

The problem is selection. The 2.5 was the maximum of 200 noisy estimates; the OOS 1.4 is the same strategy's regression toward the mean. The 2.5/1.4 degradation looks like the strategy "didn't generalize quite as well as hoped." In fact, the champion may not have been distinguishable from the median strategy in the OOS sample — it just happened to land at rank 1 in-sample by luck. PBO measures exactly this: across many random IS/OOS splits, how often does the IS winner stay above the median in OOS? If the answer is "about half the time," the selection process had no signal — your champion was random.

The engine is Combinatorially-Symmetric Cross-Validation (CSCV). The algorithm:

1. Take your N × T matrix of strategy returns. N strategies (columns), T time periods (rows).
2. Split the T rows into S equal-length chunks. S = 16 is the standard.
3. Enumerate every way to split S chunks into two equal halves. For S = 16, there are C(16, 8) = 12,870 such splits.
4. For each split, designate one half "in-sample" and the other "out-of-sample." Compute each strategy's IS Sharpe and OOS Sharpe.
5. Identify the strategy with the highest IS Sharpe — the IS champion for this split.
6. Find that champion's rank in the OOS distribution. Compute its relative rank: ω = rank_OOS / (N + 1). A relative rank of 1.0 means the champion was best OOS too; 0.5 means it was at the median; near 0 means it was the worst OOS.
7. Transform: logit(ω) = log(ω / (1 − ω)). Negative logit means the OOS rank was below median (overfit).
8. Repeat for all 12,870 splits. PBO is the fraction of splits where logit(ω) < 0 — i.e. where the IS champion ended up below the OOS median.

Two properties make this work for time-series data. Symmetry: every observation appears in IS and OOS equally often across all splits, so the procedure is balanced. Rank-based: the test uses relative rank rather than absolute Sharpe, so it is invariant to changes in volatility regime between IS and OOS halves — a different test would conflate true overfitting with regime shifts.

PBO is a probability between 0 and 1.

• PBO < 0.1 — robust. The IS ranking is highly informative for OOS ranking. Rare; typically requires either a very strong economic prior or a small parameter grid.
• 0.1 – 0.3 — healthy. The normal range for a disciplined candidate set: the IS champion generalizes more often than not.
• 0.3 – 0.5 — concerning. The IS champion fails OOS more than 30% of the time; the ranking is barely informative. Check whether N is too large for the sample or the variants are near-duplicates.
• PBO > 0.5 — overfit. The IS champion underperforms the OOS median more than half the time — picking at random would do better. 0.5 is the coin-flip baseline, not a passing grade. Common after extensive search on a sparse signal. Walk away.

A worked example, on the PBO calculator’s built-in sample so you can reproduce it:

1. Inputs. N = 20 strategies × T = 250 daily periods (19 zero-edge random walks and one with a mild positive drift), S = 16 chunks → C(16, 8) = 12,870 IS/OOS splits.
2. One split. The IS champion ranks 7th of 20 OOS: ω = 7 / (20 + 1) = 0.333, logit(ω) = ln(0.333 / 0.667) = −0.69 — negative, so this split counts as a failure. (Rank 14 would give ω = 0.667, logit +0.69: held up.)
3. All splits. The IS champion landed below the OOS median in 48.9% of the 12,870 splits; the median logit was +0.10.
4. PBO = 0.489 — concerning, a hair under the coin-flip line. One real edge among 19 random walks is barely findable on 250 periods: the selection process is close to noise.

Beyond the headline number, the full logit(ω) distribution is informative. A symmetric distribution centered at zero says the search is noise. A distribution skewed positive says the selection process has signal even where PBO does not cross a clean threshold. The PBO calculator visualizes the distribution alongside the headline probability.

Three related diagnostics, three different jobs:

• Parameter sensitivity — the cheapest. Vary each parameter ±1 step and check that Sharpe degrades smoothly. Catches narrow-spike overfits with a single afternoon of work. Pre-flight for PBO/DSR, not a replacement.
• DSR (Deflated Sharpe Ratio) — best when you have the champion's tearsheet but not the full grid. Adjusts a single Sharpe for trial count and distribution shape. One number in, one probability out.
• PBO — best when you have the full N × T matrix of strategy returns. Uses the full distribution, not just the champion. More information ⇒ tighter inference. The downside is that it requires you to have actually saved all the per-strategy return series.

For serious research, run all three. Parameter sensitivity first (cheap filter), then PBO if you have the data, then DSR on the champion as a final sanity check. They are complements, not substitutes.

Crypto research is unusually vulnerable to backtest overfitting for three reasons. Short histories: Hyperliquid has been live since 2023; many listed perps have only months of data. Small T means high variance in Sharpe estimates and a fatter null distribution under CSCV. Regime shifts: the 2024-2026 sample includes meme-coin euphoria, two major drawdowns, the Trump-era policy whiplash, and structural changes in market microstructure — strategies that worked in one regime routinely fail in the next. Survivorship: a "currently listed perp" universe systematically drops names that collapsed.

PBO partly addresses regime shift through its combinatorial symmetry — every chunk pairs with every other — but it cannot recover lost data or eliminate survivorship bias. The honest setup for crypto PBO is to use a point-in-time universe (the perps that were listed at the start of the IS window) and chunk the data finely enough that each IS/OOS pair covers multiple regimes.

A practical setting for Hyperliquid 15-minute strategies: chunk size of one calendar month, S = 16 chunks (≈ 16 months), N = 50-200 strategies. Below 16 months of history, PBO is brittle and parameter sensitivity is the more reliable diagnostic. Above two years, S = 24 with monthly chunks gives a denser combinatorial estimate.

The PBO calculator accepts a CSV of strategy returns (rows = periods, columns = strategies) and runs the full CSCV procedure in the browser. Output: PBO probability, the full logit(ω) distribution, and the IS/OOS Sharpe scatter for the champion of each split.

Keel does not ship PBO as a built-in backtest diagnostic today. The platform's parameter-sensitivity tooling is the closest available defense; PBO may be added as a native diagnostic in future releases on multi-strategy grid runs; no committed ship date.