Out-of-sample testing checks a strategy on data it was not built on. You split the history into an in-sample part, where you design and tune the rule, and an out-of-sample part you leave untouched until the end. The out-of-sample result is the one to trust. On crypto the history is short and regimes change fast, so the split matters more and costs more data.
A backtest tells you how a rule would have done on the history you ran it on. If you chose the rule and its settings by looking at that same history, part of the result is the fit, not the rule. Out-of-sample testing keeps a piece of the history away from every choice you make, so the result on that piece cannot have been fitted.
Take the MA Crossover (Golden Cross) Strategy from Keel’s strategy library, whose backtest runs from Aug 2024 to Sep 2026. To test a change to it out of sample, you would pick a cut date inside that window, try your change on the months before it, and run the one version you settle on over the months after it, once. Only that last run says whether the change helps.
Out-of-sample data has to stay out of every step:
Any contact leaks the out-of-sample data into the design. The usual leak is the honest one: a disappointing result, a small fix, and a re-run on the same data.
| Part | What it is for | What you may change | What it tells you |
|---|---|---|---|
| In-sample | Designing the rule and choosing its settings | Anything: rules, settings, which idea to keep | How well the rule fits the history it was built on |
| Out-of-sample | One check after every choice is made | Nothing. Look once, then stop | How the rule does on data it has never seen |
Expect the out-of-sample result to be weaker. The in-sample number carries the tuning; the out-of-sample number does not. A small drop is normal. A large drop, or a sign flip, means most of the in-sample result was fitting. A result that barely drops after heavy tuning is a reason to check for a leak.
Holdout. Split the history into two pieces in date order: the earlier piece for building, the later piece for one final test. It is the simplest split and the cheapest to run. Its weakness is that it is one draw: if the holdout happens to be a choppy stretch for a trend strategy, a good rule looks bad, and the reverse.
Walk-forward. Move an in-sample window and the out-of-sample window after it through the history, choosing on one and testing on the other each time. You get many out-of-sample results instead of one, so you see their spread. See walk-forward optimization for the full loop.
Time-aware k-fold. Standard k-fold cross-validation shuffles rows into folds, which lets a model learn from bars after the ones it is tested on. Time series need variants that keep the order:
Whichever split you use, keep one final holdout that none of them touch.
Keel runs a backtest over any date window you choose and keeps every run. It does not split the data for you or stop you from re-tuning on the holdout; that discipline is yours.
keel backtest run <strategy> --start-date … --end-date …, change settings and re-run as often as you like. For a stronger check, walk forward inside this part and read the windows in the walk-forward analysis tool.Further reading: Pardo, The Evaluation and Optimization of Trading Strategies, on walk-forward method; Bailey, Borwein, López de Prado and Zhu, Pseudo-Mathematics and Financial Charlatanism, on the selection bias out-of-sample testing exists to catch.
Keel is a strategy builder for Hyperliquid. Describe an idea, backtest it with fees, slippage and funding, and run it live on Hyperliquid’s crypto perps. Stock and index perps are coming soon.
Start free. Connect a Hyperliquid wallet when you’re ready to trade.
Testing a trading strategy on data that played no part in building it. You design and tune the rule on one part of the history (in-sample) and leave another part untouched until the rule is fixed (out-of-sample). The result on the untouched part is the one to trust, because the rule could not have been fitted to it.
In-sample data is where you are allowed to look, change settings and compare ideas. Out-of-sample data is held back and used once, after every choice is made. A strategy almost always looks better in-sample, because that is where it was fitted; the gap between the two results tells you how much of the in-sample number was fitting.
Enough to hold a fair number of trades and more than one kind of market, while leaving enough in-sample history to choose settings. A common starting point holds back the most recent fifth to a third of the history. On a newer crypto perp with a short history you may have to hold back less, and you should say so when you report the result.
Walk-forward is a way of doing out-of-sample testing many times. It moves an in-sample window and the out-of-sample window after it through the history, choosing settings on one and testing on the other each time. A single holdout tests once. Many traders use both: walk-forward on most of the history, and one untouched holdout at the end.
Yes. Every time you look at the out-of-sample result and go back to change the rule, some of that data leaks into the design. Run enough versions against the same holdout and one will pass by luck. Use the holdout once, after the rule is fixed, and count how many versions you tried before it.
Tune on one window, test on the next, repeat: how walk-forward works, with a worked example.
Paste a returns series and see in-sample vs out-of-sample Sharpe per window.
Run a strategy over the window you choose, with fees and funding charged, then hold back your own test window.
How to carve build, test and holdout windows on Hyperliquid perps.