Out-of-sample testing for crypto backtests

Out-of-sample testing checks a strategy on data it was not built on. You split the history into an in-sample part, where you design and tune the rule, and an out-of-sample part you leave untouched until the end. The out-of-sample result is the one to trust. On crypto the history is short and regimes change fast, so the split matters more and costs more data.

By Keel Research Team · Updated November 11, 2026

What out-of-sample testing is

A backtest tells you how a rule would have done on the history you ran it on. If you chose the rule and its settings by looking at that same history, part of the result is the fit, not the rule. Out-of-sample testing keeps a piece of the history away from every choice you make, so the result on that piece cannot have been fitted.

Take the MA Crossover (Golden Cross) Strategy from Keel’s strategy library, whose backtest runs from Aug 2024 to Sep 2026. To test a change to it out of sample, you would pick a cut date inside that window, try your change on the months before it, and run the one version you settle on over the months after it, once. Only that last run says whether the change helps.

Out-of-sample data has to stay out of every step:

  • choosing settings, by hand or by a search;
  • choosing which signals or filters to include;
  • choosing between several candidate strategies;
  • choosing how to set up the test itself, such as window lengths.

Any contact leaks the out-of-sample data into the design. The usual leak is the honest one: a disappointing result, a small fix, and a re-run on the same data.

In-sample vs out-of-sample

PartWhat it is forWhat you may changeWhat it tells you
In-sampleDesigning the rule and choosing its settingsAnything: rules, settings, which idea to keepHow well the rule fits the history it was built on
Out-of-sampleOne check after every choice is madeNothing. Look once, then stopHow the rule does on data it has never seen

Expect the out-of-sample result to be weaker. The in-sample number carries the tuning; the out-of-sample number does not. A small drop is normal. A large drop, or a sign flip, means most of the in-sample result was fitting. A result that barely drops after heavy tuning is a reason to check for a leak.

How to split the data: holdout, walk-forward and time-aware k-fold

Holdout. Split the history into two pieces in date order: the earlier piece for building, the later piece for one final test. It is the simplest split and the cheapest to run. Its weakness is that it is one draw: if the holdout happens to be a choppy stretch for a trend strategy, a good rule looks bad, and the reverse.

Walk-forward. Move an in-sample window and the out-of-sample window after it through the history, choosing on one and testing on the other each time. You get many out-of-sample results instead of one, so you see their spread. See walk-forward optimization for the full loop.

Time-aware k-fold. Standard k-fold cross-validation shuffles rows into folds, which lets a model learn from bars after the ones it is tested on. Time series need variants that keep the order:

  • Blocked k-fold: each fold is a contiguous stretch of time. Test on one fold and build on the others, then rotate.
  • Purged-and-embargoed k-fold (López de Prado): drop the training samples whose lookback overlaps the test fold (the purge), and skip a buffer of samples right after each test fold (the embargo), so serial correlation cannot carry information across the boundary.

Whichever split you use, keep one final holdout that none of them touch.

Out-of-sample testing on crypto

  • Newer perps have short histories. A market listed months ago leaves little to hold back. The out-of-sample piece may be small and may cover a single regime. Say so when you read the result, or test on the older, deeper markets first.
  • Regimes shift fast. Crypto moves between trend, chop and crash within weeks. One contiguous holdout can sit entirely inside one of them and tell you nothing about the others; walk-forward’s many windows help here.
  • The list of markets changes. Early in the history fewer markets were listed. A strategy that picks from the markets listed today is choosing with hindsight. Restrict the universe to markets that existed through the whole window, or pick it as of each date.
  • Funding is part of the result. Hyperliquid settles funding every hour. A very short out-of-sample window can be dominated by one funding spike, so give it enough time for funding to average out, and make sure the backtest charges it.

How to run an out-of-sample test on Keel today

Keel runs a backtest over any date window you choose and keeps every run. It does not split the data for you or stop you from re-tuning on the holdout; that discipline is yours.

  1. Pick a cut date. Everything after it is the holdout. Do not run anything on it yet.
  2. Build and tune before the cut. Set the backtest to end at the cut date, in the app or with keel backtest run <strategy> --start-date … --end-date …, change settings and re-run as often as you like. For a stronger check, walk forward inside this part and read the windows in the walk-forward analysis tool.
  3. Freeze the strategy. Write down the settings and how many versions you tried.
  4. Run the holdout once. Start the backtest at the cut date. If it fails, the change failed; adjusting and re-running on the same holdout uses it up.
  5. Paper trade it as the live out-of-sample test. Every day after you freeze the strategy is new data. Keel can paper trade the same strategy on live Hyperliquid prices with no wallet, which is out-of-sample by construction.

Further reading: Pardo, The Evaluation and Optimization of Trading Strategies, on walk-forward method; Bailey, Borwein, López de Prado and Zhu, Pseudo-Mathematics and Financial Charlatanism, on the selection bias out-of-sample testing exists to catch.

This article is educational. Passing an out-of-sample test does not promise live profit; it rules out the case where the in-sample number was a fitting artifact. Trading small and watching live results remains the final check.
Automate it

Trade systematically on Keel

Keel is a strategy builder for Hyperliquid. Describe an idea, backtest it with fees, slippage and funding, and run it live on Hyperliquid’s crypto perps. Stock and index perps are coming soon.

Start free. Connect a Hyperliquid wallet when you’re ready to trade.

What you can do
  • Backtest any strategy with realistic fees, slippage, and funding.
  • Change a parameter and run it again. Every backtest is kept.
  • Run it live on Hyperliquid with stop-losses and leverage caps.
  • Start from a sentence. Describe your idea and Keel builds a strategy you can test.
FAQ

Out-of-sample testing questions

What is out-of-sample testing?

Testing a trading strategy on data that played no part in building it. You design and tune the rule on one part of the history (in-sample) and leave another part untouched until the rule is fixed (out-of-sample). The result on the untouched part is the one to trust, because the rule could not have been fitted to it.

What is the difference between in-sample and out-of-sample?

In-sample data is where you are allowed to look, change settings and compare ideas. Out-of-sample data is held back and used once, after every choice is made. A strategy almost always looks better in-sample, because that is where it was fitted; the gap between the two results tells you how much of the in-sample number was fitting.

How much data should be out of sample?

Enough to hold a fair number of trades and more than one kind of market, while leaving enough in-sample history to choose settings. A common starting point holds back the most recent fifth to a third of the history. On a newer crypto perp with a short history you may have to hold back less, and you should say so when you report the result.

Is walk-forward the same as out-of-sample testing?

Walk-forward is a way of doing out-of-sample testing many times. It moves an in-sample window and the out-of-sample window after it through the history, choosing settings on one and testing on the other each time. A single holdout tests once. Many traders use both: walk-forward on most of the history, and one untouched holdout at the end.

Can an out-of-sample test be overfit?

Yes. Every time you look at the out-of-sample result and go back to change the rule, some of that data leaks into the design. Run enough versions against the same holdout and one will pass by luck. Use the holdout once, after the rule is fixed, and count how many versions you tried before it.