A magnifying glass hovering over a dense block of glowing green terminal text on a dark screen, with red warning marks highlighting a few specific lines of code, and blurred candlestick-chart fragments in the background
Original illustration generated with AI (Adobe Firefly) — not a representation of real data.

Why Backtests Lie: 5 Real Bugs an External Audit Found Before We Touched the Test Set

NodeWitness Team 6 min read

Before the 2,255-strategy tournament touched a single row of the validation partition, an external auditor read the entire simulation engine line by line. It found 5 real bugs. Two of them would have quietly invalidated the whole experiment without raising a single visible flag.

No backtest result is worth more than the audit behind it, and the overwhelming majority of backtests circulating online never get one. The trading strategy tournament covered in the previous piece of this series did, at the worst possible moment to find an error: right before spending the single test shot the protocol allowed. This article is the full catalog of what that audit found, and why two of the five findings are exactly the kind of silent failure that lets a backtest "work" without anyone noticing it's lying.

Why you audit before looking at results, not after

Fixing success criteria in writing before computing anything is only half the discipline of a pre-registered experiment. The other half, less often talked about but just as important, is auditing the simulation engine itself before spending the test partition. A test set can only be touched once, and a bug discovered after touching it can't be fixed without burning the only clean evaluation that was left. This experiment's audit followed exactly that order: engine built and tested on a training partition, external line-by-line review of the whole engine, and only then, with every finding resolved, a real run on validation and test.

Timeline with four stages: engine built and tested on train, external line-by-line audit with 2 high-severity findings and 3 medium or low ones, selection on validation with the corrected engine, and test as a single shot at the end
Original diagram built from the experiment's own verified changelog: the order matters as much as the findings themselves.

The two findings that would have invalidated the whole experiment

The first is the most dangerous of the five because it wouldn't have produced any visible error. The function that loads the candidate pool for picking winners had a default value that, with a one-line oversight (forgetting to pass which partition to use), could silently load training-partition data instead of validation data. Winner selection would have run exactly the same, producing equally plausible numbers, on data that had already been used to calibrate the engine itself. The fix removed that default (the argument is now mandatory) and added two independent checks confirming the loaded data really belongs to the requested partition: one in the loading function, one in the selection function, precisely because the latter can't assume the pool it receives already arrived clean.

The second comes down to how Python compares undefined values. When a strategy barely trades in some asset, its Sharpe ratio can come out undefined, and this wasn't a rare edge case: 1,600 of 2,400 rows for one of the five strategy families — two out of every three — in the real training partition had this problem. Without an explicit rule, an undefined value in a candidate comparison silently broke the ranking's ordering, and a single neighboring candidate with the problem could disqualify a legitimate winner as if it were a catastrophic result. Two distinct rules were ratified on purpose: a candidate with the problem in any asset is simply ineligible (it never competes on partial data), but a neighbor with the same problem is excluded from the calculation without vetoing a candidate that does have complete data. Two different situations that, before the fix, were treated as identical.

Smaller bugs, same principles

Not every finding carried the same weight, and not every one was fixed the same way. One of the five strategy families resizes its position almost daily without flipping sign, and the engine, before the fix, treated any size change as closing the trade and opening a new one, resetting the stop-loss's reference price every time. The practical result: the stop was never more than a day away from yesterday's price, nowhere near being anchored to the real entry price as the design required. Fixed, the one family affected by this went from a median of 318 recorded trades to 58. A signal that the stop had been triggering on its own resize, not on real price moves.

The other two findings didn't touch a single line of code. One was a comment in the engine promising that a take-profit order "executes at its level or better" (like a real limit order), when the code always executed exactly at the level, even when price gapped open more favorably. More conservative than what the text described, never the other way around. The comment got corrected, not the behavior: it's the right conservative bias, and changing the code to capture that favorable gap would have introduced an edge the rest of the design deliberately avoids everywhere else. The last finding documented two edge cases in how a position reversal is processed on the same day a stop triggers: real ambiguities, with no effect on the final result, written down instead of pretending they'd never existed.

Verification didn't stop at one report

The auditor went back over the already-fixed code to confirm the fixes actually worked, not just that they sounded reasonable on paper. It reproduced the most serious finding's original attack (deliberately building a dataset that mixed the two partitions) against the corrected version, and confirmed the process now stops with an explicit error before selecting anything, instead of silently continuing. It also caught, in passing, a minor gap it hadn't escalated to a finding in the first pass, and confirmed that one was resolved too. Two notes were logged without blocking anything: one about versioning validation results so future regenerations can be diffed exactly, and another acknowledging a real limit on what could be verified directly versus what was covered only by automated tests.

What these five bugs have in common

None of the five was the kind of error that makes a backtest crash with an obvious error message. All five are silent failures: they produce a number, that number looks reasonable, and only a line-by-line read of the code, not a glance at the results, brings them into the light. It's exactly the class of error a backtest published without an audit never catches, because the final result doesn't announce that something was miscalculated. Most trading strategies circulating with an "impressive" backtest behind them never go through this kind of adversarial review. Not because nobody wants to, but because finding these bugs is expensive, tedious work that doesn't change the headline if the result already "looks good."

At NodeWitness, we don't audit our own Score's backtest with this same line-by-line adversarial process. It would be dishonest to imply otherwise. What it does have is a related discipline: the data feeding the backtest gets frozen the first time it's computed, so an external source can't silently rewrite the past, and every internal attempt at improving the formula gets checked against known real cases before adoption. None has passed that bar yet. Different safeguards against the same underlying problem: a number that "looks good" isn't the same as a number that's correct.

The next piece in this series covers one of the findings with the most weight on the final verdict: why the short side loses even while collecting funding. You can read the tournament's full verdict first, or check NodeWitness's live Cycle Score, same discipline of publishing what doesn't work too.

Last updated: August 9, 2026