When a benchmark win vanishes in production

A benchmark is a promise about production. When it shows a big win, it is tempting to merge and move on. I have been caught out by this enough times to have a routine for it now, and the clearest example came from a pathfinding change in a game engine I maintain.

The win that looked enormous

Pathfinding searches have a bad case. When the goal cannot be reached, a search has to explore every tile it can reach before it gives up. On a large map that is a lot of wasted work, and it happens every time a character tries to walk somewhere it cannot get to.

The standard trick is a reachability pre-check. Label every connected region of walkable tiles once, then before a search, compare the labels of the start and the goal. Different labels mean no path, so you return straight away without searching.

On a synthetic map built to exercise the bad case, the pre-check made unreachable-goal searches about 3,000 times faster. Reachable searches were unchanged. The change was small and the tests passed.

What happened in the real game

Before merging, I replayed real goals recorded from the actual game through both versions and compared the answers. The pre-check rejected 3 of 346 goals that the old search could reach. Three wrong answers out of 346 does not sound like much, but each one is a character who refuses to walk somewhere it plainly can.

The cause was a boundary condition the synthetic map did not have. In the real game the world extends past the edge of the map data: characters can step off one edge and the search treats the space beyond as walkable. The region labelling only looked at tiles inside the map, so two areas joined only through the outside looked like separate regions. The search knew better; the labels did not.

The fix was to extend the labelling across the edge, treating the outside as one more region connected to every walkable border tile. With that change the pre-check agreed with the full search on 346 of 346 goals, and the speed-up on unreachable goals stayed.

The routine I use now

Three habits would each have caught this before it shipped.

1. Model the production boundary conditions

Synthetic inputs are good for measuring, but they are only as faithful as the assumptions behind them. Before trusting one, I list what the real inputs do at their edges: map borders, empty strings, very long lists, the largest customer, the first and last day of a month. Then I check the synthetic case has each of them. If it does not, the benchmark is measuring a different program.

2. Keep a correctness check beside the timing

An optimisation should return identical results for every case where the old code was right. I build that check into the benchmark itself, so a fast wrong answer fails loudly instead of looking like a win:

for (const goal of recordedGoals) {
  const before = searchWithoutPrecheck(start, goal);
  const after = searchWithPrecheck(start, goal);
  if (Boolean(before) !== Boolean(after)) {
    mismatches.push(goal);
  }
}
console.log(`${recordedGoals.length - mismatches.length} of ${recordedGoals.length} agree`);

The number that matters is the agreement count, and it has to be all of them.

3. Measure on real data before claiming the win

Record a sample of real inputs, from logs, a replay or a copy of production data with personal details removed, and run the benchmark on that as well as on the synthetic case. Report both. If the real-data speed-up is much smaller, that is useful to know before you tell anyone the number.

A second example: the simple version won

The same routine also works the other way, where the clever answer turns out not to be needed at all.

I was speeding up a glossary highlighter that finds known terms in article text and links them. The textbook answer for matching many terms at once is an Aho-Corasick automaton: build it once, then scan each article in a single pass. The simpler approach is a prefilter: a quick check on each word's first characters to rule most of them out, then a direct lookup for the few left.

On realistic glossary sizes the simple version won:

The automaton only pulls ahead once the glossary passes about a thousand terms, and it pays for it in memory on every PHP worker that builds it. Real glossaries I looked at were well under that size, so the prefilter stayed. A synthetic benchmark with ten thousand terms would have pointed the other way.

Before you merge a performance change

• Write down the edges of the real input and check the benchmark covers each one.
• Compare outputs, not just timings, and require full agreement on cases the old code got right.
• Run the benchmark on recorded real data as well as synthetic data, and quote the real figure.
• Note what the win costs: memory, build time, complexity, and how it scales with input size.
• Keep the correctness check in the test suite after merging, so a later change cannot quietly break it.

The headline number from a synthetic benchmark is a hypothesis. It only becomes a result when it survives real data with the answers unchanged.