The Hidden Cost of Flaky Tests

Flaky tests are tests that fail at random, without any change to the code. In a bank's delivery pipeline, they quietly do three kinds of damage: they waste engineering time on re-runs, they slow releases down, and they teach teams to ignore failing tests. That last effect is the dangerous one. When a red result usually means "just re-run it", a real defect can be re-run and dismissed in exactly the same way. 

Flaky tests rarely appear in a steering committee report, yet they decide how fast a bank can release and how far a green pipeline can be trusted as release evidence. This article explains why test-suite reliability belongs on a leadership dashboard, which numbers to track and how mature teams bring flakiness under control. 

What a flaky test is, and why it's a governance issue 

A flaky test is one that passes and fails on the same code, without any change. The cause is usually in the test or its environment, not in the product: a timing assumption, shared test data, an unstable test environment or a dependency on the current date. 

On its own, one flaky test is a nuisance. At scale, flakiness changes how an organisation behaves. Google's engineering team has published that almost 16% of their tests, roughly one in six, showed some level of flakiness, and that about 1.5% of all test runs reported a flaky result. Banking platforms, with their integrations, shared test environments and date-sensitive logic, are rarely in a better position. 

The governance problem is simple. In most banks, a passing test suite is part of the release decision: it is the evidence that a change was tested before it reached production. If the suite fails randomly, people stop treating red as a signal. Release gates are overridden "because it's probably flaky", and the test results that change management relies on lose their meaning. A control that is routinely overridden is not really a control. 

The three hidden costs 

Time. Every flaky failure triggers a re-run, an investigation or both. Re-running a 40-minute pipeline twice a day across ten teams quickly adds up to weeks of engineering time per quarter, plus the CI infrastructure cost. Developers waiting for a green build switch context and lose more time again. 

Speed. Teams slow down to protect themselves. They batch changes into bigger releases because each pipeline run is unreliable, which makes each release riskier and harder to diagnose. Release windows get missed, and regulatory or customer-facing changes slip. 

Risk. This is the most serious cost. When failures are routinely dismissed, a real defect can be dismissed with them. The test did its job, but the organisation had learned not to listen. In a payment system, that is how a defect that the suite actually caught still reaches production. 

The numbers leaders should track 

Test reliability can be measured, and a small number of metrics give leaders a clear view without going into individual tests. These are the ones we recommend reporting monthly, per product area: 

First-run pass rate on the main branch. The share of pipeline runs on unchanged, already-merged code that pass the first time. This is the single best indicator of trust. If code that already passed review fails on a re-run, the suite is the problem. 

Flake rate. The share of test failures that pass on retry with no code change. It shows how much of the "red" the team sees is noise. 

Re-run cost. CI hours and estimated engineer hours spent on re-runs and flaky-failure investigation. This translates reliability into money, which is the language budget discussions use. 

Quarantine size and age. How many tests are temporarily excluded because they are unreliable, and for how long. A quarantine that only grows is a growing gap in test coverage. 

Time to fix a flaky test. The median number of days from detection to repair. It shows whether the team treats flakiness as real work or as background noise. 

Gate overrides. How often a release went ahead with failing tests that were declared flaky. For governance, this is the number to watch most closely, because each override is a decision taken on the assumption that a failure didn't matter. 

These metrics need no special tooling. Most CI platforms already record re-runs and results, and a simple report built on that data is enough to start. 

How mature teams fix flaky tests 

Teams that get flakiness under control don't do it with a one-off clean-up sprint. They treat it as an ongoing process with clear rules. 

Detect automatically, and record retries honestly 

Automatic retries are useful, but only if they are visible. A test that fails and then passes on retry should be recorded as flaky, not as passed. The retry keeps the pipeline moving; the record keeps the problem visible. Teams that retry silently usually believe their suite is healthier than it is. 

Quarantine with an owner and a deadline 

When a test is confirmed as flaky, it moves to quarantine: it still runs and reports, but it no longer blocks the pipeline. The rule that makes quarantine work is that every quarantined test has a named owner and a fix-by date. Without both, quarantine becomes a quiet way of deleting tests. 

Fix the causes, not the symptoms 

In banking systems, most flaky tests fall into a few familiar categories, and each has a lasting fix: 

Timing assumptions. Tests that wait a fixed time for an asynchronous process, such as a payment status update, fail when the environment is slower. The fix is to wait for the expected outcome, not for a number of seconds. In one of the best-known academic studies of flaky tests, asynchronous waiting was the most common root cause, behind 45% of the fixes analysed. 

Shared test data. Two tests using the same customer or account interfere with each other, especially when run in parallel. The fix is for each test to create the data it needs. 

Unstable shared environments. A test that depends on a shared integration environment, with a core banking stub or a third-party sandbox, fails whenever that environment is down. The fix is to virtualise those dependencies for most tests, keep a small set of true end-to-end tests, and use contract tests to cover the integration points in between. 

Dates and times. This one is specific to banking. Tests that pass all month and fail on the last business day, on a public holiday, after a payment cut-off time or when clocks change usually depend on the real date. The fix is a controllable clock in the test setup, so that end-of-month, cut-off and holiday cases are tested on purpose rather than by accident. 

Order dependency. Tests that only pass when run after another test. Running tests in random order in CI exposes these early. 

Prevent new flaky tests 

New tests are run several times in a row before they are merged. A test that can't pass ten times out of ten isn't ready. This simple rule stops most new flakiness at the door. 

Budget for reliability | Mature teams reserve a fixed share of capacity, often a small percentage of each sprint, for test reliability work, and they agree a threshold for action. For example, if the first-run pass rate drops below the agreed level, reliability work takes priority over new tests until it recovers. This turns reliability from goodwill into a managed commitment. 

What leaders can do | Test reliability improves fastest when leadership treats it as part of delivery risk rather than as an engineering detail. 

Four actions make the biggest difference: 

• Put the numbers on the delivery dashboard, next to lead time and change failure rate. What gets reported gets managed. 
• Treat gate overrides as exceptions that need a reason. Not to slow teams down, but to make each decision visible and to show the pattern over time. 
• Fund reliability work explicitly. If it has to compete with features for capacity without any agreed budget, it will lose until an incident forces it. 
• Ask about trust, not just coverage. Coverage shows how much is tested. First-run pass rate shows whether the results are believed. Both matter, but only the second tells you whether the release gate is really working. 

Closing thought 

A test suite is only valuable if its results are trusted. When a team re-runs failures by reflex, the suite stops protecting releases, and the cost shows up as slower delivery, wasted engineering time and defects that were caught and still shipped. Measuring reliability, fixing flaky tests at their cause and treating overrides as governance decisions brings back the one thing a release gate needs: a red result that everyone takes seriously.