Every FinTech engineering team we work with has the same test. It fails on Tuesday, passes on Wednesday, fails again on Thursday with no code change in between. Everyone knows which one it is. Everyone knows to hit retry. Nobody has fixed it, because it is only one test.
It is never only one test. Within a few months there are a dozen, then a red build means nothing, and the team has quietly stopped treating the regression suite as evidence. They still run it. They still report on it. They just do not believe it, and a suite nobody believes is not protecting anything.
This article is about what flaky tests actually cost, why they appear in clusters rather than one at a time, and what a team in a regulated environment should do about them. The short version is that flakiness is rarely a problem with the testing tool. It is a signal that the suite has nobody accountable for it.
What a flaky test is, and what it is not
A flaky test is one that produces different results on the same code without any change in the code under test. Pass, fail, pass. The test has not found a defect. It has found something unstable in itself, its data, its environment or its timing.
That definition matters because the usual response, re-running until green, treats the symptom as noise. Sometimes it is noise. Often it is a real intermittent fault in the product that the test has caught and the team has trained itself to ignore. A race condition in a payment status update. A timeout against a bureau API under load. A date calculation that fails on the last day of the month. In financial software, intermittent is not the same as harmless.
The problem is that once a team has learned to retry, it can no longer tell the difference between a test that is unstable and a product that is.
What the research says it costs
The numbers on flaky tests are better than most testing statistics because several of them come from peer-reviewed work and large-scale engineering data rather than vendor surveys.
Google's engineering team reported that flaky tests consumed around 16 percent of its total testing compute budget, and that a single flaky test took on average several hours of engineer time to diagnose. A 2024 industrial study that tracked five years of CI logs, commits and recorded work time at a commercial project with around thirty developers put the lost productive time at roughly 2.5 percent. A separate industrial case study found developers spending about 1.3 percent of their working time repairing flaky tests. A 2025 analysis of open-source Java projects found that just over half of all projects were affected by flaky builds, and that two thirds of re-run builds showed flaky behaviour.
Those are the measured figures. Vendor estimates run much higher, with claims of six to eight hours per engineer per week, and we treat those with caution. But even the conservative end, two to three percent of engineering time plus a large slice of CI compute, is a real line item for a team of thirty. For a team of a hundred it is the cost of two or three engineers doing nothing but waiting for re-runs.
And that is the direct cost. The indirect cost is the one that matters in FinTech.
The cost that does not show up in the CI bill
When a suite becomes unreliable, three things happen in sequence, and each is worse than the last.
First, re-runs become policy. A failed build is re-triggered before anyone reads the failure. Build times double. Nobody minds, because the alternative is reading the failure.
Second, failures become invisible. Once re-running is normal, a genuine failure that happens to pass on the second attempt is indistinguishable from a flaky one. The team has built a process that systematically hides intermittent product defects. In a lending platform or a payments engine, intermittent defects are exactly the category that reaches production, because they are the category that deterministic testing was never going to catch consistently anyway.
Third, the suite loses its authority. When a release decision is being made and the regression run is red, the question is no longer "what failed" but "is it the flaky ones again". The suite has stopped being evidence. Release sign-off becomes a judgement call made under deadline pressure, which is the situation good testing exists to prevent.
We have sat in release meetings where a red regression run was waved through on the basis that it was probably the usual suspects. Sometimes it was. The time it was not, the defect was in a fee calculation and the cost of the remediation was several multiples of what fixing the suite would have taken.
Flaky tests cluster, which is the useful part
The most practically important finding in recent research is that flaky tests are not independent. A 2025 study found that around three quarters of flaky tests belonged to clusters of tests that fail together for the same underlying reason. That matches what we see when we audit a suite.
It means the right unit of work is not the test. It is the cause. A suite with sixty flaky tests usually has five or six root causes, and fixing one cause stabilises ten tests at once. The causes we find most often in FinTech suites, in rough order of frequency:
Shared mutable test data. Tests that create, modify or depend on the same customer, account or transaction records, and therefore pass or fail depending on which test ran before them. This is the single biggest cause we see and it is almost always a data strategy problem rather than a test problem.
Time and date dependence. Tests that assume the current date, the current month, business days, or a settlement cycle. They pass for twenty-eight days and fail on the twenty-ninth. Month-end, year-end and leap years are reliable sources of mysterious Monday failures.
Real external dependencies. Tests that call a live sandbox for a payment gateway, a credit bureau, an identity provider or an open banking endpoint. The sandbox is slow, rate-limited or down, and the test reports a product failure that is nothing of the kind.
Timing and asynchrony. Fixed sleeps instead of explicit waits. Assertions that run before an event has been processed. Tests that pass on a fast developer machine and fail on a loaded CI runner.
Order dependence. Tests that only pass if an earlier test has set something up. Fine until someone parallelises the suite to speed it up, at which point a quarter of it breaks.
Environment drift. Configuration, feature flags or reference data that differ between the environment the test was written against and the one CI runs on.
None of these is exotic. All of them are design decisions that were made, or not made, when the automation was built. We wrote about this earlier: most automation problems are design problems that were ignored while they were still cheap to fix.
What to do about it
The standard advice is to quarantine flaky tests and fix them later. That is better than nothing and much worse than it sounds, because later rarely arrives and a quarantine with forty tests in it is a suite with forty holes. Here is what we recommend instead, in order.
1. Measure it. Record every test outcome across every run, not just the final result of the build. A test that failed and then passed on retry is flaky, and the retry should not erase that. Most CI platforms can expose this; most teams never look. Within two weeks you will know which tests are unreliable, how often, and whether they fail together.
2. Cluster before you fix. Group the unstable tests by failure pattern, not by name. Same error message, same timing, same data. Then fix the cause of the largest cluster first. This is where the three-quarters figure pays off: a week on the shared test data problem often does more than a month of fixing tests one at a time.
3. Give every flaky test an owner and a deadline. Not a quarantine folder. A person, a ticket and a date. A test that nobody owns is a test that will be flaky forever. If the honest answer is that nobody can own it, delete it. A deleted test is a known gap. A flaky test is an unknown one.
4. Stop treating a retry as a pass. Change the CI policy so that a build only goes green if it passed first time, and a passed-on-retry is reported as a warning with the test named. This is unpopular for about a fortnight and then it is the single thing that stops the suite decaying again.
5. Fix the data strategy. Every test should create the data it needs, or use isolated fixtures it owns, and leave nothing behind. For FinTech suites this usually means investing in a proper test data layer: synthetic customers, accounts and transactions generated per test rather than a shared seeded database that every test quietly corrupts. It is the least glamorous work in automation and it is the work that makes everything else possible.
6. Treat intermittent product failures as defects. When clustering shows that a flaky test is actually catching a real race condition or timeout in the product, raise it as a defect with the same priority as any other. Intermittent is a severity, not an excuse. In payments it is often the highest one.
7. Put reliability on the dashboard. Report flake rate alongside pass rate. A suite that is 98 percent green and 15 percent flaky is not a healthy suite, and the people making release decisions should be able to see that at a glance.
The question to ask about your own suite
If your regression run goes red tomorrow, will anyone read it before they re-run it?
If the honest answer is no, the suite has already stopped doing its job, and the cost is not the CI minutes. It is that your release decisions are being made without the evidence you think you have.
Fixing that is less work than most teams expect, because the causes cluster. It does need someone to own it. If you want an independent view of where your automation actually stands, that is what our Test Automation and QA Advisory work exists for. And if the suite needs rebuilding on a design that will hold, we would rather tell you that honestly than help you add more tests to a foundation that cannot carry them.