Every team with test automation knows this scenario: the pipeline is red, someone clicks “re-run” and the second attempt passes. Nobody knows whether it was a product bug, a test bug or the environment. After a few weeks, a red status no longer means anything.
A flaky test is a test that passes one time and fails another against the same code. It is harmful for two reasons: it wastes time investigating false alarms and, worse, it teaches the team to ignore real failures.
The most common causes
Waiting for time instead of state
The most frequent culprit. A test waits a fixed two seconds for an element to load. That is enough on a fast machine but not on a busy CI runner. The fix: wait for a specific application state - an element becoming visible, a request completing, a URL changing - not for time to pass. Modern tools such as Playwright do this by default, as long as we do not fight them.
Shared test data
Two tests use the same account. One changes the password, the other tries to log in. Run sequentially, everything works; run in parallel, it randomly fails. Each test should create its own data or use an isolated data set.
Order dependency
Test B passes only if test A ran first, because A left some state behind. The check: run the suite in random order. Whatever breaks has a hidden dependency.
Unstable environments and external services
A slow test environment, a payment provider’s API limits, expiring certificates. Fixing the test itself will not help here - either stabilise the environment or replace the external service with a controlled mock in tests that do not verify the integration itself.
A real product bug
It happens. A race condition in the application looks exactly like a flaky test. That is why “it’s probably flaky” must never be the default diagnosis without analysis.
Getting it under control, step by step
- Start measuring. Collect result history: which tests fail and how often they pass on re-run. Without data, the discussion is based on impressions.
- Introduce quarantine. A test identified as flaky moves to a separate group: it still runs but does not block the pipeline. Importantly, quarantine has a deadline and an owner - it is not a graveyard.
- Fix by cost. Start with the tests that fail most often and protect the most important journeys.
- Limit automatic retries. A single retry that records it was needed is acceptable. Three silent retries only hide the problem.
- Capture evidence on failure. Screenshots, video, traces, console and network logs. A test that fails without a trace is almost impossible to fix.
- Set a rule for new tests. Before merging, a new test runs a dozen or more times in a row. If it fails once, it does not join the main suite.
How much flakiness is too much?
There is no universal threshold, but a good goal is a state in which a red pipeline almost always means a real problem. If the team regularly clicks “re-run” without looking at the result, trust has already been lost - and that is the right moment to deal with it.
Fewer stable tests give more safety than many tests nobody believes.
We also take over existing test suites: we start with a review of the repository and its stability, then propose a repair plan. If your automation gets in the way more often than it helps, tell us about it.