Debugging Flaky Tests and Race Conditions with AI
How to use AI coding assistants to diagnose intermittent test failures — the context-sharing strategies and prompting patterns that surface root causes fast.
Flaky tests are one of the most demoralizing problems in software development. A test that fails intermittently — passing on your machine, failing on CI, or failing every third run with no obvious reason — is difficult to debug because the bug is not consistently observable. It requires reasoning about timing, execution order, and shared state that most debugging tools are not well-suited for.
AI coding assistants are surprisingly effective here, not because they can observe runtime behavior, but because intermittent failures almost always follow recognizable structural patterns. If you give the model the right context, it can often identify the class of problem — and the fix — before you have reproduced it twice.
Why AI helps with non-deterministic bugs
A deterministic bug is easier to find than to fix: you narrow it with a debugger, reproduce it with a minimal case, and confirm the fix by running the test. Flaky tests are the opposite: the failure is observable but not reliably reproducible.
What makes LLMs useful is that most flaky test failures share structural causes: test state leaking between cases, async operations resolving in variable order, timers interfering with mocked time, global singletons holding state across test runs. An AI that has processed millions of test files and bug reports has encountered these patterns far more than any individual developer. You are not asking it to reason from first principles — you are asking it to pattern-match against known failure classes.
Show the model what it needs
The single biggest mistake developers make when asking AI for help with a flaky test is pasting only the test file. Effective context has three parts.
The failure output, verbatim. Paste the exact error message and stack trace. Do not summarize it — the exact error string often indicates the failure mode. AssertionError: expected null but got {...} is different from Timeout: Exceeded 2000ms, and each points to a different class of problem.
The test and the code under test, together. Flaky failures usually live at the boundary between the test and the implementation. Show both.
The observation pattern. How often does it fail? Under what conditions? Always on CI, never locally? Only on the third run in the full suite, never in isolation? This temporal and contextual pattern is often the strongest signal.
I have a flaky test that fails roughly 1 in 5 runs on CI but never locally.
It is consistently the last test in this file when run with the full suite,
but passes when run in isolation.
Failure output:
AssertionError: expected undefined but got { id: 2, name: 'legacy' }
Test file: [paste]
Module under test: [paste]
What are the most likely causes given the "passes in isolation, fails in suite" pattern?
The “passes in isolation, fails in suite” framing immediately narrows the search space to test isolation problems: shared state, module-level singletons, or teardown that is not running cleanly. The model does not need to guess at causes — you have told it the shape of the failure.
A concrete example: state leaking between tests
Here is a common pattern that AI is good at catching. Say you have a singleton cache that is not reset between test runs:
// cache.ts
let _cache: Map<string, unknown> | null = null;
export function getCache() {
if (!_cache) _cache = new Map();
return _cache;
}
And a test that passes in isolation but fails when run after another test that populates the cache:
// feature.test.ts — FLAKY
import { getCache } from './cache';
test('returns undefined for unknown key', () => {
const cache = getCache();
expect(cache.get('user:2')).toBeUndefined(); // fails if prior test set this key
});
Pasting both files and asking “what shared state could leak into this test from prior tests” surfaces the issue immediately. The fix is straightforward — reset the module-level state in afterEach:
// feature.test.ts — FIXED
import { getCache } from './cache';
afterEach(() => {
getCache().clear();
});
test('returns undefined for unknown key', () => {
const cache = getCache();
expect(cache.get('user:2')).toBeUndefined();
});
The pattern extends to database records left over from previous tests, mock implementations not reset with afterEach, and event listeners that persist across test instances. For any of these, show the setup and teardown alongside the failing test and ask specifically: “What shared state could leak into this test?”
Prompting for timing and async bugs
Async race conditions require a different approach. A test that sometimes resolves in the wrong order — or that flakes because of setTimeout and setInterval interference — is usually a symptom of a missing await, an event handler resolving before the code under test has finished, or a timer that was supposed to be mocked but is not.
When you suspect a timing issue, prompt with an explicit hypothesis:
This test is failing intermittently with a "null is not an object" error
on the line that reads response.data. I think there is a timing issue —
the response is not ready when the assertion runs, but I am not sure why
the existing await is not sufficient.
Test: [paste]
Async implementation under test: [paste]
What is wrong with the timing, and is there a cleaner way to structure this test?
Asking “what is wrong with the timing” instead of “why is this broken” focuses the model on async control flow rather than the broader implementation. You get a targeted answer about event ordering instead of a generic analysis of the whole function.
A related pattern is tests that depend on real timers. If your code calls setTimeout and your test does not mock time, the behavior depends on the CI machine’s scheduler. Ask the model explicitly: “Does this test have any dependency on real timers that should be mocked?” It will scan for setTimeout, setInterval, Date.now(), and similar calls that are easy to miss on a manual read.
Validating the proposed fix
When the model proposes a fix for a flaky test, ask for the reasoning before applying it:
You suggested adding jest.resetModules() in afterEach.
Can you explain exactly what is holding state between tests in this case,
and why resetModules fixes it rather than just resetting the mock?
I want to understand the failure before applying the fix.
This is worth doing for two reasons. First, it forces the model to produce verifiable reasoning that you can check against what you are seeing. Second, it distinguishes between a fix that addresses the root cause and a fix that changes timing enough to make the flake less frequent without actually solving it.
Flaky tests that are “fixed” by adding a small delay or increasing a timeout have not been debugged — they have been masked. The same failure will re-emerge under load or on a slower CI runner. A fix that does not come with a clear explanation of the root cause is a warning sign.
CI-only failures
Tests that fail only on CI are a distinct sub-case. The failure is real but not locally reproducible, which means the model is missing environmental context. Give it what it needs:
This test passes locally (macOS, Node 22, 8 cores) but fails on CI
(Ubuntu, Node 22, 2 cores, tests run in parallel across 4 workers).
Failure: [paste exact CI log output]
Test: [paste]
Implementation: [paste]
What execution-environment differences could cause this failure?
The most common CI-only causes are CPU throttling making timers fire in different order, parallel test workers sharing file system paths or ports, and different Date behavior between OSes. Naming the environments explicitly lets the model prioritize these rather than proposing fixes that are irrelevant to the real context.
What AI cannot do here
AI is useful for pattern recognition and structural analysis. It cannot reproduce the failure, measure actual timing, or observe runtime state. A few limits to keep in mind.
It proposes hypotheses, not confirmations. “This looks like a module-level singleton holding state” is a hypothesis you still need to verify by reading the code carefully before applying a fix.
Test infrastructure complexity defeats it quickly. If your test setup is highly abstracted — many layers of factories, fixtures, and custom matchers — the model’s ability to trace state leaks degrades. Simplify the reproducing case as much as you can before asking for help. A test that directly constructs the object under test gives the model far more signal than one that goes through six factory layers.
It does not see your CI runner configuration. If test parallelism or worker count is a factor, tell the model how many workers are running and whether tests share any external resources.
The workflow that works is iterative: use AI to narrow the search space to a few likely suspects, investigate those suspects manually, confirm the root cause, then ask for a clean fix. Think of it as a collaborator for generating and prioritizing hypotheses, not an oracle that jumps from “here is the flaky test” to “here is the fix.” For the broader discipline of giving AI the context it needs to be useful, Managing Context in Long AI Coding Sessions covers the underlying principles — and the same front-loading strategy that keeps long sessions coherent applies equally to debugging sessions with a clear scope and a known failure. And if you want a framework for evaluating any AI-proposed fix before you commit to it, Evaluating AI-Generated Code Before It Ships covers the validation workflow in detail.
The developers who get the most out of AI for flaky tests are the ones who bring structured observations — failure pattern, environment, test context, code under test — rather than just the broken test file. The more precisely you describe the shape of the failure, the more precisely you get a useful answer back.

