Flaky Test Hunter

Stop re-running. Start diagnosing.

Every flaky test has a cause, and every cause has a family. Learn to classify a flake in 5 minutes — timing, network, state, order, or environment — so your fix targets the root, not the symptom. Pair with /labs/anti-patterns (prevention) for the complete picture.

“Just hit retry” is tech debt. Automatic retries hide bugs that will escape to production. A flaky test doesn't mean the test is wrong — it often means the system under test has a real race, leak, or resource limit that users will hit too.

1. Timing — async races and premature assertions

Symptom: Test fails ~5 % of runs. Failure is always on the assertion right after a click / navigation / network call. Re-running makes it green. Slower CI machines fail more often.

Detection

// Instrument: replay the SAME test 50 times in a loop on CI
npx playwright test --repeat-each=50 --grep "flaky-test-name"

// Read the output — flakiness is a FREQUENCY, not a boolean.
// "3/50 failed" is a smoking gun; "1/50" might be noise.

// Enable trace viewer on failures to see exact timing
npx playwright test --trace on-first-retry --reporter=html

Reproduce deterministically

// Artificial CPU pressure exposes timing bugs deterministically
await page.context().setDefaultTimeout(1000);  // tight timeout
await page.emulateNetworkConditions({ latency: 500 });
// Or run under stress: parallel heavy workload on the same machine

Fix pattern

// ❌ Manual wait — the delay is a guess AND a tax on every run
await page.waitForTimeout(2000);
expect(await page.locator('.status').textContent()).toBe('Done');

// ✅ Auto-waiting assertion — polls until true or times out
await expect(page.locator('.status')).toHaveText('Done');

// For navigation, wait for the URL OR a visible element, not both
await page.waitForURL('/success');
await expect(page.getByRole('heading', { name: /thanks/i })).toBeVisible();

Rule of thumb: If the fix is 'add a longer sleep', the bug is still there — you just pushed it further out. Replace every sleep with a condition-based wait, and the race disappears AND the test gets faster.

2. Network — rate limits, retries, proxy quirks

Symptom: Test fails in bursts — several in a row go red, then green for hours. Failures cluster at specific times of day (peak hours). Isolated runs pass; suite runs fail.

Detection

// Log every outbound request + response time
page.on('request', (req) => console.log('→', req.url()));
page.on('response', async (res) => {
  console.log('←', res.status(), res.url(), res.request().timing());
});

// Grep CI logs for 429, 502, 504, 0 (connection refused)
// A 429 at random is a rate-limit issue. A 502/504 at random is infra.

Reproduce deterministically

// Hammer the endpoint from one test to trip the rate limit
test('reproduce rate-limit flakiness', async ({ request }) => {
  for (let i = 0; i < 100; i++) {
    await request.get('/api/search?q=test');
  }
  // Now run the actual suite — earlier tests poisoned the bucket
});

Fix pattern

// ✅ Per-test auth token: avoid shared-key throttling
test.use({
  extraHTTPHeaders: async ({}, use) => {
    const token = await mintFreshToken();  // unique per test
    await use({ Authorization: `Bearer ${token}` });
  },
});

// ✅ Retry 5xx responses, NOT the whole test
await expect.poll(
  async () => (await request.get('/api/item/42')).status(),
  { intervals: [100, 500, 1000, 2000] },
).toBe(200);

// ✅ Mock the unstable third party instead of calling it
await page.route('**/api.thirdparty.com/**', (route) =>
  route.fulfill({ status: 200, body: fixture }),
);

Rule of thumb: Don't retry the whole test on network flake — that hides the flake AND papers over a real rate-limit bug. Identify the unstable call, mock it if it's a third party, or isolate per-test credentials if it's your own.

3. State — shared fixtures, cross-test leakage

Symptom: Test A passes in isolation, fails in the suite. Order-sensitive — running tests in different order changes the outcome. First test always passes; later ones fail randomly.

Detection

// Randomize order — if flakiness increases, it's state leakage
npx playwright test --shuffle

// Test the suspect in isolation vs in the suite
npx playwright test --grep "suspect"           # pass
npx playwright test                             # fails at suspect

// Compare fixtures / DB state before and after — dump before+after hashes
before: SHA(db dump) = abc123
after:  SHA(db dump) = def456  ← leaked

Reproduce deterministically

// Run the full suite with --repeat-each to amplify leakage
npx playwright test --repeat-each=10 --workers=1
// workers=1 serialises, making dependency explicit

Fix pattern

// ✅ Per-test reset via Playwright fixture
export const test = base.extend<{ freshDb: void }>({
  freshDb: [
    async ({}, use) => {
      await resetDatabase();
      await seed({ users: defaultUsers });
      await use();
      // cleanup — runs even on test failure
      await resetDatabase();
    },
    { auto: true },  // applies to every test automatically
  ],
});

// ✅ Generate unique data per test instead of sharing
const email = `test+${Date.now()}+${Math.random()}@test.dev`;

// ✅ For DB-only tests, use transactional rollback
test.beforeEach(async () => await beginTx());
test.afterEach(async () => await rollbackTx());  // nothing persisted

Rule of thumb: If two tests share state, one of them is broken — even when both pass today. Fix with per-test reset (slow but bulletproof), transactional rollback (fast but DB-only), or unique-per-test data generation.

4. Order — hidden dependencies between tests

Symptom: Passing a single test directly fails: 'user not found'. The test implicitly depends on an earlier test having created that user. --shuffle breaks everything.

Detection

// Two classic signals:
// 1. A test that can't be run alone (requires previous state)
npx playwright test --grep "test-42"  // fails: user not found

// 2. --shuffle catastrophically changes pass rate
// Baseline order: 100/100 pass
// Shuffled:       73/100 pass  ← hidden dependencies

Reproduce deterministically

// Artificially re-order to expose dependencies
npx playwright test --list  # get test IDs
# pick last test and first test, swap them manually

Fix pattern

// ❌ Silent chain — test 2 depends on test 1
test('step 1: create user', async ({ request }) => {
  globalThis.userId = (await request.post('/users', { data: {...} })).id;
});

test('step 2: update user', async ({ request }) => {
  await request.patch(`/users/${globalThis.userId}`);  // breaks alone
});

// ✅ Every test provisions its own state
test('update user', async ({ request }) => {
  const userId = await createUser();            // inline setup
  await request.patch(`/users/${userId}`);
  // cleanup via fixture or afterEach
});

// ✅ Or: use test.describe.serial only when intentional
test.describe.serial('multi-step wizard', () => {
  // Explicit: these MUST run in order, documented as such
});

Rule of thumb: Order-dependent tests are tests with hidden contracts. Convert implicit chains into explicit describe.serial blocks OR break the chain entirely by duplicating setup. Never 'fix' by pinning the order — that's papering over the dependency.

5. Environment — timezone, locale, resource limits

Symptom: Passes on developer laptop, fails on CI. Passes in one region, fails in another. Fails on Tuesdays, green on Wednesdays. Fails on Docker with 2 CPUs; passes on 8 CPUs.

Detection

// Compare environments explicitly
// Local:     Node 22.13.1, Linux 6.5, 16 CPUs, 32 GB RAM, TZ=CET
// CI runner: Node 22.13.0, Linux 6.2,  2 CPUs,  7 GB RAM, TZ=UTC

// Check for ambient dependencies
echo "Date: $(date)"
echo "TZ:   $TZ"
echo "Locale: $LANG"
echo "CPU count: $(nproc)"
echo "RAM:  $(free -m | awk '/Mem/ {print $2}')"
echo "DST:  $(date +%z)"

Reproduce deterministically

// Pin environment differences deterministically
docker run -e TZ=Pacific/Auckland --cpus=1 --memory=1g \
  myapp:test \
  npx playwright test

// Or reproduce locally with a Docker-in-Docker setup

Fix pattern

// ✅ Freeze the clock inside tests
await page.clock.install({ time: new Date('2026-04-21T10:00:00Z') });

// ✅ Pin locale + timezone on the browser context
const context = await browser.newContext({
  locale: 'en-US',
  timezoneId: 'UTC',
});

// ✅ Resource-aware assertions
const isLowCPU = os.cpus().length <= 2;
const timeout = isLowCPU ? 15_000 : 5_000;
await expect(widget).toBeVisible({ timeout });

// ✅ Don't assume GPU presence for canvas / WebGL tests
test.skip(os.platform() === 'linux' && !process.env.DISPLAY,
  'Requires display server');

Rule of thumb: Every implicit dependency on the environment is a future flake. Pin timezone, locale, and clock explicitly. Scale timeouts with available resources. Skip (not fail) tests that need capabilities the environment lacks.

Failure-frequency triage matrix

Flakiness is a frequency, not a boolean. Measure over 50–100 runs, then act based on the rate:

Failure rate (over 50+ runs)	Action
< 1 %	Acknowledge, log. Consider if it's noise.
1 – 5 %	Investigate when touched. Quarantine if unresolvable in 1 day.
5 – 20 %	Fix within the sprint. File a ticket immediately.
> 20 %	Top-priority. Disable suite until fixed — it's worse than no test.

Quarantine = move to a separate “known-flaky” suite that doesn't gate merges but surfaces for review. Never delete flaky tests — they often point at real bugs.

← Back to all Labs

Back to Labs

Flaky Test Hunter

Stop re-running. Start diagnosing.

1. Timing — async races and premature assertions

Symptom: Test fails ~5 % of runs. Failure is always on the assertion right after a click / navigation / network call. Re-running makes it green. Slower CI machines fail more often.

Detection

// Instrument: replay the SAME test 50 times in a loop on CI
npx playwright test --repeat-each=50 --grep "flaky-test-name"

// Read the output — flakiness is a FREQUENCY, not a boolean.
// "3/50 failed" is a smoking gun; "1/50" might be noise.

// Enable trace viewer on failures to see exact timing
npx playwright test --trace on-first-retry --reporter=html

Reproduce deterministically

// Artificial CPU pressure exposes timing bugs deterministically
await page.context().setDefaultTimeout(1000);  // tight timeout
await page.emulateNetworkConditions({ latency: 500 });
// Or run under stress: parallel heavy workload on the same machine

Fix pattern

// ❌ Manual wait — the delay is a guess AND a tax on every run
await page.waitForTimeout(2000);
expect(await page.locator('.status').textContent()).toBe('Done');

// ✅ Auto-waiting assertion — polls until true or times out
await expect(page.locator('.status')).toHaveText('Done');

// For navigation, wait for the URL OR a visible element, not both
await page.waitForURL('/success');
await expect(page.getByRole('heading', { name: /thanks/i })).toBeVisible();

2. Network — rate limits, retries, proxy quirks

Symptom: Test fails in bursts — several in a row go red, then green for hours. Failures cluster at specific times of day (peak hours). Isolated runs pass; suite runs fail.

Detection

// Log every outbound request + response time
page.on('request', (req) => console.log('→', req.url()));
page.on('response', async (res) => {
  console.log('←', res.status(), res.url(), res.request().timing());
});

// Grep CI logs for 429, 502, 504, 0 (connection refused)
// A 429 at random is a rate-limit issue. A 502/504 at random is infra.

Reproduce deterministically

// Hammer the endpoint from one test to trip the rate limit
test('reproduce rate-limit flakiness', async ({ request }) => {
  for (let i = 0; i < 100; i++) {
    await request.get('/api/search?q=test');
  }
  // Now run the actual suite — earlier tests poisoned the bucket
});

Fix pattern

// ✅ Per-test auth token: avoid shared-key throttling
test.use({
  extraHTTPHeaders: async ({}, use) => {
    const token = await mintFreshToken();  // unique per test
    await use({ Authorization: `Bearer ${token}` });
  },
});

// ✅ Retry 5xx responses, NOT the whole test
await expect.poll(
  async () => (await request.get('/api/item/42')).status(),
  { intervals: [100, 500, 1000, 2000] },
).toBe(200);

// ✅ Mock the unstable third party instead of calling it
await page.route('**/api.thirdparty.com/**', (route) =>
  route.fulfill({ status: 200, body: fixture }),
);

3. State — shared fixtures, cross-test leakage

Symptom: Test A passes in isolation, fails in the suite. Order-sensitive — running tests in different order changes the outcome. First test always passes; later ones fail randomly.

Detection

// Randomize order — if flakiness increases, it's state leakage
npx playwright test --shuffle

// Test the suspect in isolation vs in the suite
npx playwright test --grep "suspect"           # pass
npx playwright test                             # fails at suspect

// Compare fixtures / DB state before and after — dump before+after hashes
before: SHA(db dump) = abc123
after:  SHA(db dump) = def456  ← leaked

Reproduce deterministically

// Run the full suite with --repeat-each to amplify leakage
npx playwright test --repeat-each=10 --workers=1
// workers=1 serialises, making dependency explicit

Fix pattern

// ✅ Per-test reset via Playwright fixture
export const test = base.extend<{ freshDb: void }>({
  freshDb: [
    async ({}, use) => {
      await resetDatabase();
      await seed({ users: defaultUsers });
      await use();
      // cleanup — runs even on test failure
      await resetDatabase();
    },
    { auto: true },  // applies to every test automatically
  ],
});

// ✅ Generate unique data per test instead of sharing
const email = `test+${Date.now()}+${Math.random()}@test.dev`;

// ✅ For DB-only tests, use transactional rollback
test.beforeEach(async () => await beginTx());
test.afterEach(async () => await rollbackTx());  // nothing persisted

4. Order — hidden dependencies between tests

Symptom: Passing a single test directly fails: 'user not found'. The test implicitly depends on an earlier test having created that user. --shuffle breaks everything.

Detection

// Two classic signals:
// 1. A test that can't be run alone (requires previous state)
npx playwright test --grep "test-42"  // fails: user not found

// 2. --shuffle catastrophically changes pass rate
// Baseline order: 100/100 pass
// Shuffled:       73/100 pass  ← hidden dependencies

Reproduce deterministically

// Artificially re-order to expose dependencies
npx playwright test --list  # get test IDs
# pick last test and first test, swap them manually

Fix pattern

// ❌ Silent chain — test 2 depends on test 1
test('step 1: create user', async ({ request }) => {
  globalThis.userId = (await request.post('/users', { data: {...} })).id;
});

test('step 2: update user', async ({ request }) => {
  await request.patch(`/users/${globalThis.userId}`);  // breaks alone
});

// ✅ Every test provisions its own state
test('update user', async ({ request }) => {
  const userId = await createUser();            // inline setup
  await request.patch(`/users/${userId}`);
  // cleanup via fixture or afterEach
});

// ✅ Or: use test.describe.serial only when intentional
test.describe.serial('multi-step wizard', () => {
  // Explicit: these MUST run in order, documented as such
});

5. Environment — timezone, locale, resource limits

Symptom: Passes on developer laptop, fails on CI. Passes in one region, fails in another. Fails on Tuesdays, green on Wednesdays. Fails on Docker with 2 CPUs; passes on 8 CPUs.

Detection

// Compare environments explicitly
// Local:     Node 22.13.1, Linux 6.5, 16 CPUs, 32 GB RAM, TZ=CET
// CI runner: Node 22.13.0, Linux 6.2,  2 CPUs,  7 GB RAM, TZ=UTC

// Check for ambient dependencies
echo "Date: $(date)"
echo "TZ:   $TZ"
echo "Locale: $LANG"
echo "CPU count: $(nproc)"
echo "RAM:  $(free -m | awk '/Mem/ {print $2}')"
echo "DST:  $(date +%z)"

Reproduce deterministically

// Pin environment differences deterministically
docker run -e TZ=Pacific/Auckland --cpus=1 --memory=1g \
  myapp:test \
  npx playwright test

// Or reproduce locally with a Docker-in-Docker setup

Fix pattern

// ✅ Freeze the clock inside tests
await page.clock.install({ time: new Date('2026-04-21T10:00:00Z') });

// ✅ Pin locale + timezone on the browser context
const context = await browser.newContext({
  locale: 'en-US',
  timezoneId: 'UTC',
});

// ✅ Resource-aware assertions
const isLowCPU = os.cpus().length <= 2;
const timeout = isLowCPU ? 15_000 : 5_000;
await expect(widget).toBeVisible({ timeout });

// ✅ Don't assume GPU presence for canvas / WebGL tests
test.skip(os.platform() === 'linux' && !process.env.DISPLAY,
  'Requires display server');

Failure-frequency triage matrix

Flakiness is a frequency, not a boolean. Measure over 50–100 runs, then act based on the rate:

Failure rate (over 50+ runs)	Action
< 1 %	Acknowledge, log. Consider if it's noise.
1 – 5 %	Investigate when touched. Quarantine if unresolvable in 1 day.
5 – 20 %	Fix within the sprint. File a ticket immediately.
> 20 %	Top-priority. Disable suite until fixed — it's worse than no test.

Quarantine = move to a separate “known-flaky” suite that doesn't gate merges but surfaces for review. Never delete flaky tests — they often point at real bugs.

← Back to all Labs