CI/CD for Testing

From 30-minute local suite to 3-minute CI signal

You've written the tests. Now make them actually gate merges — fast, reliably, with actionable failures. Five patterns that scale a test suite from “dev tool” to “release gate”, with concrete YAML for GitHub Actions and explicit trade-offs.

The CI triangle: speed, cost, confidence. Pick two — the third pushes back. Every pattern below trades one for another. Knowing which you're trading away is the skill.

1. Parallelization — many workers, one suite

Split tests across N workers on a single machine. Linear speedup until workers exceed available CPU.

When to reach for it: Your suite takes more than 5 minutes. The easy first-order win — no infra change, just configuration.

Example (GitHub Actions)

# .github/workflows/e2e.yml
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '22', cache: 'pnpm' }
      - run: pnpm install --frozen-lockfile
      - run: pnpm playwright install --with-deps chromium

      # Parallel workers — tune to CPU count of the runner
      - run: pnpm playwright test --workers=4

# Playwright config
// playwright.config.ts
export default defineConfig({
  workers: process.env.CI ? 4 : '50%',  // 50% of CPUs locally
  fullyParallel: true,
});

Trade-off

Linear speedup up to CPU count, diminishing returns after. More workers means more memory pressure — 4 Chromium instances can burn 8 GB RAM easily. Requires test isolation: if tests share state, parallelization breaks them.

Common pitfall

Assuming 'more workers = faster' without measuring. 8 workers on a 2-CPU runner is SLOWER than 2 workers because of context switching. Profile first — `--workers=1` baseline vs N workers, pick the inflection point.

2. Sharding — split the suite across machines

Distribute test files across M machines. Each machine runs 1/M of the suite. Parallel machines, not just workers.

When to reach for it: Parallelization within one machine has maxed out but you still need to go faster. Or: tests need different environments (browser matrix, OS matrix).

Example (GitHub Actions)

# .github/workflows/e2e-sharded.yml
jobs:
  e2e:
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]     # 4 shards
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: pnpm install --frozen-lockfile
      - run: pnpm playwright install --with-deps chromium

      # Playwright built-in sharding
      - run: pnpm playwright test --shard=${{ matrix.shard }}/4

      - name: Upload blob report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report
          retention-days: 7

  # Merge reports after all shards complete
  merge-reports:
    needs: e2e
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with: { path: all-blobs, pattern: blob-report-* }
      - run: pnpm playwright merge-reports --reporter html all-blobs

Trade-off

Scales further than parallelization (dozens of machines). Costs more (N machines instead of 1). Added complexity: you need to merge reports from all shards afterwards, or lose per-shard visibility.

Common pitfall

Uneven shards — if one shard has all the slow tests, it becomes the bottleneck. Playwright's --shard distribution is file-based, not runtime-weighted. Use `--shard=i/N` with explicit test-file groupings if some files are 10× longer than others.

3. Retries — safety net OR flakiness hideout

Re-run failed tests automatically. Right approach for infra flakes, wrong approach for code bugs.

When to reach for it: On CI only, 1-2 retries max, as a last line of defence. NEVER on local dev — flakes must be visible there.

Example (GitHub Actions)

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,  // 2 retries on CI, 0 locally

  // Separate project for the retry pass — different timeouts
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
  ],
});

# Or more explicit: retry strategy per-failure-type
- name: Run tests (first pass)
  run: pnpm playwright test || echo "first-pass failed, will retry"
  continue-on-error: true

- name: Run failed tests (retry pass)
  run: pnpm playwright test --last-failed --workers=2

Trade-off

Retries hide genuine infrastructure flakes — good. Retries ALSO hide real bugs that would have been caught on first run — bad. Log every retry, track retry rate as a health metric, alert when it climbs.

Common pitfall

Retrying until green and considering the suite 'stable'. If 20 % of tests retry at least once, you have 20 % hidden bugs — not a stable suite. Surface retry-rate as a dashboard metric and treat regressions the same as test failures.

4. Reporting — logs, traces, screenshots, trends

Make failures actionable. Artifacts on every failure; trend charts over time.

When to reach for it: From day one. A failing CI without artifacts is a coin-flip — half the time it fails again locally, half the time you can't reproduce at all.

Example (GitHub Actions)

# .github/workflows/e2e.yml (reporting bits)
- run: pnpm playwright test
  continue-on-error: true

# Always upload artifacts, even on success (for trend analysis)
- name: Upload trace
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: playwright-trace
    path: test-results/
    retention-days: 30

- name: Upload HTML report
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: playwright-report
    path: playwright-report/
    retention-days: 30

- name: Publish to Allure / Slack / Datadog
  if: failure()
  run: |
    ./scripts/upload-to-allure.sh
    curl -X POST "$SLACK_WEBHOOK" \
      -d "{\"text\":\"🔴 E2E failed on ${{ github.ref }}\"}"

Trade-off

Artifacts cost storage (GitHub free tier: 2 GB retention). Trace files for every test is the gold standard but expensive. Compromise: trace on-first-retry, screenshots on failure only, full video only for the top 5 hardest-to-reproduce suites.

Common pitfall

Uploading the HTML report without context — a 500-test report without a status dashboard is unreadable. Pair with a per-file summary (which file failed, retry count, p95 duration) in the job output itself — one screen of CLI text beats one megabyte of HTML.

5. Flake detection — measure, don't just retry

Automated cron that re-runs stable tests and flags the ones that fail intermittently. Quarantine them before they erode trust.

When to reach for it: After you have a working suite. The first 100 tests are fine; the 100th flaky test starts the 'ignore red' culture that kills CI credibility.

Example (GitHub Actions)

# .github/workflows/flake-detection.yml
on:
  schedule:
    - cron: '0 3 * * *'  # 3 AM daily
  workflow_dispatch:

jobs:
  flake-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install --frozen-lockfile

      # Run each test 10 times, record pass/fail
      - run: pnpm playwright test --repeat-each=10 --max-failures=0

      # Parse results, compute per-test failure rate
      - run: ./scripts/compute-flake-rate.sh

      # Open or update GitHub Issues for flakes > 5%
      - run: ./scripts/file-flake-tickets.sh

# Playwright quarantine — tag known flakes, exclude from blocking runs
test('suspect test', { tag: ['@quarantined', '@flaky'] }, async () => {});

# On PR: skip quarantined tests
pnpm playwright test --grep-invert "@quarantined"

Trade-off

Running the suite N× every night costs compute (and CO₂). But the alternative — a slowly decaying CI where 'it was already red' becomes normal — is far more expensive in lost velocity and missed regressions.

Common pitfall

Quarantining tests into a void — they get tagged @flaky and never looked at again. Set an SLA: every @flaky test must be fixed or deleted within N days, or it auto-deletes. A pile of indefinitely-quarantined tests is dead weight, not a safety net.

Economics cheat-sheet

For each axis you want to push, here is the primary lever, what it costs, and where it stops working:

Axis	Lever	Cost	Ceiling
Speed	Parallelization + Sharding	Linear compute cost (N× workers = N× machine-minutes)	Amdahl's law — setup/install phases do not parallelize
Confidence	No retries + flake detection cron	Slower time-to-merge while flakes surface	Zero-retry only works if the suite is actually stable
Cost	Run fewer tests on PRs, full suite nightly	Some bugs merge and are caught at night, not pre-merge	Critical-path tests must still run on every PR

← Back to all Labs

From 30-minute local suite to 3-minute CI signal

The CI triangle: speed, cost, confidence. Pick two — the third pushes back. Every pattern below trades one for another. Knowing which you're trading away is the skill.

# .github/workflows/e2e.yml jobs: e2e: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: { node-version: '22', cache: 'pnpm' } - run: pnpm install --frozen-lockfile - run: pnpm playwright install --with-deps chromium # Parallel workers — tune to CPU count of the runner - run: pnpm playwright test --workers=4 # Playwright config // playwright.config.ts export default defineConfig({ workers: process.env.CI ? 4 : '50%', // 50% of CPUs locally fullyParallel: true, });

# .github/workflows/e2e-sharded.yml jobs: e2e: strategy: fail-fast: false matrix: shard: [1, 2, 3, 4] # 4 shards runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 - run: pnpm install --frozen-lockfile - run: pnpm playwright install --with-deps chromium # Playwright built-in sharding - run: pnpm playwright test --shard=${{ matrix.shard }}/4 - name: Upload blob report if: always() uses: actions/upload-artifact@v4 with: name: blob-report-${{ matrix.shard }} path: blob-report retention-days: 7 # Merge reports after all shards complete merge-reports: needs: e2e if: always() runs-on: ubuntu-latest steps: - uses: actions/download-artifact@v4 with: { path: all-blobs, pattern: blob-report-* } - run: pnpm playwright merge-reports --reporter html all-blobs

// playwright.config.ts export default defineConfig({ retries: process.env.CI ? 2 : 0, // 2 retries on CI, 0 locally // Separate project for the retry pass — different timeouts projects: [ { name: 'chromium', use: { ...devices['Desktop Chrome'] } }, ], }); # Or more explicit: retry strategy per-failure-type - name: Run tests (first pass) run: pnpm playwright test || echo "first-pass failed, will retry" continue-on-error: true - name: Run failed tests (retry pass) run: pnpm playwright test --last-failed --workers=2

# .github/workflows/e2e.yml (reporting bits) - run: pnpm playwright test continue-on-error: true # Always upload artifacts, even on success (for trend analysis) - name: Upload trace if: always() uses: actions/upload-artifact@v4 with: name: playwright-trace path: test-results/ retention-days: 30 - name: Upload HTML report if: always() uses: actions/upload-artifact@v4 with: name: playwright-report path: playwright-report/ retention-days: 30 - name: Publish to Allure / Slack / Datadog if: failure() run: | ./scripts/upload-to-allure.sh curl -X POST "$SLACK_WEBHOOK" \ -d "{\"text\":\"🔴 E2E failed on ${{ github.ref }}\"}"

# .github/workflows/flake-detection.yml on: schedule: - cron: '0 3 * * *' # 3 AM daily workflow_dispatch: jobs: flake-scan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: pnpm install --frozen-lockfile # Run each test 10 times, record pass/fail - run: pnpm playwright test --repeat-each=10 --max-failures=0 # Parse results, compute per-test failure rate - run: ./scripts/compute-flake-rate.sh # Open or update GitHub Issues for flakes > 5% - run: ./scripts/file-flake-tickets.sh # Playwright quarantine — tag known flakes, exclude from blocking runs test('suspect test', { tag: ['@quarantined', '@flaky'] }, async () => {}); # On PR: skip quarantined tests pnpm playwright test --grep-invert "@quarantined"

Economics cheat-sheet

For each axis you want to push, here is the primary lever, what it costs, and where it stops working:

Axis	Lever	Cost	Ceiling
Speed	Parallelization + Sharding	Linear compute cost (N× workers = N× machine-minutes)	Amdahl's law — setup/install phases do not parallelize
Confidence	No retries + flake detection cron	Slower time-to-merge while flakes surface	Zero-retry only works if the suite is actually stable
Cost	Run fewer tests on PRs, full suite nightly	Some bugs merge and are caught at night, not pre-merge	Critical-path tests must still run on every PR