Skip to main content

Fixing tests

A test failed, broke, or flakes. Diagnose from evidence, not guesses given by Claude based off of code instead of a live system. Never paper over flakiness with retries or sleeps.

1. Read the trace, not the tea leaves​

Every failure records a Playwright trace. Open it and step through what the browser actually did:

playwright show-trace test-results/<path>/trace.zip

Or the hosted viewer: https://plrview.int.octostar.com (Tailscale). On CI, trace-viewer deep links are baked into each test's Allure attachments.

2. Tail the per-test log​

Logs land in results/logs/<test>_<timestamp>_<env>_env.log — one per test. Tailing it while a test runs is also how you catch an LLM going down a rabbit hole: re-trying the same broken selector, inventing fixtures, looping. If you see that, stop it and re-scope — don't let it churn.

3. Diagnose flakiness properly​

Flaky = passes sometimes, fails sometimes. The cause is almost always a missing wait condition: the test asserts before the app is ready.

  • ✅ Wait on the real signal — an element state, a readiness flag, a count settling. Example: PR #279 gated a linkchart fixture on edge-discovery readiness instead of a sleep.
  • ❌ Do not add time.sleep(...) or blind retries to "stabilise" a test. That hides the bug and slows the suite. Ask before adding any retry — there's almost always a real condition to wait on instead.

4. Is it the test, the product, or infra?​

  • Test bug — wrong/brittle selector, missing wait, bad assumption about seed data.
  • Product bug — the test is right; the app regressed. That's a find, not a nuisance; file it.
  • Infra — Tailscale, health check, environment. Reproduce locally to tell them apart.
  • Github Action outage - Github is regularly having issues that cause test failures, we now surface them to the qa channel Slack message