TurnSignal

TurnSignal for Playwright: what it is, who it's for, what it costs

In short

TurnSignal is a hosted test-results service for Playwright Test (the Node.js test runner). A reporter in your CI sends results as tests run; TurnSignal keeps each test's history across runs and branches and tells you which failures are new (likely caused by the change), already failing on the default branch, flaky, or missing because a runner crashed.

Who it is for

  • Built for: Teams running Playwright Test in CI who need history across runs: which failures are new, which tests are flaky and how often, and which results went missing. Teams whose tests are written or fixed by AI agents can also review those changes with it.
  • Not needed for: A small suite where one run's HTML report answers every question, or single-run reporting only.
  • Not a test generator: TurnSignal records and reviews tests; it does not write them.

What it tells you after every run

Every run opens with a plain verdict. Each test gets a label with the reason behind it:

  • Failed: The last attempt of the test failed.
  • New failure: Fails here but passed in its latest result on the default branch: most likely caused by this change.
  • Also failing on main: Its latest result on the default branch also failed: not caused by this change.
  • Known flaky: It was flaky in at least one of its previous 20 results (any branch). Check before blaming the change.
  • No history: No default-branch result to compare against yet.
  • Quarantined: Someone marked the test as known broken, with an owner and an end date. Its failures are not counted as new; CI still fails.
  • No new failures: On the runs list: the run failed, but only in tests already failing on the default branch, known to be flaky or quarantined. Nothing new to fix in this change.
  • Failing on main: Tests failing on the latest default-branch run, with how many runs in a row. Every branch shows them as "Also failing on main" until they are fixed.
  • Flaky: Failed at least once, then passed on a retry or when the CI job was re-run.
  • Missing: Planned for the run but never reported: the runner crashed, was killed (often out of memory), or the run stopped first. The reason is shown.
  • Incomplete: A run where some tests never reported a result, or a machine stopped without finishing.

Key facts

  • Setup: npm i -D turnsignal, add ['turnsignal'] to reporter in playwright.config.ts next to existing reporters, and set the project token as TURNSIGNAL_TOKEN in CI.
  • The reporter never changes Playwright's exit code. Results are written to disk in CI first; npx turnsignal upload sends anything that could not be delivered.
  • CI: GitHub Actions, GitLab CI, Jenkins, CircleCI, Azure Pipelines, Buildkite, Bitbucket Pipelines and Argo Workflows are detected automatically (shards merge into one run); any other CI or scheduler works with TURNSIGNAL_RUN_KEY.
  • The pull request comment (npx turnsignal comment) works on GitHub only for now. Slack, Discord and signed webhook alerts work everywhere.
  • Also included: Playwright trace viewer, videos and screenshots in the browser; flaky tests over 7-90 days; report-only quarantine; read-only API and an MCP server (npx turnsignal mcp).
  • npx turnsignal governance reviews test changes in a pull request (weakened assertions, skipped tests, added waits) and npx turnsignal analyze flags weak tests; both run locally and upload nothing.
  • Secrets in errors and traces are redacted by pattern in CI before upload, and errors are checked again on the server; screenshots and videos can't be redacted and can be turned off.
  • Argo Workflows is detected automatically. Kubernetes Jobs (Indexed Jobs, CronJobs, Argo CD sync hooks): set TURNSIGNAL_RUN_KEY from the pod label batch.kubernetes.io/controller-uid so all shards merge into one run, and start tests with exec npx turnsignal run -- npx playwright test so results are uploaded before the pod exits. Images without the .git folder set TURNSIGNAL_BRANCH (and TURNSIGNAL_COMMIT); without a branch a run still shows missing and flaky tests but is not compared with the default branch.
  • Scheduled and nightly runs on the default branch get the same labels as any run: new since the previous run there, already failing, known flaky, missing. Only the pull request comparison and comment need a pull request.
  • Results are sent while the tests run (and saved to disk in CI first), not only at the end of the run.
  • Not for regulated data today: TurnSignal is a hosted beta with no BAA or HIPAA terms and no self-hosted version. Test titles, file paths and error messages are always sent (credentials are redacted; names, member ids and other personal data are not, unless they match your own TURNSIGNAL_REDACT patterns). Use it only with tests that run on synthetic data.
  • Not supported today: Playwright for Python, Java or .NET; Cypress.

Setup in three steps

  1. Install the reporter: npm i -D turnsignal.
  2. Add it next to your reporters in playwright.config.ts: reporter: [['list'], ['turnsignal']].
  3. Sign in, create a project, and set its token as the TURNSIGNAL_TOKEN secret in CI.

The docs have the setup for GitHub Actions, GitLab CI and other CI. Is it safe to add? See security and data.

Free during the public beta

TurnSignal is in public beta: every feature is free while we work with the first teams. No credit card.

  • Each organization can record up to 3,000 runs and 300,000 test results a month, in up to 5 projects. Need more? email hello@turnsignal.ai and we'll raise it.
  • Organization owners are warned at 80% and 100% of the limit: in the dashboard, and by email when we have their address. Over the limit, runs that already started still finish; new runs aren't recorded until the 1st of the next month (UTC), and your CI passes or fails exactly as it would without TurnSignal.
  • Paid plans will come later. We'll announce them at least 30 days before anything changes. There's no card on file, so nothing can be charged automatically: you decide whether to move to a paid plan.
  • The plans we intend to launch, and the founding-team offer (12 months free, then half price for life), are on the pricing page. Open-source projects stay free.
  • Beta means some things will change as we learn from you. See the beta terms.

Questions people ask

Does TurnSignal replace the Playwright HTML report?

No. It runs as one more reporter next to the ones you have; the HTML report keeps working.

When do the labels start being useful?

From the first run on your default branch. Until then a failure is labelled No history, because there is nothing to compare it with. Known flaky needs earlier results of the same test: it looks at that test's previous 20 results.

Who can see our results?

Only members of your organization. Owners manage projects, tokens and members; members can read. Other organizations cannot see your projects.

What will it cost after the beta?

The planned plans are Free, Team at $44 a month and Business at $74 a month, flat per organization. Nothing is charged today; see pricing.

Can AI coding agents read TurnSignal results?

Yes. npx turnsignal mcp starts a read-only MCP server, so an agent can ask which failures are new, which tests are flaky and what a test's history looks like.

Try it on your next run

Free during the beta. No credit card, no repository access.

Get started free Read the docs