TurnSignal

Help

Everything you need to send Playwright results to TurnSignal and read them.

Get started

  1. Sign in and create a project

    One project per Playwright test suite. You'll get a token; it's shown once.

    Sign in

  2. Install the reporter

    npm i -D turnsignal
  3. Add it to playwright.config.ts

    reporter: [['list'], ['turnsignal']],
  4. Give CI the token

    Store it as a secret named TURNSIGNAL_TOKEN. Results appear on the project's runs page while tests are still running.

CI setup

Add the secret under Settings → Secrets and variables → Actions. Then:

- name: Run Playwright
  run: npx playwright test
  env:
    TURNSIGNAL_TOKEN: ${{ secrets.TURNSIGNAL_TOKEN }}

# Recovers results if a runner crashed or the API was unreachable.
- name: Upload any undelivered results
  if: always()
  uses: sekharsdet/turnsignal-action@v1
  with:
    token: ${{ secrets.TURNSIGNAL_TOKEN }}

Optional PR comment: a final job (needs your test job, if: always(), pull-requests: write) with the same action and command: comment posts the verdict on the pull request. Prefer plain commands? npx turnsignal upload and npx turnsignal comment do the same. Action details

Optional test-change review: a job running npx turnsignal governance flags pull requests that weaken an assertion, change an expected value or disable a test (e.g. an AI "fix"). It needs no token and uploads nothing. How

Add TURNSIGNAL_TOKEN under Settings → CI/CD → Variables (masked). Then:

e2e:
  parallel: 4                     # optional sharding
  script:
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  after_script:
    - npx turnsignal upload       # runs even when tests fail or the job crashes

The PR comment (npx turnsignal comment) works on GitHub only for now; on GitLab, find each pipeline's run on this project's Runs page.

Jenkins, CircleCI, Azure Pipelines, Buildkite, Bitbucket Pipelines and Argo Workflows are detected automatically, so all shards of a pipeline merge into one run. On any other CI, and for Kubernetes Jobs (recipe), also set TURNSIGNAL_RUN_KEY to a value shared by all shards of one pipeline.

export TURNSIGNAL_TOKEN=<from your CI secret store>
npx playwright test
npx turnsignal upload             # run this step even when tests fail

The PR comment (npx turnsignal comment) works on GitHub only for now.

Reading a run

  1. Read the verdict

    The first lines answer "did my change break something?". Click a line to jump to the tests behind it.

  2. Triage in "Needs attention"

    Every failed, missing and flaky test on one screen, in the order to look at them: new failures first. Filter by label or Playwright project. "Same error ×N" means several tests failed with the same message, often from one cause.

  3. Open a test

    You get the error, the code where it failed, its earlier results (new, or failing again?), Open trace and Watch video, View code at this commit (GitHub and GitLab) and the command that runs just this test locally.

While a run is in progress the page updates every 10 seconds and keeps your scroll position, open sections and filter.

What the labels mean

FailedThe last attempt of the test failed.
New failureFails here but passed in its latest result on the default branch: most likely caused by this change.
Also failing on mainIts latest result on the default branch also failed: not caused by this change.
Known flakyIt was flaky in at least one of its previous 20 results (any branch). Check before blaming the change.
No historyNo default-branch result to compare against yet.
QuarantinedSomeone marked the test as known broken, with an owner and an end date. Its failures are not counted as new; CI still fails.
No new failuresOn the runs list: the run failed, but only in tests already failing on the default branch, known to be flaky or quarantined. Nothing new to fix in this change.
Failing on mainTests failing on the latest default-branch run, with how many runs in a row. Every branch shows them as "Also failing on main" until they are fixed.
FlakyFailed at least once, then passed on a retry or when the CI job was re-run.
MissingPlanned for the run but never reported: the runner crashed, was killed (often out of memory), or the run stopped first. The reason is shown.
IncompleteA run where some tests never reported a result, or a machine stopped without finishing.

Screenshots, videos and traces

Playwright decides what is recorded; TurnSignal (reporter 0.3.0+) uploads it and your project setting decides what is kept: failed and flaky tests by default (30 days), or all tests (passed ones 7 days), or nothing. Open a failed test on the run page to watch the video or open the trace in the hosted trace viewer.

use: {
  trace: 'retain-on-failure',
  video: 'retain-on-failure',
  screenshot: 'only-on-failure',
},
  • Secrets: traces are redacted in your CI before upload (authorization headers, cookies, secret values) and don't include your test source. Screenshots and videos can't be redacted: choose what to send with TURNSIGNAL_ARTIFACTS=trace,screenshot (or none).
  • CI time: uploads run in the background; at the end the reporter waits at most 30 s (TURNSIGNAL_ARTIFACTS_TIMEOUT_MS). Anything left is sent by npx turnsignal upload.

Containers and Kubernetes

Results are sent while the tests run. When a pod or container stops, its disk goes with it, so wrap the test command: whatever is still waiting is uploaded before it exits, and the exit code is always your test command's.

npx turnsignal run -- npx playwright test

Argo Workflows is detected automatically: every pod of one workflow joins the same run.

Kubernetes Jobs (an Indexed Job, a CronJob, an Argo CD sync hook) aren't detected as CI, so give every pod of one Job the same TURNSIGNAL_RUN_KEY; otherwise each pod shows up as its own run and the missing-test check can't see the other shards (the reporter warns about this). Use the Job's UID: it is new for every CronJob run and every Argo CD sync, even when a hook Job keeps its name. Run the command with exec so the stop signal reaches TurnSignal, and leave time to upload when a pod is stopped:

spec:
  completionMode: Indexed
  completions: 5
  parallelism: 5
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: e2e
          env:
            - name: TURNSIGNAL_RUN_KEY
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['batch.kubernetes.io/controller-uid']
            - name: TURNSIGNAL_BRANCH
              value: main
          command: ["sh", "-c", "exec npx turnsignal run -- npx playwright test --shard=$((JOB_COMPLETION_INDEX + 1))/5"]

Branch and commit come from the CI or the git checkout in the pod. An image without the .git folder needs TURNSIGNAL_BRANCH (and optionally TURNSIGNAL_COMMIT); without a branch a run still shows missing and flaky tests but is not compared with your default branch. npx turnsignal doctor in the pod shows the run key and branch it will use.

Compared with your default branch

Every finished run is compared with the latest finished run on the default branch: failures are labelled new or already failing there, and the run lists tests that were fixed (failed there, pass here) and tests that got much slower (at least 50% and 3 s slower than the test's usual time: the median of its last 10 passing runs on the default branch, at least 3 needed). The same lists appear in the PR comment and the API.

Scheduled and nightly runs on the default branch are compared with the previous run there, so they get the same labels; only the PR comment needs a pull request (GitHub only for now).

Quarantine a known-broken test

When a test is broken or flaky and someone is on it, quarantine it from its history page (or from the Flaky tests page): say why, who owns it and when the quarantine ends (default 14 days, at most 90). Any member of the organization can quarantine, renew or remove; every change is in the audit log.

  • What changes: its failures are labelled "Quarantined" and are not counted as new failures in the verdict, the runs list, the PR comment or notifications.
  • What doesn't: Playwright still runs the test and your CI still fails when it fails. TurnSignal never changes test results or exit codes. To stop a test failing CI, use test.fixme() in your code.
  • It ends: after the end date the test counts normally again and shows "Quarantine expired" until someone renews or removes it.
  • API: GET /v1/quarantine lists them; each test in GET /v1/runs/:id carries its quarantine.

Test changes in a pull request

AI agents and "healers" fix failing tests fast, and sometimes the fix makes the test stop checking the bug. npx turnsignal governance (turnsignal 0.6.0+) compares every test changed in a pull request with the base branch and posts one sticky comment: what needs a human review, what is worth a look, and what is safe.

Needs reviewAssertion weakened (e.g. toHaveText → toBeVisible, expect → expect.soft, an assertion removed), expected value changed, test disabled or deleted.
Worth a lookSleep, longer timeout, force: true or test.slow() added; snapshot baseline updated.
InformationNew tests (with static checks such as "no assertion"), other step changes.
SafeOnly a locator changed, or the test was renamed with the same steps and assertions.

Each change says who made it when git knows: a person, an agent, a bot, or AI-assisted (a Co-authored-by trailer such as Claude or Copilot). It runs with git on your CI machine, needs no TurnSignal token, and uploads nothing.

test-changes:
  if: github.event_name == 'pull_request'
  runs-on: ubuntu-latest
  permissions:
    contents: read
    pull-requests: write
  steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0              # needs the base branch history
    - run: npx -y turnsignal governance
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  • Make it a required check: add --strict; the job then fails when a change needs review. Without it the job always passes.
  • Other CI or locally: npx turnsignal governance --base origin/main prints the same report; --json for scripts.
  • Not yet: page objects and fixtures are listed but not classified; snapshot updates are flagged by file.

Slack, Discord and webhooks

In a project's setup page, owners add a Slack incoming webhook, a Discord channel webhook or any HTTPS endpoint. By default you hear about new failures, tests that never reported and anything failing on the default branch; you can choose every failed run or turn it off. JSON webhooks are signed: x-turnsignal-signature: sha256=HMAC(secret, timestamp + "." + body).

AI coding agents (MCP)

Let Claude Code, Cursor or Copilot ask why CI failed (turnsignal 0.4.0+, read-only, uses a project token):

{
  "mcpServers": {
    "turnsignal": {
      "command": "npx",
      "args": [
        "-y",
        "turnsignal",
        "mcp"
      ],
      "env": {
        "TURNSIGNAL_TOKEN": "<your token>"
      }
    }
  }
}

Your data

  • Retention: runs are deleted after 90 days (or your project's setting); screenshots, videos and traces after 30 days (7 for passed tests).
  • Deletion: owners delete a project or a whole organization; anyone deletes their account from the Account page. It's immediate and includes stored files.
  • Export: download your account data from the Account page, and an organization's data from its Members page (JSON).
  • Limits: each organization can record a fixed number of runs and test results per month (shown on the Members page). A run that started always completes; over the limit new runs are not recorded, and your CI is never affected.
  • Details: privacy notice.

Troubleshooting

  • Nothing shows up. Run npx turnsignal doctor in the same environment: it checks the token, the connection, a proxy and results waiting to be sent.
  • The runner crashed or had no network. Results are saved to .turnsignal/ first. npx turnsignal upload in an always-run step sends them later; running it twice is safe.
  • "Invalid character in header content" or doctor shows "Token valid: ✖". The token secret contains an invisible character, often from pasting or piping it on Windows. Set the secret again from the setup page's Copy button.
  • Behind a corporate proxy. HTTPS_PROXY, HTTP_PROXY and NO_PROXY are respected.
  • Shards show up as separate runs. On a CI we don't detect automatically, set TURNSIGNAL_RUN_KEY to one value shared by all shards of a pipeline.
  • Will it break my CI? No. The reporter never changes Playwright's exit code and gives up quickly if our API is unreachable.

Use the API

Read-only JSON API for dashboards, scripts and AI agents. Send the project token as a bearer token; it only ever sees this project.

curl -H "Authorization: Bearer $TURNSIGNAL_TOKEN" https://app.turnsignal.ai/v1/runs?limit=5
GET /v1/whoamiThe project this token belongs to
GET /v1/runs?branch=&page=&limit=Runs, newest first: status, counts, verdict, link
GET /v1/runs/:idFull run report: failures with labels, missing tests, machines
GET /v1/run-by-key?key=The same report by CI run key
GET /v1/tests/:testKey/historyOne test's results across recent runs
GET /v1/flaky?days=14Tests flaky in the last N days, with counts
GET /v1/quarantineQuarantined tests: reason, owner, expiry, active or expired

AI coding agents (MCP)

Let Claude Code, Cursor or Copilot ask why CI failed: add TurnSignal as an MCP server (turnsignal 0.4.0+). Read-only; uses this project's token.

{
  "mcpServers": {
    "turnsignal": {
      "command": "npx",
      "args": [
        "-y",
        "turnsignal",
        "mcp"
      ],
      "env": {
        "TURNSIGNAL_TOKEN": "<your token>"
      }
    }
  }
}

Guides