Playwright Azure DevOps Pipeline Setup

Stream Playwright results from Azure DevOps Pipelines to TestDino live, with sharded jobs grouped into one run.

Set up an Azure DevOps pipeline that streams Playwright test results to TestDino during the run and view aggregated analytics, failure analysis, and flaky test detection on your dashboard.

Prerequisites

• A TestDino account with a project created
• A TestDino API key (Generate API Keys)
• An Azure DevOps project with pipelines enabled
• A Playwright test project (or clone the TestDino Example Repository to get started)
• @testdino/playwright installed and configured in playwright.config.ts|js|mjs:

npm install @testdino/playwright

playwright.config.ts

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['@testdino/playwright', {
      token: process.env.TESTDINO_TOKEN,
      serverUrl: "https://reporter.testdino.com",
    }],
  ],
});

Set Up Your API Key

Store your TestDino API key as a secret pipeline variable so it is available to your pipeline without exposing it in logs or config files.

1. Open your Azure DevOps pipeline
2. Click Edit on the pipeline
3. Click Variables
4. Click New variable
5. Set the name to TESTDINO_TOKEN
6. Paste your TestDino API key as the value
7. Check Keep this value secret
8. Save the pipeline

Warning Never commit your API key directly in pipeline files. Always use secret variables. Secret variables are not exposed in pipeline logs.

Basic Pipeline Config

For a simple setup without sharding, run your tests with TESTDINO_TOKEN mapped into the script. Results stream to TestDino during the run, so no separate upload step is needed.

azure-pipelines.yml: Basic pipeline

azure-pipelines.yml

trigger:
  branches:
    include:
      - main

pr:
  branches:
    include:
      - main

variables:
  CI: "true"

pool:
  vmImage: ubuntu-latest

steps:
  - checkout: self

  - task: UseNode@1
    inputs:
      version: "20.x"
    displayName: Install Node.js

  - script: npm ci
    displayName: Install dependencies

  - script: npx playwright install --with-deps
    displayName: Install Playwright browsers

  - script: npx playwright test
    displayName: Run tests with TestDino
    env:
      TESTDINO_TOKEN: $(TESTDINO_TOKEN)
      TESTDINO_SERVER_URL: https://reporter.testdino.com

Tip The env block maps the secret variable to an environment variable @testdino/playwright reads. Results stream live even when tests fail, so failures land on the dashboard without a condition: always() upload step.

Sharded Test Runs

For larger test suites, the Azure DevOps matrix strategy splits tests across multiple jobs. Each shard streams its own results, and TestDino groups them into one run when they share a ci-run-id.

How it works

1. Azure DevOps runs Playwright across 4 shards using a matrix strategy
2. Each shard streams its results to TestDino during the run
3. Every shard passes the same --ci-run-id so TestDino merges them into a single run
4. No merge or upload stage runs after the shards finish

Full sharded config

azure-pipelines.yml: Sharded pipeline

azure-pipelines.yml

trigger:
  branches:
    include:
      - main

pr:
  branches:
    include:
      - main

variables:
  CI: "true"

pool:
  vmImage: ubuntu-latest

jobs:
  - job: Playwright
    strategy:
      matrix:
        shard1:
          SHARD: 1/4
        shard2:
          SHARD: 2/4
        shard3:
          SHARD: 3/4
        shard4:
          SHARD: 4/4
    steps:
      - checkout: self

      - task: UseNode@1
        inputs:
          version: "20.x"
        displayName: Install Node.js

      - script: npm ci
        displayName: Install dependencies

      - script: npx playwright install --with-deps
        displayName: Install Playwright browsers

      - script: npx tdpw test --ci-run-id "$(Build.BuildId)" -- --shard=$(SHARD)
        displayName: Run shard $(SHARD) with TestDino
        env:
          TESTDINO_TOKEN: $(TESTDINO_TOKEN)
          TESTDINO_SERVER_URL: https://reporter.testdino.com

Info Every shard passes the same --ci-run-id "$(Build.BuildId)", so TestDino merges them into a single run. The env block maps the secret variable into each shard’s script.

Key details in the sharded config

Config Block	What It Does
strategy: matrix	Defines 4 shards with SHARD: 1/4 through 4/4
--shard=$(SHARD)	Passes the shard value directly to Playwright (Azure DevOps uses 1-based values)
--ci-run-id "$(Build.BuildId)"	Groups all shards into one run on the dashboard
env: TESTDINO_TOKEN	Maps the secret variable to an environment variable @testdino/playwright reads

Set the ciRunId reporter option in playwright.config to $(Build.BuildId) if you prefer running npx playwright test directly instead of npx tdpw test.

Pipeline execution

After the pipeline runs, Azure DevOps shows all shard jobs in the pipeline view. Each shard streams its results, and TestDino merges them into one run.

Results in TestDino

The test run appears in your TestDino dashboard with full failure details, flaky detection, and trend data as tests complete.

Troubleshooting

Results not appearing on the dashboard

• Confirm @testdino/playwright is in your playwright.config reporter array and TESTDINO_TOKEN is mapped into the script via the env block
• Results stream during the run, so no separate upload step is required

Sharded runs show up as separate runs

• Pass the same --ci-run-id "$(Build.BuildId)" (or ciRunId reporter option) to every shard
• Different values create one run per shard

TESTDINO_TOKEN not available in script

• Secret variables in Azure DevOps are not automatically available as environment variables. You must map them explicitly using the env block in the script step
• Verify the variable name matches exactly: TESTDINO_TOKEN: $(TESTDINO_TOKEN)

Related

CI Optimization

Reduce CI time with smart reruns

Environment Mapping

Route test results to Dev, Staging, or Production by branch

Integrations

Connect Slack, Jira, Linear, Asana, and more

Azure DevOps Extension

View test runs inside Azure DevOps