Playwright GitHub Status Checks

Configure GitHub status checks from Playwright results, set quality gate thresholds, and make the TestDino check required to merge.

GitHub CI Checks are automated quality gates that block merges when Playwright test results do not meet your configured rules. Define pass rate thresholds, mandatory tags, and flaky test handling for each environment. TestDino evaluates every run and posts a clear pass or fail signal on the PR.

Quick Reference

Setting	Default	Purpose
Pass Rate	90%	Minimum percentage of tests that must pass
Flaky Handling	Neutral	How flaky tests affect the check (Strict or Neutral)
Mandatory Tags	None	Tags that must pass regardless of the overall rate
Environment Overrides	None	Custom rules per branch environment

What are GitHub CI Checks?

GitHub CI Checks are automated quality gates that run on your pull requests and commits. TestDino shows a pass or fail signal in GitHub based on the test rules you set. If a required check fails, GitHub blocks the merge.

• When your tests finish, TestDino compares the run against your quality gate settings.
• TestDino posts a green check (passed) or a red check (failed) on the PR or commit.
• The result appears in GitHub, so nobody has to open the dashboard to see it.

TestDino posts these checks through the GitHub App. If you have not connected it yet, start with the GitHub Integration.

Why do CI checks matter?

• Stop unstable or failing code from being merged
• Enforce strict rules for critical branches (like main)
• Use different rules for PROD, STAGE, and DEV
• See failures instantly inside GitHub
• Combine real test signals with GitHub’s protection rules

Quality Gate Settings

These rules determine whether TestDino marks a check as pass or fail.

Default Settings

By default, the Pass % and Flaky settings apply to all branches. You can override them later for specific environments such as PROD, STAGE, or DEV.

Pass Rate

Minimum percentage of tests that must pass for the check to succeed.

• Range: 0-100%
• Default: 90%
• Example: If set to 90%, at least 90% of your tests must pass for the check to be green

Mandatory Tags

All tests carrying these tags must pass. If even one fails, the whole check fails, no matter how high the overall pass rate is.

• Use the @ prefix (for example @critical, @payment, @auth)
• Useful for login, payments, security, or any flow that you cannot risk breaking

For example, if you set @critical as mandatory and one critical test fails, the check is red even when everything else passes.

Tags come from your Playwright test files. Learn how to add them in Test Annotations.

Flaky Handling

How flaky tests are treated:

• Strict: flaky tests count as failures. Use this for production branches where stability is critical.
• Neutral: flaky tests are excluded from the pass rate calculation (default). Use this for development branches to focus on actual failures.

To see how TestDino decides that a test is flaky, read Flaky Tests.

Environment Overrides

You can create different quality gates for different environments.

How does it work?

1. Set up branch environments in your project (such as Production, Staging, and Development).
2. Each environment appears as a row in the CI Checks settings table.
3. For each environment, set a custom Pass %, Flaky Handling, and whether Tags apply.
4. If you do not override a setting, the default applies.

Environments come from your branch mappings. Set those up in Environment Mapping.

Example Configuration

In this example the team lowered its own default to 80%, then tightened the rules for Production.

Environment	Pass Rate	Flaky Handling	What it means
Default	80%	Neutral	Every branch without its own row follows this rule
Development	70%	Neutral	70% of tests must pass, flaky tests are ignored
Staging	85%	Neutral	85% must pass, suitable before a deploy
Production	95%	Strict	95% must pass and no flaky failures are allowed

Understanding Check Results

Each GitHub check shows green or red based on your rules.

Passed: Green Check

Your code meets all quality gate requirements:

• Pass rate meets or exceeds the threshold
• All mandatory tag tests passed

You can merge your PR.

Failed: Red Check

Your code does not meet quality gate requirements because:

• Pass rate is below the threshold, or
• One or more mandatory tag tests failed

Fix the failing tests before merging.

Check Details

Click Details on the GitHub check to open the full report. It opens on the test results table, where every row links to the matching view in your TestDino dashboard.

The report has up to 3 sections. The last 2 appear only when the environment has mandatory tags configured.

Section	Shows	Effect on the check
Test results table	Pass, fail, and flaky counts with links to the dashboard	Drives the pass rate verdict
Mandatory tag analysis	Which mandatory tags passed and which failed	A failed tag fails the check
Tags not found	Mandatory tags configured here but absent from your tests	None, they are skipped

A tag under “Tags not found” usually means a typo, or a tag that was renamed in the test files. The check still passes, so fix the configuration or that coverage stays unenforced.

Making CI Checks Required

This step tells GitHub which checks must pass before a pull request can be merged.

1. Go to Repository Settings → Rulesets
2. Create or edit a rule
3. Enable Require status checks to pass
4. Click Add checks
5. Select TestDino
6. Set target branches (for example, main)
7. Save the rule

GitHub now stops merges unless the TestDino CI Check is green.

Common Scenarios

High Pass Rate, but the Check Failed

Situation: 95% of tests passed, but the check is still red.

Reason: A mandatory tag test failed.

Solution: Fix the mandatory tag test first. Mandatory tags override the pass rate completely.

Flaky Tests Causing Failures

Situation: The check fails because flaky tests are counted as failures.

Solution: Switch the environment to Neutral flaky handling, fix the flaky tests, or use Strict only on stable branches like Production.

Different Rule Requirements by Branch

Situation: You want strict rules for Production but lighter rules for Development.

Solution: Use Environment Overrides. Set Development to a 70% pass rate with Neutral flaky handling, and Production to 95% with Strict.

Best Practices

Area	Recommendation
Starting thresholds	Begin at an 80-90% pass rate with Neutral flaky handling
Mandatory tags	Reserve them for critical user flows, security features, and data integrity operations. If everything is mandatory, nothing is
Environment rules	Production 95% and Strict, Staging 85-90%, Development 70-80% and Neutral
Failed checks	Review the failed tests in the check details, then open the dashboard for the full error context
Test stability	Fix flaky tests rather than working around them, because they point at real stability issues

Troubleshooting

Symptom	Likely causes	Fix
Check not appearing on the PR	GitHub Checks are not enabled; no commit SHA in the test run metadata; repository mismatch between the TestDino project and the GitHub connection	Verify the GitHub connection and enable CI Checks
Check always failing	Pass rate set too high; mandatory tags misconfigured; flaky handling too strict	Review your quality gate settings and adjust the thresholds
Mandatory tags not working	Tag names do not match (they are case-sensitive); missing @ prefix; the tests do not carry the tags	Check tag spelling and case, then review “Tags not found” in the check details. TestDino adds the @ prefix when it is missing, but confirm the tag exists in your tests
Environment override not applied	The branch pattern does not match; the environment is not configured in project settings	Verify the branch environment mapping in project settings

Related

GitHub Integration

Install the GitHub App and connect a repository

Environment Mapping

Map branches to the environments used by overrides

Flaky Tests

How TestDino classifies a flaky result

Test Annotations

Add the tags that mandatory tags rely on