Practically everyone that has dealt with testing web applications has been confronted with flaky tests. These are the tests that can fail, but pass once retried and without any code changes. It's one of the most challenging obstacles with testing, and tremendous engineering effort can be invested (and wasted) to resolve these issues.
Cypress was architected with flaky tests in mind, providing users with the ability to automatically retry assertions, await for visibility and actionability of DOM elements, as well as its deterministic command execution model. To further enhance Cypress's ability to handle flake, we recently released test retries in Cypress v5.0.0, which enables automatic retrying of failing tests to minimize the impact of flake on CI/CD pipelines.
Flake Detection
Empowered with test retrying capabilities, we're excited to introduce test flake detection for the Cypress Dashboard, which will enable users to track and monitor flaky test runs in their CI. The first step in combating flake is to triage and assess the severity of flaky tests so you can appropriately prioritize the work needed to fix them. The new "Flaky tests" Analytics page provides the exact context to do just that, let's go ahead and take a look.
The flaky tests analytics pages provides a bird-eye-view on the state of flake for your project. Its main page visualizes the number of flaky test cases over time, lists a filterable log of all flaky tests, and provides you with an overall flakiness level.
Each flaky test case is given a severity level which is currently determined by the frequency of flake or flake rate of the test. You can leverage the severity level to help prioritize the work needed to resolve flake issues.
Once you've identified the flaky test case to focus on, the Dashboard will provide you with additional details about the flaky situation by showing the latest flaky test runs, most common errors for the test case, and information about how the flake and failure rate of the test case has changed over time.
To see the full capabilities of this flaky tests analytics page, please check out the Flaky Test Management docs.
Flake Alerting
To enhance flake detection and to further assist easy tracking of flaky test occurrences, the Cypress Dashboard can also alert you about flake right within your GitHub PR workflow, as well as a Slack channel of your choice. Let's take a look.
The Dashboard GitHub integration will highlight flaky tests via PR comments and status checks, making sure you are always on top of flake before you merge in changes:
Outside of your GitHub workflow, you can leverage Slack alerts via the Dashboard Slack integration to stay on top of flake issues:
To learn more about test flake alerting, please check out the Flaky Test Management docs.
One of the concerns some of our users had with the advent of test retries was that real failures in CI would be hidden and potentially never addressed. This is a valid concern and one that the Cypress Dashboard resolves by providing the best of both worlds. By enabling test retries, you can keep your CI pipeline moving smoothly, while the Dashboard will immediately highlight occurrences of flake and the full context around every failure caused during retry attempts.
Test retries, flake test detection and alerting, are just the beginning of our larger effort to combat flake. We're excited for everyone to start leveraging these features, and we'd appreciate your feedback along the way.
Happy Testing!