Since we've opened up free trials of Cypress Accessibility in Cypress Cloud, one question we've been answering is: how is this different to just running accessibility checks using an open source package like cypress-axe?
At a high level, the approach taken in Cypress Accessibility helps teams stop dealing with some of the slower and more manual aspects of collecting, triaging, and communicating about accessibility problems, so they get back to focusing on their primary tasks faster.
Customers have reported:
- faster functional test execution by removing existing in-test usage of accessibility scanners
- improved external accessibility audit results because more issues are detected and fixed by their teams in between audits
- a more effective debugging experience due to full-page live DOM snapshots and Test Replays in the artifacts, instead of code snippets or static images
For teams where accessibility is a priority, leveraging your existing automated test pipeline to scan accessibility issues is an obvious win, though it can be challenging to scale the approach up if you are depending on checks to run within your tests themselves.
In-test accessibility checks
If you have previously worked with any kind of accessibility checks in your automation pipeline, you've been working with the "in-test" variety. In-test accessibility checks usually involve running a specific command each time you want to check accessibility, or building the checks into the test execution in other ways, such as scanning the initial state of the page, or even watching for DOM changes.
In a typical setup, your in-test checks will look something like this:
This kind of accessibility automation can lead to disappointing results at scale, despite the time and effort spent on setting up accessibility checks. In some cases, adoption fails completely.
These are some of the challenges we've heard about with in-test checks:
- Often just the first page load or step in a workflow is tested, so important issues are missed that show up after interactions
- “Testing everything all the time”, helps capture missing workflow steps, but can lead to pipeline flakiness and increased testing time and resource consumption
- This tradeoff requires training all testers to know about accessibility testing to make sure they test enough states while managing performance
- Results are noisy and disorganized – you can fail a test but it's difficult to get useful artifacts out or do analysis beyond the single moment in time where something fails
- Preventable issues still get to production without anybody being aware of them
Despite these challenges, many companies have built up custom workflows or created dedicated accessibility scanning test suites that are maintained specifically to run these kinds of checks, isolated from the main test suites.
What is out-of-test accessibility automation?
In Cypress Accessibility, no accessibility checks take place during test execution. Instead, we process the Test Replay data that is already part of a standard Cypress Cloud recording. This is possible because Cypress Accessibility is baked into the processing pipelines of Cypress Cloud itself, instead of running on your CI machines. We use a combination of rules from the Axe-Core® library and custom logic implemented by Cypress to create reports.
With Cypress Accessibility enabled, your tests will look and behave exactly the same as before, but without any explicit accessibility assertions:
This means there's never a tradeoff between how many times you run an accessibility check and the overall performance of your functional test pipeline. The out-of-test execution then enables a lot of benefits that would be too slow (or just impossible) to achieve in a standard test context:
- No setup or installation
- No training on when to run accessibility checks
- Inspectable, debuggable full-page HTML & CSS snapshots for every issue
- Page/component reports with rule breakdowns from every run - not individual test failures
- Results in Cypress Cloud, GitHub, Slack… + APIs for custom workflows and merge-blocking
- Built-in analytics and tracking across all projects
- On-demand diffs between any two reports to see only the net-new problems with a specific PR, and what's been fixed
- No impact on test performance or flakiness
- Easy-to-share reports with complete reproduction steps and visual presentation of issues, all tied back to specific test runs, branches, and builds
- New: additional accessibility feedback based on how your tests actually use the elements in your page, not just on their static properties
Because we avoid fighting for resources with your functional tests, we are able to run hundreds or thousands of accessibility scans, covering every step of every user journey, and then merge the data together into an actionable, deduplicated list of issues.
How to get started
There's no effort required on your part. Results are available immediately when you start your trial, bringing you to full adoption on day one, with a consistent set of standards tested on all your Cypress projects. Connect with us to set up a trial or get a personalized demo and walkthrough with reports from your own application.
Note: AXE-CORE® is a trademark of Deque Systems, Inc. in the US and other countries.