A 2024 GitHub survey found that nearly all enterprise developers (97%) are using Generative AI coding tools. The majority of companies surveyed actively encourage the use of AI. At the same time, GitClear reports declining code quality metrics, and more frequent revisions of existing code, when code is written with the help of AI assistants.
All of this puts more pressure on the test automation pipeline as the gatekeeper for quality, and the early warning system for when code isn’t ready to release. But the same things that make testing more valuable – the increasing pace and scope of code changes – also make it more challenging than ever to understand.
In complex applications with large test suites, even small changes in the application or the test code can have large effects on the actual confidence provided by your test pipeline.
How UI Coverage can help during AI-powered development cycles
With UI Coverage in Cypress Cloud, you can gain some important insights into your test suite:
- Understand which parts of any given CI build are covered by your automated tests, for any test run, at any moment in time
- Compare across builds to see when new interactive elements have been added that do not have coverage, or when tests were changed in a way that reduced coverage
- Spot redundant test behavior where the same user flows may be tested repeatedly without adding any extra value
We’ve written in the past about the difference between UI Coverage and code coverage and what it can offer teams. We've also hosted a webinar earlier this year on how Test Generation works in UI Coverage, and what we are building next within Studio in the Cypress App.
Here we'll talk about how all of this helps you manage risks related to an accelerated level of AI-driven code changes.
Audit your tests to find important missing coverage
The idea with UI Coverage is not that 100% of your application needs to be interacted with in Cypress, but that 100% of what you intend to test, is actually interacted with. As you audit your test coverage, you’ll find yourself making decisions about what actually matters to test or not, and you’ll be able to ignore items that don’t require testing by the team who owns the Cypress project. Through configuration, you can ignore entire pages, or specific elements and groups of elements and focus on what matters.
This ensures you can be informed about the current state of your coverage at any moment in time and sets you up for success when changes come in.
Note: During the free UI Coverage trials, we’ll often join you on this first pass to kickstart the config process, so you can see the results that best match your own priorities. Our next blog post will go into more details about the kinds of configuration available and how this helps you shape the reports, but it’s worth knowing that you don’t have to go it alone - we’ve helped many teams get up and running.
Add coverage quickly where you need it
UI Coverage Test Generation (currently in Beta) allows you to generate a new test to interact with any element that’s highlighted in your UI Coverage report. The new test will be based on your existing test code practices and conform to the surrounding patterns you already use for things like locating elements and setting up state, letting you get right to work adding the steps and assertions you need.
React to changes in the application
UI Coverage can surface new untested buttons, links or other ways of interacting with the page or submitting information. This level of detailed reporting helps you keep up with the changes in the application to understand if the test coverage that existed before is still adequate. This is possible when you compare UI Coverage reports using Branch Review.
If you are the developer who is also writing Cypress tests, this cross-checking is incredibly useful and may be impossible to get in any other way. If you work in QA, and changes are deployed to a specific environment that you test against, this is a great way to understand the reality of what is changing in the application, which may be different to what was expected based on initial requirements and test plans.
Verify the outcome of a test refactor
Accepting individual suggestions when using AI to write code can lead to duplication and inconsistencies that will need to be cleaned up later. As a test suite grows, you’ll often find chances to refactor and reorganize the testing to avoid repeating the same test code in multiple places.
This kind of consolidation is important for maintainability and consistency reasons, but there is not always a good way to know that the tests are functionally the same before and after a reorganization pass.
Comparing runs with UI Coverage is a fast and efficient way to understand if test refactoring was effective at maintaining the same tested elements before and after, or important coverage was lost in the process.
Detect redundant coverage
In addition to tracking the elements that are not interacted with during your tests, UI Coverage provides traceability for all the interactions that did take place. This allows you to sort elements by interactions to see where repeated, redundant tests are taking place, to optimize and consolidate your testing efforts.
If you are generating parts of your test suit using generic AI-powered code assistants that are not informed about your testing conventions, you can end up with redundant sets of steps that add time and cost to your test execution, without adding any new value. UI Coverage will show you the interaction counts for each element and map them back to the tests that performed the interaction, so you can keep an eye on over-testing.
Measure the impact of skipping or quarantining tests
All code, including AI-generated code, needs to account for the stability and consistency of the application. If this is not considered at the time of generating the code, it’s possible that tests may become flaky, because the application itself is less stable. A typical example of this would be a hidden race condition that reveals itself under the resource constraints in CI. This may also occur on your users’ devices, but it’s unlikely to happen on a more powerful development machine.
When this happens, teams will sometimes skip the newly flaky or failing test for a while, intending to solve the problem later and then reactivate the missing test, so that the pipeline can remain stable. In practice, skipped tests can linger much longer than intended as teams prioritize new work.
Comparing runs before and after skipped tests is a great way to understand what is actually lost when those tests don’t run.This lets you identify what might need more manual testing until the missing coverage is restored, and helps to frame the urgency of restoring those tests. On the other hand, it may also reveal that a certain test wasn’t contributing much and can be safely deleted.
How it works
UI Coverage captures all of the user interactions with your application's UI elements during test runs. These interactions are then mapped to your application's DOM structure, giving you a clear picture of which UI components are being tested.
The reports are highly visible and interactive, which means they work well for a wide audience - if somebody understands your application itself, they will understand the coverage report. It does not require knowing about the code or anything else technical.
Best of all, UI Coverage requires no code changes or instrumentation by your team. Reports are generated in Cypress Cloud in a fully server-side process, using the Test Replay artifacts that are already a part of the standard Cloud recording process. This means there is no performance overhead, no startup effort, and nobody has to do anything "right" in the tests to get this to work.
Individual pages and components are automatically detected and displayed, and related elements are grouped by default – but everything can be configured and customized to your particular application and your goals for coverage.
Takeaways
Understanding how test suites move through your application is critical to understanding the value and impact of your test pipeline.
UI Coverage provides a unique perspective on the overall test coverage of your projects, and always reflects the reality of what actually happens in tests – how the application is actually rendered, and what the tests actually check. It provides high-quality visibility and prioritization capabilities that would be prohibitively time-consuming and expensive to generate manually, especially on an ongoing run-by-run basis.
With the rising use of AI to generate code, the pace and scope of code changes is increasing. The ongoing visibility and change detection offered by UI Coverage helps you track the state of your tests as things change, and offers AI-powered test suggestions to make it easier to keep your test coverage up to date.
If your team would like to try out UI Coverage with Test Generation, reach out to us to request a free trial.