End-to-End Snapshot Testing

Snapshot testing has taken the JavaScript unit testing world by storm. It removes much of the manual coding and much of the boilerplate, leaving the test runner to save or compare produced values. This blog post shows how the Cypress Test Runner can bring the same power to your end-to-end tests.

Experimental feature

This document shows a very early proof of concept. The API and features shown here are likely to change. Use at your own risk.

I will start by reviewing snapshot testing as it is used in unit testing. Then I will show how we can take the same approach in end-to-end tests to snapshot data objects. Finally, we will look at snapshot testing DOM elements.

Snapshot testing

First, a quick intro to snapshot testing. Imagine a unit test. You probably thought of something like this:

const add = (a, b) => a + b

it('adds numbers', () => {
  expect(add(2, 3)).toBe(5)
})

The test computes a value using a call to add(2, 3) and compares it to a hard coded value of 5. How did we get this expected value 5? Well, I (the human developer) took a piece of paper and a pencil and added 2 + 3. Easy.

Imagine a real world unit test. Would computing the expected value be as simple as adding two small numbers? No. A more complex algorithm would generate a large object as a result, not a single number. For example, if we are unit testing an API response, we would probably get something like this:

const api = require('./api')

api.topSeller()
  .then(console.log)
/*
{
  "id": "1234-5678...",
  "name": "Nerf Gun Zombie Blaster Dominator",
  "displayName": "Zombie Nerf Gun",
  "alias": "zombie-nerf-gun",
  "SKU": "...",
  "price": ...,
  "currency": "...",
  "availability": "...",
  "promotion". "..."
  and many other fields
}
*/

Instead of coding the expected value by hand, I often do the following trick in my tests (well, this was before snapshot testing became a thing!)

it('returns top seller item', () => {
  api.get().then(item => console.log(JSON.stringify(item)))
})

Then I would copy the result from the terminal and would paste it into the unit test, removing the print statement.

it('returns top seller item', () => {
  const expected = {
    id: '1234-5678-...',
    name: 'Nerf Gun Zombie Blaster Dominator',
    displayName: '...'
    // and the rest of the properties
  }
  return api.get()
    .then(item => expect(item).toDeepEqual(expected))
})

Printing, copying, and pasting values into the test quickly becomes tedious and the tests blow up in size. Luckily, snapshot testing (see links 1, 2, 3 for more examples) became a major thing, thanks to the Jest test runner that includes it by default. The same test above becomes:

it('returns top seller item', () =>
  api.get().then(item =>
    expect(item).toMatchSnapshot()
  )
)

The first time the Jest (or Ava, or Mocha with snap-shot-it plugin) test runner executes this test, it will save the received value into a plain JavaScript snapshot file. In this case it would look something like this:

exports[`returns top seller item 1`] = `
{
  id: '1234-5678-...',
  name: 'Nerf Gun Zombie Blaster Dominator',
  displayName: '...'
  // and the rest of the properties
}
`;

You, the developer, can review this file to make sure the saved value is correct. Then you should commit the snapshot file together with the spec files. Next time the test runner executes the test, it will load the value from the snapshot and compare it with the received value. If they are equal (deep equality is used), the assertion passes. If they are different, the assertion throws an error. Jest can intelligently show the difference, thanks to picking a diff plugin appropriate for specific data type (for example JSON object vs HTML text).

In the above image, the generated HTML component is different from the saved snapshot value. You can either find why the code produced different HTML or update the snapshot file if this is what it should be now.

End-to-end snapshot testing

In my previous blog post I showed examples of user interface tests, data store tests and even API tests against a server. Taken together, these tests give me tremendous confidence that this TodoMVC application is reliable and correct. But the tests in the repo have a major flaw - they are full comparisons of generated values against verbose objects. The test below is a typical verbose example. It adds a few todo items by dispatching actions to the data store and then asserts that the central data store has the expected state object.

it('can be driven by dispatching actions', () => {
  store.dispatch('setNewTodo', 'a new todo')
  store.dispatch('addTodo')
  store.dispatch('clearNewTodo')

  // assert UI
  getTodoItems().should('have.length', 1).first().contains('a new todo')

  // assert store
  getStore().should('deep.equal', {
    loading: false,
    todos: [
      {
        title: 'a new todo',
        completed: false,
        id: '1'
      }
    ],
    newTodo: ''
  })
})

There are many places like this in the spec files, noticeable with the should('deep.equal') assertion. This would be a great opportunity to replace coded (by hand!) values with automatic snapshot saving and comparing. But these end-to-end tests are running inside a real browser (Electron or any Chrome-based one), and just saving and loading a snapshot file synchronously is impossible. Can we do something about this?

Yes, we can!

In fact, we can add snapshot testing to our end-to-end test runner without modifying the Cypress core code itself - it is all possible via 3rd party code. We have published a snapshot add-on as the @cypress/snapshot npm module. Just do these three steps to get started:

1. Install as a dev dependency

    npm i -D @cypress/snapshot
   

2. Load and register the add-on

    // in your cypress/support/commands.js
    // add the following line
    require('@cypress/snapshot').register()
    // to get new .snapshot() command
   

3. There is no step 3!

We now have a new command that can operate on any subject: object, string, array, or DOM element. The simplest example could be our ‘addition’ unit test.

const add = (a, b) => a + b
it('adds numbers', () => {
  cy.wrap(add(2, 3)).snapshot()
})

The saved snapshot file looks like this after the test run:

module.exports = {
  "adds numbers": {
    "1": 5
  }
}

Each snapshot is saved under the full test name (including its parent describe suite names). Plus there is an index in case a single test has multiple snapshots. You can overwrite the name with more memorable string:

const add = (a, b) => a + b
it('adds numbers', () => {
  cy.wrap(add(2, 3)).snapshot()
  cy.wrap(add(1, 10)).snapshot()
  cy.wrap(add(-6, -3)).snapshot({ name: 'negatives' })
})

The above test produces the following snapshot file:

module.exports = {
  "adds numbers": {
    "1": 5,
    "2": 11,
    "negatives": -9
  }
}

The Cypress Test Runner is not very interesting when snapshots are matching saved values. But if we change some numbers in our example to make the test fail, then it gets interesting. Let’s change the test:

// before (correct)
cy.wrap(add(1, 10)).snapshot()
// after (incorrect)
cy.wrap(add(1, 100)).snapshot()

The computed value will be 101 rather than the saved value, 11.

The Command Log area on the left shows the difference. When I click on the “SNAPSHOT” command, it outputs the expected and current values to the DevTools. This allows me to quickly determine what went wrong; or at least it gives me as much information as the test runner itself knows.

Under the hood the @cypress/snapshot module is loading the entire set of snapshots before all tests start running, and during the run it is working against an object store, thus avoiding file reading and writing problems. I took this idea from the karma-snapshot plugin and implemented it using cy.readFile() and cy.writeFile() commands.

Now we can go to town, cutting our end-to-end tests to the bare minimum. Each test should modify the application, grab the central data store or server response and snapshot it. We can use .snapshot() almost anywhere. Here is one example that confirms the shape of an object (the names of its properties).

const getStore = () => cy.window().its('app.$store')

it('has loading, newTodo and todos properties', () => {
  // before
  getStore().its('state').should('have.keys', ['loading', 'newTodo', 'todos'])
  // after
  getStore().its('state').then(Object.keys).snapshot()
})

The snapshot file for this test:

module.exports = {
  "UI to Vuex store": {
    "has loading, newTodo and todos properties": {
      "1": [
        "loading",
        "todos",
        "newTodo"
      ]
    }
  }
}

The larger the object - the better! Our previously verbose test has become extremely small. It now has two separate sections: actions, assertions and nothing else.

it('can be driven by dispatching actions', () => {
  // actions
  store.dispatch('setNewTodo', 'a new todo')
  store.dispatch('addTodo')
  store.dispatch('clearNewTodo')

  // assert UI
  getTodoItems().should('have.length', 1).first().contains('a new todo')

  // assert store
  getStore().snapshot()
})

Important: Do not forget to inspect the snapshots from the Cypress Test Runner or in the saved snapshots.js file to make sure they are correct - they are becoming part of the test!

Note that in the Cypress Test Runner, when you click on the “SNAPSHOT” command, the test frame shows the temporary DOM snapshot at that exact moment!

Element snapshots

Important: before we proceed, let me clarify some Cypress nomenclature because it might get confusing.

Caution

The names below are likely to change in the future, we are still working to make the terminology as clear as possible.

• DOM Snapshot - temporary copy of the test application’s DOM saved after each step of a test while running in the GUI mode. When you hover over a test step the corresponding DOM snapshot is displayed in the test area showing you how the page looked before and after that test step.
• snapshot - any object stored on disk when we execute cy.wrap(...).snapshot() command.
• element snapshot - serialized DOM element stored on disk when we execute cy.get('<selector>').snapshot() command. The element could be a <div>, a <main class="todo"> or even the entire <body>.

So this section talks about how we can extend the idea of test snapshots to capture DOM elements and why this is so useful.

Let us go through a complex test scenario. We will add a couple of items to the list, then we will mark two of them completed. How many tests and assertions do we need to write? Well, thanks to the extracted utility functions, manipulating the page is easy.

it('marks completed items', () => {
  // several actions
  enterTodo('first item')
  enterTodo('second item')
  enterTodo('item 3')
  enterTodo('item 4')
  toggle('item 3')
  toggle('item 4')
})

But making assertions can become tiresome. We can write so many individual assertions: to test the number of items, to test checked status on each item, etc.

it('marks completed items', () => {
  // several actions
  // ...
  // assertions
  getTodoItems().should('have.length', 4)
  getCompleted().should('have.length', 2)
  getTodo('first item').find('[type="checkbox"]').should('not.be.checked')
  getTodo('second item').find('[type="checkbox"]').should('not.be.checked')
  getTodo('item 3').find('[type="checkbox"]').should('be.checked')
  getTodo('item 4').find('[type="checkbox"]').should('be.checked')
})

This quickly gets tedious. But, just like before, we can replace lots of individual assertions by capturing a value describing the entire page, or its header, or the list items. We will compare the entire element using cy.get('ul.todo-list').snapshot(). By default the DOM element’s HTML will be saved as a snapshot (with transient information stripped, like React the “id” attribute). Alternatively, you might want to serialize DOM structure as a JSON object. Look at this beautiful test:

it('marks completed items', () => {
  // actions
  enterTodo('first item')
  enterTodo('second item')
  enterTodo('item 3')
  enterTodo('item 4')
  toggle('item 3')
  toggle('item 4')
  // make sure app has rendered the check marks
  getCompleted().should('have.length', 2)
  // single snapshot of entire <ul class="todo-list"> element
  cy.get('ul.todo-list')
    .snapshot({ name: 'todo-list with 2 completed items' })
})

Important: we do not know when the application renders the list, but want to take its snapshot after the checkboxes have finished rendering. Thus we place the Cypress assertion getCompleted().should('have.length', 2) which is retried by the test runner until two items have checked input boxes present. Then we take the element snapshot. If we click on the “SNAPSHOT” step we can see the captured HTML.

Just like before I encourage you to inspect any new snapshot closely before committing it to source control.

Imagine something has changed, and our application stopped marking the items as completed. (In reality, I will just comment out the test commands). The element snapshot will no longer match the saved value. The test runner shows the difference.

The Command Log is showing (using git-diff like algorithm from module called disparity) that the line <li class="todo completed"> is gone, and instead there is line <li class="todo">. If you click on the “SNAPSHOT” command in the Command Log, the full expected and received values are printed to the DevTools console.

Final thoughts

You can find the source code for this blog post in cypress-example-recipes repository. In some cases the spec files became smaller (store-spec.js), but in other cases snapshot testing has allowed me to write much more complex test scenarios, while keeping the test code short and easy to read.

For now, we have released @cypress/snapshot as a 3rd party module. But we have big plans to bring snapshot testing into the core test runner. We have been using unit test snapshots in Mocha extensively in ther Cypress Test Runner repository itself, and think that the developer experience should be substantially improved by giving the snapshot review and comparison a nice graphical user interface! Stay tuned and follow the GitHub issue #772.