Streaming Test Results

In this blog post, I will show how to receive individual test results as soon as each test is finished, without waiting for the entire run to finish.

A single Cypress test might be fast, but if you wait for an entire large test run to finish, you might as well watch the paint dry. What if the first test fails, and yet you wait for 30 minutes to find out about it? Not good. Let's see if we can find out about each test's result as soon as it is done, without waiting for the entire test run to finish.

Note: you can find the source code in the recipe "Stream of tests" in the cypress-example-recipes repository.

Instead of cypress run we will execute the test from index.js Node script. We can start an inter process communication server with id cypressListener - this listener will receive individual test results.

// index.js
const cypress = require('cypress')
const ipc = require('node-ipc')

ipc.config.id = 'cypressListener'
ipc.serve(() => {
  cypress.run().then(...)

  // receive stream of events
  ipc.server.on('test:after:run', (data) => {
    console.log('test finsihed: "%s" %s %dms',
      data.title, data.state, data.duration)
  })
})

The above process uses node-ipc to start listening for messages, and then runs Cypress via its NPM module API. For now it just prints the test results, but could forward the messages to external notification system, or simply stop the Cypress run after the first failed test.

Let's look at the Cypress test code that runs in the browser. We want to know what each test reports, and we can find out by listening to the built-in Cypress events. We are interested in test:after:run event, but unfortunately we can't simply forward the event via IPC socket to the cypressListener process - because the test runs in the browser, and the IPC listener runs using Node on the host machine. Thus we need an intermediate process - this is the perfect job for cypress/plugins/index.js process - it runs in Node background process!

// cypress/support/index.js
let testAttributesToSend

// sends test results to the plugins process
// using cy.task https://on.cypress.io/task
const sendTestAttributes = () => {
  if (!testAttributesToSend) {
    return
  }

  console.log('sending test attributes: %s %s',
    testAttributesToSend.title, testAttributesToSend.state)

  const attr = testAttributesToSend

  testAttributesToSend = null

  cy.task('testFinished', attr)
}

beforeEach(sendTestAttributes)

after(sendTestAttributes)

// you cannot execute async code from event callbacks
// thus we need to be patient and send the test results
// when the next test starts, or after all tests finish
Cypress.on('test:after:run', (attributes, test) => {
  testAttributesToSend = attributes
})

After each test, the test attributes (state, duration, title) are saved in a object. On the next test, or after all tests have finished, this object is sent using cy.task to the plugins process.

The plugins process receives the test object and can send an IPC message to cypressListener process, which is its final destination in this recipe.

// cypress/plugins/index.js
const ipc = require('node-ipc')

ipc.connectTo('cypressListener', () => {
  ipc.of.cypressListener.on('connect', () => {
    ipc.log('## connected to Cypress listener ##')
  })
})

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
  on('task', {
    testFinished (attributes) {
      // console.log(name)
      console.log('%s: "%s" %dms', 
        attributes.state, attributes.title, attributes.duration)
      ipc.of.cypressListener.emit('test:after:run', {
        state: attributes.state,
        title: attributes.title,
        duration: attributes.duration,
      })

      return null
    },
  })
}

The cypressListener process receives each test object, and prints it to the console. We can see each test being reported below:

To summarize, here is the diagram of separate processes and their messages.

• node . creates IPC server process with ID cypressListener, then starts the test run using cypress.run()
• Cypress plugins file runs in Node and can relay messages received from the browser to the node . process using IPC emit commands
• Cypress Test Runner processes in the browser sends individual test results using cy.task commands. The communication from the browser to the plugins file happens over the WebSockets mechanism built into Cypress

Hope you find this recipe useful, let us know how it could be improved. If you have a suggestion, open an issue!