Visual Testing for Emails Sent Using 3rd Party Dynamic Templates

June 11, 2021

By Gleb Bahmutov

This blog post will teach you how to visually verify the emails sent by a 3rd party service. Such verification is necessary to ensure the emails have the expected style, and include the necessary information. The post teaches you how to deal with the difficult email markup generated from dynamic templates when doing image comparison.

Note: you can find the source code for this blog post in the repository cypress-sendgrid-visual-testing-example, and see the visual snapshots at its Percy page.

Dynamic templates

If your application is sending HTML emails using a 3rd party service like SendGrid, you might be using "dynamic templates" - meaning your email design is stored on the service, and your application triggers the send by providing the template data.

For example, I have designed the following "Confirmation code" email in SendGrid

Dynamic SendGrid template

To send an email, the application needs to provide the confirmation code, the application URL to open, and the user name. We can see the email preview while designing the template:

Previewing the email as it would be seen by the user

Because the template lives in a 3rd party system, it is simple for someone to modify it, and accidentally break the data to markup mapping, leading to bugs such as the missing user name in the greeting bug below:

An email that accidentally used a wrong field in the greeting line

In order to guarantee our users are going to get a nice-looking and correct confirmation email, we need to test the entire confirmation flow, and we need to verify the received email using visual testing.

Sending emails

Our application no longer needs to have HTML markup for every email. Instead it makes a call to the SendGrid API and provides the template ID and the template data object.

// pages/api/register.js
const initEmailer = require('../../src/emailer')
const emailer = await initEmailer()
await emailer.sendTemplateEmail({
  to: email,
  // the ID of the dynamic template we have designed
  template_id: 'd-9b1e07a7d2994b14ae394026a6ccc997',
  dynamic_template_data: {
    code: confirmationCode,
    username: name,
    confirm_url: 'http://localhost:3000/confirm',
  },
})
console.log('sent a confirmation email to %s', email)

The emailer function uses the NPM package @sendgrid/client to contact SendGrid

// src/emailer.js
const sgClient = require('@sendgrid/client')
sgClient.setApiKey(process.env.SENDGRID_API_KEY)

/**
  * Sends an email by using SendGrid dynamic design template
  * @see Docs https://sendgrid.api-docs.io/v3.0/mail-send/v3-mail-send
  */
async sendTemplateEmail({ from, template_id, dynamic_template_data, to }) {
  const body = {
    from: {
      email: from || process.env.SENDGRID_FROM,
      name: 'Confirmation system',
    },
    personalizations: [
      {
        to: [{ email: to }],
        dynamic_template_data,
      },
    ],
    template_id,
  }

  const request = {
    method: 'POST',
    url: 'v3/mail/send',
    body,
  }
  const [response] = await sgClient.request(request)

  return response
}

We can print the email before posting it by digging into the @sendgrid/client code and adding a console command:

'{\n' +
  "  url: 'v3/mail/send',\n" +
  "  method: 'POST',\n" +
  '  headers: {\n' +
  "    Accept: 'application/json',\n" +
  "    'Content-Type': 'application/json',\n" +
  "    'User-Agent': 'sendgrid/7.4.3;nodejs',\n" +
  "    Authorization: 'Bearer <SENDGRID_API_KEY>'\n" +
  '  },\n' +
  '  maxContentLength: Infinity,\n' +
  '  maxBodyLength: Infinity,\n' +
  '  data: {\n' +
  '    from: {\n' +
  "      email: '<SENDGRID_FROM>',\n" +
  "      name: 'Confirmation system'\n" +
  '    },\n' +
  '    personalizations: [\n' +
  '      {\n' +
  "        to: [ { email: '[email protected]' } ],\n" +
  '        dynamic_template_data: {\n' +
  "          code: '8a82b408',\n" +
  "          username: 'Gleb',\n" +
  "          confirm_url: 'http://localhost:3000/confirm'\n" +
  '        }\n' +
  '      }\n' +
  '    ],\n' +
  "    template_id: 'd-9b1e07a7d2994b14ae394026a6ccc997'\n" +
  '  },\n' +
  "  baseURL: 'https://api.sendgrid.com/'\n" +
  '}'

Note that to run our application we need to provide the SENDGRID_API_KEY and SENDGRID_FROM environment variables.

Receiving emails

Every time we call the SendGrid API it sends an email to the to recipient. Our test needs to "catch" these emails. Luckily there is a free Ethereal email service with a great programmatic API we can use.

When the Cypress project starts, we can create a new email user.

// cypress/plugins/index.js
const makeEmailAccount = require('./email-account')

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = async (on) => {
  const emailAccount = await makeEmailAccount()

  on('task', {
    getUserEmail() {
      return emailAccount.email
    },

    getLastEmail() {
      return emailAccount.getLastEmail()
    },
  })
}

The email-account.js uses the nodemailer to create the user and retrieve the last email received.

// cypress/plugins/email-account.js
// use Nodemailer to get an Ethereal email inbox
// https://nodemailer.com/about/
const nodemailer = require('nodemailer')
// used to check the email inbox
const imaps = require('imap-simple')
// used to parse emails from the inbox
const simpleParser = require('mailparser').simpleParser

...
// Generate a new Ethereal email inbox account
const testAccount = await nodemailer.createTestAccount()
console.log('created new email account %s', testAccount.user)
console.log('for debugging, the password is %s', testAccount.pass)

In the spec file we use the testAccount.email address when registering the new user.

// cypress/integration/spec.js
describe('Email confirmation', () => {
  let userEmail

  before(() => {
    // get and check the test email only once before the tests
    cy.task('getUserEmail').then((email) => {
      expect(email).to.be.a('string')
      userEmail = email
    })
  })

  it('sends confirmation code', () => {
    const userName = 'Joe Bravo'

    cy.visit('/')
    cy.get('#name').type(userName)
    cy.get('#email').type(userEmail)
    cy.get('#company_size').select('3')
    cy.get('button[type=submit]').click()
  })
})

The test types the new Ethereal user into the form and clicks the "Submit" button

The registration form uses the test email account

The plugin file prints the created test account email and password in the terminal. Using these credentials we can log into the Ethereal inbox to see the received email.

Use the printed test email and password to log into the Ethereal email web interface

In the inbox we should see the email from SendGrid service with the random confirmation code generated by the application.

The received email looks nice

Tip: notice the "From" field above - I am using another random Ethereal email as a verified sender in the SendGrid settings. Even though the Ethereal email cannot send any emails, it can receive the sender verification email from SendGrid, and I can click the "confirm" link after logging into the Ethereal Web interface. This is exactly the same confirmation flow we are trying to verify in our application.

Cypress email test

Let's complete our test - we want to access the Ethereal inbox, find the last email, load its HTML inside the Cypress' browser (it is an HTML email after all!), click on the "Enter the confirmation code ..." button and enter the code in our application's page. Here is our complete test:

// cypress/integration/spec.js
const { recurse } = require('cypress-recurse')

describe('Email confirmation', () => {
  let userEmail

  before(() => {
    // get and check the test email only once before the tests
    cy.task('getUserEmail').then((email) => {
      expect(email).to.be.a('string')
      userEmail = email
    })
  })

  it('sends confirmation code', () => {
    const userName = 'Joe Bravo'

    cy.visit('/')
    cy.get('#name').type(userName)
    cy.get('#email').type(userEmail)
    cy.get('#company_size').select('3')
    cy.get('button[type=submit]').click()

    cy.log('**shows message to check emails**')
    cy.get('[data-cy=sent-email-to]')
      .should('be.visible')
      .and('have.text', userEmail)

    // retry fetching the email
    recurse(
      () => cy.task('getLastEmail'), // Cypress commands to retry
      Cypress._.isObject, // keep retrying until the task returns an object
      {
        timeout: 60000, // retry up to 1 minute
        delay: 5000, // wait 5 seconds between attempts
      },
    )
      .its('html')
      .then((html) => {
        cy.document({ log: false }).invoke({ log: false }, 'write', html)
      })
    cy.log('**email has the user name**')
    cy.contains(`Dear ${userName},`).should('be.visible')
    cy.log('**email has the confirmation code**')
    cy.contains('a', 'Enter the confirmation code')
      .should('be.visible')
      .invoke('text')
      .then((text) => Cypress._.last(text.split(' ')))
      .then((code) => {
        cy.log(`**confirm the code ${code} works**`)
        expect(code, 'confirmation code')
          .to.be.a('string')
          .and.have.length.gt(5)

        // add synthetic delay, otherwise the email
        // flashes very quickly
        cy.wait(2000)

        // unfortunately we cannot confirm the destination URL
        // via <a href="..."> attribute, because SendGrid changes
        // the href to its proxy URL

        // before we click on the link, let's make sure it
        // does not open a new browser window
        // https://glebbahmutov.com/blog/cypress-tips-and-tricks/#deal-with-target_blank
        cy.contains('a', 'Enter the confirmation code')
          // by default the link wants to open a new window
          .should('have.attr', 'target', '_blank')
          // but the test can point the open back at itself
          // so the click opens it in the current browser window
          .invoke('attr', 'target', '_self')
          .click()

        cy.get('#confirmation_code', { timeout: 10000 }).should('be.visible')
        // confirm the URL changed back to our web app
        cy.location('pathname').should('equal', '/confirm')

        cy.get('#confirmation_code').type(code)
        cy.get('button[type=submit]').click()
        // first positive assertion, then negative
        // https://glebbahmutov.com/blog/negative-assertions/
        cy.get('[data-cy=confirmed-code]').should('be.visible')
        cy.get('[data-cy=incorrect-code]').should('not.exist')
      })
  })
})

The test runs through the entire flow

The full confirmation email flow

Visual test for the registration form

Let's start with a visual assertion for the very first registration form page. We have an array of choices for visual testing - you can do visual testing yourself, or use a 3rd-party commercial service. In this blog post I will use Percy visual testing service - it is simple to set up and has a generous free tier. For setup steps, see the official Percy Cypress docs. In the spec itself, we can use cy.percySnapshot(<title>) command.

// cypress/integration/spec.js
// https://docs.percy.io/docs/cypress
require('@percy/cypress')
describe('Email confirmation', () => {
  it('sends confirmation code', () => {
    const userName = 'Joe Bravo'

    cy.visit('/')
    cy.get('#name').type(userName)
    cy.get('#email').type(userEmail)
    cy.get('#company_size').select('3')
    cy.percySnapshot('1 - registration screen')
    cy.get('button[type=submit]').click()
    ...
  })
})

Percy visual service runs asynchronously - it sends the DOM and CSS snapshots to its image rendering and comparing cloud, and then posts the status to GitHub. Thus we can have commits and pull requests where functional Cypress tests finish successfully, while visual Percy tests fail.

Percy posts a failing status check because we have a new visual snapshot to approve

As soon as we run Percy for the very first time in this project, it notices a new visual snapshot - we need to verify the image to accept it (which turns the failing Percy status check into a passing one).

The first Percy build with 1 unreviewed snapshot

By default, Percy renders the DOM + CSS snapshot in both Chrome and Firefox browsers, and in two resolutions.

Percy has rendered the registration form in Chrome and Firefox

As you can see, Firefox renders the form almost identically, thus we do not need the extra images. But we should keep the two rendered resolutions, since the form's layout is responsive and should be tested in both viewports. We can control the browsers from the project's settings page in Percy.

Rendering the images in the Chrome browser only should be enough for our project

Random data

We feel good - our registration form is from now on going to stay the same until we explicitly allow it to change. We run the test again ... and it fails!

Visual snapshot does not match because our test account email is different

In order to have exactly the same rendered images, the application page must have exactly the same data each time. We generated a random test email, thus the input field has different value. Ughh, what should we do?

We could try hiding the email input field from Percy by providing custom CSS like this:

// we need to ignore the userEmail region when doing
// the visual diff, since the text is dynamic
cy.percySnapshot('1 - registration screen', {
  percyCSS: '#email { display: none; }',
})

This does not solve the problem - it removes the element completely, creating an image that looks wrong.

Percy does not render the email input field due to display: none CSS

Instead I will do the following trick:

cy.visit('/')
cy.get('#name').type(userName)
cy.get('#company_size').select('3')
// avoiding visual difference due to new email
cy.get('#email').type('[email protected]')
cy.percySnapshot('1 - registration screen')
// type the real email
cy.get('#email').clear().type(userEmail)
cy.get('button[type=submit]').click()

Now every build will have the same form data producing the same image.

Visual snapshot uses the same email before typing the real Ethereal email address

Visual test for HTML email

Let's add a visual test for the HTML we retrieve from the server and write into the browser.

cy.contains('a', 'Enter the confirmation code')
  .should('be.visible')
  .as('@codeLink') // save under an alias
  .invoke('text')
  .then((text) => Cypress._.last(text.split(' ')))
  .then((code) => {
    cy.log(`**confirm the code ${code} works**`)
    expect(code, 'confirmation code')
      .to.be.a('string')
      .and.have.length.gt(5)
    
    // add synthetic delay, otherwise the email
    // flashes very quickly
    cy.wait(2000)
    cy.percySnapshot('2 - email')
    ...

The two second delay is very useful for making sure the rendered HTML appears in the video captured by Cypress during cypress run. Percy build shows the new snapshot for approval. We immediately can see the problem - we are showing the confirmation code. Because the code is dynamic data, even if we approve the current snapshot, every new snapshot will fail.

Email snapshot shows the dynamic data. It will cause problems later.

Luckily, we can play the same data trick - by now we grabbed the confirmation code, and can safely change it in the page to something constant. Unfortunately, SendGrid dynamic templates are strict with HTML markup, and we cannot add data attributes following our best practices for selecting elements. We can still use cy.contains to find and replace element's text.

// add synthetic delay, otherwise the email
// flashes very quickly
cy.wait(2000)
// replace the dynamic confirmation code with constant text
// since we already validated the code
cy.get('@codeLink').invoke(
  'text',
  `Enter the confirmation code abc1234`,
)
cy.contains('strong', new RegExp('^' + code + '$')).invoke(
  'text',
  'abc1234',
)
cy.percySnapshot('2 - email')

Time-traveling debugger helps us confirm the test changes the dynamic text in two elements correctly.

Replacing the dynamic text in the email before sending to Percy

With these changes, the visual snapshot is going to be consistent.

Consistent email snapshot

The email still works the same after our text changes, because the functional Cypress test confirms it. Let's make sure the confirmation screen is tested visually too.

Visual test for confirmation page

The localhost:3000/confirm is the page where the user enters the confirmation code. Our test verifies that the code is correct. We can add a visual assertion to the test, but after changing the dynamic data in the input field to a constant.

// confirm the URL changed back to our web app
cy.location('pathname').should('equal', '/confirm')

cy.get('#confirmation_code').type(code)
cy.get('button[type=submit]').click()
// first positive assertion, then negative
// https://glebbahmutov.com/blog/negative-assertions/
cy.get('[data-cy=confirmed-code]').should('be.visible')
cy.get('[data-cy=incorrect-code]').should('not.exist')

cy.get('#confirmation_code').clear().type('correct code')
cy.percySnapshot('3 - correct code')
Visual test for confirmed code screen

We also need to confirm the rejection flow. If the user enters the wrong confirmation code, the application should show an error. We can create a separate test for this:

it('rejects wrong code', () => {
  cy.visit('/confirm')
  cy.get('#confirmation_code').type('wrong code')
  cy.get('button[type=submit]').click()
  // first positive assertion, then negative
  // https://glebbahmutov.com/blog/negative-assertions/
  cy.get('[data-cy=incorrect-code]').should('be.visible')
  cy.get('[data-cy=confirmed-code]').should('not.exist')
  cy.percySnapshot('incorrect code')
})

There is no need for data manipulation in this case - the form has no dynamic data.

Visual test for rejected confirmation code screen

Recap

  • Visual testing is a powerful mechanism for quickly confirming the entire browser page looks the same as before.
  • Cypress can interact with remote services to fetch delivered emails and render them in its browser. Then we can use visual testing to confirm the email looks the same as expected.
  • Dynamic data can pose challenges for visual testing, and we should ensure the page's DOM has the same data before taking a snapshot.

Happy email testing!

See also

We have shown alternative email testing approaches in the following blog posts: