Testing for reliability of a Cypress test

Overview

When writing automation tests, you want to ensure the test is reliable, and doesn't provide false positives for you and your team. This is especially important when the test makes it into the continuous integration(CI) pipeline. For Cypress, we do two things to try and address reliability.

1. The first is to ensure the test can run x number of times without breaking. 
2. The second is for end-to-end(e2e) tests that require a full run-through, but the backend might be a little bit flaky.

Method 1

I usually run the tests I write a few times before submitting it for a pull request to go into the main CI pipeline. Since Cypress tests are written in javascript, I wrap the entire test into a loop! I run this test 2-3 times to ensure it passes consistently. I found that if you increase the number, the browser will crash sometimes. Make sure you remove this after validating that the test is reliable!

var i = 0;
for (i = 0; i < 20 ; i++) {   
  describe('Is my test actually reliably?', () => {
    it('should do that thing', function() {
      // the new test I want to validate is working
    })
  })
}

Method 2

The second method has been written up before, but its basically retrying only failed tests to account for odd backend instability. You can view the write up here.

Hopefully some of these tips and tricks will help you on your automation journey. Good luck!