At Panorama, we strongly believe in automated tests. No code gets deployed without passing both a thorough code review and a battery of thousands of automated tests, and no code makes it through code review without updating tests for the new features it’s adding. Automated tests help us build new features and refactor old code without introducing bugs, and are a big part of the reason we are able to confidently deploy new changes to our production apps multiple times a day.
What are flaky tests?
But with an automated test suite comes the dreaded possibility of tests that fail nondeterministically—that is, most of the time they pass, but every once in a while they fail for no obvious reason. When the tests are retried, they pass again. Flaky tests.