Skip to main content

Analyzing Failure Patterns Across Your Test Suite


Failure Analysis is designed to help you optimize test efficiency and efficacy. The proprietary machine learning algorithms review pass/fail data along with Selenium and Appium command logs to unearth common failures and their impact on the test suite as a whole. It then presents a report with tabs that aggregate patterns that are predictive of failure, helping you avoid similar or duplicate failures in future tests. Using Failure Analysis:

  • Improves developer efficiency, streamlining detection and triage of the most pervasive errors
  • Validates investment in test automation by showing larger patterns as a source of failure, allowing for global mitigation and faster time-to-market with better quality

How it Works#


Failure Analysis can only be effective if your automation tests are configured to report a pass/fail outcome.

Failure Analysis leverages your test data and identifies potential failure patterns based on aggregate test errors. More specifically, the tool:

  • Identifies failed tests
  • Aggregates failures on test names
  • Detects common failure patterns
  • Ranks and prioritizes patterns by most pervasive impact

For example, the image below shows a failed build where each test contains a bad, or outdated, web element locator. Failure Analysis detects any failure patterns and attributes a percentage to show how pervasive this failure is within this particular build.

Failed Tests View

To see the specifics of each failure pattern, go to Insights > Failure Analysis, or select Failure Patterns when viewing data about your build. As you can see in the next image, a pattern of failures due to invalid element locators has emerged that is impacting 25% of the tests in the build.

Failed Tests View

You can optimize the power of the Failure Analysis tool by Providing Context for Selenium Commands with the JavaScript Executor.