Jenkins

Sauce Labs and Jenkins Tutorial

This tutorial explains how to integrate Jenkins with tests run with the Sauce Labs cloud of Selenium servers.

We assume that you have some familiarity with Jenkins and the fundamentals of automated testing. However, even if this is your first time using Jenkins and automated testing you should be able to successfully follow these step-by-step instructions.

Installing the Jenkins Sauce OnDemand plugin

The Sauce OnDemand plugin for Jenkins can be installed through the Jenkins Administration page.

To access this page, click Manage Jenkins from the left-hand navigation pane.

Manage Jenkins

Then click on the Manage Plugins link:

Manage Plugins

Select the Available tab:

Available Tab

Scroll down the list of plugins to find the Sauce OnDemand plugin, select the check box and click the Download and install after restart buton:

Install Plugin

This will download the Sauce OnDemand plugin for Jenkins. The plugin file is quite large, so the download may take some time to complete. Select the Restart Jenkins when installation is complete and no jobs are running, and when the download has finished, the Jenkins instance will restart.

Restart Jenkins

Configuring the Jenkins Sauce OnDemand plugin

After the Sauce plugin has been installed, the username and access key of the Sauce OnDemand user account must be entered within the Jenkins administration interface.

To configure the authentication, first click Manage Jenkins from the left-hand navigation pane.

Manage Jenkins

Click the Configure System link

Configure System

Scroll down to the Sauce OnDemand section.

Sauce Admin

This section contains the fields required to configure how the authentication for the Sauce plugin. Enter the values of the username and access key you wish the Sauce plugin to use in the Username and API Access Key fields.

By default, the Sauce plugin will use the user home directory as the working directory when extracting the Sauce Connect binary file. You can override this behaviour by specifying a directory location in the Sauce Connect Working Directory field.

The plugin also supports reading authentication details from a .sauce-ondemand file located in the user home directory (e.g. /Users/Shared/Jenkins). The contents of the file should be in the following format:

username:sauceUsername
key:sauceAccessKey

If you wish for the plugin to use this file, then select the Use authentication details in ~/.sauce-ondemand? checkbox.

Once the authentication details have been entered, clicking on the Test Connection button will connect to Sauce OnDemand to verify that the plugin can authenticate with the entered details.

Once the authentication details have been entered and saved on the Configure System screen, you can then enable Sauce OnDemand support on the Configuration screen for a Jenkins Job.

In addition to the authentication fields, the following fields are displayed:

Jenkins Configuration for a Java-based Project

To demonstrate the Sauce plugin for Jenkins, let's create a new Jenkins Freestyle project for a Java project.

From the Jenkins dashboard page, click New Job

New Job

Enter 'Sauce Java Demo' in the Job Name field and select Build a free-style software project.

New Freestyle Project

Our sample code is located in github, so select Git in the Source Code Management section, enter https://github.com/rossrowe/sauce-ci-java-demo as the repository URL and enter master in the branch specifier.

Git Setup

Now let's add a build step which will run our tests. Click on the Add Build Step menu in the Build section, and select Invoke top-level Maven targets

Invoke Maven

Enter test in the Goals field.

Maven Goals

Now let's enable the Sauce OnDemand support for a Jenkins Job can be enabled by checking the Sauce OnDemand Support checkbox.

Sauce Configure

Select the Enable Sauce Connect? check box. When selected, the Sauce plugin will launch an instance of Sauce Connect prior to the running of your Job. This instance will be closed when the Job completes.

The Sauce Connect Launch Condition allows you to set fine-grained rules which define when Sauce Connect should be launched. By default this is set to Always.

Click on the WebDriver radio button and select a browser to run our tests against (let's pick Chrome 35 running in Mac OS X 10.8)

Sauce Configure

Sauce OnDemand supports a wide range of browsers, but some browser combinations are only supported for SeleniumRC or WebDriver tests. The multi-select lists beneath the Appium, SeleniumRC and WebDriver radio buttons are populated by retrieving the list of respective supported browsers via the Sauce REST API.

If a single browser is selected, then the SELENIUM_PLATFORM, SELENIUM_VERSION, SELENIUM_BROWSER and SELENIUM_DRIVER environment variables will be populated to contain the details of the selected browser. In addition, the SAUCE_ONDEMAND_BROWSERS environment variable will be populated with a JSON-formatted string containing the attributes of the selected browsers. An example of the JSON string is:

[
    {
    "platform":"LINUX",
    "os":"Linux",
    "browser":"firefox",
    "url":"sauce-ondemand:?os=Linux&browser=firefox&browser-version=16",
    "browserVersion":"16"
    },
    {
    "platform":"VISTA",
    "os":"Windows 2008",
    "browser":"iexploreproxy",
    "url":"sauce-ondemand:?os=Windows 2008&browser=iexploreproxy&browser-version=9",
    "browserVersion":"9"
    }
]

When the Use latest versions of browser checkbox is selected, the Sauce plugin will populate the environment variables with the version information for the latest available version that corresponds to the selected browser and operating system.

The plugin will set a series of environment variables based on the information provided on the Job Configuration. These environment variables can either be explicitly referenced by your unit tests, or through the use of the [selenium-client-factory] library.

By default, the plugin will use the authentication details specified in the Jenkins administration section. However, this can be overriden at the Job level by enabling the Override default authentication? checkbox and specifying values in the Username and Access key fields.

Note: These values are set automatically by the Jenkins plugin. If the Enable Sauce Connect? checkbox is selected, then the SELENIUM_HOST and SELENIUM_PORT variables will default to localhost:4445. If the checkbox is not set, then the SELENIUM_HOST and SELENIUM_PORT variables will be set to ondemand.saucelabs.com:4444.

The values for the SELENIUM_HOST and SELENIUM_PORT environment variables can be overridden by explicitly specifying the host and port in the Sauce Host and Sauce Port fields, which can be displayed by clicking on the Sauce Connect Advanced Options button.

In addition to the Sauce Host and Port fields, there are several other options included within the Sauce Connect Advanced Options section:

Sauce Configure

Embedding Sauce Reports

The plugin also supports the embedding of Sauce Job reports within the display of test results. This requires the tests executed by the Jenkins job to produce result files in the JUnit XML report format.

To enable this, select the Add post-build Action within the Post-build Actions section.

Add Post-build action

From the pop-up menu, select the Publish JUnit test result report option.

JUnit Post-build action

Enter target/surefire-reports/*.xml as the path to the test reports that are produced by your Jenkins Job, and check the Embed Sauce OnDemand reports checkbox.

Embed Sauce Reports

That's it, our configuration is all setup, let's run the tests!

Integrating tests with the Jenkins Sauce OnDemand plugin

To run the tests, click the Build Now link on the Job navigation pane. This should compile and run three tests.

Once the build has finished, navigate to the Build Summary page.

Sauce Build Summary

On this page, you will see that links to the Sauce Jobs executed as part of the build are listed.

Sauce Summary Links

Clicking on one of the links will present a page containing the Sauce report, which will allow you to view the steps performed and watch a video of the test.

Sauce Report

The embedded Sauce reports can also be displayed on the individual test result pages. To view these, click on the Test Result link.

Sauce Test Result

Navigate to individual test result, and the embedded report will be displayed with the test details.

Sauce Embed Test Result

That's it, we've successfully configured Jenkins to run our tests against Sauce OnDemand!

To summarise, in order to make the most use out of the Sauce Jenkins plugin, the following steps should be performed:

Referencing Job Configuration

As mentioned previously, the Sauce Jenkins plugin will set a series of environment variables that reflect the values entered on the Jenkins Job Configuration screen.

Your test code will need to be updated to reference these environment variables.

Below is some sample Java code which demonstrates how to reference the environment variables that are set by the Jenkins plugin

DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
desiredCapabilities.setBrowserName(System.getenv("SELENIUM_BROWSER"));
desiredCapabilities.setVersion(System.getenv("SELENIUM_VERSION"));
desiredCapabilities.setCapability(CapabilityType.PLATFORM, System.getenv("SELENIUM_PLATFORM"));
WebDriver driver = new RemoteWebDriver(
            new URL("http://sauceUsername:sauceAccessKey@ondemand.saucelabs.com:80/wd/hub"),
                desiredCapabilities);

As part of the post build activities, the Sauce plugin will parse the test result files. It attempts to identify lines in the stdout or stderr that are in the following format:

SauceOnDemandSessionID=<some session id> job-name=<some job name>

The session id can be obtained from the RemoteWebDriver instance and the job-name can be any string, but is generally the name of the test class being executed.

Below is a Java sample that demonstrates outputting the session id to the Java stdout.

private void printSessionId() {

    String message = String.format("SauceOnDemandSessionID=%1$s job-name=%2$s", 
    (((RemoveWebDriver) driver).getSessionId()).toString(), "some job name");
    System.out.println(message);
}

Selenium Client Factory

An alternative to explicitly referencing the environment variables in your test code is to use the Selenium Client Factory library. This allows you to construct your SeleniumRC or WebDriver instances in a single line, eg.

WebDriver webDriver = SeleniumFactory.createWebDriver();

Implementations of the library exist for Java and Python

Maven Configuration for a Java-based Project

The setup for Maven projects is essentially the same as for Freestyle projects, but there's a couple of extra steps. Let's create a new Jenkins Maven project for our Java demo project.

From the Jenkins dashboard page, click New Job

New Job

Enter 'Sauce Java Maven Demo' in the Job Name field and select Build a maven 2/3 project.

New Freestyle Project

Our sample code is located in github, so select Git in the Source Code Management section, enter https://github.com/rossrowe/sauce-ci-java-demo as the repository URL and enter master in the branch specifier.

Git Setup

Enter test in the Goals and options field.

Maven Goals

Now let's enable the Sauce OnDemand support for a Jenkins Job can be enabled by checking the Sauce OnDemand Support checkbox.

Sauce Configure

Select the Enable Sauce Connect? check box. When selected, the Sauce plugin will launch an instance of Sauce Connect prior to the running of your Job. This instance will be closed when the Job completes.

Click on the WebDriver radio button and select a browser to run our tests against (let's pick Firefox 15 running in Windows 2008)

Sauce Configure

To enable the embedding of Sauce OnDemand reports for test results, select the Add post-build Action within the Post-build Actions section, and select the Additional test report feature (Sauce OnDemand) option.

Select the Add post-build Action action again, and select the Additional test report feature option, and check the Embed Sauce OnDemand reports checkbox.

Maven Post-build action

That's it, our configuration is all setup, let's run the tests!

Jenkins Configuration for a Python-based Project

Now let's create a new Jenkins Freestyle project for a Python project.

The test results will need to be in the JUnit XML format in order for them to be displayed in Jenkins user interface and so that the Sauce plugin can parse the results. There are several Python libraries which can produce the appropriate XML output, for this tutorial we are going to use NoseXUnit.

From the Jenkins dashboard page, click New Job.

New Job

Enter 'Sauce Java Demo' in the Job Name field and select Build a free-style software project.

New Freestyle Project

Our sample code is located in github, so select Git in the Source Code Management section, enter https://github.com/rossrowe/sauce-ci-python-demo as the repository URL and enter master in the branch specifier.

Git Setup

Now let's add a build step which will run our tests. Click on the Add Build Step menu in the Build section, and select Execute Shell

Execute Shell

Enter nosetests -s --with-xunit simple_test.py in the Command field.

Nose command

Now let's enable the Sauce OnDemand support for a Jenkins Job can be enabled by checking the Sauce OnDemand Support checkbox.

Sauce Configure

Select the Enable Sauce Connect? check box. When selected, the Sauce plugin will launch an instance of Sauce Connect prior to the running of your Job. This instance will be closed when the Job completes.

Click on the WebDriver radio button and select a browser to run our tests against (let's pick Firefox 15 running in Windows 2008)

Sauce Configure

Further details on the environment variables set by the Sauce plugin can be found on the Java sample project tutorial page

To enable this, select the Add post-build Action within the Post-build Actions section.

Add Post-build action

From the pop-up menu, select the Publish JUnit test result report option.

JUnit Post-build action

Enter nosetests.xml as the path to the test reports that are produced by your Jenkins Job, and check the Embed Sauce OnDemand reports checkbox.

Embed Sauce Reports

That's it, our configuration is all setup, let's run the tests!

Multi-configuration projects

Jenkins Multi-configuration projects allow you to run the same build with different input parameters. The Sauce plugin for Jenkins provides an additional option for multi-configuration projects to specify the browser combination for each build.

To configure this on a multi-configuration job, click on the Add Axis button and select either the Sauce OnDemand WebDriver tests or `Sauce OnDemand SeleniumRC tests' item.

Sauce Configure

This will present a list of browser combinations that are supported by Sauce Labs for WebDriver and SeleniumRC.

WebDriver browsers

For each selected browsers, a separate job will run when the build is invoked. This job will include SELENIUM_PLATFORM, SELENIUM_VERSION, SELENIUM_BROWSER, and SELENIUM_DRIVER system properties which will be set to the corresponding browser.

Troubleshooting Jenkins

The source code for the plugin is available from the sauce-ondemand-plugin github project

Bugs/enhancements/feature requests can be recorded within the Jira instance, or you can raise a support request