Skip to main content

Configuring Your Espresso Tests

saucectl relies on a YAML specification file to determine exactly which tests to run and how to run them. To customize saucectl to run your Espresso tests, simply modify the properties of the YAML file accordingly. This page defines each of the configuration properties specific to running Espresso tests.

Setting an Alternative Configuration File#

By default, saucectl looks for the config.yml file in the .sauce folder of your project root, but you can actually specify a different file, or if you are using multiple frameworks or need to configure different sets of tests to run separately, you may choose to have multiple configuration files that you can direct saucectl to reference as necessary.

Use the following configuration at runtime to direct saucectl to use any configuration file you choose:

saucectl run -c ./path/to/{config-file}.yml
YAML Required

While you can use multiple files of different names or locations to specify your configurations, each file must be a *.yml and follow the saucectl syntax. Our IDE Integrations (e.g. Visual Studio Code) can help you out by validating the YAML files and provide handy suggestions, so make sure to check them out!

Example Configuration#

.sauce/config.yml
loading...

Each of the properties supported for running Espresso tests through saucectl is defined below.

apiVersion#

| REQUIRED | STRING |

Identifies the version of the underlying configuration schema. At this time, v1alpha is the only supported value.

apiVersion: v1alpha

kind#

| REQUIRED | STRING/ENUM |

Specifies which framework is associated with the automation tests configured in this specification.

kind: espresso

showConsoleLog#

| OPTIONAL | BOOLEAN |

Generates the console.log as local output and as a test asset in Sauce Labs for all tests. By default, console.log is only included in results for failed tests.

showConsoleLog: true

defaults#

| OPTIONAL | OBJECT |

Specifies any default settings for the project.

defaults:  timeout: 15m

timeout#

| OPTIONAL | DURATION |

Instructs how long (in ms, s, m, or h) saucectl should wait for each suite to complete. You can override this setting for individual suites using the timeout setting within the suites object. If not set, the default value is 0 (unlimited).

  timeout: 15m

sauce#

| OPTIONAL | OBJECT |

The parent property containing all settings related to how tests are run and identified in the Sauce Labs platform.

sauce:  region: eu-central-1  metadata:    tags:      - e2e      - release team      - other tag    build: Release $CI_COMMIT_SHORT_SHA  concurrency: 5

region#

| OPTIONAL | STRING/ENUM |

Specifies through which Sauce Labs data center tests will run. Valid values are: us-west-1 or eu-central-1.

  region: eu-central-1

metadata#

| OPTIONAL | OBJECT | VIRTUAL ONLY |

The set of properties that allows you to provide additional information about your project that helps you distinguish it in the various environments in which it is used and reviewed, and also helps you apply filters to easily isolate tests based on metrics that are meaningful to you, as shown in the following example:

metadata:  build: RC 10.4.a  tags:    - e2e    - Android    - beta    - featurex

concurrency#

| OPTIONAL | INTEGER |

Sets the maximum number of suites to execute at the same time. If the test defines more suites than the max, excess suites are queued and run in order as each suite completes.

caution

Set this value to equal or less than your Sauce concurrency allowance, as setting a higher value may result in jobs dropped by the server.

  concurrency: 5

Alternatively, you can override the file setting at runtime by setting the concurrency flag as an inline parameter of the saucectl run command:

saucectl run --ccy 5

retries#

| OPTIONAL | INTEGER |

Sets the number of times to retry a failed suite.

  retries: 1

Alternatively, you can override the file setting at runtime by setting the retries flag as an inline parameter of the saucectl run command:

saucectl run --retries 1

tunnel#

| OPTIONAL | OBJECT |

saucectl supports using Sauce Connect to establish a secure connection with Sauce Labs. To do so, launch a tunnel; then provide the name and owner (if applicable) in this property.

sauce:  tunnel:    name: your_tunnel_name    owner: tunnel_owner_username

name#

| OPTIONAL | STRING |

Identifies an active Sauce Connect tunnel to use for secure connectivity to the Sauce Labs cloud.

note

This property replaces the former id property, which is deprecated.

sauce:  tunnel:    name: your_tunnel_name

owner#

| OPTIONAL | STRING |

Identifies the Sauce Labs user who created the specified tunnel, which is required if the user running the tests did not create the tunnel.

note

This property replaces the former parent property, which is deprecated.

sauce:  tunnel:    name: your_tunnel_name    owner: tunnel_owner_username

reporters#

| OPTIONAL | OBJECT |

Configures additional reporting capabilities provided by saucectl.

reporters:  junit:    enabled: true    filename: saucectl-report.xml

artifacts#

| OPTIONAL | OBJECT |

Specifies how to manage test output, such as logs, videos, and screenshots.

artifacts:  cleanup: true  download:    when: always    match:      - junit.xml    directory: ./artifacts/

cleanup#

| OPTIONAL | BOOLEAN |

When set to true, all contents of the specified download directory are cleared before any new artifacts from the current test are downloaded.

  cleanup: true

download#

| OPTIONAL | OBJECT |

Specifies the settings related to downloading artifacts from tests run by saucectl.

  download:    when: always    match:      - junit.xml    directory: ./artifacts/

when#

| OPTIONAL | STRING |

Specifies when and under what circumstances to download artifacts. Valid values are:

  • always: Always download artifacts.
  • never: Never download artifacts.
  • pass: Download artifacts for passing suites only.
  • fail: Download artifacts for failed suites only.
    when: always

match#

| OPTIONAL | STRING/ARRAY |

Specifies which artifacts to download based on whether they match the name or file type pattern provided. Supports the wildcard character * (use quotes for best parsing results with wildcard).

  match:    - junit.xml    - "*.log"

directory#

| OPTIONAL | STRING |

Specifies the path to the folder location in which to download artifacts. A separate subdirectory is generated in this location for each suite for which artifacts are downloaded.

    directory: ./artifacts/

notifications#

| OPTIONAL | OBJECT |

Specifies how to set up automatic test result alerts.

notifications:  slack:    channels:      - "saucectl-results"      - "espresso-tests"    send: always

slack#

| OPTIONAL | OBJECT |

Specifies the settings related to sending tests result notifications through Slack. See Slack Integration for information about integrating your Sauce Labs account with your Slack workspace.

  slack:    channels: "saucectl-espresso-tests"    send: always

channels#

| OPTIONAL | STRING/ARRAY |

The set of Slack channels to which the test result notifications are to be sent.

  slack:    channels:      - "saucectl-results"      - "espresso-team"    send: always

send#

| OPTIONAL | STRING |

Specifies when and under what circumstances to send notifications to specified Slack channels. Valid values are:

  • always: Send notifications for all test results.
  • never: Do not send any test result notifications.
  • pass: Send notifications for passing suites only.
  • fail: Send notifications for failed suites only.
  slack:    channels: "saucectl-espresso-tests"    send: always

espresso#

| REQUIRED | OBJECT |

The parent property containing the details specific to the Espresso project.

espresso:  app: ./apps/calc.apk  testApp: ./apps/calc-success.apk  otherApps:    - ./apps/pre-installed-app1.apk    - ./apps/pre-installed-app2.apk

app#

| REQUIRED | STRING |

The path to the app. The default directory is {project-root}/apps/filename.apk, and the property supports expanded environment variables to designate the path, as shown in the following examples, or an already uploaded app reference. Supports *.apk and *.aab files.

AAB App Signing

To install an *.apk app that is extracted from an *.aab file, Sauce Labs must sign the *.apk using its own signature. In such cases, Sauce Labs signs both the app and testApp to ensure matching signatures, even if instrumentation is disabled. Otherwise, the app installation will fail.

  app: ./apps/calc.apk
  app: $APP
  app: storage:099557f6-aabb-f8b3-6ad1-8f6200898b92
  app: storage:filename=calc.apk

testApp#

| REQUIRED | STRING |

The path to the testing app. The relative file location is {project-root}/apps/testfile.apk, and the property supports expanded environment variables to designate the path, as shown in the following examples, or an already uploaded test app reference. Supports *.apk and *.aab files.

AAB App Signing

To install an *.apk app that is extracted from an *.aab file, Sauce Labs must sign the *.apk using its own signature. In such cases, Sauce Labs signs both the app and testApp to ensure matching signatures, even if instrumentation is disabled. Otherwise, the app installation will fail.

  testApp: ./apps/calc-success.apk
  testApp: $TEST_APP
  testApp: storage:fbd59e8e-2555-0d3c-5583-1bba2cd17b64
  testApp: storage:filename=calc-success.apk

otherApps#

| OPTIONAL | ARRAY | REAL DEVICES ONLY |

Set of up to seven apps to pre-install for your tests. You can upload an *.apk or *.aab app file from your local machine by specifying a filepath (relative location is {project-root}/apps/app1.apk) or an expanded environment variable representing the path, or you can specify an app that has already been uploaded to Sauce Labs App Storage by providing the reference storage:<fileId> or storage:filename=<filename>.

note

Apps specified as otherApps inherit the configuration of the main app under test for Device Language, Device Orientation, and Proxy, regardless of any differences that may be applied through the Sauce Labs UI, because the settings are specific to the device under test. For example, if the dependent app is intended to run in landscape orientation, but the main app is set to portrait, the dependent app will run in portrait for the test, which may have unintended consequences.

  otherApps:    - ./apps/pre-installed-app1.apk    - $PRE_INSTALLED_APP2    - storage:d6aac80c-2000-a2f1-4c4e-539266e93ee6    - storage:filename=pre-installed-app3.apk

suites#

| REQUIRED | OBJECT |

The set of properties providing details about the test suites to run. May contain multiple suite definitions. See the full example config for an illustration of multiple suite definitions.

Configure RDC and VMC

You can configure tests for both Real Devices and Virtual Machines in a single configuration file.


name#

| REQUIRED | STRING |

The name of the test suite, which will be reflected in the results and related artifacts.

  - name: "saucy test"

timeout#

| OPTIONAL | DURATION |

Instructs how long saucectl should wait for the suite to complete, potentially overriding the default project timeout setting.

note

Setting 0 reverts to the value set in defaults.

  timeout: 15m

emulators#

| OPTIONAL | OBJECT |

The parent property that defines details for running this suite on virtual devices using an emulator.

emulators:  - name: "Android GoogleApi Emulator"    orientation: portrait    platformVersions:      - "11.0"      - "10.0"

name#

| OPTIONAL | STRING |

The name of the device to emulate for this test suite. To ensure name accuracy, check the list of supported virtual devices. If you are using emulators for this test suite, this property is REQUIRED.

  - name: "Android GoogleApi Emulator"

orientation#

| OPTIONAL | ENUM |

The screen orientation to use while executing this test suite on this virtual device. Valid values are portrait or landscape.

  orientation: portrait

platformVersions#

| OPTIONAL | ARRAY |

The set of one or more versions of the device platform on which to run the test suite. Check the list of supported virtual devices for compatible versions.

  platformVersions:    - "11.0"    - "10.0"

devices#

| OPTIONAL | OBJECT |

The parent property that defines details for running this suite on real devices. You can request a specific device using its ID, or you can specify a set of criteria to choose the first available device that matches the specifications.

When an ID is specified, it supersedes the other settings.

devices:  - name: "Google Pixel.*"    platformVersion: 8.1    options:      carrierConnectivity: true  - id: Google_Pixel_2_real_us

id#

| OPTIONAL | STRING |

Request a specific device for this test suite by its ID. You can look up device IDs on device selection pages or by using our Get Devices API request.

        id: Google_Pixel_2_real_us

name#

| OPTIONAL | STRING |

Find a device for this test suite that matches the device name or portion of the name, which may provide a larger pool of available devices of the type you want.

Use Complete Name
      - name: Google Pixel 4 XL
Use Pattern Matching
        name: Google Pixel.*

platformVersion#

| OPTIONAL | STRING |

Request that the device matches a specific platform version.

        platformVersion: 8.0

options#

| OPTIONAL | OBJECT |

A parent property to further specify desired device attributes within the pool of devices that match the name and version criteria.


carrierConnectivity#

| OPTIONAL | BOOLEAN |

Request that the matching device is also connected to a cellular network.

  options:      carrierConnectivity: true

deviceType#

| OPTIONAL | STRING |

Request that the matching device is a specific type of device. Valid values are: ANY, TABLET, or PHONE.

  options:      deviceType: TABLET

private#

| OPTIONAL | BOOLEAN |

Request that the matching device is from your organization's private pool.

  options:      private: true

testOptions#

| OPTIONAL | OBJECT |

A set of parameters allowing you to provide additional details about which test class should be run for the suite and how to apply them.

suites:  testOptions:    class:      - com.example.android.testing.androidjunitrunnersample.CalculatorAddParameterizedTest    notClass:      - com.example.android.testing.androidjunitrunnersample.CalculatorInstrumentationTest    size: small    package: com.example.android.testing.androidjunitrunnersample    notPackage: com.example.android.testing.androidMyDemoTests    annotation: com.android.buzz.MyAnnotation    notAnnotation: com.android.buzz.NotMyAnnotation    numShards: 4    clearPackageData: true    useTestOrchestrator: true

class#

| OPTIONAL | ARRAY |

Instructs saucectl to only run the specified classes for this test suite.

  class:    - com.example.android.testing.androidjunitrunnersample.CalculatorAddParameterizedTest

notClass#

| OPTIONAL | ARRAY |

Instructs saucectl to run all classes for the suite except those specified here.

  notClass:    - com.example.android.testing.androidjunitrunnersample.CalculatorInstrumentationTest

size#

| OPTIONAL | ENUM |

Instructs saucectl to run only tests that are annotated with the matching size value. Valid values are small, medium, or large. You may only specify one value for this property.

  size: small

package#

| OPTIONAL | STRING |

Instructs saucectl to run only tests in the specified package.

  package: com.example.android.testing.androidjunitrunnersample

notPackage#

| OPTIONAL | STRING | REAL DEVICES ONLY |

Instructs saucectl to run run all tests except those in the specified package.

  notPackage: com.example.android.testing.androidMyDemoTests

annotation#

| OPTIONAL | STRING |

Instructs saucectl to run only tests that match a custom annotation that you have set.

  annotation: com.android.buzz.MyAnnotation

notAnnotation#

| OPTIONAL | STRING |

Instructs saucectl to run all tests except those matching a custom annotation that you have set.

  notAnnotation: com.android.buzz.NotMyAnnotation

numShards#

| OPTIONAL | INTEGER |

Sets the number of separate shards to create for the test suite. Read more about shard tests on the Android developer site.

When sharding is configured, saucectl automatically creates the sharded jobs for each of the devices defined for the suite based on the number of shards you specify. For example, for a suite testing a single emulator version that specifies 2 shards, saucectl clones the suite and runs one shard index on the first suite, and the other shard index on the identical clone suite. For a suite that is testing 2 emulator version and two real devices, saucectl must clone the suite to run each shard index for each emulator and device, so 8 jobs in total for the suite.

note

Espresso may not distribute tests evenly across the number of shards specified, especially if the number of shards is near or equivalent to the number of tests in the suite. In such cases, it is not unusual to see jobs with no tests at all because they were already executed in other shard jobs.

  numShards: 2

clearPackageData#

| OPTIONAL | BOOLEAN | REAL DEVICES ONLY |

Removes all shared states from the testing device's CPU and memory at the completion of each test.

note

The flag clearPackageData has to be used in conjunction with useTestOrchestrator.

  clearPackageData: true  useTestOrchestrator: true

useTestOrchestrator#

| OPTIONAL | BOOLEAN | REAL DEVICES ONLY |

Run each of your tests in its own Instrumentation instance to remove most of the app's shared state from the device CPU and memory between tests. Use this setting in conjunction with clearPackageData: true to completely remove all shared state.

When set, the instrumentation starts with Test Orchestrator version 1.1.1 in use. This property applies only to real devices, not emulators.

  useTestOrchestrator: true

Advanced Configuration Considerations#

The configuration file is flexible enough to allow for any customizations and definitions that are required for any of the supported frameworks. The following sections describe some of the most common configurations.

Setting up a Proxy#

If you need to go through a proxy server, you can set it through the following variables:

  • HTTP_PROXY: Proxy to use to access HTTP websites
  • HTTPS_PROXY: Proxy to use to access HTTPS websites
PS> $Env:HTTP_PROXY=http://my.proxy.org:3128/PS> $Env:HTTPS_PROXY=http://my.proxy.org:3128/
$> export HTTP_PROXY=http://my.proxy.org:3128/$> export HTTPS_PROXY=http://my.proxy.org:3128/

Try Espresso with Cucumber#

saucectl supports Espresso using Cucumber, and the Espresso demo repo includes an example Cucumber setup.