Skip to main content

Configuring Your XCTest Tests for Real Devices

saucectl uses a YAML specification file to determine exactly which tests to run and how to run them. To customize saucectl for running your XCTest tests, adjust the properties of the YAML file as needed. This page outlines the configuration properties specific to running XCTest tests.

Utilize the following configuration at runtime to instruct saucectl to use any configuration file of your choice:

saucectl run -c ./path/to/{config-file}.yml
YAML Required

While you can utilize multiple files with different names or locations to specify your configurations, each file must be a *.yml and adhere to the saucectl syntax. Our IDE integrations (for example: Visual Studio Code) can assist you by validating the YAML files and offering suggestions.

Example Configuration​

.sauce/xctest_1.yaml
loading...

Each property supported for running XCTest tests through saucectl is defined below.

apiVersion​

| REQUIRED | STRING |

Identifies the version of the underlying configuration schema. Currently, v1alpha is the only supported version value.

apiVersion: v1alpha

kind​

| REQUIRED | STRING/ENUM |

Specifies the framework associated with the automation tests configured in this specification.

kind: xctest

showConsoleLog​

| OPTIONAL | BOOLEAN |

Controls whether the contents of console.log are always displayed in the local output of saucectl. By default (false), console.log is only visible for failed suites.

showConsoleLog: true

defaults​

| OPTIONAL | OBJECT |

Specifies any default settings for the project.

defaults:
  timeout: 15m

timeout​

| OPTIONAL | DURATION |

Specifies the duration (in ms, s, m, or h) that saucectl should wait for each suite to finish. If not configured, the default value is 0 (unlimited).

Real Device Max Duration

When setting the timeout values for your suites, note that native framework tests on real devices enforce a maximum test duration limit of 90 minutes.

defaults:
  timeout: 15m

sauce​

| OPTIONAL | OBJECT |

The parent property that encompasses all settings concerning how tests are executed and identified in the Sauce Labs platform.

sauce:
  region: eu-central-1
  metadata:
    tags:
      - e2e
      - release team
      - other tag
    build: Release $CI_COMMIT_SHORT_SHA
  concurrency: 5

region​

| OPTIONAL | STRING/ENUM |

Specifies through which Sauce Labs data center tests will run. Valid values are: us-west-1 or eu-central-1.

note

If you do not specify a region in your configuration file, you must set it with the -region flag when executing your command.

sauce:
  region: eu-central-1

metadata​

| OPTIONAL | OBJECT |

The set of properties that enable you to provide additional information about your project that helps distinguish it across the various environments in which it is used and reviewed. It also assists in applying filters to isolate tests based on metrics that are meaningful to you.

sauce:
  metadata:
    build: RC 10.4.i
    tags:
      - e2e
      - iPad
      - beta
      - featurex

concurrency​

| OPTIONAL | INTEGER |

Sets the maximum number of suites that can run simultaneously. If the test specifies more suites than the maximum, the excess suites are placed in a queue and executed in order as each suite completes.

caution

Set this value to be equal to or less than your Sauce concurrency allowance, as setting a higher value may lead to jobs being dropped by the server.

sauce:
  concurrency: 5

Alternatively, you can override the file setting at runtime by specifying the concurrency flag as an inline parameter for the saucectl run command:

saucectl run --ccy 5

retries​

| OPTIONAL | INTEGER |

Sets the number of attempts to retry a failed suite. Please refer to passThreshold for additional settings.

sauce:
  retries: 1

Alternatively, you can override the file setting at runtime by specifying the retries flag as an inline parameter of the saucectl run command:

saucectl run --retries 1

tunnel​

| OPTIONAL | OBJECT |

saucectl supports the use of Sauce Connect to establish a secure connection with Sauce Labs. To achieve this, launch a tunnel and then provide the name and owner (if applicable) in this property.

sauce:
  tunnel:
    name: your_tunnel_name
    owner: tunnel_owner_username

name​

| OPTIONAL | STRING |

Identifies an active Sauce Connect tunnel for secure connectivity to the Sauce Labs cloud.

note

This property supersedes the previous id property, which is deprecated.

sauce:
  tunnel:
    name: your_tunnel_name

owner​

| OPTIONAL | STRING |

Identifies the Sauce Labs user who created the specified tunnel, which is required if the user running the tests did not create the tunnel.

note

This property supersedes the previous parent property, which is deprecated.

sauce:
  tunnel:
    name: your_tunnel_name
    owner: tunnel_owner_username

launchOrder​

| OPTIONAL | STRING |

Specifies the sequence in which your test suites are executed. When set to fail rate, test suites with the highest failure rate will run first. If not specified, test suites will run in the order they are listed in the configuration file.

sauce:
  launchOrder: fail rate

reporters​

| OPTIONAL | OBJECT |

The parent property that encompasses all reporting capabilities provided by saucectl.

reporters:
  junit:
    enabled: true
    filename: saucectl-report.xml
    webhookURL: https://my-webhook-url

spotlight​

| OPTIONAL | OBJECT |

The spotlight reporter highlights failed or otherwise interesting jobs. It may include an excerpt of failed tests or additional information that could be useful for troubleshooting.

reporters:
  spotlight:
    enabled: true

junit​

| OPTIONAL | OBJECT |

The JUnit reporter collects JUnit reports from all jobs and merges them into a single report.

reporters:
  junit:
    enabled: true

json​

| OPTIONAL | OBJECT |

The JSON reporter collects test results from all jobs and merges them into a single report.

reporters:
  json:
    enabled: true

enabled​

| OPTIONAL | BOOLEAN |

Toggles the reporter on/off.

reporters:
  junit: # or json
    enabled: true

webhookURL​

| OPTIONAL | STRING |

Specifies the webhook URL. When saucectl test is finished, it'll send an HTTP POST with a JSON payload to the configured webhook URL.

reporters:
  junit: # or json
    webhookURL: https://my-webhook-url

filename​

| OPTIONAL | STRING |

Specifies the report filename. Defaults to "saucectl-report.json".

reporters:
  junit: # or json
    filename: my-saucectl-report.json

artifacts​

| OPTIONAL | OBJECT |

The parent property indicating how to manage test output, including logs, videos, and screenshots.

artifacts:
  cleanup: true
  download:
    when: always
    match:
      - junit.xml
    directory: ./artifacts/

cleanup​

| OPTIONAL | BOOLEAN |

When set to true, all contents of the specified download directory are cleared before any new artifacts from the current test are downloaded.

artifacts:
  cleanup: true

download​

| OPTIONAL | OBJECT |

Specifies the settings for downloading artifacts from tests run by saucectl.

artifacts:
  download:
    when: always
    match:
      - junit.xml
    directory: ./artifacts/

when​

| OPTIONAL | STRING |

Specifies when and under which circumstances to download artifacts. Valid values are:

  • always: Always download artifacts.
  • never: Never download artifacts.
  • pass: Download artifacts for passing suites only.
  • fail: Download artifacts for failed suites only.
artifacts:
  download:
    when: always

match​

| OPTIONAL | STRING/ARRAY |

Specifies which artifacts to download based on whether they match the provided name or file type pattern. Supports the use of the wildcard character * (use quotes for optimal parsing results with wildcard).

artifacts:
  download:
    match:
      - junit.xml
      - "*.log"

directory​

| OPTIONAL | STRING |

Specifies the path to the folder where artifacts will be downloaded. A new subdirectory is created in this location for each suite from which artifacts are downloaded. The subdirectory's name will correspond to the suite name. If a directory with the same name already exists, a serial number will suffix the new one.

artifacts:
  download:
    directory: ./artifacts/

XCTest​

| REQUIRED | OBJECT | Real Devices Only

The parent property containing the details specific to the XCTest project.

XCTest:
  app: ./apps/SauceLabs-Demo-App.app # This is an example app for real devices.
  appDescription: My demo app
  xcTestRunFile: ./apps/SauceLabs-Demo-App.xctestrun
  otherApps:
    - ./apps/SauceLabs-Demo-AppUI-Runner.app # If your test plan contains a XCUITest you need to provide it in the other apps.
    - ./apps/SomeOtherApp.app # An example of another app that might be part of your test plan.
caution

saucectl supports running XCTests on Real Devices only. Simulators only support XCUITest: You need to configure your devices via:

app​

| REQUIRED | STRING |

Specifies a local path, URL, or storage identifier for the app under test. This property supports expanded environment variables and accepts *.ipa, *.app, and *.zip file types.

XCTest:
  # Local paths
  ## Real Devices
  app: ./apps/XCTest/SauceLabs-Demo-Real-Device-App.app

# Remote locations
  ## Real Devices
  app: https://example.app.download.url/SauceLabs-Demo-Real-Device-App.app

  # Using an environment variable
  app: $APP

  # Storage identifiers
  app: storage:c78ec45e-ea3e-ac6a-b094-00364171addb

  ## Real Devices
  app: storage:filename=SauceLabs-Demo-App.app

appDescription​

| OPTIONAL | STRING |

Specifies the description for the uploaded app.

XCTest:
  appDescription: My demo app

otherApps​

| OPTIONAL | ARRAY |

If your test plan spans multiple app targets or includes XCUITests, please provide them here. Refer to app for additional app details.

note
  1. Apps specified as otherApps inherit the configuration of the main app under test for Device Language, Device Orientation, and Proxy, regardless of any differences that may be applied through the Sauce Labs UI, since the settings are specific to the device being tested. For instance, if the dependent app is designed to operate in landscape orientation while the main app is set to portrait, the dependent app will run in portrait during the test, which may lead to unintended consequences.
XCTest:
  otherApps:
    - ./apps/pre-installed-app1.app                           # This is an example app for real devices.
    - https://example.app.download.url/pre-installed-app1.app # This is an example app for real devices.
    - $PRE_INSTALLED_APP2
    - storage:8d250fec-5ecb-535c-5d63-aed4da026293
    - storage:filename=pre-installed-app3.app                 # This is an example app for real devices.

suites​

| REQUIRED | OBJECT |

The set of properties providing details about the test suites to run. May contain multiple suite definitions. Refer to the full example config for an illustration of multiple suite definitions.

name​

| REQUIRED | STRING |

The name of the test suite, which will be reflected in the results and related artifacts.

suites:
  - name: My Saucy Test

app​

| OPTIONAL | STRING |

Sets the test application at the suite level. Refer to the full usage. If this property is not set, saucectl will use the default app.

suites:
  - name: My Saucy Test
    app: ./apps/SauceLabs-Demo-App.app # This is an example app for real devices.

appDescription​

| OPTIONAL | STRING |

Specifies the description for the uploaded test app at the suite level. If app is not set at the suite level, saucectl will use the default appDescription.

suites:
  - name: My Saucy Test
    appDescription: My Demo app

timeout​

| OPTIONAL | DURATION |

Instructs how long saucectl should wait for the suite to complete, potentially overriding the default project timeout setting.

When the suite reaches the timeout limit, its status is set to '?' in the CLI. This does not reflect the actual status of the job in the Sauce Labs web UI or API.

note

Setting 0 reverts to the value set in defaults.

suites:
  - name: My Saucy Test
    timeout: 15m

passThreshold​

| OPTIONAL | INTEGER |

Specifies the minimum number of successful attempts for a suite to be deemed passed. It should be used in conjunction with retries.

note

For example, setting retries to 3 and passThreshold to 2 means the maximum attempts would be 4. If the test passes twice, it will stop and be marked as passed. Otherwise, it will be marked as failed.

sauce:
  retries: 3
suites:
  - name: My Saucy Test
    passThreshold: 2

smartRetry​

| OPTIONAL | OBJECT |

Specifies the retry strategy to apply to that suite. It should be used in conjunction with retries.

sauce:
  retries: 3
suites:
  - name: My Saucy Test
    smartRetry:
      failedOnly: true

failedOnly​

| OPTIONAL | BOOLEAN |

When set to true, saucectl collects any failed tests from the previous run and automatically retries them.

suites:
  - name: My Saucy Test
    smartRetry:
      failedOnly: true

failedClassesOnly​

| OPTIONAL | BOOLEAN |

failedClassesOnly is deprecated. Use failedOnly instead.

appSettings​

| OPTIONAL | OBJECT | Real Devices Only |

Application settings for real device tests.

suites:
  - name: My Saucy Test
    appSettings:
      instrumentation:
        networkCapture: true

resigningEnabled​

| OPTIONAL | BOOLEAN | Real Devices Only |

Enables resigning the app to allow it to run on RDC devices. By default, it is set to true. It must remain true for instrumentation features to function. It can be set to false for private devices to install the app on the device as is. This section describes the resigning enabled capability in more detail

suites:
  - name: My Saucy Test
    appSettings:
      resigningEnabled: true

instrumentation​

| OPTIONAL | OBJECT | Real Devices Only |

Instrumentation settings for real device tests.

suites:
  - name: My Saucy Test
    appSettings:
      instrumentation:
        networkCapture: true
vitals​

| OPTIONAL | BOOLEAN | Real Devices Only |

Configure app settings for real device to enable vitals.

suites:
   - name: My Saucy Test
     appSettings:
        instrumentation:
          vitals: true
networkCapture​

| OPTIONAL | BOOLEAN | Real Devices Only |

Record network traffic for HTTP/HTTPS requests during app tests on real devices.

suites:
  - name: My Saucy Test
    appSettings:
      instrumentation:
        networkCapture: true
biometrics​

| OPTIONAL | BOOLEAN | Real Devices Only |

Configure app settings for real devices to intercept biometric authentication.

suites:
   - name: My Saucy Test
     appSettings:
        instrumentation:
          biometrics: true
groupDirectory​

| OPTIONAL | BOOLEAN | Real Devices Only |

Configure app settings for real devices to enable group directory access.

suites:
   - name: My Saucy Test
     appSettings:
        instrumentation:
          groupDirectory: true
sysAlertsDelay​

| OPTIONAL | BOOLEAN | Real Devices Only |

Configure app settings for real devices to delay system alerts.

suites:
   - name: My Saucy Test
     appSettings:
        instrumentation:
          sysAlertsDelay: true
imageInjection​

| OPTIONAL | BOOLEAN | Real Devices Only |

Configure app settings for real devices to inject the provided images into the user app.

suites:
   - name: My Saucy Test
     appSettings:
        instrumentation:
          imageInjection: true

devices​

| OPTIONAL | OBJECT |

The parent property that defines how to select real devices on which to run the test suite. You can request a specific device using its ID or specify a set of criteria to choose the first available device that matches the specifications.

When an ID is specified, it supersedes the other settings.

caution

A device or a simulator is required to run your tests. You CANNOT combine them into a single suite.

suites:
  - name: My Saucy Test
    devices:
      - name: "iPhone 11"
        platformVersion: "14.3"
        options:
          carrierConnectivity: true
      - id: iPhone_11_14_5_real_us

id​

| OPTIONAL | STRING |

Request a specific device for this test suite by its ID. You can look up device IDs on device selection pages or by using our Get Devices API request.

suites:
  - name: My Saucy Test
    devices:
      - id: iPhone_11_14_5_real_us

name​

| OPTIONAL | STRING |

Find a device for this test suite that matches the device name or part of the name (Dynamic Device Allocation), which might offer a broader selection of available devices of the type you want.

Use Complete Name
suites:
  - name: My Saucy Test
    devices:
      - name: iPhone 11
Use Dynamic Allocation
suites:
  - name: My Saucy Test
    devices:
      - name: iPhone.*

platformVersion​

| MANDATORY for Virtual Devices | OPTIONAL for Real Devices | STRING |

Allows you to select the mobile OS platform version you wish to use in your test.

The platformVersion configuration is optional for real devices. There are three options available to determine which version you want to use for your automated Appium, Espresso, or XCTest tests:

  1. Do not provide a platformVersion; this will result in using any available Android or iOS device, regardless of the version.
  2. Provide a platformVersion that begins with the platformVersion string you provided:
    • 12: matches all minor and patch versions for platformVersion: "12". For example 12.1.0|12.1.1|12.2.0|...
    • 12.1: matches all patch versions for platformVersion: "12.1". For example 12.1.0|12.1.1 will not match 12.2.x|12.3.x and higher
    • 12.1.1: matches all devices that have this exact platform version
  3. Include or exclude a specific version and/or a range of versions using a regular expression (regex). You don't need to provide the forward slashes (/{your-regex}/) as you typically would with regular expressions. Remember that the regular expression must match the format MAJOR.MINOR.PATCH. The possibilities are endless; here are just a few examples:
    • ^1[3-4|6].*: This will match 13, 14, and 16, but not 15. See example.
    • ^(?!15).*: This will exclude version 15 along with all its minor and patch versions while matching all other versions. See example.
note

The stricter the platformVersions configuration, the narrower the selection of available devices will be, and the longer you may have to wait for a device. We recommend using only the major version or the regular expression option to achieve the best results and to access an available device more quickly.

Use complete version for Real Devices
suites:
  - name: My Saucy Test
    devices:
      - name: iPhone.*
        platformVersion: 14.3
Use complete version for Virtual Devices
suites:
  - name: My Saucy Test
    simulators:
      - name: iPhone 14 Pro Simulator
        platformVersion: 16.2
Use dynamic platformVersion allocation. Real Devices Only
suites:
  - name: My Saucy Test
    devices:
      platformVersion: '^1[3-4|6].*'

options​

| OPTIONAL | OBJECT | Real Devices Only |

A parent property that further specifies the desired device attributes within the selection of devices that match the name and version criteria.

carrierConnectivity​

| OPTIONAL | BOOLEAN | Real Devices Only |

Request that the matching device be connected to a cellular network.

suites:
  - name: My Saucy Test
    devices:
      - name: iPhone.*
        options:
          carrierConnectivity: true
deviceType​

| OPTIONAL | STRING | Real Devices Only |

Request that the matching device must be a particular type. Valid values are ANY, TABLET, or PHONE.

suites:
  - name: My Saucy Test
    devices:
      - name: iPhone.*
        options:
          deviceType: TABLET
private​

| OPTIONAL | BOOLEAN | Real Devices Only |

Request that the matching device be from your organization's private pool.

suites:
  - name: My Saucy Test
    devices:
      - name: iPhone.*
        options:
          private: true

env​

| OPTIONAL | OBJECT | Virtual Devices Only |

A property containing one or more environment variables that can be referenced in the tests for this suite. Expanded environment variables are supported. Values specified here will be overwritten by those set in the global env property.

  env:
    FOO: bar

Advanced Configuration Considerations​

The configuration file is flexible enough to accommodate any necessary customizations and definitions for the supported frameworks. The following sections outline some of the most common configurations.

Setting up a Proxy​

If you need traffic to pass through a proxy server, you can configure it using the following variables:

  • HTTP_PROXY: Proxy to use to access HTTP websites
  • HTTPS_PROXY: Proxy to use to access HTTPS websites
PS> $Env:HTTP_PROXY=http://my.proxy.org:3128/
PS> $Env:HTTPS_PROXY=http://my.proxy.org:3128/
$> export HTTP_PROXY=http://my.proxy.org:3128/
$> export HTTPS_PROXY=http://my.proxy.org:3128/