Appium Testing with Real Devices

This topic describes automated Appium testing on the Sauce Labs Real Device Cloud (RDC). You can accelerate your test execution by running parallel automated tests across thousands of mobile device/OS combinations, in our public real device cloud and/or a private device pool of your own.

What You'll Need#

  • Your Sauce Labs username and access key.
  • Ensure that your mobile app and project setup meet our real device cloud requirements.
  • Have your mobile app file (.ipa for iOS, .apk for Android) and mobile test file on hand.

If you don't have an app and would like to try out our test functionality, feel free to download and use our Sauce Labs demo app.

Uploading and Accessing Your Apps on Real Devices#

Select your preferred testing platform for uploading your app in order to view the instructions.

Installing Mobile Apps from a Remote Location

There may be situations where you want to install an app from a downloadable remote location (e.g., AWS S3 bucket, a GitHub repository). The app is completely removed from the real device after the test completes, providing an added layer of security for your app.

Please review the following guidelines below before uploading your app:

  1. Make sure your app meets the requirements for Android and iOS Mobile App Testing.
  2. Upload your app to the hosting location.
  3. Ensure Sauce Labs has READ access to the app URL.
  4. In your Appium test script, enter the URL for the app as the "app" desired capability. Example Java Snippet:
    caps.setCapability("app", "https://github.com/saucelabs/sample-app-mobile/releases/download/2.3.0/Android.SauceLabs.Mobile.Sample.app.2.3.0.apk?raw=true");

Installing your App on Private Devices

In some cases, you may need to upload / install your app to a private device and also prevent the device from broad internet access while under test. The steps to achieve this are:

  • Upload your app to an internal git repository, or private hosting solution with the necessary permissions (e.g. Amazon S3 with a strict bucket policy).
  • Ensure the hosted app URL is available to the machine running the automated test.
  • Ensure that you've enabled the Require Sauce Connect/VPN setting in your organization's security settings.

NOTE: Each session is a "fresh" installation of your app, meaning, you will not be able to access information about previous versions of your app.

Configuring Appium Tests for Real Devices#

This section describes parameters that we recommend using to configure your Appium tests on Sauce Labs real devices. See Appium for Real Devices CLI Reference for the full list of required and optional parameters.

Please Read

Certain Appium capabilities behave differently when running Appium tests on our RDC versus a local Appium server. On our RDC:

  • Some Appium capabilities are not supported.
  • Emulator-only capabilities will not work.
  • The app capability will be always be overwritten; it will point to the app file you uploaded to our system.
  • The noReset capability will only work if device caching is enabled.
  • Different setups may have different ways of handling capabilities and/or different requirements. Check to make sure you're providing all of the required capabilities.

Set appiumVersion#

If you omit the appiumVersion in your test configuration, your test will be running with our default Appium version. We recommend that you specify one of the newer Appium versions that provides a more extended API and fixes to known bugs.

Check the Appium Version for Your Test#

  1. Log in to Sauce Labs.
  2. Go to Test Details.
  3. Find and select the test that you ran using Appium.
  4. Click the Metadata tab.
  5. Look for the Logs row and select Appium Log. The first line should indicate the Appium version. For example, 2019-05-05T17:45:07.541Z - info: Welcome to Appium v1.10.1.

Set the browserName#

When testing a native mobile app, the value for browserName is an empty string, as in caps.setCapability("browserName", "");.

Set the Location of Your Mobile App#

If the app you want to test has been uploaded to a location other than Sauce Storage, you need to specify this location for app, and make sure that this location is accessible to Sauce Labs browsers. For example, caps.setCapability("app","sauce-storage:mapp.zip");.

Set the automationName for Android Apps#

If you're testing a native mobile app against Android versions 4.0-4.1, or a hybrid mobile against Android versions 4.0 - 4.2, you need to set the capability "automationName","selendroid".

These Android versions are only supported via Appium’s bundled version of Selendroid, which utilizes Instrumentation. Later versions of Android are supported via Appium’s own UiAutomator library.

Example Configuration Code Snippets#

Example: Appium capabilities for an iPhone project using iOS version 12.2.

DesiredCapabilities caps = DesiredCapabilities();
caps.setCapability("username", "SAUCE_USERNAME");
caps.setCapability("accessKey", "SAUCE_ACCESS_KEY");
caps.setCapability("deviceName","iPhone .*");
caps.setCapability("deviceOrientation", "portrait");
caps.setCapability("platformVersion","12.2");
caps.setCapability("platformName", "iOS");
caps.setCapability("browserName", "");
caps.setCapability("app", "storage:filename=<file-name>");

Example: Appium capabilities for an iPhone project using iOS version 12.2.

DesiredCapabilities caps = DesiredCapabilities();
caps.setCapability("username", "SAUCE_USERNAME");
caps.setCapability("accessKey", "SAUCE_ACCESS_KEY");
caps.setCapability("deviceName","Samsung.*Galaxy.*");
caps.setCapability("deviceOrientation", "portrait");
caps.setCapability("browserName", "");
caps.setCapability("platformVersion","8.1");
caps.setCapability("platformName","Android");
caps.setCapability("app", "sauce-storage:<upload_filename>");

Native vs. Hybrid Apps#

iPhone 6 Real Device

{
deviceName:'iPhone 6 Device',
platformName:'iOS',
platformVersion:'8.4',
app:'sauce-storage:SampleIOSApp.ipa',
"appium-version":"1.4.16"
}

Samsung Galaxy S5 Real Device

{
deviceName:'Samsung Galaxy S5 Device',
platformVersion:'4.4',
platformName:'Android',
"appActivity": ".ContactManager",
"appPackage": "com.example.android.contactmanager",
app:"http://saucelabs.com/example_files/ContactManager.apk",
"appium-version":"1.4.16"
}

Samsung Galaxy S4

{
deviceName:'Samsung Galaxy S4 Device',
platformVersion:'4.4',
platformName:'Android',
"appActivity": ".ContactManager",
"appPackage": "com.example.android.contactmanager",
app:"http://saucelabs.com/example_files/ContactManager.apk",
"appium-version":"1.4.16"
}

Dynamic Device Allocation#

Dynamic Allocation involves providing basic parameters for the platform and operating system, or the type of device you want to use in your tests, and a device with those specifications is selected from the device pool. While static allocation allows you more fine-grained control over the device used in your tests, it can also cause delays in your test execution if that device isn't available when you run your tests. If you only need to test on a particular platform and OS version, such as an Android 4.1, or on a particular type of device, you should use dynamic allocation, and we recommend that you use dynamic allocation for all automated mobile application testing in CI/CD environments.

Required Capabilities for Dynamic Allocation#

CapabilityCapability Explanation
platformName

Defines the type of mobile platform to use in your tests (i.e., Android or iOS). The values for capabilities are not case-sensitive, so android is the same as Android, and ios is the same as iOS.

platformVersion

The platform version to use in your tests, for example "4" or "4.1". This is a substring match. You can specify both major versions and incremental versions of an operating system.

For example, if you set only a major version 4, you also have access to all devices running incremental versions (e.g., "4.1"," 4.2", "4.2.1", "4.4.4").

This also extends to minor and point versions. For example, if you specify "11.4", it will match "11.4.0", "11.4.1".

deviceName

The display name of the device to use, such as "Samsung S7". You can also use regular expressions for setting the deviceName. Some examples:

To allocate any iPhone:

"iPhone.*", "iPhone .*"

To allocate any device with the word "nexus" in its display name.

".*nexus.*"

To allocate either "iPhone 7" or "iPhone 6" device.

"iPhone [67]" or "iPhone [6-7]"

To allocate either "iPhone 7S" or "iPhone 6S" device.

"iPhone [67]S" or "iPhone [6-7]S"

To allocate "iPhone 7" or "iPhone 7S", or any device that starts with the display name "iPhone 7".

"iPhone 7.*"

NOTE: Regular expressions are not case sensitive.

Optional Capabilities for Dynamic Allocation#

CapabilityDescription
privateDeviceOnly

If you have access to both private and public devices, you can request allocation of private devices only with "true".

tabletOnly

Select only tablet devices with "true".

phoneOnly

Select only phone devices with "true".

carrierConnectivityOnly

Only allocate devices if they are connected to a carrier network with "true".

Static Device Allocation#

With Static Allocation, you can specify the device to use in your tests, but if that device is not available when you run your tests, it will delay test execution. For this reason, you should always make sure that the device you want to use is available before launching your tests. The platformName and platformVersion capabilities will be set by default, and if these are included in your test script, they will be ignored.

CapabilitySetting
deviceNameThe ID of the device you want to use in your test, for example LG_Nexus_5X_real. You can find the ID for a device by locating it in the real device selection menu of the Sauce Labs application, and then click the Details link for the device.

Device Caching#

cacheId Capability#

By default, every time you complete a test session, the real device cloud uninstalls your app, performs device cleaning, and de-allocates the device. This means that if you're running multiple tests on the same device, you would need to wait for this cleaning process to complete between every test.

To get around this, you can use the capability cacheId, which keeps the device allocated to you for 10 seconds after each test completes. If you immediately start another test on the device, you won't need to wait for the allocation and device cleaning process to be repeated. In this case, no device cleaning will take place in between sessions, with the only exception being the application under test and the data it owns.

CapabilitySetting
cacheId

A random string. This value for cacheId must be the same for all test methods that you want to run on the cached device. In addition, the application and project ID used for the tests must remain the same, along with the values for these capabilities:

  • deviceName
  • platformName
  • platformVersion
  • tabletOnly
  • phoneOnly
  • privateDevicesOnly
  • automationName
  • autoGrantPermissions
  • appiumVersion

Using Device Caching with noReset#

You can also use the cacheId capability in conjunction with the standard noReset Appium capability. In the default case, where noReset is set to false, your application will be uninstalled and reinstalled after every test. If noReset is set to true, the application you are testing won't be reinstalled after every test run. This might save you further time, but it won't be suitable for test setups that require the application's state to be reset between tests. Note that then cacheId is set, no device cleaning will take place in between sessions, regardless of noReset value.

NOTE: Our legacy Real Device Cloud platform used the capability testobject_cache_device – specific to static allocation – to keep the device allocated to you during the cleaning process. This capability has been deprecated and replaced with cacheId, which works for both static and dynamic allocation. If you have scripts that use testobject_cache_device, they will still work for static allocation, and the 10-second limit on cached devices is still the same.

Additional Test Configuration Options#

Once you're up and running with your real device tests, check out our Best Practices and Tips for making the most of your testing. Here are some examples:

Full Example Scripts#

These Appium script examples can help streamline your real device testing process. They use the pytest test framework. Feel free to clone these scripts directly from GitHub, and follow the instructions in the README file.

Visit our sample test frameworks GitHub repository for more detailed language-specific examples.

NOTE: For Example Purposes Only The code in these scripts is provided on an "AS-IS” basis without warranty of any kind, either express or implied, including without limitation any implied warranties of condition, uninterrupted use, merchantability, fitness for a particular purpose, or non-infringement. Your tests and testing environments may require you to modify these scripts. Issues regarding these scripts should be submitted through Sauce Labs GitHub. These scripts are not maintained by Sauce Labs Support.

Monitoring Real Device Performance for Appium Tests#

By including the desired capability recordDeviceVitals in your Appium test script, you can collect performance statistics for the real devices used in your tests, including CPU, Network, Memory, and Thread performance. This topic describes how to set up the desired capability in your tests, and collect the statistics when the test completes.

Setting recordDeviceVitals

Include this setting as part of your Appium test script capabilities:

capabilities.setCapability("recordDeviceVitals", true);

Collecting Performance Metrics

  1. When your test completes, log into the Real Device Testing web interface, and select the app that you used in the test from your dashboard.
  2. Click Automated Testing > Appium.
  3. Select the Test Results for your test.
  4. Under Mobile Vitals, click the link to download the CSV file containing your test performance metrics.

Performance Metrics for Android Devices

The CSV file will contain these performance metrics for Android devices.

MetricDescription
unix_epoch_millisecondsUnix epoch timestamp in milliseconds, which can be matched to events within your automated tests
device_local_time Device’s local date and time
cpu_totalSystem-wide CPU usage in percentage across all CPU cores. 4 cores at max use would be shown as a value of 400%.
cpu_userCPU usage for user processes in percentage across all CPU cores. 4 cores at max use would be shown as a value of 400%.
cpu_kernelAndroid OS CPU usage in percentage across all CPU cores. 4 cores at max use would be shown as a value of 400%.
n_processorsAmount of available CPU cores. Use this to divide the cpu values into per core
n_threadsTotal threads in use by the app
memory_size_kb Total memory currently used by device in kilobytes
memory_resident_kbMemory currently in use by application in kilobytes
memory_shared_kbAnonymous shared memory currently in use by system shared between application(s) and system
network_wifi_receive_bData in bytes received over wifi connection
network_wifi_sent_bData in bytes sent over wifi connection
network_mobile_receive_bData in bytes received from the mobile carrier network
network_mobile_sent_b Data in bytes sent over mobile carrier network

Performance Metrics for iOS Devices

The CSV file will contain these performance metrics for iOS devices.

MetricDescription
Unix_epoch_millisecondsUnix epoch timestamp in milliseconds, which can be matched to events within your automated tests
device_local_timeDevice’s local date and time
cpu_totalSystem-wide CPU usage in percentage. Values between 0 and 100%.
cpu_userUser processes CPU usage in percentage. Values between 0 and 100%.
cpu_processApp under test CPU usage in percentage. Values between 0 and 100%.
n_threadsTotal threads in use by the process
process_memory_used_bMemory used in bytes by the app that is under test
net_bytes_inTotal data in bytes received
net_bytes_outTotal data in bytes sent
net_packets_inTotal packets received
net_packets_outTotal packets sent
disk_write_opsDisk write operations during time period
disk_bytes_writtenBytes written to disk during time period

TestObject (Legacy RDC)#

warning

This information applies specifically to TestObject, our Legacy Real Device Platform.

For APIs and authorization credentials, use the Legacy Real Device Cloud REST API (app.testobject.com) and TestObject Data Center Endpoints.

Uploading Using TestObject#

Creating a Real Device Project on TestObject

When you create a project, you'll need to provide information about the website or app you want to test, and the device settings you want to use in your tests. You can also create versions of the project to reflect changes in the app or website throughout your development process.

Versioning Real Device Projects on TestObject

Once you've created a real device project, you can create versions of it. Each of these versions will be stored in your project, and you can run tests against the current Active version, or a previous version.

TestObject API Examples

Mac OSX / Linux

$ curl -u "$TEST_OBJECT_USERNAME:$TEST_OBJECT_API_KEY" -X POST \
"https://app.testobject.com:443/api/storage/upload" -H \
"Content-Type: application/octet-stream" --data-binary @/path/to/Android.SauceLabs.Mobile.Sample.app.x.x.x.apk

Windows

> curl -u "%TEST_OBJECT_USERNAME%:%TEST_OBJECT_API_KEY%" -X POST \
"https://app.testobject.com:443/api/storage/upload" -H \
"Content-Type: application/octet-stream" --data-binary @\path\to\Android.SauceLabs.Mobile.Sample.app.x.x.x.apk

Code Snippets#

These code snippets show how to configure Appium Tests for TestObject, our legacy real device cloud platform, which you can find under SAUCE APPS > Legacy RDC.

Example: iPhone project using iOS version 12.2.

DesiredCapabilities caps = DesiredCapabilities();
caps.setCapability("testobject_api_key", "project_api_key");
caps.setCapability("testobject_app_id", "1");
caps.setCapability("deviceName","iPhone .*");
caps.setCapability("deviceOrientation", "portrait");
caps.setCapability("platformVersion","12.2");
caps.setCapability("platformName", "iOS");
caps.setCapability("browserName", "");

Example: Samsung Galaxy project using Android version 8.1.

DesiredCapabilities caps = DesiredCapabilities();
caps.setCapability("testobject_api_key", "project_api_key");
caps.setCapability("testobject_app_id", "1");
caps.setCapability("deviceName","Samsung.*Galaxy.*");
caps.setCapability("deviceOrientation", "portrait");
caps.setCapability("browserName", "");
caps.setCapability("platformVersion","8.1");
caps.setCapability("platformName","Android");

Additional Resources#

:::Tip Sauce Labs provides access to our real device cloud and virtual device cloud (emulators, simulators) in the same place. This allows you to use the same features – APIs, endpoints, reporting, secure tunnels, analytics, and more – for both clouds. :::

Last updated on by Kim