Live Mobile Application Testing

Mobile Real Device Testing#

With Sauce Labs, you can test your mobile applications on a variety of real Android and iOS mobile devices. If you do not have an app, consider using the Sauce Labs Swag Labs sample app for validating your account functionality as well as your tests.

What You'll Need#

Uploading an App#

You can upload your app via the Sauce Labs UI or via the REST API. For information about uploading via the API, see Upload Files with the REST API.

To upload an app via the Sauce Labs UI:

  1. In Sauce Labs, in the left panel, click LIVE and then click Mobile App.
  2. Click App Upload. You can either drag and drop an application, or browse for and select the file. We currently support APK and IPA files up to 4 GB. Non-app file uploads are not supported in the UI at this time, but can be uploaded through the API.
Upload an application

NOTE: The Sauce Labs UI currently supports live testing on real devices only. To test on virtual devices, use the REST API.

Deleting an App#

Deleting an app in Sauce Labs will delete the whole application (i.e., the group of builds belonging to the same app package). Files associated with app identifiers (i.e., belonging to the same platform and accessible to the same team) are indicated by the + symbol next to version number. Also, the version number shown is the most recently updated file, not necessarily the latest version of the application.

To delete an application, on the Mobile App test page, hover over the test and then click Delete.

Delete an application

App Settings#

To view or change the application settings, on the Mobile App test page, hover over the test and then click Settings.

Application settings

To easily copy a test's file name or ID, hover over the test and then click the clipboard icon.

Copy a file name or ID

Default App Settings

SettingDescription
Device LanguageUse the dropdown to select the device language.
Device OrientationUse the dropdown to set the device orientation (Landscape or Portrait).
ProxyEnable/disable the use of a proxy. Enter the Hostname and Port and then click Update.
InstrumentationEnable/disable device instrumentation. Enabling allows you to use advanced features when testing your app in the real device cloud, like image injection and taking screenshots of secure views.
Image InjectionEnable/disable image injection. Image injection allows you to mimic camera behavior when testing applications by letting you upload an image and present it to the application as if it were read by the device camera.
Bypass Screenshot Restriction

ANDROID ONLY

Enable/disable Bypass Screenshot Restriction (not supported on applications uploaded to the legacy sauce storage). Enabling Bypass Screenshot Restriction allows you to take screenshots of your app during tests, even if your app does not allow screenshots for security reasons.
System Alerts Display

iOS Only

Enable/disable a system alerts delay. Enabling delays alerts, such as asking for permission to access the camera, to prevent app crashes at startup.
Biometrics Interception

iOS Only

Enable/disable biometrics. Enabling allows you to choose authentication options if your mobile app requires a biometric authentication, such as Touch ID or Face ID.
Group Folder Redirect

iOS Only

Enable/disable a group directory redirect. Enabling allows you to use your app's private app container directory instead of the shared app group container directory. When your app gets resigned, the shared directory is not accessible.

NOTE: Any changes you make to the application settings will affect all uploaded versions of the application.

Example Settings - iOS

Application settings - iOS

Example Settings - Android

Application settings - Android

Most settings update automatically, however, when you make changes to the proxy setting, click Update to finish.

Selecting a Real Device#

You must select a device prior to launching a session.

  1. On the App Selection page, hover over the app you want to test and then click Choose Device.
Choose a device
  1. Use the search box and filters to find the device you want to test on, or select the device in the grid.

Public vs. Private Devices#

There is a distinction between Public Devices and Private Devices.

  • Public devices are accessible by anyone with a Sauce Labs account and are subject to availability. If a device is in use, you will see a yellow Busy flag across the thumbnail.
Busy public device
  • Private devices are associated with your account and are an enterprise only feature. Private devices are indicated by a person icon.

Launching a Test#

You can launch a test from the following screens:

  • Hover over the device in the grid and then click Launch.

  • Hover over the device in the grid and then click Details. On the Details screen, click Launch.

    Launch a test from the Details screen

    You'll see a loading screen, and then the app will launch in a live test window using the device you selected.

    Mobile real device test interface

Live Test Interface#

IconNameDescription
Unpin/Pin Toolbar iconUnpin/Pin ToolbarUnpins or pins the live testing toolbar.
Session Info iconSession InfoOpens the Current Session window, which includes app and device details.
Take Screenshot iconTake ScreenshotTakes a screenshot of the current device screen. The image downloads automatically as a .png file.
Share Session iconShare SessionOpens the Share Device window. For a sharable link to the device, click Get Link.
NOTE: Users must be logged in to be able to view the test.
Rotate Device iconRotate DeviceRotates the device between portrait and landscape.
Home iconHomeOpens the device home screen.
More Device Options iconMore Device OptionsSet Location - Set the GPS location using coordinates or by dropping a pin on the map.
Camera Injection - Opens the Camera Injection window. See Camera Image Injection for more information.
Biometric Injection - Opens the Biometric Authentication window.
Restart App iconRestart AppRestarts the app.
Switch App Version iconSwitch App VersionOpens the Switch App Version window. To change the version of the app you are testing, hover over the version and then click Choose version.
Clipboard iconClipboardOpens the Paste Content Into Device window.
Install Dependency iconInstall DependencyOpens the Install Dependent App window.
Developer Options iconDeveloper OptionsOpens the Developer Options panel, which includes the Device Log, Device Vitals, and Dev Tools tabs.

Device Log#

Device Log
IconNameDescription
Log LevelLog Level
  • VERBOSE
  • DEBUG
  • INFO
  • WARN
  • ERROR
  • ASSERT
Search Log iconSearch LogOpens the Search log interface. Enter a term or terms in the search box and select or deselect the following checkboxes as necessary:
  • REGEX
  • IGNORE CASE
  • INVERT
Pause Log iconPause LogPauses the log feed.
Wrap Text iconWrap TextWraps text in the log for easier reading.
Clear Log iconClear LogClears the log feed.
Download Log iconDownload LogDownloads the log as a .txt file.

Device Vitals#

Device Vitals is a feature that collects useful performance data in real time from a device during a live session. Data such as network, CPU, and memory usage helps users understand the general performance of a device and the application under test. Users can view a graph of this performance data in real time as the app is processing.

Device Vitals
IconNameDescription
Graph HeightGraph HeightChange the size of the graphs (S, M, L).
Grid Layout iconGrid LayoutDisplays graphs side by side.
Row Layout iconRow LayoutDisplays graphs as as a list.
Download Vitals iconDownload VitalsDownloads the device vitals as a .txt file.

Performance Metrics for Android/iOS Devices

The graph and csv file will contain these performance metrics for devices.

MetricDescription
cpu_totalSystem-wide CPU usage in percentage across all CPU cores. 4 cores at max use would be shown as a value of 400%
cpu_userCPU usage for user processes in percentage across all CPU cores. 4 cores at max use would be shown as a value of 400%
cpu_kernalAndroid OS CPU usage in percentage across all CPU cores. 4 cores at max use would be shown as a value of 400%
n_threadsTotal threads in use by the app
memory_size_kbTotal threads in use by the app
memory_resident_kbMemory currently in use by application in kilobytes
memory_shared_kbAnonymous shared memory currently in use by system shared between application(s) and system
network_wifi_receive_bData in bytes received over wifi connection
network_wifi_sent_bData in bytes sent over wifi connection
network_mobile_receive_bData in bytes received from the mobile carrier network
network_mobile_sent_bData in bytes sent over mobile carrier network

NOTE: Device Vitals for live testing is currently in beta state, which means we are testing it on a few devices first. We will roll out to all devices in the coming weeks, please refer to this page for updates. See the full list of combinations available for beta below.

App Tests

Device/PlatformiPhone 11iPhone XRiPhoneXSiPhone XiPhone 8iPhone 7iPhone 6iPhone 6 PlusiPhone 6S PlusiPhone 5SiPhone SEiPad Pro 11 2018iPad ProiPad Air 2019iPad 9.7 2017iPad 4
iOS 9.3.2
iOS 10.0.2
iOS 10.1
iOS 10.3.3
iOS 11.4
iOS 11.4.1
iOS 12.2
iOS 12.4.1
iOS 13.0
iOS 13.1

Changing an App Version#

Sometimes you need to conduct A/B testing, or document and validate feature parity between different versions of the same application. You can change the app version, as well as the real device, and launch a new test session.

  1. On the App Upload page, click the +n in the Version column.
App with multiple versions
  1. On the Settings page, in the versions list, hover over the version you want to launch.
  2. Click Choose Device.
Change the version of an app

Testing Apple Pay in Mobile Apps#

There are three ways to test Apple Pay with Sauce Labs:

  • Using simulators
  • Using real private devices with an Apple Pay Sandbox Testing account
  • Using real private devices with a real production account and real credit cards

Apple Certificates#

Apple certificates are used to ensure security in their systems, and they are much more strict about them than Android. This level of security makes certificates a very complex part of making Apple Pay work with devices in a cloud.

To give you an example, Android apps can be installed without any specific signing on whatever real device you want. With Apple you have two options, or you need to add a remote device to your developer certificate and the provisioning profile, so you are allowed to install the app on that specific device. Or you need to use an enterprise certificate where the Apple device that has that certificate installed allows you to install the app. Similarly, when you install an iOS app on a device, we re-sign the app with a Sauce Labs enterprise certificate so you can install your app on all Sauce Labs public/private devices.

NOTE: Apple Pay has a limitation that it cannot work with an enterprise certificate. You need to use the developer certificate where the device has been added to the provisioning profile in order to make this work. This can only be done for Sauce Labs private devices on which you have disabled the resigning.

Apple Pay on Real Private Devices#

To make Apple Pay work on Sauce Labs real private devices:

  1. Follow Apple’s steps to enable Apple Pay (see Setting Up Apple Pay Requirements). Apple is strict about certificates, so they require you to follow very specific steps:

  2. Set up Apple Pay integration in your app.

  3. Register the Merchant ID in your Apple developer account.

  4. Set up an Apple sandbox tester account (see Create a sandbox tester account for more information).

  5. Build your app. Apple Pay doesn’t work with enterprise certificates, so it will not work with Sauce Labs out of the box. The first step is to add the Sauce Labs real private devices to your Apple developer certificate before building the app. You can do that in one of the following ways:

  6. Manually adding the device and its UDID to the device list for your developer certificate. NOTE: Your device list can be found on Apple’s Certificates, Identifiers & Profiles page for your developer account, and you can get the UDID of your private device by contacting your Sauce Labs CSM.

  7. Using the Sauce Labs Virtual USB solution:

    1. Start a session with Virtual USB (see Testing with Virtual USB on Real Devices for more information).

    2. When the connection is established, open XCODE.

    3. Select the device from the device list.

      Apple Pay setup - device list

      On the Signing & Capabilities tab, you will see that the device has not yet been added.

      Apple Pay setup - device not added
    4. Click Register Device to add the device to your developer certificate.

      Apple Pay setup - add device to certificate
    5. Once the UDID of the device is added to the developer certificate, you can build the application (manually or automatically):

      1. Select your build scheme and then select Generic iOS Device.
      2. To build the application, click Product and then click Archive.
      3. Click Distribute App.
      4. Distribute the application with Ad Hoc and Automatically manage signing.
      5. Store the application on your local machine.

    If the application has been built, you should not yet upload it to Sauce Labs. The device to be tested needs to be prepared. If you have already prepared the device, then you can skip to step 4.

  8. Prepare the device. Set up the first Sauce Labs private device to use Apple Pay with the Apple sandbox account that was created in step 1.

Disable the Passcode#

Apple Pay requires that you have set a passcode on your phone, and you can’t add cards to your wallet without it. But setting a passcode on a device can break Appium automation because Appium can’t automate the passcode screen. To prevent the testing device from displaying the passcode screen:

  1. On the device, go to Settings > Display & Brightness and disable Auto-Lock.
  2. Ask your CSM to disable rebooting the device by providing them with the unique device name, found in the device details. NOTE: There is no guarantee that the device won’t reboot or show the passcode screen. The test run on the device might be less reliable if the passcode screen appears during the automated session.

Add the Testing Account#

  1. On the device, go to Settings and then click Sign in to your iPhone. Sign in using your Apple sandbox tester account.
Apple Pay setup - sign in to account
  1. If prompted, enter the device’s passcode.
Apple Pay setup - passcode

If you weren’t prompted for a passcode, set it by going to Settings > Face ID & Passcode and tapping Turn Passcode On.

Add Apple Sandbox Test Cards#

Apple test cards can be found on Apple’s Sandbox Testing page.

  1. On your device, go to Wallet. If you didn’t set a passcode, Apple will show a notification.
Apple Pay setup - passcode notification
  1. In Wallet, tap the plus sign to add a new card. Use the card information on Apple’s Sandbox Testing page.
Apple Pay setup - Add new card
  1. Prepare Sauce Labs. As mentioned before, Sauce Labs uses an enterprise certificate to install an app on public and private devices. But Apple Pay can’t work with the enterprise certificate, so the app needs to be signed with the developer certificate. You need to instruct Sauce Labs to not re-sign the app when it is installed.

Disable Re-Signing#

  1. In Sauce Labs, in the left navigation, click Live and then click Mobile-App.
Apple Pay setup - Sauce login

You will see an overview of the already uploaded apps. If no app has been uploaded, then upload the app. Once uploaded, open the app settings by hovering over the row until you see this:

Apple Pay setup - Settings
  1. Click Settings.
Apple Pay setup - Settings
  1. Under Default settings, toggle Instrumentation to Disabled.
Apple Pay setup - Disable instrumentation

Disabling this allows the app to use Apple Pay and the developer certificate and provisioning profile that you used when you built the app.

NOTE: Disabling re-signing will break the installation of the app on public devices. The app will only be allowed to be installed on private devices that have been added to the developer certificate and provisioning profile.

Once the app has been uploaded and re-signing has been disabled, you can start the device and let Sauce Labs install the application on the device.

  1. Test the app. View the Sauce Labs Demo Payments app:
Apple Pay setup - Demo appApple Pay setup - Demo appApple Pay setup - Demo appApple Pay setup - Demo app

Camera Image Injection#

Camera Image Injection is a core feature built into our RDC functionality and available for use with public and private devices. Your mobile app accesses the camera and instead of getting back the picture of the device camera, it'll retrieve your uploaded image for your test. You employ the built-in device camera in your live and automated testing and perform test cases that require taking images with any of the device cameras.

Key Specs#

Supported

  • All iOS and Android devices available in the RDC
  • Front-facing and rear-facing system device cameras
  • Image file sizes up to 5MB
  • JPG, JPEG, PNG image file formats

Not Supported

  • Ephemeral apps (i.e., app with temporary messages that disappear after a certain timeframe)
  • Testing with emulators, simulators

What You'll Need#

You'll need to upload your app to Sauce Labs prior to testing (see Uploading an App).

Testing with Camera Image Injection#

  1. In Sauce Labs, click LIVE and then click Mobile App.
  2. On the App Selection test page, hover over the test and then click Settings.
Mobile app settings navigation
  1. On the Settings page, ensure that Image Injection is enabled and then return to the App Selection page.
  2. On the App Selection test page, hover over the test and then click Choose Device.
  3. On the device selection page, hover over a device and then click Launch.
  4. When you want to capture an image of the test, in the right toolbar, click the Camera icon.
  5. Click Choose Image and navigate to the image you want to use.
Camera image injection navigation
  1. Activate the camera inside of your app. The device will show your uploaded image in the app as if the image was taken by the device camera. The image will continue to be available, should you go back to the camera during your test session, or you can upload another image and capture it with the camera.
Last updated on by Kim