The Speedo Node JS package is a tool that allows you to enter a simple command from a command line to measure basic performance of your Sauce app and validate any regressions based on previously executed tests. Speedo also easily integrates with any CI/CD system, allowing you to plug Sauce Performance into your CI/CD pipeline and start capturing performance within seconds.
- Why the Speedo Node JS package is an easy way to get basic performance data about your app
- What's required to use Speedo
- How to install Speedo
- How to format Speedo commands to return performance metrics
- How to interpret the Speedo results
- How to use Speedo commands with your CI/CD pipeline
- Google Chrome (no older than 3 versions from latest) as the test browser
- Node.js v8 or later (for NPM installations)
- SAUCE_USERNAME and SAUCE_ACCESS_KEY defined for your environment
You can install the Speedo package for Sauce using NPM or Docker.
Install the package on your system to make the speedo command available.
Pull the speedo package and run the installation command using Docker.
Sauce Performance supports the following Speedo commands:
- speedo run - Get the set of standard performance metrics for any pre-defined URL
- speedo analyze - Validate the performance of URLs that are accessed by one of the previously run WebDriver tests.
NOTE: The Speedo commands require your SAUCE_USERNAME and SAUCE_ACCESS_KEY. Export the values into your environment in order to avoid passing them as parameters each time you call a Speedo command. For example:$ export SAUCE_USERNAME=Slavocado$ export SAUCE_ACCESS_KEY=XXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX\
The Speedo run command initiates a series of tests on Sauce Labs against a single URL defined in the query and returns the corresponding performance results. As part of the execution, the command facilitates the following functions:
- Establishes a baseline for each metric upon initial instantiation by running a diagnostic 10 times
- Downloads the performance logs as test artifacts, the local directory of which is provided in the command line output
Stored performance logs in /var/folders/11/p0wfqdkd4wgct7jdpfzxk4j40000gn/T/tmp-8379w4yCSzRBXqN2
- Automatically updates the job status to PASS or FAIL based on previously established baseline
- Outputs reference URLs to the job in the Sauce Labs UI and the Google Lighthouse report
- Exits with a proper exit code so that your pipeline can potentially block the release of your web application in the event a performance regression was introduced
$ speedo run <url> [params]
Kick off a Speedo test by calling:
The output returned for a passing test may look as follows:
The Speedo analyze command allows you to compare the performance of multiple URLs that are accessed by a test script performed using automation tool like Selenium or WebdriverIO. Once the test completes, run the analyze command, specifying the name of the test as an inline attribute, to evaluate the page performance for each of the URLs accessed during the test. For example, a Login test would include each of the URL pages associated with successful authentication.
$ speedo analyze "<test name>"
NOTE: The specified test must have the appropriate options set for Performance.
You can apply parameter specifications to customize the test execution either by including the parameters as inline arguments or through a config file (speedo.config.js) located in the directory from which Speedo is called. Following is a list of common inline arguments. NOTE: Call $ speedo analyze -h to view the complete list of run command options.
Before you analyze the performance of page loads in an automation script, make sure you have a named test that has run to completion.
The following example is based on a WebdriverIO test in which a user logs into Instagram and successfully accesses the home screen.
The following screenshot shows an example of the returned output for the analysis of the automation test.
The Speedo package is designed to organically fit into your existing continuous integration and delivery pipeline, as illustrated in the following examples.
- Jenkins with Docker
- GitLabs with Docker
This example uses a
sauce attribute to pass the Sauce Labs credentials into the environment.
This example defines a Speedo Docker container and validates that Speedo is available.
By default, Speedo downloads a log containing all the recorded metrics for your test in a local file, the path to which is provided in the output of the call. If you prefer to specify a different location, use the
--logDir (-l) parameter. You can also include the
--traceLogs (-t) parameter to generate a log with trace level details in addition to the standard log in the same directory location. The following example is highlighted to show the parameter settings in the command and the log location in the output.
Once the test has completed, you can access the data for further use, such as pushing it to a third part analysis service.
The Speedo CLI tool supports a high degree of customization to ensure you are only capturing data that is relevant to your organization. You can customize the execution of either Speedo command by passing applicable parameters:
- as inline command arguments to customize only the current command execution
- through a config file (speedo.config.js) located in the directory from which Speedo is called to apply the customization every time the command is called
|Run||Identify your performance test with a build ID.|
|Run||Identify your performance test with a name.|
|Both||Pass your Sauce Labs access key to authenticate.|
|Both||Pass your Sauce Labs username to authenticate.|
|Analyze||Identify the Sauce Labs data center for your account. Valid values include |
|Both||Record trace level data in addition to the default log. If specified, the trace.json file is saved to the same folder as the performance log. If not specified, trace level data is not captured.|
|Both||Specify a local directory in which to save the performance log file (performance.json). If not specified, the log file is written to a default location in the directory in which you installed Speedo and the path is provided in the output of the call.|
|Analyze||The number of pages accessed during the test.|
|Analyze||If only measuring performance for one page in the test, specify the URL of the page to analyze.|
|Run||Specify the platform on which to run the test. If not specified, the default value is Windows 10.|
|Run||Specify the version of Chrome (only browser supported at this time) on which to run the test. If not specified, the default value is latest, which resolves to the most current version.|
|Both||Tailor the command to measure performance for only certain metrics. See Metric Values for the list of supported metrics. If not specified, the test defaults to |
|Run||Tailor the metric measurement standards based on the CPU at which the pages are being accessed. See CPU Settings for the list of supported values for this parameter. If not specified, the test defaults to 4 (4X).|
|Run||Tailor the metric measurement standards based on the specific network environment in which the pages are being accessed. See Network Conditions Settings for the list of supported network profile values for this parameter. If not specified, the test defaults to Good 3G.|
|Analyze||Set this boolean to |
The following values can be used with the
-m parameter of either the
|An aggregate of unexpected movement of content as a page loads. The score is calculated as the percentage of space impacted by the movement times the percentage of distance the content moved on the screen.||percentage|
|The point at which visual content is fully rendered and backend scripts begin to execute.||seconds|
|The amount of time the page takes to respond to user input.||milliseconds|
|The time from when the page starts loading to when any part of the page's content is rendered on the screen. In this context, "content" can be text, images, elements, or canvas (non-white) elements. This does not mean that the page is fully rendered.||seconds|
| The amount of time it takes for a page's primary body of content to rendeR. This metric is replaced by ||seconds|
|The time it takes to render the first pixel on the page once the URL has been called.||seconds|
|The time it takes for anything to be visually painted in the viewport. Calculated by video analysis, this is an alternative metric to firstPaint that is browser agnostic.||seconds|
|The amount of time it takes for the page's largest visual element to display. This metric is considered a more accurate reflection of when the main content of a page has loaded.||seconds|
|The amount of time it takes for the final visual element to display.||seconds|
|The amount of time it takes for all page objects and dependent resources to be loaded.||seconds|
|The average time is takes the contents of a page to fully render.||seconds|
|The amount of time it takes for a page to be able to reliably respond to user input.||seconds|
|The amount of time that elapses between ||seconds|
The following profiles can be used with the
--throttleNetwork parameter of the
run command to simulate various network conditions for your test.
|300ms||31.25 KB/s||6.25 KB/s|
|100ms||93.75 KB/s||31.25 KB/s|
|150ms||56.25 KB/s||18.75 KB/s|
|40ms||192.00 KB/s||93.75 KB/s|
|20ms||512.00 KB/s||384.00 KB/s|
|5ms||256.00 KB/s||128.00 KB/s|
|2ms||3.75 MB/s||1.88 MB/s|
You can configure your test to run under the network condition of a predefined profile from the table, as in the following example:
$ speedo run https://www.saucelabs.com --throttleNetwork online
Alternatively, you can specify custom values in the format
download, upload, latency (in KB/s):
$ speedo run https://www.saucelabs.com --throttleNetwork "1000,500,40"
The following profiles can be used with the
--throttleCpu parameter of the
run command to simulate various load conditions for your test.
|2||2x CPU throttling|
|3||3x CPU throttling|
|4||4x CPU throttling (default)|
Emulates mobile performance
$ speedo run https://www.saucelabs.com --throttleCpu 2