Set up suites

A suite in Tradefed refers to a setup where several tests are running under a common test runner that drives the overall execution.

In Tradefed, suites are driven through the ITestSuite class, which lets tests be added and removed independently of how they are run.

Definitions

  • Suite: Set of test modules configured to run under a similar top-level setup to report their results under a single invocation.
  • Top-level setup: Setup applied to the devices before running any of the test modules.
  • Main configuration: The suite-level Tradefed XML configuration that describes which modules should run and which top-level setup should be used.
  • Module-level setup: Setup applied to the devices right before running the module. These are also known as module-specific setups.
  • Module configuration: Refers to the AndroidTest.xml Tradefed XML configuration that describes the modules and which module-level setup should be done.
  • Module: Test unit composed of a setup step (module-level setup), a test execution step and a tear down step.
  • Intra-module retry: Automatic retry done by the harness inside the module.
  • Suite retry: Full rerun of the suite's previously failed tests.

ITestSuite structure

ITestSuite in Tradefed refers to the common base class driving a suite execution. It's shared by all major test suites, specifically the Android Compatibility Test Suite (CTS) and Android Vendor Test Suite (VTS), and ensures a consistent execution experience across all suites.

We sometimes refer to ITestSuite as the suite runner.

The suite runner follows these steps when executing:

  1. Load the module's configuration and determine which set should run.

  2. Run each module:

    1. Run module-level setup.
    2. Run module tests.
    3. Run module-level tear down.

  3. Report the results.

Top-level setup

From a Tradefed point of view, ITestSuite is just another test. It's a complex one but is still just a test like any other IRemoteTest. So when specifying the suite runner in a Tradefed configuration, Tradefed follows the usual pattern of the configuration: running build_provider, target_preparer, test (our suite in this case), and target_cleaner.

This sequence in the Tradefed configuration containing the ITestSuite is the top-level setup.

Example:

<configuration description="Common config for Compatibility suites">

    <build_provider class="com.android.compatibility.common.tradefed.build.CompatibilityBuildProvider" />
    <!-- Setup applied before the suite: so everything running in the suite will
    have this setup beforehand -->
    <target_preparer class="com.android.tradefed.targetprep.RunCommandTargetPreparer">
        <option name="run-command" value="settings put global package_verifier_enable 0" />
        <option name="teardown-command" value="settings put global package_verifier_enable 1"/>
    </target_preparer>

    <!-- Our ITestSuite implementation -->
    <test class="com.android.compatibility.common.tradefed.testtype.suite.CompatibilityTestSuite" />

    <result_reporter class="com.android.compatibility.common.tradefed.result.ConsoleReporter" />
</configuration>

Module metadata

We call module metadata extra information specified in the test module AndroidTest.xml. This metadata lets you specify additional information about the module, and modules can be filtered using the metadata.

Example metadata:

<option name="config-descriptor:metadata" key="component" value="framework" />
<option name="config-descriptor:metadata" key="parameter" value="instant_app" />

Example filter on metadata:

--module-metadata-include-filter component=framework

The above would run all the modules with a framework as component metadata.

Full AndroidTest.xml example:

<configuration description="Config for CTS Gesture test cases">
    <option name="test-suite-tag" value="cts" />
    <!-- Metadata -->
    <option name="config-descriptor:metadata" key="component" value="framework" />
    <option name="config-descriptor:metadata" key="parameter" value="instant_app" />
    <!-- End: metadata -->
    <target_preparer class="com.android.tradefed.targetprep.suite.SuiteApkInstaller">
        <option name="cleanup-apks" value="true" />
        <option name="test-file-name" value="CtsGestureTestCases.apk" />
    </target_preparer>
    <test class="com.android.tradefed.testtype.AndroidJUnitTest" >
        <option name="package" value="android.gesture.cts" />
        <option name="runtime-hint" value="10m50s" />
    </test>
</configuration>

Parameterized module

A special metadata type is parameter.

<option name="config-descriptor:metadata" key="parameter" value="instant_app" />

This metadata specifies that the module needs to be executed in a different mode, for example as an instant app, instead of a standard app mode.

All the possible modes or parameters are described by ModuleParameters and have an associated handler in ModuleParametersHelper that lets you change the module setup to execute in the particular mode.

For example, the instant app mode forces the APK installation as instant mode.

In order for the parameterization to occur, the command line needs to enable it with:

--enable-parameterized-modules

It's also possible to run a single given mode with:

--enable-parameterized-modules --module-parameter <Mode>

--enable-parameterized-modules --module-parameter INSTANT_APP

When a parameterized version of a module runs, it reports its results under a parameterized module name, for example CtsGestureTestCases[instant] versus base CtsGestureTestCases.