Test suite
For a test to be a part of VTS, it must have the following setting
in Android.bp.
test_suites: ["vts"],
Additionally, adding the test to the suite general-tests allows it to be
part of a Test Mapping suite used in presubmit checks.
Test configuration
In most cases, test configuration, which is an XML file used by Trade Federation to run a VTS test, is automatically generated during the build. However, you can customize the test configuration.
Create a customized test configuration file
Creating a new test XML file from scratch can be complicated, as it involves undestanding how the test harness works, as well as the difference between each test runner. The autogenerated test config file is designed to make this process easier.
If you must customize the test XML file, you can use the autogenerated file as a starting point.
To locate the autogenerated test config file, first
run the make command to build the config, as shown below.
$ m VtsHalUsbV1_1TargetTest
In your build out directory, you can search for the config file based on the module name, as shown below.
$ find out/ -name VtsHalUsbV1_1TargetTest.config
There can be multiple copies of the file and you can use any of them.
Copy the .config file to the directory
where the Android.bp file is.
If there's only one test module in the Android.bp file, you can
rename the XML file to AndroidTest.xml, and the build system automatically
uses that as the test module’s configuration file. Otherwise,
add a test_config attribute to the module, as shown in the
example below.
test_config: "VtsHalUsbV1_1TargetTest.xml",
Now you have a test configuration file to work with and implement the customization.
Force the test to run with adb root
Most VTS tests require root privilege to run. If the test configuration
file is autogenerated, you can add following attribute to Android.bp.
require_root: true,
If the test configuration file is customized, add following to the test XML file.
<target_preparer class="com.android.tradefed.targetprep.RootTargetPreparer"/>
Stop framework during the test
Many VTS tests don't require Android framework to run, and running the
test with framework stopped allows the test to run with stability
without being affected by device flakes. If the test configuration
file is autogenerated, you can add following attribute to Android.bp.
disable_framework: true,
If the test configuration file is customized, add following to the test XML file.
<target_preparer class="com.android.tradefed.targetprep.StopServicesSetup"/>
Add additional test arguments
Some gtest tests might require more time to run. In such cases, you can add test runner options in the XML file.
For example, the native-test-timeout setting in the following
entry allows the test to run with a timeout of 3 minutes, instead of the
default of 1 minute.
<test class="com.android.tradefed.testtype.GTest" >
<option name="native-test-device-path" value="/data/local/tmp" />
<option name="module-name" value="VtsHalNfcV1_0TargetTest" />
<option name="native-test-timeout" value="180000"/>
</test>
Require minimum API level
Some VTS tests can run only on devices with minimum API level. If the
test configuration file is autogenerated, you can add following
attribute to Android.bp.
min_shipping_api_level: 29,
or
vsr_min_shipping_api_level: 202404,
If the test configuration file is customized, add the following command to the test XML file.
<object type="module_controller" class="com.android.tradefed.testtype.suite.module.ShippingApiLevelModuleController">
<option name="min-api-level" value="29" />
</object>
or
<object type="module_controller" class="com.android.tradefed.testtype.suite.module.ShippingApiLevelModuleController">
<option name="vsr-min-api-level" value="202404" />
</object>
API level properties
Android 12 defines ro.board.first_api_level and
ro.board.api_level properties to show the API level of the vendor images on
these devices. Combining these properties with ro.product.first_api_level,
test suites choose proper test cases for the devices.
Android 13 defines ro.vendor.api_level that is
automatically set by calculating the required vendor API level using the
ro.product.first_api_level, ro.board.first_api_level, and
ro.board.api_level properties.
For more details, see Vendor API level.
ro.board.first_api_level
The ro.board.first_api_level property is the API level when the vendor images
for an SoC that is qualified for
vendor freeze are first
released with this property. This does not depend on the device's launching API
level but only depends on the SoC's first API level that defines this value. The
value is permanent for the lifetime of the SoC.
To set ro.board.first_api_level, device manufacturers can define
BOARD_SHIPPING_API_LEVEL in their device.mk file, as shown in the following
example:
# BOARD_SHIPPING_API_LEVEL sets ro.product.first_api_level to indicate
# the first api level that the device has been commercially launched on.
BOARD_SHIPPING_API_LEVEL := 23
It will automatically define the ro.board.first_api_level property to
vendor/build.prop on the device. The property is set by the vendor init
process.
ro.board.api_level
The ro.board.api_level property is the current vendor API level of the vendor
images that has YYYYMM format in which the vendor API was frozen. It is
automatically set by the build system.
ro.vendor.api_level
The ro.vendor.api_level property was introduced in
Android 13 to show the API level that the vendor images
are required to support. This is automatically set to the
ro.product.first_api_level, or ro.board.api_level if
ro.board.first_api_level is present and ro.board.api_level is set to an
earlier API level than ro.product.first_api_level. The version will be
substitute with the corresponding vendor API level if it is set to the version
from ro.product.first_api_level which is greater than or equal to 35. Tests
for vendor implementation that require vendor image upgrade might be excluded
from the vendor requirements for the SoC by referring to this property.
Sharding process using VTS
For Android version 10 or higher, you can perform the sharding process on multiple devices while testing with both VTS and CTS-on-GSI plans by following the instructions below.
run vts --shard-count <number of devices> -s <device serial> ...
This command splits the VTS plan into shards and runs them on multiple devices.
run cts-on-gsi --shard-count <number of devices> -s <device serial> -s ...
This command splits the CTS-on-GSI plan into shards and runs them on multiple devices.