A test might fail for any reason, and sometimes simply re-running the test is enough to make it pass again (due to flakiness, such as from issues in the underlying infrastructure). You can configure Tradefed to conduct the retry automatically.
The core of auto-retry is to avoid re-running all of the tests; it re-runs only the failed tests, resulting in large savings in execution time.
Tradefed also supports running tests multiple times in order to detect flakiness via the iterations feature. In this case, all tests will be re-run, and the test will fail if any of the iterations fail.
Enable automatic retry
The automatic retry is controlled via the
RetryDecision object
which provides two options to enable the feature: max-testcase-run-count
and retry-strategy.
max-testcase-run-count drives the number of retries or iterations that will
be attempted. It sets an upper bound to avoid retrying forever.
retry-strategy drives the decision of how to retry; see the following sections
for more details.
Disable automatic retry
Use the following option:
--retry-strategy NO_RETRY
Retry failures
To retry test failures, use the following options:
--retry-strategy RETRY_ANY_FAILURE --max-testcase-run-count X
This will retry the failure until it passes or until the max number of retries is reached, whichever comes first.
Iterations
To re-run tests for a number of time, the following options can be used:
--retry-strategy ITERATIONS --max-testcase-run-count X
What do the results look like?
Result reporters by default will receive aggregated results of all attempts.
For example: a Fail and a Pass for RETRY_ANY_FAILURE will result in an
aggregated Pass since the retry managed to clear the failure.
It is possible for reporters to receive the non-aggregated results. To do so, they need to extend the ISupportGranularResults interface that declares support for the granular (non-aggregated) results.
Implementation details
To enable auto-retry to rerun failures at the test case level, implement ITestFilterReceiver.
If you can't implement ITestFilterReceiver, you can implement IAutoRetriableTest to manually control how rerun operates. InstalledInstrumentationsTest is an example implementation of that interface.