The latest version of this document is available at.
There are currently three tools used in testing:
run_tests.py, for testing a specific configuration.
validate.py, for testing many configurations.
test_libcxx.py, for testing libc++.
A test configuration is a tuple of (ABI, target platform, toolchain, device).
At some point the three of these will most likely merge into one script.
For targeted testing during development,
run_tests.py can be used to verify a single test configuration, as well as run specific tests.
Running the NDK tests requires a complete NDK package (see README.md for building instructions).
From the NDK source directory (not the extracted package):
$ python run_tests.py --abi $ABI_TO_TEST
If you're testing a downloading NDK, the path to the NDK to test may optionally be passed to
run-tests.py. Otherwise the path will be assumed to be the install location in the out directory.
The default toolchain for testing is Clang. To run the tests with GCC, use the option
--toolchain 4.9..
When testing multiple configurations,
validate.py will automatically detect connected devices/emulators and choose a subset of them to fit our QA configuration. This script will run all of the tests across all available ABIs on several devices, and thus will take a long time (takes ~75 minutes on my machine, even with a few configurations unavailable). As such, this isn't suitable for active development, but should be run for QA and for any changes that might have a wide impact (compiler updates,
ndk-build changes, sysroot changes, etc).
To use this script, connect any devices and launch any emulators you need for testing (make sure ADB has authorization to control them), then run:
$ python validate.py
If you're testing a downloading NDK, the path to the NDK to test may optionally be passed to
run-tests.py. Otherwise the path will be assumed to be the install location in the out directory.
By default, test logs will be placed in $PWD/test-logs. This can be controlled with the
--log-dir flag. user:
def build_broken(abi, platform, toolchain): if abi == 'arm64-v8a': return abi, '' return None, None def run_unsupported(abi, device_api, toolchain, subtest): if device_api < 21: return device_api. Can be
None, in which case the default API level for that ABI should be considered.
device_platform: The platform version of the device the test will be run on. Note that this parameter is ommitted for build tests. In a
--skip-runconfiguration, this is set to the minimum supported API level for the given API (9 for LP32, 21 for LP64).
toolchain: The toolchain being used.
'clang'if we‘re using clang (the default), or
'4.9'if we’re using GCC.
subtest: This is
Nonefor build tests and for the build step of device tests, but will be set to the name of the executable for the run step of device tests. If the test builds fine but fails at runtime, you must gate your check with this.
The libc++ tests are not currently integrated into the main NDK tests. To run the libc++ tests:
$ ./test_libcxx.py --abi $ABI --platform $API_LEVEL
Note that these tests are far from failure free. In general, most of these test failures are locale related and fail because we don't support anything beyond the C locale.
To run the NDK tests, you will need:
adbin your path.
adb. Note that the version of
adbin the SDK manager might be surprisingly old. It's best to use a version built fresh from AOSP.
For testing a release, make sure you're testing against the released builds of Android.
For Nexus devices, factory images are available here:. Googlers, you may want to use the flash station to get a userdebug image since that is needed for ASAN testing. You should still make sure you also test on user builds because that is what all of our users have.
For emulators, use emulator images from the SDK rather than from a platform build. Again, these are what our users will be using.
After installing the emulator images from the SDK manager, they can be configured and launched for testing with (assuming the SDK tools directory is in your path):
$ android create avd --name $NAME --target android-$LEVEL --abi $ABI $ emulator -avd $NAME -no-window -no-audio -no-skin
This will create a new virtual device and launch it in a headless state.
The current configuration we use to test NDK releases is as written in qa_config.yaml:
Each API level/ABI pair will be checked with both Clang and GCC, unified and deprecated headers.
Note that there are no ARM64 emulators whatsoever in the SDK manager. Testing ARM64 will require a physical device.