Skip to content

CMake: Update README.md with CTest/greentea instructions #15001

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

CMake: Update README.md with CTest/greentea instructions #15001

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 45 additions & 15 deletions tools/cmake/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,27 +82,57 @@ If you're running CMake directly, you may need to pass it in yourself as follows
cmake -S <source-dir> -B <build-dir> -DCMAKE_BUILD_TYPE=debug
```

## How to build a greentea test

Install prerequisites suggested in the previous section and follow the below steps to build:
* Set your current directory to the test suite directory

* Run the following command for the configuration CMake module to be generated
```
mbedtools configure -t <TOOLCHAIN> -m <MBED_TARGET> --mbed-os-path /path/to/mbed-os
```
* Build the test binary with the full profile
## How to build and run greentea tests

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we need a note that migration of all tests is still in progress?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the last remaining question. Can we merge this to get into the upcoming release (code freeze today).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we wait until the migration of all tests is complete before merging this? I can see it causing confusion for users. This PR also removes the previous instructions for running greentea tests, which users could've been relying on. Rather than adding caveats or calling out both methods of running the tests, maybe we wait until the work to migrate all tests is complete, then the documentation is correct (and doesn't need to be changed again later)?

### Set up
Install the [mbed-hosts-tests](https://pypi.org/project/mbed-host-tests/) python package.

Also install the prerequisites suggested in the previous section.

### Configuring the build system
1. Set your current directory to the mbed-os root.
1. Connect your target and detect it using `mbed-tools`
```
mbedtools detect
```
_Make a note of the mount point and serial port. We will need to pass them to CMake in a later step._

1. Generate an mbed_config.cmake configuration file for your target using `mbed-tools`
```
mbedtools configure -t <TOOLCHAIN> -m <MBED_TARGET> --mbed-os-path . --output-dir __build
```
1. Generate the build system with CMake, configuring it to build greentea tests.
Here you will pass the serial port and mount point you noted in an earlier
step
```
cmake -S . -B __build -G Ninja -DBUILD_GREENTEA_TESTS=ON -DCMAKE_CTEST_ARGUMENTS="--output-on-failure;-V" -DMBED_HTRUN_ARGUMENTS="-p;<serial-port-from-detect>;-d;<mount-point-from-detect>"
```
The `MBED_HTRUN_ARGUMENTS` argument is a semicolon separated list of arguments
to forward to `mbedhtrun`. You can pass any of the optional flags you would
usually pass to `mbedhtrun` by adding them to this list.
Comment on lines +110 to +112
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth noting that each argument is already quoted by CTest so the user should not add quotation marks or escape any characters.
For example, a mount point has a space inside, in which case -DMBED_HTRUN_ARGUMENTS="-p;/dev/tty.usbmodem14402;-d;/Volumes/NuMicro MCU" just works, but -DMBED_HTRUN_ARGUMENTS="-p;/dev/tty.usbmodem14402;-d;/Volumes/NuMicro\ MCU" causes the backslash to be treated as a literal character, and -DMBED_HTRUN_ARGUMENTS="-p;/dev/tty.usbmodem14402;-d;'/Volumes/NuMicro\ MCU'" causes the single quotes to be treated as literal characters.


The `CMAKE_CTEST_ARGUMENTS` argument is a semicolon separated list of
arguments to forward to CTest, for more details see [the CMake documentation.](https://cmake.org/cmake/help/latest/variable/CMAKE_CTEST_ARGUMENTS.html)

If you want to build with the "bare-metal" profile, append
`-DMBED_GREENTEA_TEST_BAREMETAL=ON` to the above command.

### Build and run tests
To build and run the tests run the following commands
```
cd cmake_build/<MBED_TARGET>/<PROFILE>/<TOOLCHAIN>/ && cmake ../../../.. -G Ninja && cmake --build .
cmake --build __build && cmake --build __build --target test
```
Or build the test binary with the baremetal profile
Alternatively you can run ctest directly
```
cd cmake_build/<MBED_TARGET>/<PROFILE>/<TOOLCHAIN>/ && cmake ../../../.. -G Ninja -DMBED_GREENTEA_TEST_BAREMETAL=ON && cmake --build .
cmake --build __build
cd __build
ctest -V
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth adding a note on how to run an individual test?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add a link to the ctest documentation to avoid repeating it here.

Copy link
Contributor

@LDong-Arm LDong-Arm Sep 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though how to filter an individual test(s) is in the CTest documentation, it only covers the "run" stage. It's still good to let users know how to build one test. Presumably using --target <test-name> instead of --target test? Since building & testing of one test is a very commonly done task, giving a special mention of both in one place could be useful in my opinion.

Especially that the build time for all tests has increased quite a lot by now (e.g. over 10K object files as of today). Even though it's partially because INTERFACE libraries does lots of duplicated compilation than STATIC (a bit off-topic now...)

```
This will build the tests and run them using `mbedhtrun` to communicate with the target.
For more information on `ctest` and the options it offers, see [the ctest documentation.](https://cmake.org/cmake/help/latest/manual/ctest.1.html)

Notes:
* These steps will change when `mbedtools` implements a sub-command to invoke Greentea tests
* Some Greentea tests require specific application configuration files in order to build and run successfully. For example, the `connectivity/mbedtls/tests/TESTS/mbedtls/sanity` test requires the configuration file found at `TESTs/configs/experimental.json`.
* Some Greentea tests require specific application configuration files in order to build and run successfully. For example, the `connectivity/mbedtls/tests/TESTS/mbedtls/sanity` test requires the configuration file found at `TESTs/configs/experimental.json`. The config should be passed to `mbed-tools configure` using the `--app-config` argument.

## Naming convention

Expand Down