diff --git a/TESTING.md b/TESTING.md index 50cc9bea3e..19965157fb 100644 --- a/TESTING.md +++ b/TESTING.md @@ -7,9 +7,9 @@ gem5's testing infrastructure has the following goals: * Fast execution in the simple case * High coverage of gem5 code -# Running unit tests +## Running the CPP unit tests -gem5 comes with unit tests, created using the Google Test framework. These can +gem5 comes with unit tests for CPP, created using the Google Test framework. These can be built through SCons. To build and run all the unit tests: @@ -41,9 +41,30 @@ To run a specific test function (e.g., BitUnionData.NormalBitfield): ./build/ALL/base/bitunion.test.opt --gtest_filter=BitUnionData.NormalBitfield ``` -# Running system-level tests +## Running the Python unit tests -Within the `tests` directory we have system-level tests. These tests run +gem5 comes with Python unit tests. +These are built using the [Python unit testing framework](https://docs.python.org/3/library/unittest.html). +These tests can be found in "tests/gem5/pyunit". + +To run these tests a gem5 binary must first be compiled. +We recommend, `build/ALL/gem5.opt`: + +```sh +scons build/ALL/gem5.opt -j {number of compilation threads} +``` + +Then the Pyunit tests may be executed using: + +```sh +./build/ALL/gem5.opt tests/run_pyunit.py +``` + +**Note**: These tests are also run via the 'quick' system-level tests, explained below. + +## Running system-level tests + +Within the "tests/gem5" directory we have system-level tests. These tests run the gem5 framework against various hardware configurations, with different ISAs, then verify the simulations execute correctly. These should be seen as high-level, coarse-grained tests to compliment the unit-tests. @@ -77,56 +98,37 @@ arguments: This will load every test in directory1 and directory2 (and their subdirectories). -## Specifying a subset of tests to run +### 'quick', 'long', and 'very-long' tests -You can use the tag query interface to specify the exact tests you want to run. -For instance, if you want to run only with `gem5.opt`, you can use +There are three categoties of tests which may be run from the "tests" directory: -```shell -./main.py run --variant opt +1. **'quick' tests**. This suite of tests are designed to finish execution in a few hours, inclusive of compilation of gem5. +We run these as part of our continuous integration tests on pull requests made to our repository. +These tests all utilize a binary build `scons build/ALL/gem5.opt`, and thus only rely on a single compilation for the tests to run. +2. **'long' tests**. This suite of tests are designed to finish execution in around 12 hours. +They incorporate longer running tests which are unsuitable to run as part of the 'quick' tests. +We run these daily via a scheduled job. +3. **'very-long' tests**. This suite of tests are designed to finish execution in days. +They incorporate tests which are too long to run frequntly +We run these daily via a scheduled job. + +When executing `./main.py run` the 'quick' tests are executed. +To run the 'long' tests execute: + +```sh +./main.py run --length=long ``` -Or, if you want to just run quick tests with the `gem5.opt` binary: +and to run the 'very-long' tests execute: -```shell -./main.py run --length quick --variant opt +```sh +./main.py run --length=very-long ``` +In most cases we recommend running the 'quick' tests for most changes. +Only in some cases, such as contributions which significantly change the codebase, do we recommend running the 'long' or 'very-long' suite. -To view all of the available tags, use - -```shell -./main.py list --all-tags -``` - -The output is split into tag *types* (e.g., isa, variant, length) and the -tags for each type are listed after the type name. - -Note that when using the isa tag type, tests were traditionally sorted based -on what compilation it required. However, as tests have switched to all be -compiled under the ALL compilation, which includes all ISAs so one doesn't -need to compile each one individually, using the isa tag for ISAs other than -ALL has become a less optimal way of searching for tests. It would instead -be better to run subsets of tests based on their directories, as described -above. - -You can specify "or" between tags within the same type by using the tag flag -multiple times. For instance, to run everything that is tagged "opt" or "fast" -use - -```shell -./main.py run --variant opt --variant fast -``` - -You can also specify "and" between different types of tags by specifying more -than one type on the command line. For instance, this will only run tests with -both the "ALL" and "opt" tags. - -```shell -./main.py run --isa All --variant opt -``` - -## Running tests in batch +### Running tests in batch The testing infrastructure provides the two needed methods to run tests in batch. First, you can list all of the tests based on the same tags as above in @@ -160,7 +162,7 @@ run more than one uid, you must call `./main.py` multiple times. Currently, you must specify `--skip-build` if you want to run a single suite or run in batch mode. Otherwise, you will build gem5 for all architectures. -## Rerunning failed tests +### Rerunning failed tests While developing software a common practice is to run tests, make a change, and assert that the tests still pass. If tests fail you'll likely want to @@ -178,7 +180,7 @@ using the `rerun` command. ./main.py rerun ``` -## If something goes wrong +### If something goes wrong The first step is to turn up the verbosity of the output using `-v`. This will allow you to see what tests are running and why a test is failing. @@ -186,7 +188,7 @@ allow you to see what tests are running and why a test is failing. If a test fails, the temporary directory where the gem5 output was saved is kept and the path to the directory is printed in the terminal. -## Debugging the testing infrastructure +### Debugging the testing infrastructure Every command takes an option for the verbosity. `-v`, `-vv`, `-vvv` will increase the verbosity level. If something isn't working correctly, you can @@ -197,7 +199,7 @@ contains the base code for tests, suites, fixtures, etc. The code in tests/gem5 is *gem5-specific* code. For the most part, the code in tests/gem5 extends the structures in ext/testlib. -## Common errors +### Common errors You may see a number of lines of output during test discovery that look like the following: @@ -213,45 +215,7 @@ test library executes each python file it finds searching for tests. It's okay if the file causes an exception. This means there are no tests in that file (e.g., it's not a new-style test). - -## Binary test applications - -The code for some test binaries that are run in the gem5 guest during -testing can be found in `tests/test-progs`. -There's one directory per test application. -The source code is under the `source` directory. - -You may have a `bin` directory as well. -The `bin` directory is automatically created when running the test case that -uses the test binary. -This is not the case when a test is run via the --bin-path option. -In that scenario a bin directory will be created in the selected path -rather than in `tests/test-progs`. -The binary is downloaded from the gem5 servers the first -time it is referenced by a test. - -Some other tests (like Linux-boot) don't have sources inside gem5 and -are simply downloaded from gem5 servers. - -## Updating the test binaries - -The test infrastructure should check with the gem5 servers to ensure you have -the latest binaries. However, if you believe your binaries are out of date, -simply delete the `bin` directory and they will be re-downloaded to your local -machine. - -## Building (new-style) test binaries - -In each `src/` directory under `tests/test-progs`, there is a Makefile. -This Makefile downloads a docker image and builds the test binary for some ISA -(e.g., Makefile.x86 builds the binary for x86). Additionally, if you run `make -upload` it will upload the binaries to the gem5 server, if you have access to -modify the binaries. *If you need to modify the binaries for updating a test or -adding a new test and you don't have access to the gem5 server, contact a -maintainer (see MAINTAINERS).* - - -## Running Tests in Parallel +### Running Tests in Parallel Whimsy has support for parallel testing baked in. This system supports running multiple suites at the same time on the same computer. To run @@ -261,41 +225,24 @@ For example, to run up to three test suites at the same time:: ./main.py run --skip-build -t 3 +### Testing resources + +By default binaries and testing resources are obtained via the [gem5 resources infrastructure](https://www.gem5.org/documentation/general_docs/gem5_resources/). +The downloaded resources are cached in "tests/gem5/resources". +The resources are cached to avoid re-downloading when tests are run multiple times, though some of these resources, such as disk images, are large. +It is therefore recommended you remove the "tests/gem5/resources" directory when you are done testing. + ## Running Tests within GitHub Actions -To run these tests within GitHub Actions, we use the format of running -tests by directory as shown above in the "Running Tests from Multiple -Directories" section. These tests are run within workflow files, -which can be found in the .github directory of this repository. -You can learn more about workflows -[here](https://docs.github.com/en/actions/using-workflows/about-workflows). +These tests outlined here are run as part of [GitHub Actions](https://github.com/features/actions). +These are outlined in [workflow files](https://docs.github.com/en/actions/using-workflows/about-workflows), which can be found in the repo's ".github" directory. +Each workflow is made up of individual jobs where ecch job consists of a series of steps which are executed within a [GitHub Runner](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners). -Each workflow is made up of individual jobs, where each job is a set -of tests that is executed on a runner within GitHub. In each -workflow, each version of gem5.opt needed is first built, and then -stored as an artifact for main.py to use. +### Adding Tests to GitHub Actions -There are two sets of runners within the gem5 repository: builders and -runners. The builders have more resources available to allow for a -quicker compilation of gem5, while the runners have less, and are -meant to only run tests. +To ensure tests added are run in GitHub Actions you may need to modify the worklfow files. +For tests run via `./main.py` we split up the tests via the subdirectories in "tests/gem5". +For example, all tests under "test/gem5/cpu_tests" are run as one job. +Therefore tests added to existing directories are likely to be included, but modifications to the workflow files may be needed if new directories are added. -After the gem5 artifact has been uploaded, a runner can then download -the versions needed for their tests. For example, in the daily-tests.yaml, -in order to run the multi_isa tests, you need artifacts of ARM, RISCV, -and VEGA_X86. - -## Adding Tests to GitHub Actions - -In order to add new tests to our GitHub Actions testing infastructure, -follow the format currently shown in the existing workflows. If the -new tests were added to an already existing directory (ex. A very-long -test in the gem5_library_example_tests), it will automatically be -included into the weekly testing, since weekly-tests.yaml already -contains a job for the gem5_library_example_tests. - -However, if a new directory is added to the tests, you need to manually -add a new step to the GitHub workflows. This would consist of both a -step to build whatever version of gem5 was required if it wasn't -already included in the file, as well as a step to run main.py -in the given directory after downloading the gem5 artifact. +We strongly recommend that when adding or ammending tests, that contributors check the ".github/workflows" files to ensure the tests they specify will be run as intended.