The x86-parsec gem5 Resource (v1.0.0,
http://resources.gem5.org/resources/x86-parsec?version=1.0.0) is 33GB.
The gem5 GitHub Actions self-hosted runners do not have enough Disk
Space in the VMs they are run to download this. Ergo we skip it.
Change-Id: I290fe265f03ceca65b2bed87e9f4a4ad601e0fc1
This argument allows the passing of IDs of resources which should be
skipped for this check.
Note: A current limitation here is you cannot specify the version of a
resource. Passing the ID of a resource to this will skip the downloading
for all versions of that resource.
Change-Id: Ifdb7c2b71553126fd52a3d286897ed5dd8e98f7c
These tests are disabled due our GitHub Actions self-hosted Runners
having a 60GB of disk space. The PARSEC Disk Image Resource (v1.0.0,
http://resources.gem5.org/resources/x86-parsec?version=1.0.0) is 33GB
and is simply too big to download and unzip for these tests.
These tests can be reenabled when this issue is resolved.
Change-Id: I9a63aa1903cea3ce7942bdc85bcd0b24761d2f29
There was another missing file in the configs for the fs tests,
so this should allow the switcheroo tests to pass.
Change-Id: Ic4e26cceeb9209f176158b80eaaba88b47968c39
The configs directory for the fs tests was missing the
checkpoint.py file, causing some of the CI tests to fail.
Change-Id: Ifbd775ad658f96d06bea7bee554fe3bedcf5a5b5
"tests/gem5/configs/download_check.py" is used by the
"test-resource-downloading" test (defined in
"tests/gem5/gem5-resources/test_download_resources.py" and ran as part
of the "very-long" suite).
Prior to this change "download_check.py" would download each resource,
check it's md5, then at the end of the script remove all the downloaded
resources. This is inefficient on disk space and was causing our
"very-long" suite of tests to require a machines with a lot of disk
space to run.
This change alters 'download_check.py" to remove each resource after the
md5 check. Thus, only one resource is ever downloaded and present at any
given time during the running of this script.
Change-Id: I38fce100ab09f66c256ccddbcb6f29763839ac40
This changeset reorganizes the testing directory within gem5,
removing the bigger config folders, then replacing them with
smaller configs folders within each directory containing only
the scripts necessary for that set of tests. It also changes
the locations of the config scripts used in each set of tests,
and updates the tests accordingly.
Change-Id: I38297d4496f72bd5cf7200471acd5c4d93002b27
This change adds READMEs to each directory within tests/gem5,
with a short description of the test, as well as how to run it.
Change-Id: I574ebcdc837848b52f21e8c0f8856ff09463284b
* stdlib,configs,tests: Remove `Resource` class use
This class is deprecated, but was still used in various example
configuration scriots and tests. This patch replaces it with the
`obtain_resource` function.
Change-Id: I0c89bf17783ccaaafc18072aaeefb5d1e207bc55
* configs: Remove `CustomDiskImageResource` use
The class is deprecated but was still used in the SPEC example scripts.
This patch replaces it with the `DiskImageResource` class.
Change-Id: Ie0697fe59a3d737b05eb45ff3bc964f42b0387e0
* configs,tests: Remove `CustomResource` use
This class is deprecated but was still used in example scripts and
mentioned, incorrectly, in comments in the pyunit tests. This patch
removes these.
Change-Id: Icb6d02f47a5b72cd58551e5dcd59cc72d6a91a01
* stdlib: Remove '\' in Workload docstring example
This example shows how to use the Workload. The backslash is not correct Python and would fail if used in this way.
Co-authored-by: Jason Lowe-Power <jason@lowepower.com>
---------
Co-authored-by: Jason Lowe-Power <jason@lowepower.com>
This patch includes several changes to the gem5 tools interface to the
gem5-resources infrastructure. These are:
* The old download and JSON query functions have been removed from the
downloader module. These functions were used for directly downloading
and inspecting the resource JSON file, hosted at
https://resources.gem5.org/resources. This information is now obtained
via `gem5.client`. If a resources JSON file is specified as a client,
it should conform to the new schema:
https//resources.gem5.org/gem5-resources-schema.json. The old schema
(pre-v23.0) is no longer valid. Tests have been updated to reflect
this change. Those which tested these old functions have been removed.
* Unused imports have been removed.
* For the resource query functions, and those tasked with obtaining the
resources, the parameter `gem5_version` has been added. In all cases
it does the same thing:
* It will filter results based on compatibility to the
`gem5_version` specified. If no resources are compatible the
latest version of that resource is chosen (though a warning is
thrown).
* By default it is set to the current gem5 version.
* It is optional. If `None`, this filtering functionality is not
carried out.
* Tests have been updated to fix the version to “develop” so the
they do not break between versions.
* The `gem5_version` parameters will filter using a logic which will
base compatibility on the specificity of the gem5-version specified in
a resource’s data. If a resource has a compatible gem5-version of
“v18.4” it will be compatible with any minor/hotfix version within the
v18.4 release (this can be seen as matching on “v18.4.*.*”.) Likewise,
if a resource has a compatible gem5-version of “v18.4.1” then it’s
only compatible with the v18.4.1 release but any of it’s hot fix
releases (“v18.4.1.*”).
* The ‘list_resources’ function has been updated to use the
“gem5.client” APIs to get resource information from the clients
(MongoDB or a JSON file). This has been designed to remain backwards
compatible to as much as is possible, though, due to schema changes,
the function does search across all versions of gem5.
* `get_resources` function was added to the `AbstractClient`. This is a
more general function than `get_resource_by_id`. It was
primarily created to handle the `list_resources` update but is a
useful update to the API. The `get_resource_by_id` function has been
altered to function as a wrapped to the `get_resources` function.
* Removed “GEM5_RESOURCE_JSON” code has been removed. This is no longer
used.
* Tests have been cleaned up a little bit to be easier to read.
* Some docstrings have been updated.
Things that are left TODO with this code:
* The client_wrapper/client/abstract_client abstractions are rather
pointless. In particular the client_wrapper and client classes could
be merged.
* The downloader module no longer does much and should have its
functions merged into other modules.
* With the addition of the `get_resources` function, much of the code in
the `AbstractClient` could be simplified.
Change-Id: I0ce48e88b93a2b9db53d4749861fa0b5f9472053
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/71506
Reviewed-by: Kunal Pai <kunpai@ucdavis.edu>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
(cherry picked from commit 82587ce71b)
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/71739
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
This patch makes changes to the stdlib based on the gem5 Vision project.
Firstly, a MongoDB database is supported.
A JSON database's support is continued.
The JSON can either be a local path or a raw GitHub link.
The data for these databases is stored in src/python
under "gem5-config.json".
This will be used by default.
However, the configuration can be overridden:
- by providing a path using the GEM5_CONFIG env variable.
- by placing a gem5-config.json file in the current working directory.
An AbstractClient is an abstract class that implements
searching and sorting relevant to the databases.
Clients is an optional list that can be passed
while defining any Resource class and obtain_resource.
These databases can be defined in the config JSON.
Resources now have versions. This allows for a
single version, e.g., 'x86-ubuntu-boot', to have
multiple versions. As such, the key of a resource is
its ID and Version (e.g., 'x86-ubuntu-boot/v2.1.0').
Different versions of a resource might be compatible
with different versions of gem5.
By default, it picks the latest version compatible with the gem5 Version
of the user.
A gem5 resource schema now has additional fields.
These are:
- source_url: Stores URL of GitHub Source of the resource.
- license: License information of the resource.
- tags: Words to identify a resource better, like hello for hello-world
- example_usage: How to use the resource in a simulation.
- gem5_versions: List of gem5 versions that resource is compatible with.
- resource_version: The version of the resource itself.
- size: The download size of the resource, if it exists.
- code_examples: List of objects.
These objects contain the path to where a resource is
used in gem5 example config scripts,
and if the resource itself is used in tests or not.
- category: Category of the resource, as defined by classes in
src/python/gem5/resources/resource.py.
Some fields have been renamed:
- "name" is changed to "id"
- "documentation" is changed to "description"
Besides these, the schema also supports resource specialization.
It adds fields relevant to a specific resource as specified in
src/python/gem5/resources/resource.py
These changes have been made to better present
information on the new gem5 Resources website.
But, they do not affect the way resources are used by a gem5 user.
This patch is also backwards compatible.
Existing code doesn't break with this new infrastructure.
Also, refs in the tests have been changed to match this new schema.
Tests have been changed to work with the two clients.
Change-Id: Ia9bf47f7900763827fd5e873bcd663cc3ecdba40
Co-authored-by: Kunal Pai <kunpai@ucdavis.edu>
Co-authored-by: Parth Shah <helloparthshah@gmail.com>
Co-authored-by: Harshil Patel <harshilp2107@gmail.com>
Co-authored-by: aarsli <arsli@ucdavis.edu>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/70858
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
The Looppoint checkpoint feature is still under-development.
These tests are stalling indefinitely and causing the Weekly tests to
timeout. They are therefore disabled until looppoint functionality is
complete and this bug is resolved.
Change-Id: I2c16abdaec639120c0ba349d90f5cc2fe747f8e6
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/69240
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
The script run by 'test_replacement_policies.py' was named
'run_replacement_policy_test.py'. This caused the TestLib framework to
fail as any file with the string "test" in it was attempted to parse as
a test. As this is a gem5 Python config file, this caused a failure.
To fix this the file was renamed 'run_replacement_policy.py'.
Change-Id: I6f0a09e4f5056a9c94e258c9aea26bf277206733
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/69237
Reviewed-by: Matt Sinclair <mattdsinclair@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Matt Sinclair <mattdsinclair@gmail.com>
The current GPU Ruby Random tester tests only test for WT L2 caches,
meaning that some transitions (specific to WB caches) are never
tested. To help ensure better coverage, this commit adds a separate
test that tests WB GPU L2 caches to the per-checkin and nightly
regressions.
Change-Id: I539ece3b825b9a38630027d947dc11ebef588752
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/69258
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
This patches does the following:
- Adds 'SimpointResource' which encapsulates Simpoint data and
functionality. It replaces the old 'gem5.util.simpoint.SimPoint'
class. Simpoints can be loaded from gem5-resources using the
`obtain_resource` function.
- Adds 'SimpointDirectoryResource'. This inherits form
'SimpointResource'. While 'SimpointResource' takes raw Simpoint data
via parameters, 'SimpointDirectoryResource' assumes the data exists
in files, in a directory.
- Updates the
"configs/example/gem5_library/checkpoints/simpoints-se-checkpoint.py"
and
"configs/example/gem5_library/checkpoints/simpoints-se-restory.py"
example files to utilize this new Simpoint resource classes.
**Note**: While the old "SimPoint" class
("src/python/gem5/util/simpoint.py") is marked as deprecated, it may be
difficult to utilize given updates to the APIs in the gem5 stdlib Cores
and Simulator modules.
Change-Id: I9bed5c643ffc735838c9f22a58c53547941010e7
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/67339
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
This commit specializes the Resource class into specific sub-types.
The `Resource`, `CustomResource` and `CustomDiskImageResource` classes
have been deprecated in favor of the `AbstractResource` subclasses.
Custom Resources can be created via the resource specialization
constructor. Resources can be obtained via the gem5-resource
infrastructure with the `obtain_resource` function.
Fully implemented:
- DiskImageResource
- BinaryResource
- KernelResource
- BootloaderResource
- FileResource
- DirectoryResource
Partially implemented:
- SimpointResource
- CheckpointResource
While the schema of the resource.json file has changed, efforts have
been made to ensure backwards compatibility is maintained during this
transition.
Tests are included in this commit to verify this feature works as
expected.
**Note:** The Simpoint tests are disabled in this commit, to be
reenabled when Simpoint resource specialization is fully incorporated
here:
https://gem5-review.googlesource.com/c/public/gem5/+/67339
Change-Id: I77277ecaffc7abc86db08526aacc0b606ef04fe8
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/67175
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Add tests to test the correctness of replacement policies using
TrafficGen.
Enable debug-flags to print the hit and miss messages so that you
can compare the results with the comments in the tests. Even though
the tests are targeting specific replacement policies, they can be
reused to test all replacement policies.
Change-Id: I3a8013fbcb19adae25b0818ac9e4b0be60be0689
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/60389
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
This commit adds the following functions to the `m5` python module:
- setMaxTick(tick) -> None
- getMaxTick() -> int
- getTicksUntilMax() -> int
- scheduleTickExitFromCurrent(tick, exit_string) -> None
- scheduleTickExitAbsolute(tick, exit_string) -> None
Until this patch the only way to set an exit at a particular tick was
via `simulate.run` which would reschedule the maximum tick. This
functionality has been explicity exposed via the new `setMaxTick`
function. However, as this is only rescheduling the maximum tick, it
stops scheduling exits at multiple different ticks.
To get around this problem the `scheduleTickExit` functions have been
added. These allow a user to schedule multiple exit events. The
functions contain a `exit_string` parameter that provides the string
the simulator is to return when the specified tick is met. By default
this string is "Tick exit reached" which is used by the stdlib
Simulator module to declare a new `SCHEDULED_TICK` exit event (Note:
this has been deliberatly kept seperate from the `MAX_TICK` exit event.
This commit serves as an attempt to decouple these are two concepts).
Tests are provided in this patch to ensure these new functions work as
intended.
Additional notes:
- The `simulate` function has been fixed to match the documentation. If
the `num_cycles` is -1 then the maximum ticks is set to MaxTicks.
Otherwise the max ticks is set to `curTicks() + num_cycles`. The
functionality of this function will remain unchanged to the end-user.
- Full integration into the Simulator module is not complete as of this
patch. Users must us the m5 python module to set these exit events.
Change-Id: I6c92b31dd409dc866152224600ea8166cfcba38b
Issue-on: https://gem5.atlassian.net/browse/GEM5-1131
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/66231
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
An example case,
```python
mem_side_port = RequestPort(
"This port sends requests and " "receives responses"
)
```
This is the residue of running the python formatter.
This is done by finding all tokens matching the regex `"\s"(?![.;"])`
and manually replacing them by empty strings.
Change-Id: Icf223bbe889e5fa5749a81ef77aa6e721f38b549
Signed-off-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/66111
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
The "test-gem5-library-example-riscvmatched-fs" test, which runs
"configs/example/gem5_library/riscvmatched-fs.py", was running the
script in full. This takes a very long time. Given we already have boot
tests for RISCV, it's better to just run this configuration to just the
end of the Linux boot (significantly faster than a full OS boot). This
patch adds this feature to the config script and modifies the test to
utilize it.
Change-Id: I1e37a26aab5e9a127ebd64590be79fbc16fe53aa
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/65853
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Hi, we are security researchers from the Advanced Research Center at Trellix.
We have began a campaign to patch a widespread bug named CVE-2007-4559.
CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using
extract() or extractall() on a tarfile object without sanitizing input,
a maliciously crafted .tar file could perform a directory path traversal
attack. We found at least one unsantized extractall() in your codebase
and are providing a patch for you via pull request. The patch essentially
checks to see if all tarfile members will be extracted safely and throws
an exception otherwise. We encourage you to use this patch or your own
solution to secure against CVE-2007-4559.
If you have further questions you may contact us through this
projects lead researcher Kasimir Schulz.
Change-Id: I891ac6652cfbd479aed51d64ef6d4e0fe740e06d
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/65271
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
The ARM Boot Tests required the compilation of ARM/gem5.opt to run a
quick test that the CHI protocol was functioning correctly with ARM and
the ArmBoard. This test has been removed and the test refactored
slightly to use the ALL/gem5.opt.
The CHI protocol is already tested nightly.
Change-Id: Ibe406348caefa2493860036eb89a20681478ea66
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/65195
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
The only tests requiring the the compilation of SPARC, MIPS, and POWER
for the long/nightly suite were
"tests/gem5/multi_isa/test_multi_isa.py" and
"gem5/stdlib/test_requires.py". As compilation of gem5 is quite costly,
it'd best we simply remove these tests. They are minor and not very
important.
Compilation of these ISAs will continue to be tested via the compilation
tests.
Change-Id: I98b33eec5d0adb144109d32851033380f1641ad4
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/65193
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
The long/nightly tests are failing due to timeout (e.g.,:
https://jenkins.gem5.org/job/nightly/398/). We must therefore be more
careful about what we test on a nightly basis.
Each of these X86 boot tests takes an hour and, generally, are largely
the same (just with different CPU cores, cache hierarchy, and memory
system). Given this is largely redundant, some of these tests have been
remove dto save on testing time.
Change-Id: I761fca1aa5e111a03183f83d4e326aaea1bdbc3a
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/65192
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
The long/nightly tests are failing due to timeout (e.g.:
https://jenkins.gem5.org/job/nightly/398/). We must therefore be more
careful about what we test on a nightly basis.
Each of these ARM boot tests takes an hour and, generally, are largely
the same (just with different CPU cores, cache hierarchy, and memory
system). Given this is largely redundant, some of these tests have been
removed to save on testing time.
Change-Id: I8d80d3e0869aca67aa7279a164bdce85d20f3682
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/65191
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The "config/example/gem5_library/arm-ubuntu-boot-exit.py" script is
renamed to "config/example/gem5_library/arm-ubuntu-run.py". This makes
it more consistent with similar scripts in the
"config/example/gem5_library" directory: "x86-ubuntu-run.py" and
"riscv-ubuntu-run.py".
Change-Id: I9d96fd68e122f2841573b1717b0969cd44972771
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/65132
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Through working with the gem5 stdlib there have been instances where
connecting the memory, processor, and cache hierarchy to the board (via
the AbstractBoard's `_connect_things` function) at the point of the
AbstractBoard's construction is problematic as the memory, processor,
and cache hierarchy may require information to connect correctly that is
only known to the AbstractBoard after construction. In particular this
can occur when a Workload contains information needed to configure
correctly.
To resolve this problem the `_connect_things` function has been moved to
run as a pre-initialization step. That is, run immediately before
`m5.instantiate`. This is done in the Simulator module.
This will break cases where a user utilizes the stdlib AbstractBoard but
does not use the stdlib Simulator module. As such, an Exception is
raised in these cases explaining the fix to the user. This is done via a
hack where the boards' `createCCObject` function (inheritted
from SimObject) is overriden with a check to ensure `_connect_things`
has been run. To fix the `_pre_instantiate` function must be executed
prior to `m5.instantiate` in the Python configuration script. Test and
config scripts in the gem5 repo have been updated accordingly.
Change-Id: Ibaef36eb7433ce104b861b1da80fc600f08f715a
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/65051
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
In the case where no regex was specified (i,e., `regex == None`), the
`_iterable_regex` function returned `None`. For the testlib to work this
function must return an iterable.
Without this patch it is not possible to compare two files without a
specifying a regex.
Change-Id: Ibc8a2f783c3b786dbdca6a4284850b594024f2f9
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/64851
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
