Commit Graph

946 Commits

Author SHA1 Message Date
Kaustav Goswami
68af3f45c9 tests: updated the nightly tests to use SST 13.0.0 (#441)
PR https://github.com/gem5/gem5/pull/396 updates the gem5 SST bridge to
use StandardMem in SST. This change updates the nightly tests to use SST
13.0.0 instead of SST 11.1.0. It also updates the dockerfile.

Change-Id: I5c109c40379d2f09601a1c9f19c51dd716c6582e

---------

Signed-off-by: Kaustav Goswami <kggoswami@ucdavis.edu>
Co-authored-by: Bobby R. Bruce <bbruce@ucdavis.edu>
2023-10-13 14:31:35 -07:00
Bobby R. Bruce
298119e402 misc,python: Run pre-commit run --all-files
Applies the `pyupgrade` hook to all files in the repo.

Change-Id: I9879c634a65c5fcaa9567c63bc5977ff97d5d3bf
2023-10-10 21:47:07 -07:00
Bobby R. Bruce
3f5d7d647a misc: Run pre-commit autoupdate (#419)
1. Runs `pre-commit autoupdate`.
2. Runs `pre-commit run --all-files`.
3. Adds (2.) to ".git-blame-ignore-rev".
2023-10-10 21:41:33 -07:00
Bobby R. Bruce
25b2786db8 misc,python: Add requirements-txt-fixer to pre-commit (#422) 2023-10-10 14:30:39 -07:00
Bobby R. Bruce
ddf6cb88e4 misc: Run pre-commit run --all-files
This is reflect the updates made to black when running `pre-commit
autoupdate`.

Change-Id: Ifb7fea117f354c7f02f26926a5afdf7d67bc5919
2023-10-10 14:01:58 -07:00
Bobby R. Bruce
dc38a801b7 Merge branch 'develop' into workflows-for-new-runners 2023-10-09 23:10:18 -07:00
Bobby R. Bruce
da212e04b5 Merge branch 'develop' into requirements-fixer-hook 2023-10-09 22:37:25 -07:00
Bobby R. Bruce
fa8c9414b2 misc,python: Run pre-commit run --all-files
This applies the automatical formatting to the .yaml files.

Change-Id: I10d067ba65722aca8aaf64a62b42ae57de468e75
2023-10-09 13:20:25 -07:00
Bobby R. Bruce
402ec3a57c misc,python: Run pre-commit run --all-files
This applies the `requirement-txt-fixer` to the repo.

Change-Id: I23b1d26ad8deb49ec0019095efc6d253ac1c817c
2023-10-09 13:10:23 -07:00
Bobby R. Bruce
d5e454138a util: Remove 'run' and 'build' tags from runners
Change-Id: Ib7b2eba5f08a1d8a311dc20cb55f540a5cd7dc7b
2023-10-09 09:56:32 -07:00
Bobby R. Bruce
df3bcaf143 util: Make all runs "build" and "run"
Change-Id: If9ecf467efa5c7118d34166953630e6c436c55a4
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
53219bf827 util: Add Troubleshooting for "Vagrant failed..."
Change-Id: I01e637f09084acb6c5fbd7800b3e578a43487849
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
6571a54a65 util: Use a Multi-Machine Vagrantfile
This patch removed the bespoke "vm_manager.sh" script in favor of a
Multi-Machine Vagrantfile.

With this the users needs to only change the variables in Vagrantfile
then use the standard `vagrant` commands to launch the VMs/Runners.

Change-Id: Ida5d2701319fd844c6a5b6fa7baf0c48b67db975
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
0e5c6d9f50 util: Resize VM root partition max size to ~128GB
Prior to this change we were limited to a root partition with only 60GB
of space which caused issues when running larger simulations (see:
https://github.com/gem5/gem5/issues/165).

There are two factors in this issue which this patch resolves:

1. The root partition in the VM was capped at 60GB despite the virtual
machines size being capped at 128GB. This resulted in libvirt giving the
VM free space it couldn't use. To fix this `lvextend` was added to the
"provision_root.sh" script to resize the root partition to fill the
available space.
2. The virtual machine size can be set via the `machine_virtual_size`
parameter. The minimum and default value is 128GB. This wasn't exposed
previously. Now, if we required, we can increase the size of the VM/Root
partition if we require (though I believe 128GB is more than sufficient
for now).

Fixes: https://github.com/gem5/gem5/issues/165
Change-Id: I82dd500d8807ee0164f92d91515729d5fbd598e3
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
f36449be98 util: Add missing copyright notices
Change-Id: I243046c17264eb5c522285096ecf9c7e5e968322
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
8c2d414223 util: Cleanup the provision_root.sh
Change-Id: I58215dddc34476695c7aedc77b55d338e0304198
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
c0cb16ba89 util: Create HOSTNAME variable
Change-Id: Ia68f1bef2bb9e4e5e18476b6100be80f8cf1c799
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
529423f47a util: Remove note about ssh use
This is confusing and setting the ssh username and password is normal.

Change-Id: Ic925e92ade47f455c86a461a267b8cad7aa6d7ba
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
a924fa3bdc util: Add action-run.sh to run Action Runners
The "action-run.sh" action replaces inline scripting in the Vagrantfile.

The major improvement is this script runs an infinite loop and
configures the runners to be ephemeral. This means they cease to exist
after a job is complete. The script then cleans the VM workspace and the
loop restarts by configuring and setting up another runner. This means
our VMs no longer accumulate files that eventually lead to the VM
running out of space.

Change-Id: Iba6dc9a480f5805042602f120fc84bdc47a96d55
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
3e1c0b0714 util: Move runners from gem5 repository to gem5 org
There are two places self-hosted runners can exist on GitHub:

1. At the level of the repository: In this case the runners can only be
used by that repository and runners can only be distinguished from one
another by labels.
2. At the level of the organization: In this case the runners can be
used by any repository in the organization, thus increasing their
versatility. In addition to labels, runners in the level of the
organization can be organized into groups.

While we do not use our self-hosted runners on other repositories, there
may be future use for this, so we might as well enable it now.

Change-Id: Id5e113194314336221dcdc8c2858b352afcbaf6e
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
d1f9f98747 util: Make all runners the same type
Having two types of GitHub Action Runners has not yielded much benefit
and caused confusion and inefficiencies. This change simplifies things
to having just one runner with 8-cores and 16GB of memory. It is
sufficient to build gem5 and run most simulations.

Change-Id: Ic49ae5e98b02086f153f4ae2a4eedd8a535786c8
2023-10-06 15:53:57 -07:00
Jason Lowe-Power
6f5d877b1a misc: Update gem5 to use clang-15 and clang-16 (#365)
This introduces the changes necessary for clang-15 and clang-16 to run
within gem5, and adds them to the compiler tests.

This also updates the dockerfiles for ubuntu 22.04 to include the steps
necessary to compile clang-15 and clang-16.
2023-10-04 09:51:12 -07:00
Bobby R. Bruce
e211674625 util-docker: Fix/Improve ubuntu-22.04_clang-16
* Removes `+` symbol accidently left in (this broke building).
* Removes `ARG` thus making the Docker exclusively for clang-16.
* Adds "llvm.sh" to the repo. This stops us being dependent on the url
  download. The script is under the apache license therefore compatible.
* Merges several `apt install` commands into one.

Change-Id: Iaf411656aac83f67f5395b20efd96ecc1eabb263
2023-09-29 13:12:19 -07:00
Melissa Jost
a79dc3f23c util: Add steps to compile clang-15 and clang-16
This updates the dockerfiles for ubuntu 22.04 to include the
steps necessary to compile clang-15 and clang-16.

Change-Id: I2bba6393ab93a6ce05a2c3ce31f3bbc71bcdca7c
2023-09-29 08:32:01 -07:00
Bobby R. Bruce
62d34ef374 misc: 'sim{out/err}' -> 'sim{out/err}.txt' (#250)
By default, the `--stderr-file` and `--stdout-file` arguments were
directing the simulator to output files named "simerr" and "simout"
respectively if an output redirect was requested.

A small annoyance is these files lack an extension meaning programs
refuse to open them, or don't do so withou additional effort. On many
systems they are assumed to scripts.

This patch adds the `.txt` extension to both, thus clearly indicating to
other programs these are text files and can be opened and read as such.
2023-09-27 17:36:03 -07:00
Bobby R. Bruce
391f62b213 misc: 'sim{out/err}' -> 'sim{out/err}.txt'
By default, the --stderr-file and --stdout-file arguments were
directing the simulator output to files named "simerr" and
"simout" respectively if an output redirect was requested.

A small annoyance is these files lack an extension meaning programs
refuse to open them, or to do so without some additional effort. On
many systems they are assumed to scripts.

This patch adds the .txt extension to both, thus clearly indicating
to other programs these are text files and can be opened to be read
as such.

Change-Id: Iff5af4a9e6966b4467d005a029dbf401099fbd35
2023-09-21 12:57:43 -07:00
Bobby R. Bruce
3f9afe96c6 python,util: Add Python MyPy Stubgen to enable Pylance IntelliSense (#307)
This allows us to generate stubs for the modules in gem5. The output
will be a "typings" directory which can be used by Pylance (Python
IntelliSense) to infer typings in Visual Studio Code.

Note: A "typings" directory in the root of the workspace is the default
location for Pylance to look for typings. This can be changed via
`python.analysis.stubPath` in "settings.json".

Usage
=====

```
pip3 install -r requirements.txt
scons build/ALL/gem5.opt -j$(nproc)
./build/ALL/gem5.opt util/gem5-stubgen.py
```
2023-09-21 11:52:16 -07:00
Bobby R. Bruce
600ea81031 util: Add 'obtain-resource.py' utility
This can be used to obtain a resource from gem5-resources.

Change-Id: I922d78ae0450bf011f18893ffc05cb1ad6c97572
2023-09-14 15:33:17 -07:00
Giacomo Travaglini
f95e1505b8 util: Fix TLM configs making use of TraceCPU replayer
A recent PR [1] moved the TraceCPU away from the BaseCPU hierarchy.
While the common etrace_replayer.py has been amended, I missed these
hybrid TLM + TraceCPU example scripts.

[1]: https://github.com/gem5/gem5/pull/302

Change-Id: I7e9bc9a612d2721d72f5881ddb2fb4d9ee011587
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
2023-09-13 13:36:33 +01:00
Bobby R. Bruce
bceac5d951 util: Allow MyPy stubgen to aid Pylance IntelliSense
Change-Id: I42fe177e5ae428afd0f23ea482b6af5b7d3ecaf9
2023-09-12 15:19:49 -07:00
Roger Chang
e41184fafc util: Update the RISC-V PCState checkpoint
Change-Id: I64b6a3e1706173a001f5f8fb06756bd50d65f5bd
2023-09-12 17:54:35 +08:00
Bobby R. Bruce
a217c218e0 util: Update gcn-gpu Dockerfile (#290)
This adds the setting up of environment variables to the gcn-gpu
Dockerfile from the halofinder Dockerfile in gem5-resources so that we
don't need to use a separate Docker image in the gpu tests.
2023-09-11 11:19:49 -07:00
Bobby R. Bruce
7091a8b7a0 util: Revert "Add docker prune cron to GitHub..."
This reverts commit 0249d47acc,
https://github.com/gem5/gem5/pull/271.

This solution doesn't work. GitHub runners pull the images they need at
the start of job (i.e., all the images they may need for each step).
They then create the containers later, at the step they are needed.
This solution therefore breaks in the case a cleanup happens during the
running of a job. I.e., a `docker system prune` happens after setup,
therefore deleting all the images, then the job tries to use one of the
images during a step.

This crontab solution may work if we can only do it when the runner is
in an idle state. Whether this is possible is unknown.

Change-Id: I7cb5b2d98d596e9380ae1525c7d66ad97af1b59b
2023-09-10 20:32:49 -07:00
Melissa Jost
16e8c95091 util: Update gcn-gpu Dockerfile
This adds the setting up of environment variables to the gcn-gpu
Dockerfile from the halofinder Dockerfile in gem5-resources
so that we don't need to use a seperate Docker image in the
gpu tests.

Change-Id: Ifcc7a4c6bbcd5289ce9561923366e9ed193f170c
2023-09-08 10:25:09 -07:00
Bobby R. Bruce
9cdd6093bd util: Add docker prune cron to GitHub runners (#271)
Issue-on: https://github.com/gem5/gem5/issues/254

This has been implemented in the runners. The regular pruning of the
docker images should fix the issue.
2023-09-07 12:06:41 -07:00
Bobby R. Bruce
0249d47acc util: Add docker prune cron to GitHub runners
Change-Id: Ic90ebc650b6a89606eaf9e8feafddfe15c44e578
Issue-on: https://github.com/gem5/gem5/issues/254
2023-09-06 15:09:32 -07:00
Bobby R. Bruce
1b0bb678ab util-docker: Proof-of-concept using Docker buildx
Introduced in https://github.com/gem5/gem5/pull/236 the
"docker-build.yaml" file will allow us to build and push docker images
to the GitHub Container Registry. This allows for both automation of
docker image building and allows us to utilize Github's zero-cost
pulling policy for downloads to GitHub Actions runners.

In this PR https://github.com/gem5/gem5/pull/236 has been altered to
use Docker `buildx` which allows for multi-platform Docker Image builds.
A multi-platform Docker image pull automatically pull the correct image
for your platform from a single URL. In this prototype the images are
build to both `linux/arm64` and `linux/amd64` have been set.

Docker `buildx` has it's own file format for specifying image builds
called `bake`. "util/dockerfiles/docker-bake.hcl" has been added with
the goal of replacing "util/dockerfiles/docker-compose.yaml".

In this proof-of-concept doesn't build all our docker images, just
enough to ensure it works inside our actions as intended.

Change-Id: Id0debed216c91ec514aa4fce3bc2ff4fc2ea669b
2023-09-05 17:59:06 -07:00
Matthew Poremba
3520c83673 util: Add gdb to gcn-gpu Dockerfile
gdb was originally part of the ROCm 1.6 Dockerfile a few years ago. It
got removed when updating to ROCm 4.0. This adds it back as being able
to debug things is quite useful.

Change-Id: I3f8148cde79e6cc5233fa3c8c830b64817f01d3a
2023-08-31 11:08:30 -05:00
atrah22
99fc5de3fb util: Update & fix bug in m5stats2streamline.py
1) writeBinary function binary_list can have either string or ints and it needs to be properly converted to bytes
2) packed32(x) function can have x as int or float. Incase of float it needs to be converted to int
3) encode lines to string using .decode() or else TypeError will be invoked during run

Change-Id: I678169f191901f02a80187418a17adbc1240c7d3
2023-08-27 19:07:45 -07:00
atrah22
fab458daa2 util: Update & fix bug in m5stats2streamline.py
1) writeBinary function binary_list can have either string or ints and it needs to be properly converted to bytes
2) packed32(x) function can have x as int or float. Incase of float it needs to be converted to int

Change-Id: I6a52aa59e1582dd6bb06b2d1c49ddaf8fe61c997
2023-08-27 19:07:29 -07:00
Bobby R. Bruce
74f6fa34af util-docker: Fix clang-version-8 docker container
clang v8, when installed in this manner via Docker, did not install the
libstdc++. This caused compilation errors. This patch adds the
libstdc++-10-dev package to this Dockerfile.

Change-Id: Ia0f41e82b3df2d4bf32b418b0cb78111a35e0b9f
2023-08-16 10:00:45 -07:00
Bobby R. Bruce
cfea9afae3 misc: Update MAINTAINERS.yaml (#155)
- Updates the list of current gem5 Maintainers.
- Updates the MAINATAINERS.yaml comments to reflect gem5's contribution
policy and maintainer responsibilities since moving to GitHub.
- Updates the subsystem maintainers, based on maintainer's stated
preferences.
- Adds the optional 'expert' field to subsystems.
2023-08-11 02:05:26 -07:00
Bobby R. Bruce
a41c6f8d84 misc: Remove PMC/Maintainers list from MAINTAINERS.yaml
Change-Id: I772fa31d0aeea5534355731d841cf2d118fa0df4
2023-08-10 15:48:13 -07:00
Adrià Armejach
f777cc143c util: fix cpt upgrader for rvv changes in PR #83
* Solves issue #106 by updating the cpts with the necessary vector
    registers.

Change-Id: Ifeda90e96097f0b0a65338c6b22a8258c932c585

util: clear vector_element field

Change-Id: I6c9ec4e71f66722b26de030fa139cd626bdb24dc
2023-08-04 13:59:23 +02:00
Hoa Nguyen
9ec7a1c14a util: Ignore line length check for #include pragma in C/C++ files
The length of the path of the #include pragma can be more than
79-character long.

Change-Id: Id72250c166370c7f456bd1f7d05589a49c14c33d
Signed-off-by: Hoa Nguyen <hn@hnpl.org>
2023-07-26 19:41:04 -07:00
Bobby R. Bruce
65fc9a6bfa misc: Drop older compilers and Ubuntu 18.04 (#80)
* tests,util-docker,misc: Drop compiler support for GCC 7

Change-Id: I8b17b77c92b88e78a8cb6d38cd5f045dbe80a643

* tests,util-docker,misc: Drop compiler support for clang 6.0

Change-Id: Ie3b6bfe889ad1d119cee0c9ffb04c5996517922e

* util-docker,tests,misc: Remove Ubuntu 18.04 support

18.04 is no longer supported. This patch removes specific 18.04 compiler
tests and removes our 18.04 dockerfiles. Images will no longer be
produced for specific 18.04 tasks.

Compiler images for GCC and Clang, which used 18.04 have been updated to
use 20.04.

Change-Id: I6338ab47af3287a25a557dbbeaeebcfccfdec9fc
2023-07-18 10:28:09 -07:00
Bobby R. Bruce
6062214d87 util: Add "Improving stability" sec to github-vagrant-runner (#87)
Change-Id: I9812a21523b5b29bd7f570df4f1e90dbeabea085
2023-07-17 10:42:28 -07:00
mbabaie
037e6fe33c misc: update gem5 links
This change updates all of the gerrit links to use github.

Change-Id: I2a020dafac0bd2ba99b26c6a9cd4f0c585e253f8
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/71719
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
2023-07-10 22:57:28 +00:00
KUNAL PAI
9b9dc09f6e resources: Add the gem5 Resources Manager
A GUI web-based tool to manage gem5 Resources.

Can manage in two data sources,
a MongoDB database or a JSON file.

The JSON file can be both local or remote.

JSON files are written to a temporary file before
writing to the local file.

The Manager supports the following functions
on a high-level:
- searching for a resource by ID
- navigating to a resource version
- adding a new resource
- adding a new version to a resource
- editing any information within a searched resource
(while enforcing the gem5 Resources schema
found at: https://resources.gem5.org/gem5-resources-schema.json)
- deleting a resource version
- undo and redo up to the last 10 operations

The Manager also allows a user to save a session
through localStorage and re-access it through a password securely.

This patch also provides a
Command Line Interface tool mainly for
MongoDB-related functions.

This CLI tool can currently:
- backup a MongoDB collection to a JSON file
- restore a JSON file to a MongoDB collection
- search for a resource through its ID and
view its JSON object
- make a JSON file that is compliant with the
gem5 Resources Schema

Co-authored-by: Parth Shah <helloparthshah@gmail.com>
Co-authored-by: Harshil2107 <harshilp2107@gmail.com>
Co-authored-by: aarsli <arsli@ucdavis.edu>
Change-Id: I8107f609c869300b5323d4942971a7ce7c28d6b5
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/71218
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
2023-07-08 02:01:02 +00:00
Bobby R. Bruce
84b457c6aa util: Add 'swapspace' daemon to runner VM.
As these VMs, particularly the runners, don't have much memory, the
'swapspace' daemon allows for dynamic swap spaces to be created for when
more memory is required.

Change-Id: Ie8e734a8fde54e122df33dda187c6c4aafdcd006
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/71680
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
2023-07-07 02:31:50 +00:00