Commit Graph

962 Commits

Author SHA1 Message Date
Rajarshi Das
f71450d26d python,util: Fix magic number check in decode_inst_dep_trace.py (#560)
The decode_inst_dep_trace.py opens the trace file in read mode, and
subsequently reads the magic number from the trace file. Once the number
is read, it is compared against the string 'gem5' without decoding it
first. This causes the comparison to fail.
The fix addresses this by calling the decode() routine on the output of
the read() call. Please find the details in the associated issue #543
2023-11-14 03:47:04 -08:00
Bobby R. Bruce
cb62b08989 util-docker: Update Ubuntu 20.04 to use GCC-10
GCC-9 is no longer supported.

Change-Id: I09bf8f744546908b1c06615b458b31b9b814b61a
2023-11-13 01:36:52 -08:00
Bobby R. Bruce
c40c4450f5 util-docker: Remove GCC Version 9 from Dockerfiles
As we are no longer testing for GCC Version 9, we no longer need to
compile these Docker images.

Change-Id: Ia8fc712043ce211ff46da47fdce691a67ecdbb54
2023-11-13 01:29:56 -08:00
Bobby R. Bruce
b6ce2d0db8 misc: Add GitHub Runner API rate limiting (#497)
This stops the 'action-run.sh' from exhausting the GitHub API by adding
sleeps.
2023-10-24 14:51:31 -07:00
Bobby R. Bruce
b670ed9fba util: Add 'sudo' to rm WORK_DIR command (#496)
Unfortunately Actions uses docker contaienrs to create files on the
system with root permissions. The 'vagrant' user which we login to run
the Actions Runner, can't remove these files. However, 'vagrant' is part
of the sudo group and can therefore use sudo to remove these files.

I don't like this, but it works.
2023-10-24 14:51:19 -07:00
Jason Lowe-Power
73c48a4828 arch-riscv: Add dynamic VLEN and ELEN configuration support to RVV path (#171)
At this moment, VLEN and ELEN RVV parameters are set as constants that
need to be modified at compile time if you want to experiment with
different values. With this patch, I want to set a first point to
discuss how to configure these parameters dynamically.

Also, I have modified some data types that were provoking wrong
behaviour in particular instructions when using a large enough VLEN
value in the considered range inside the specification.
2023-10-19 07:41:39 -07:00
Bobby R. Bruce
62e5198796 docker-images: Use GitHub Container Registry (#418)
This PR aims to enhance our Docker image build and registry management
by implementing multi-platform support and migrating from the Google
Docker registry to the GitHub Container Registry. Issue:
[#336](https://github.com/gem5/gem5/issues/336)
2023-10-18 22:08:01 -07:00
Adrià Armejach
bfb295ac3f util: cpt_upgrader fix vregs size for #PR171
* Make cpt_upgrader set vregs of size MaxVecLenInBytes

Change-Id: Ie7e00d9bf42b705a0fb30c9d203933fc2e9bdcd9
2023-10-19 00:27:58 +02:00
Bobby R. Bruce
3783afff5d util: Enable KVM on VMs and ensure working in Runners
This patch:

1. Adds setup scripting to "provision_root.sh" to setup and enable KVM,
   for the 'vagrant' user, for VMs which are capable of this.
2. Runs a check on each VM to see if KVM can be run sucessfully within a
   docker container. If so, the GitHub Actions runner is given a 'kvm'
   label. It is unknown at this time if GitHub Runners can utlized KVM
   but it is open to their processes.

Change-Id: Idfcbb7bfa3e5b7cc47d29aea50fb1ebcafdb7acc
2023-10-16 21:21:31 -07:00
Bobby R. Bruce
d18087af96 util: Add halt-helper.sh
This script helps use safely halt vagrant VMs.

Change-Id: I2f2f36b93f82e07756d069334db178604a9915b3
2023-10-16 21:14:09 -07:00
Bobby R. Bruce
4b9c4e1e17 misc: Add --all to Runner docker system prune
Without `--all` `docker prune --force --volumes` will remove everything
exception non-dangling images. For an image to be considered dangling it
must be untagged and/or not used by a container at that time. As most of
the images we download are tagged (e.g., `:latest`) then most of our
images are never removed without the inclusion of `--all` which will
remove any image not currently used by a container.

Images were starting to accumulate on runners. This will ensure they do
not and are cleaned after each job run.

Change-Id: I6d8441a11d22fdcf827e9c44422dbcf02cf600e0
2023-10-16 13:33:30 -07:00
Bobby R. Bruce
cfef2ac23b util-docker: Fix end-of-line error in docker-bake.hcl
Change-Id: I2c792f35d8c74e29cf0dc0bc1287b6b5f3e4d6c8
2023-10-16 12:06:01 -07:00
Ivana Mitrovic
cb078f14c6 docker-bake: Changed compilers names to be more descriptive 2023-10-16 11:53:45 -07:00
Ivana Mitrovic
5b721b033f docker-bake: modified .hcl file
Migrated all the image build definitions from docker-compose.yaml to the bake file.
2023-10-16 11:47:49 -07:00
Ivana Mitrovic
df471092d9 dockerfiles: multi-platform setup (#336)
Updated Dockerfiles to work with multi-platform setups
2023-10-16 11:47:49 -07:00
Bobby R. Bruce
5240c07d3c util: Fix runners to extent to max disk size (#460)
THe `lvextend` command extends the logical volume. However, the
`resize2fs` command is needed to extend the filesystem to fill the
logical volume.

Prior to this patch the filesystem ran out of space despite there being
enough room in the volume. This was just wasted free space.
2023-10-16 09:20:13 -07:00
Kaustav Goswami
68af3f45c9 tests: updated the nightly tests to use SST 13.0.0 (#441)
PR https://github.com/gem5/gem5/pull/396 updates the gem5 SST bridge to
use StandardMem in SST. This change updates the nightly tests to use SST
13.0.0 instead of SST 11.1.0. It also updates the dockerfile.

Change-Id: I5c109c40379d2f09601a1c9f19c51dd716c6582e

---------

Signed-off-by: Kaustav Goswami <kggoswami@ucdavis.edu>
Co-authored-by: Bobby R. Bruce <bbruce@ucdavis.edu>
2023-10-13 14:31:35 -07:00
Bobby R. Bruce
298119e402 misc,python: Run pre-commit run --all-files
Applies the `pyupgrade` hook to all files in the repo.

Change-Id: I9879c634a65c5fcaa9567c63bc5977ff97d5d3bf
2023-10-10 21:47:07 -07:00
Bobby R. Bruce
3f5d7d647a misc: Run pre-commit autoupdate (#419)
1. Runs `pre-commit autoupdate`.
2. Runs `pre-commit run --all-files`.
3. Adds (2.) to ".git-blame-ignore-rev".
2023-10-10 21:41:33 -07:00
Bobby R. Bruce
25b2786db8 misc,python: Add requirements-txt-fixer to pre-commit (#422) 2023-10-10 14:30:39 -07:00
Bobby R. Bruce
ddf6cb88e4 misc: Run pre-commit run --all-files
This is reflect the updates made to black when running `pre-commit
autoupdate`.

Change-Id: Ifb7fea117f354c7f02f26926a5afdf7d67bc5919
2023-10-10 14:01:58 -07:00
Bobby R. Bruce
dc38a801b7 Merge branch 'develop' into workflows-for-new-runners 2023-10-09 23:10:18 -07:00
Bobby R. Bruce
da212e04b5 Merge branch 'develop' into requirements-fixer-hook 2023-10-09 22:37:25 -07:00
Bobby R. Bruce
fa8c9414b2 misc,python: Run pre-commit run --all-files
This applies the automatical formatting to the .yaml files.

Change-Id: I10d067ba65722aca8aaf64a62b42ae57de468e75
2023-10-09 13:20:25 -07:00
Bobby R. Bruce
402ec3a57c misc,python: Run pre-commit run --all-files
This applies the `requirement-txt-fixer` to the repo.

Change-Id: I23b1d26ad8deb49ec0019095efc6d253ac1c817c
2023-10-09 13:10:23 -07:00
Bobby R. Bruce
d5e454138a util: Remove 'run' and 'build' tags from runners
Change-Id: Ib7b2eba5f08a1d8a311dc20cb55f540a5cd7dc7b
2023-10-09 09:56:32 -07:00
Bobby R. Bruce
df3bcaf143 util: Make all runs "build" and "run"
Change-Id: If9ecf467efa5c7118d34166953630e6c436c55a4
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
53219bf827 util: Add Troubleshooting for "Vagrant failed..."
Change-Id: I01e637f09084acb6c5fbd7800b3e578a43487849
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
6571a54a65 util: Use a Multi-Machine Vagrantfile
This patch removed the bespoke "vm_manager.sh" script in favor of a
Multi-Machine Vagrantfile.

With this the users needs to only change the variables in Vagrantfile
then use the standard `vagrant` commands to launch the VMs/Runners.

Change-Id: Ida5d2701319fd844c6a5b6fa7baf0c48b67db975
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
0e5c6d9f50 util: Resize VM root partition max size to ~128GB
Prior to this change we were limited to a root partition with only 60GB
of space which caused issues when running larger simulations (see:
https://github.com/gem5/gem5/issues/165).

There are two factors in this issue which this patch resolves:

1. The root partition in the VM was capped at 60GB despite the virtual
machines size being capped at 128GB. This resulted in libvirt giving the
VM free space it couldn't use. To fix this `lvextend` was added to the
"provision_root.sh" script to resize the root partition to fill the
available space.
2. The virtual machine size can be set via the `machine_virtual_size`
parameter. The minimum and default value is 128GB. This wasn't exposed
previously. Now, if we required, we can increase the size of the VM/Root
partition if we require (though I believe 128GB is more than sufficient
for now).

Fixes: https://github.com/gem5/gem5/issues/165
Change-Id: I82dd500d8807ee0164f92d91515729d5fbd598e3
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
f36449be98 util: Add missing copyright notices
Change-Id: I243046c17264eb5c522285096ecf9c7e5e968322
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
8c2d414223 util: Cleanup the provision_root.sh
Change-Id: I58215dddc34476695c7aedc77b55d338e0304198
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
c0cb16ba89 util: Create HOSTNAME variable
Change-Id: Ia68f1bef2bb9e4e5e18476b6100be80f8cf1c799
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
529423f47a util: Remove note about ssh use
This is confusing and setting the ssh username and password is normal.

Change-Id: Ic925e92ade47f455c86a461a267b8cad7aa6d7ba
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
a924fa3bdc util: Add action-run.sh to run Action Runners
The "action-run.sh" action replaces inline scripting in the Vagrantfile.

The major improvement is this script runs an infinite loop and
configures the runners to be ephemeral. This means they cease to exist
after a job is complete. The script then cleans the VM workspace and the
loop restarts by configuring and setting up another runner. This means
our VMs no longer accumulate files that eventually lead to the VM
running out of space.

Change-Id: Iba6dc9a480f5805042602f120fc84bdc47a96d55
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
3e1c0b0714 util: Move runners from gem5 repository to gem5 org
There are two places self-hosted runners can exist on GitHub:

1. At the level of the repository: In this case the runners can only be
used by that repository and runners can only be distinguished from one
another by labels.
2. At the level of the organization: In this case the runners can be
used by any repository in the organization, thus increasing their
versatility. In addition to labels, runners in the level of the
organization can be organized into groups.

While we do not use our self-hosted runners on other repositories, there
may be future use for this, so we might as well enable it now.

Change-Id: Id5e113194314336221dcdc8c2858b352afcbaf6e
2023-10-06 15:53:57 -07:00
Bobby R. Bruce
d1f9f98747 util: Make all runners the same type
Having two types of GitHub Action Runners has not yielded much benefit
and caused confusion and inefficiencies. This change simplifies things
to having just one runner with 8-cores and 16GB of memory. It is
sufficient to build gem5 and run most simulations.

Change-Id: Ic49ae5e98b02086f153f4ae2a4eedd8a535786c8
2023-10-06 15:53:57 -07:00
Jason Lowe-Power
6f5d877b1a misc: Update gem5 to use clang-15 and clang-16 (#365)
This introduces the changes necessary for clang-15 and clang-16 to run
within gem5, and adds them to the compiler tests.

This also updates the dockerfiles for ubuntu 22.04 to include the steps
necessary to compile clang-15 and clang-16.
2023-10-04 09:51:12 -07:00
Bobby R. Bruce
e211674625 util-docker: Fix/Improve ubuntu-22.04_clang-16
* Removes `+` symbol accidently left in (this broke building).
* Removes `ARG` thus making the Docker exclusively for clang-16.
* Adds "llvm.sh" to the repo. This stops us being dependent on the url
  download. The script is under the apache license therefore compatible.
* Merges several `apt install` commands into one.

Change-Id: Iaf411656aac83f67f5395b20efd96ecc1eabb263
2023-09-29 13:12:19 -07:00
Melissa Jost
a79dc3f23c util: Add steps to compile clang-15 and clang-16
This updates the dockerfiles for ubuntu 22.04 to include the
steps necessary to compile clang-15 and clang-16.

Change-Id: I2bba6393ab93a6ce05a2c3ce31f3bbc71bcdca7c
2023-09-29 08:32:01 -07:00
Bobby R. Bruce
62d34ef374 misc: 'sim{out/err}' -> 'sim{out/err}.txt' (#250)
By default, the `--stderr-file` and `--stdout-file` arguments were
directing the simulator to output files named "simerr" and "simout"
respectively if an output redirect was requested.

A small annoyance is these files lack an extension meaning programs
refuse to open them, or don't do so withou additional effort. On many
systems they are assumed to scripts.

This patch adds the `.txt` extension to both, thus clearly indicating to
other programs these are text files and can be opened and read as such.
2023-09-27 17:36:03 -07:00
Bobby R. Bruce
391f62b213 misc: 'sim{out/err}' -> 'sim{out/err}.txt'
By default, the --stderr-file and --stdout-file arguments were
directing the simulator output to files named "simerr" and
"simout" respectively if an output redirect was requested.

A small annoyance is these files lack an extension meaning programs
refuse to open them, or to do so without some additional effort. On
many systems they are assumed to scripts.

This patch adds the .txt extension to both, thus clearly indicating
to other programs these are text files and can be opened to be read
as such.

Change-Id: Iff5af4a9e6966b4467d005a029dbf401099fbd35
2023-09-21 12:57:43 -07:00
Bobby R. Bruce
3f9afe96c6 python,util: Add Python MyPy Stubgen to enable Pylance IntelliSense (#307)
This allows us to generate stubs for the modules in gem5. The output
will be a "typings" directory which can be used by Pylance (Python
IntelliSense) to infer typings in Visual Studio Code.

Note: A "typings" directory in the root of the workspace is the default
location for Pylance to look for typings. This can be changed via
`python.analysis.stubPath` in "settings.json".

Usage
=====

```
pip3 install -r requirements.txt
scons build/ALL/gem5.opt -j$(nproc)
./build/ALL/gem5.opt util/gem5-stubgen.py
```
2023-09-21 11:52:16 -07:00
Bobby R. Bruce
600ea81031 util: Add 'obtain-resource.py' utility
This can be used to obtain a resource from gem5-resources.

Change-Id: I922d78ae0450bf011f18893ffc05cb1ad6c97572
2023-09-14 15:33:17 -07:00
Giacomo Travaglini
f95e1505b8 util: Fix TLM configs making use of TraceCPU replayer
A recent PR [1] moved the TraceCPU away from the BaseCPU hierarchy.
While the common etrace_replayer.py has been amended, I missed these
hybrid TLM + TraceCPU example scripts.

[1]: https://github.com/gem5/gem5/pull/302

Change-Id: I7e9bc9a612d2721d72f5881ddb2fb4d9ee011587
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
2023-09-13 13:36:33 +01:00
Bobby R. Bruce
bceac5d951 util: Allow MyPy stubgen to aid Pylance IntelliSense
Change-Id: I42fe177e5ae428afd0f23ea482b6af5b7d3ecaf9
2023-09-12 15:19:49 -07:00
Roger Chang
e41184fafc util: Update the RISC-V PCState checkpoint
Change-Id: I64b6a3e1706173a001f5f8fb06756bd50d65f5bd
2023-09-12 17:54:35 +08:00
Bobby R. Bruce
a217c218e0 util: Update gcn-gpu Dockerfile (#290)
This adds the setting up of environment variables to the gcn-gpu
Dockerfile from the halofinder Dockerfile in gem5-resources so that we
don't need to use a separate Docker image in the gpu tests.
2023-09-11 11:19:49 -07:00
Bobby R. Bruce
7091a8b7a0 util: Revert "Add docker prune cron to GitHub..."
This reverts commit 0249d47acc,
https://github.com/gem5/gem5/pull/271.

This solution doesn't work. GitHub runners pull the images they need at
the start of job (i.e., all the images they may need for each step).
They then create the containers later, at the step they are needed.
This solution therefore breaks in the case a cleanup happens during the
running of a job. I.e., a `docker system prune` happens after setup,
therefore deleting all the images, then the job tries to use one of the
images during a step.

This crontab solution may work if we can only do it when the runner is
in an idle state. Whether this is possible is unknown.

Change-Id: I7cb5b2d98d596e9380ae1525c7d66ad97af1b59b
2023-09-10 20:32:49 -07:00
Melissa Jost
16e8c95091 util: Update gcn-gpu Dockerfile
This adds the setting up of environment variables to the gcn-gpu
Dockerfile from the halofinder Dockerfile in gem5-resources
so that we don't need to use a seperate Docker image in the
gpu tests.

Change-Id: Ifcc7a4c6bbcd5289ce9561923366e9ed193f170c
2023-09-08 10:25:09 -07:00