misc: Merge branch 'release-staging-v23-1' into develop

This is just a stanity check to ensure all changes are in the `develop`
branch.

Change-Id: I7f6e6709338ae9a386bc7527d5cf4daf10d768c2
This commit is contained in:
Bobby R. Bruce
2023-12-27 11:42:56 -08:00
10 changed files with 314 additions and 35 deletions

View File

@@ -1,3 +1,159 @@
# Version 23.1
gem5 Version 23.1 is our first release where the development has been on GitHub.
During this release, there have been 362 pull requests merged which comprise 416 commits with 51 unique contributors.
## Significant API and user-facing changes
### The gem5 build can is now configured with `kconfig`
- Most gem5 builds without customized options (excluding double dash options) (e.g. , build/X86/gem5.opt) are backwards compatible and require no changes to your current workflows.
- All of the default builds in `build_opts` are unchanged and still available.
- However, if you want to specialize your build. For example, use customized ruby protocol. The command `scons PROTOCOL=<PROTOCAL_NAME> build/ALL/gem5.opt` will not work anymore. you now have to use `scons <kconfig command>` to update the ruby protocol as example. The double dash options (`--without-tcmalloc`, `--with-asan` and so on) are still continue to work as normal.
- For more details refer to the documentation here: [kconfig documentation](https://www.gem5.org/documentation/general_docs/kconfig_build_system/)
### Standard library improvements
#### `WorkloadResource` added to resource specialization
- The `Workload` and `CustomWorkload` classes are now deprecated. They have been transformed into wrappers for the `obtain_resource` and `WorkloadResource` classes in `resource.py`, respectively.
- Code utilizing the older API will continue to function as expected but will trigger a warning message. To update code using the `Workload` class, change the call from `Workload(id='resource_id', resource_version='1.0.0')` to `obtain_resource(id='resource_id', resource_version='1.0.0')`. Similarly, to update code using the `CustomWorkload` class, change the call from `CustomWorkload(function=func, parameters=params)` to `WorkloadResource(function=func, parameters=params)`.
- Workload resources in gem5 can now be directly acquired using the `obtain_resource` function, just like other resources.
#### Introducing Suites
Suites is a new category of resource being introduced in gem5. Documentation of suites can be found here: [suite documentation](https://www.gem5.org/documentation/gem5-stdlib/suites).
#### Other API changes
- All resource object now have their own `id` and `category`. Each resource class has its own `__str__()` function which return its information in the form of **category(id, version)** like **BinaryResource(id='riscv-hello', resource_version='1.0.0')**.
- Users can use GEM5_RESOURCE_JSON and GEM5_RESOURCE_JSON_APPEND env variables to overwrite all the data sources with the provided JSON and append a JSON file to all the data source respectively. More information can be found [here](https://www.gem5.org/documentation/gem5-stdlib/using-local-resources).
### Other user-facing changes
- Added support for clang 15 and clang 16
- gem5 no longer supports building on Ubuntu 18.04
- GCC 7, GCC 9, and clang 6 are no longer supported
- Two `DRAMInterface` stats have changed names (`bytesRead` and `bytesWritten`). For instance, `board.memory.mem_ctrl.dram.bytesRead` and `board.memory.mem_ctrl.dram.bytesWritten`. These are changed to `dramBytesRead` and `dramBytesWritten` so they don't collide with the stat with the same name in `AbstractMemory`.
- The stats for `NVMInterface` (`bytesRead` and `bytesWritten`) have been change to `nvmBytesRead` and `nvmBytesWritten` as well.
## Full-system GPU model improvements
- Support for up to latest ROCm 5.7.1.
- Various changes to enable PyTorch/TensorFlow simulations.
- New packer disk image script containing ROCm 5.4.2, PyTorch 2.0.1, and Tensorflow 2.11.
- GPU instructions can now perform atomics on host addresses.
- The provided configs scripts can now run KVM on more restrictive setups.
- Add support to checkpoint and restore between kernels in GPUFS, including adding various AQL, HSA Queue, VMID map, MQD attributes, GART translations, and PM4Queues to GPU checkpoints
- move GPU cache recorder code to RubyPort instead of Sequencer/GPUCoalescer to allow checkpointing to occur
- add support for flushing GPU caches, as well as cache cooldown/warmup support, for checkpoints
- Update vega10_kvm.py to add checkpointing instructions
## SE mode GPU model improvements
- started adding support for mmap'ing inputs for GPUSE tests, which reduces their runtime by 8-15% per run
## GPU model improvements
- update GPU VIPER and Coalescer support to ensure correct replacement policy behavior when multiple requests from the same CU are concurrently accessing the same line
- fix bug with GPU VIPER to resolve a race conflict for loads that bypass the TCP (L1D$)
- fix bug with MRU replacement policy updates in GPU SQC (I$)
- update GPU and Ruby debug prints to resolve various small errors
- Add configurable GPU L1,L2 num banks and L2 latencies
- Add decodings for new MI100 VOP2 insts
- Add GPU GLC Atomic Resource Constraints to better model how atomic resources are shared at GPU TCC (L2$)
- Update GPU tester to work with both requests that bypass all caches (SLC) and requests that bypass only the TCP (L1D$)
- Fixes for how write mask works for GPU WB L2 caches
- Added support for WB and WT GPU atomics
- Added configurable support to better model the latency of GPU atomic requests
- fix GPU's default number of HW barrier/CU to better model amount of concurrency GPU CUs should have
## RISC-V RVV 1.0 implemented
This was a huge undertaking by a large number of people!
Some of these people include Adrià Armejach who pushed it over the finish line, Xuan Hu who pushed the most recent version to gerrit that Adrià picked up,
Jerin Joy who did much of the initial work, and many others who contributed to the implementation including Roger Chang, Hoa Nguyen who put significant effort into testing and reviewing the code.
- Most of the instructions in the 1.0 spec implemented
- Works with both FS and SE mode
- Compatible with Simple CPUs, the O3, and the minor CPU models
- User can specify the width of the vector units
- Future improvements
- Widening/narrowing instructions are *not* implemented
- The model for executing memory instructions is not very high performance
- The statistics are not correct for counting vector instruction execution
## ArmISA changes/improvements
- Architectural support for the following extensions:
* FEAT_TLBIRANGE
* FEAT_FGT
* FEAT_TCR2
* FEAT_SCTLR2
- Arm support for SVE instructions improved
- Fixed some FEAT_SEL2 related issues:
- [Fix virtual interrupt logic in secure mode](https://github.com/gem5/gem5/pull/584)
- [Make interrupt masking handle VHE/SEL2 cases](https://github.com/gem5/gem5/pull/430)
- Removed support for Arm Jazelle and ThumbEE
- Implementation of an Arm Capstone Disassembler
## Other notable changes/improvements
- Improvements to the CHI coherence protocol implementation
- Far atomics implemented in CHI
- Ruby now supports using the prefetchers from the classic caches, if the protocol supports it. CHI has been extended to support the classic prefetchers.
- Bug in RISC-V TLB to fixed to correctly count misses and hits
- Added new RISC-V Zcb instructions https://github.com/gem5/gem5/pull/399
- RISC-V can now use a separate binary for the bootloader and kernel in FS mode
- DRAMSys integration updated to latest DRAMSys version (5.0)
- Improved support for RISC-V privilege modes
- Fixed bug in switching CPUs with RISC-V
- CPU branch preditor refactoring to prepare for decoupled front end support
- Perf is now optional when using the KVM CPU model
- Improvements to the gem5-SST bridge including updating to SST 13.0
- Improved formatting of documentation in stdlib
- By default use isort for python imports in style
- Many, many testing improvements during the migration to GitHub actions
- Fixed the elastic trace replaying logic (TraceCPU)
## Known Bugs/Issues
- [RISC-V RVV Bad execution of riscv rvv vss instruction](https://github.com/gem5/gem5/issues/594)
- [RISC-V Vector Extension float32_t bugs/unsupported widening instructions](https://github.com/gem5/gem5/issues/442)
- [Implement AVX xsave/xstor to avoid workaround when checkpointing](https://github.com/gem5/gem5/issues/434)
- [Adding Vector Segmented Loads/Stores to RISC-V V 1.0 implementation](https://github.com/gem5/gem5/issues/382)
- [Integer overflow in AddrRange subset check](https://github.com/gem5/gem5/issues/240)
- [RISCV64 TLB refuses to access upper half of physical address space](https://github.com/gem5/gem5/issues/238)
- [Bug when trying to restore checkpoints in SPARC: “panic: panic condition !pte occurred: Tried to execute unmapped address 0.”](https://github.com/gem5/gem5/issues/197)
- [BaseCache::recvTimingResp can trigger an assertion error from getTarget() due to MSHR in senderState having no targets](https://github.com/gem5/gem5/issues/100)
# Version 23.0.1.0
This minor release incorporates documentation updates, bug fixes, and some minor improvements.
## Documentation updates
* "TESTING.md" has been updated to more accurately reflect our current testing infrastructure.
* "README" has been replaced with "README.md" and includes more up-to-date information on using gem5.
* "CONTRIBUTING.md" has been updated to reflect our migration to GitHub and the changes in policy and proceedures.
* Where needed old references to Gerrit have been removed in favor of GitHub.
## Bug Fixes
* Fixes an assert failure when using ARM which was trigged when `shiftAmt` is 0 for a UQRSH instruction.
* Fixes `name 'fatal' is not defined` being thrown when tracing is off.
* Fixes a bug in ARM in which the TLBIOS instructions were decoded as normal MSR instructions with no effect on the TLBs.
* Fixes invalid `packet_id` value in flit.
* Fixes default CustomMesh for use with Garnet.
## Minor Improvements
* The gem5 resources downloader now outputs more helpful errors in the case of a failure.
* "util/github-runners-vagrant" has been added. This outlines how to setup a GitHub Action's set-hosted runner for gem5.
* The PyUnit tests have been refactored to no longer download large resources during testing.
* Using Perf is now optional when utilizing KVM CPUs.
# Version 23.0.0.1
**[HOTFIX]** Fixes compilation of `GCN3_X86` and `VEGA_X85`.

View File

@@ -597,14 +597,6 @@ for variant_path in variant_paths:
env.Append(LINKFLAGS=['-Wl,--no-keep-memory'])
else:
error("Unable to use --no-keep-memory with the linker")
# Treat warnings as errors but white list some warnings that we
# want to allow (e.g., deprecation warnings).
env.Append(CCFLAGS=['-Werror',
'-Wno-error=deprecated-declarations',
'-Wno-error=deprecated',
])
else:
error('\n'.join((
"Don't know what compiler options to use for your compiler.",

View File

@@ -31,7 +31,7 @@ PROJECT_NAME = gem5
# This could be handy for archiving the generated documentation or
# if some version control system is used.
PROJECT_NUMBER = [DEVELOP-FOR-23.0]
PROJECT_NUMBER = v23.1.0.0
# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute)
# base path where the generated documentation will be put.

View File

@@ -389,10 +389,8 @@ decode QUADRANT default Unknown::unknown() {
if (status.fs == FPUStatus::OFF)
return std::make_shared<IllegalInstFault>("FPU is off",
machInst);
status.fs = FPUStatus::DIRTY;
xc->setMiscReg(MISCREG_STATUS, status);
Fc1_bits = Mem;
}}, {{
EA = rvZext(sp + offset);
@@ -420,10 +418,8 @@ decode QUADRANT default Unknown::unknown() {
if (status.fs == FPUStatus::OFF)
return std::make_shared<IllegalInstFault>("FPU is off",
machInst);
status.fs = FPUStatus::DIRTY;
xc->setMiscReg(MISCREG_STATUS, status);
freg_t fd;
fd = freg(f32(Mem_uw));
Fd_bits = fd.v;

View File

@@ -32,6 +32,6 @@ namespace gem5
/**
* @ingroup api_base_utils
*/
const char *gem5Version = "DEVELOP-FOR-23.1";
const char *gem5Version = "23.1.0.0";
} // namespace gem5

View File

@@ -47,7 +47,6 @@
#define __MEM_PORT_HH__
#include <memory>
#include <sstream>
#include <stack>
#include <string>

View File

@@ -67,26 +67,6 @@ You can check the status of the runner here: https://github.com/organizations/{G
If the runner ever shows as offline, you can rerun the `vagrant up --provider=libvirt` command to make sure everything is working properly.
## Troubleshooting
### The default libvirt disk image storage pool is on the wrong drive
By default libvirt will store disk images in "/var/lib/libvirt/images".
This is not ideal as it is on a small root partition.
A solution to this is to change the default storage location.
To do so, do the following:
```sh
virsh pool-list --all # Confirm here a "default" pool exist. We'll modify this.
virsh pool-dumpxml default >default-pool.xml # We take a dump of the default then removed it.
virsh pool-destroy default
virsh pool-undefine default
vim default-pool.xml # Change the image path to the desired path
virsh pool-define default-pool.xml # From here we re-add the default.
virsh pool-start default
virsh pool-autostart default
```
### Error: "Vagrant failed to initialize at a very early stage"
W set the `VAGRANT_HOME` environment variable to the CWD.

View File

@@ -0,0 +1,57 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
config.vm.box = "generic/ubuntu2204"
config.vm.box_check_update = true
config.vm.define "<VM NAME>"
config.vm.hostname = "<VM NAME>"
# allows us to ssh into the machine, addressing the problem below
# https://www.reddit.com/r/vagrant/comments/sb7hfl/new_to_vagrant_getting_efault_warning/
config.ssh.username = "vagrant"
config.ssh.password = "vagrant"
config.vm.provider "libvirt" do |vb|
# Customize the amount of cpus and memory on the VM:
vb.cpus = "4".to_i
vb.memory = "16384".to_i
end
# sets up vm
config.vm.provision :shell, path: "provision_root.sh"
config.vm.provision :shell, privileged: false, path: "provision_nonroot.sh"
# To ensure we don't run out of memory, we enable dynamic Swap Space. This is
# done via the "swapspace" daemon: https://pqxx.org/development/swapspace/
config.vm.provision :shell, inline: "sudo apt install swapspace -y"
# The provision_root.sh adds the vagrant user to the docker group, so we need to reload the VM.
config.vm.provision :reload
config.vm.provision :shell, run: 'always', inline: <<-SHELL
# When running gem5 in SE mode we must overcommit memory.
# This is run on every startup of the VM.
/sbin/sysctl vm.overcommit_memory=1
SHELL
config.vm.provision :shell, privileged: false, run: 'always', inline: <<-SHELL
if [ -d ~/actions-runner ]; then
# This will be run everytime the VM is run (once created).
cd actions-runner
nohup ./run.sh &
else
# This will be run the first time the VM is created.
mkdir ~/actions-runner && cd ~/actions-runner
curl -so actions-runner-linux-x64-2.304.0.tar.gz -L \
https://github.com/actions/runner/releases/download/v2.304.0/actions-runner-linux-x64-2.304.0.tar.gz
tar xzf ./actions-runner-linux-x64-2.304.0.tar.gz
# configure the runner
# echo automatically sets the name of the runner, and the tags
# create a personal access token with admin permission and copy it into the curl command
echo -ne '\n\nbuild\n\n' | ./config.sh --url https://github.com/<GITHUB REPO> --token $(curl -L \
-X POST -H "Accept: application/vnd.github+json" -H "Authorization: Bearer <PERSONAL ACCESS TOKEN>" \
-H "X-GitHub-Api-Version: 2022-11-28" \
https://api.github.com/repos/<GITHUB REPO>/actions/runners/registration-token | jq -r '.token')
# start the runner
nohup ./run.sh &
fi
SHELL
end

View File

@@ -0,0 +1,56 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
config.vm.box = "generic/ubuntu2204"
config.vm.box_check_update = true
config.vm.define "<VM NAME>"
config.vm.hostname = "<VM NAME>"
# allows us to ssh into the machine, addressing the problem below
# https://www.reddit.com/r/vagrant/comments/sb7hfl/new_to_vagrant_getting_efault_warning/
config.ssh.username = "vagrant"
config.ssh.password = "vagrant"
config.vm.provider "libvirt" do |vb|
# Customize the amount of cpus and memory on the VM:
vb.cpus = "1".to_i
vb.memory = "8192".to_i
end
# sets up vm
config.vm.provision :shell, path: "provision_root.sh"
config.vm.provision :shell, privileged: false, path: "provision_nonroot.sh"
# To ensure we don't run out of memory, we enable dynamic Swap Space. This is
# done via the "swapspace" daemon: https://pqxx.org/development/swapspace/
config.vm.provision :shell, inline: "sudo apt install swapspace -y"
# The provision_root.sh adds the vagrant user to the docker group, so we need to reload the VM.
config.vm.provision :reload
config.vm.provision :shell, run: 'always', inline: <<-SHELL
# When running gem5 in SE mode we must overcommit memory.
# This is run on every startup of the VM.
/sbin/sysctl vm.overcommit_memory=1
SHELL
config.vm.provision :shell, privileged: false, run: 'always', inline: <<-SHELL
if [ -d ~/actions-runner ]; then
# This will be run everytime the VM is run (once created).
cd actions-runner
nohup ./run.sh &
else
# This will be run the first time the VM is created.
mkdir ~/actions-runner && cd ~/actions-runner
curl -so actions-runner-linux-x64-2.304.0.tar.gz -L \
https://github.com/actions/runner/releases/download/v2.304.0/actions-runner-linux-x64-2.304.0.tar.gz
tar xzf ./actions-runner-linux-x64-2.304.0.tar.gz
# configure the runner
# echo automatically sets the name of the runner, and the tags
# create a personal access token with admin permission and copy it into the curl command
echo -ne '\n\nrun\n\n' | ./config.sh --url https://github.com/<GITHUB REPO> --token $(curl -L \
-X POST -H "Accept: application/vnd.github+json" -H "Authorization: Bearer <PERSONAL ACCESS TOKEN>" \
-H "X-GitHub-Api-Version: 2022-11-28" \
https://api.github.com/repos/<GITHUB REPO>/actions/runners/registration-token | jq -r '.token')
# start the runner
nohup ./run.sh &
fi
SHELL
end

View File

@@ -0,0 +1,43 @@
#!/bin/bash
NUM_RUNNERS=20
NUM_BUILDERS=3
RUNNER_PREFIX="$(hostname)-runner-"
BUILDER_PREFIX="$(hostname)-builder-"
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
export VAGRANT_HOME=${SCRIPT_DIR}
param="up"
if [[ $# -ge 1 ]]; then
param=$1
if [[ "${param}" != "destroy" ]] && [[ "${param}" != "shutdown" ]]; then
echo "Only valid parameters are 'destroy' and 'shutdown' to destroy all VMs or shutdown all VMs"
exit 1
fi
fi
for (( i=1; i<=NUM_RUNNERS; i++ )); do
sed -i "s/ config.vm.define.*/ config.vm.define \"${RUNNER_PREFIX}${i}\"/g" Vagrantfile-runner
sed -i "s/ config.vm.hostname.*/ config.vm.hostname = \"${RUNNER_PREFIX}${i}\"/g" Vagrantfile-runner
if [[ "${param}" == "destroy" ]]; then
VAGRANT_VAGRANTFILE=Vagrantfile-runner vagrant destroy -f
elif [[ "${param}" == "shutdown" ]]; then
VAGRANT_VAGRANTFILE=Vagrantfile-runner vagrant halt -f
else
VAGRANT_VAGRANTFILE=Vagrantfile-runner vagrant up --provider=libvirt
fi
done
for (( i=1; i<=NUM_BUILDERS; i++ )); do
sed -i "s/ config.vm.define.*/ config.vm.define \"${BUILDER_PREFIX}${i}\"/g" Vagrantfile-builder
sed -i "s/ config.vm.hostname.*/ config.vm.hostname = \"${BUILDER_PREFIX}${i}\"/g" Vagrantfile-builder
if [[ "${param}" == "destroy" ]]; then
VAGRANT_VAGRANTFILE=Vagrantfile-builder vagrant destroy -f
elif [[ "${param}" == "shutdown" ]]; then
VAGRANT_VAGRANTFILE=Vagrantfile-builder vagrant halt -f
else
VAGRANT_VAGRANTFILE=Vagrantfile-builder vagrant up --provider=libvirt
fi
done