The environment variable HSA_ENABLE_INTERRUPT controls if Interrupt or
busy wait signals are used in the ROCm runtime. Interrupts are not being
sent in gem5 causing simulations to hang indefinitely in certain
situations. To fix this, always disable interrupts to fall back to busy
wait signals. Using interrupts is an old and simple optimization to not
waste CPU cycles, but from the perspective of simulation this is not
important. Disabling interrupt-based HSA signals therefore increases the
number of applications working within gem5.
Change-Id: I1ae21d7ee01548a4d00a8972642079b90278f9a2
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/61652
Reviewed-by: Matt Sinclair <mattdsinclair@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Matt Sinclair <mattdsinclair@gmail.com>
The supported version of ROCm in GPUFS only supports the DKMS build of
the amdgpu driver. However, since a gem5 user can potentially pass in
any Linux kernel as a parameter, it is possible that the DKMS package
for that kernel was not installed on the disk image. This would result
in the simulation appearing to work when in reality it is just spinning
waiting for commands from the driver. This check exits gem5 early in the
simulation and outputs an error on the console to sanity check the
correct driver is being used.
Change-Id: I708912e5625e47eba15dcb2f722772a3b2928b98
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/58129
Reviewed-by: Matt Sinclair <mattdsinclair@gmail.com>
Maintainer: Matt Sinclair <mattdsinclair@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The amdgpu driver supports fetching instructions from pages which reside
in system memory rather than device memory. This changeset adds support
to do this by adding the system hub object added in a prior changeset to
the fetch unit and issues requests to the system hub if the system bit
in the memory page's PTE is set. Otherwise, the requestor ID is set to
be device memory and the request is routed through the Ruby network /
GPU caches to fetch the instructions.
Change-Id: Ib2fb47c589fdd5e544ab6493d7dbd8f2d9d7b0e8
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/57652
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Add the constructors for the Vega TLB and TLB coalescers in the python
config. These need a pointer to the gpu device which is added as a
parameter. The last level TLB's page table walker is added as a dma
device to the system so that the port is connected to the GPU device
memory in the disjoint VIPER configuration file.
A portion of the the GPUFS system configuration file needs to be
shuffled around so that the shader CPU is created before the TLBs are
created so they can be connected to the shader's ports. This means the
real CPU init code needs to break once reaching the shader. The vendor
string must also be set after createThreads is called on real CPUs.
Change-Id: I36ed93db262b21427f3eaf4904a1c897a2894835
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/57649
Reviewed-by: Matt Sinclair <mattdsinclair@gmail.com>
Maintainer: Matt Sinclair <mattdsinclair@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The CorePairs in MOESI_AMD_Base round up the number of CPUs when
creating CPU sequencers. If the CPU count is an odd number, this was
causing the Disjoint_VIPER config to connect a sequencer that does not
exist. As a result the crossbar was waiting for a range change from the
sequencer but it never arrived, causing an assert.
This patch fixes this by conditionally connecting CPU sequencers to the
PIO port only if the ID is less than the number of CPUs.
Change-Id: I2280c0048492d43528429a947a726871f1c23ca7
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/57531
Reviewed-by: Matt Sinclair <mattdsinclair@gmail.com>
Maintainer: Matt Sinclair <mattdsinclair@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Make the necessary changes to connect Vega pagetable walkers for
full-system mode. Previously the CP and HSA packet processor could only
read AQL packets from system/host memory using proxy port. This allows
for AQL to be read from device memory which is used for non-blit
kernels.
Change-Id: If28eb8be68173da03e15084765e77e92eda178e9
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/53077
Reviewed-by: Matt Sinclair <mattdsinclair@gmail.com>
Maintainer: Matt Sinclair <mattdsinclair@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The disjoint VIPER configuration creates completely disconnected CPU and
GPU Ruby networks which can communicate only via the PCI bus. Either
garnet or simple network can be used. This copies most of the Ruby setup
from Ruby.py's create_system since creating disjoint networks is not
possible using Ruby.py.
Change-Id: Ibc23aa592f56554d088667d8e309ecdeb306da68
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/53072
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Connect the --second-disk option in GPUFS. Typically this is used as a
benchmarks disk image. If the disk is unmounted at the time of
checkpoint, a new disk image can be mounted after restoring the
checkpoint for a simple way to add new benchmarks without recreating a
checkpoint.
Change-Id: I57b31bdf8ec628006d774feacff3fde6f533cd4b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/53071
Reviewed-by: Matt Sinclair <mattdsinclair@gmail.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Matt Sinclair <mattdsinclair@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
A valid CPU vendor string (i.e., not "M5 Simulator") needs to be passed
to CPUID in order for Linux to create the sysfs files needed for ROCm's
Thunk interface to initialize properly. If these are no created
hipDeviceProperties and other basic GPU code APIs will error out.
Change-Id: I6e3f459162e4673860a8f0a88473e38d5d7be237
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/53070
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The PM4 packet processor is handling all non-HSA GPU packets such
as packets for (un)mapping HSA queues. This commit pulls many
Linux structs and defines out into their own files for clarity.
Finally, it implements the VMID related functions in AMDGPU device.
Change-Id: I5f0057209305404df58aff2c4cd07762d1a31690
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/53068
Reviewed-by: Matt Sinclair <mattdsinclair@gmail.com>
Maintainer: Matt Sinclair <mattdsinclair@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Add the devices that have been added in previous changesets to the
config file. Forward MMIO writes to the appropriate device based
on the MMIO address. Connect doorbells and forward rings to the
appropriate device based on queue type.
Change-Id: I44110c9a24559936102a246c9658abb84a8ce07e
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/53065
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Remove the line "For use for simulation and test purposes only" in files
were AMD is the only copyright holder listed in the header. This happens
to be the case for all files where this line exists, removing it
completely from gem5.
Change-Id: I623f266b002f564301b28774f49081099cfc60fd
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/53943
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The flow for Full System amdgpu is the use KVM to boot linux and begin
loading the driver module. However, the amdgpu module requires reading
the VGA ROM located at 0xc0000 in X86. KVM does not support having a
small 128KiB hole at this location, therefore we take a checkpoint and
switch to a timing CPU to continue loading the drivers before the VGA
ROM is read.
This creates a checkpoint just before the first MMIOs. This is indicated
by three interrupts being sent to the PCI device. After three interrupts
in a row are counted a checkpoint exit event occurs. The interrupt
counter is reset if a non-interrupt PCI read is seen.
Change-Id: I23b320abe81ff6e766cb3f604eca2979339938e5
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/46161
Reviewed-by: Matt Sinclair <mattdsinclair@gmail.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Matt Sinclair <mattdsinclair@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This is an initial configuration capable of booting Linux and
registering a PCI device which registers as an AMD Vega 10 (Frontier
Edition) GPU. It it loosely based on the the example/fs.py and gem5 book
full system example scripts. The top-level file is meant to be modular
such that convenience scripts can be created to set arguments
automatically and then call the main run function.
This will evolve over time as more full-system GPU components are added
and the network topology needed for disjoint address spaces is created
for the VIPER protocol.
Change-Id: I7002213ca8de5eb73919e49fb11840a688744012
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/44907
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Matt Sinclair <mattdsinclair@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
