derek/gem5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gpu-compute,dev-hsa: Send vendor packet completion signal

Browse Source 
gem5 does not currently implement any vendor-specific HSA packets.
Starting in ROCm 5.5, vendor packets appear to end with a completion
signal. Not sending this completion causes gem5 to hang. Since these
packets are not documented anywhere and need to be reverse engineered we
send the completion signal, if non-zero, and finish the packet as is the
current behavior.

Testing: HIP examples working on most recent ROCm release (5.7.1).

Change-Id: Id0841407bec564c84f590c943f0609b17e01e14c
This commit is contained in:
Matthew Poremba
2023-10-21 12:36:35 -05:00
parent 06bf783a85
commit d05433b3f6
2 changed files with 24 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
src/dev/hsa/hsa_packet.hh
View File

@@ -100,6 +100,14 @@ struct _hsa_barrier_or_packet_t
uint64_t completion_signal;
};
struct _hsa_generic_vendor_pkt
{
uint32_t padding[14];
Addr completion_signal;
};
// All HSA AQL packets are 64 bytes. Confirm that here.
static_assert(sizeof(_hsa_generic_vendor_pkt) == 64);
} // namespace gem5
#endif // __DEV_HSA_HSA_PACKET_HH__

23
src/gpu-compute/gpu_command_processor.cc
View File

@@ -473,18 +473,27 @@ GPUCommandProcessor::driver()
*/
/**
* TODO: For now we simply tell the HSAPP to finish the packet,
* however a future patch will update this method to provide
* the proper handling of any required vendor-specific packets.
* In the version of ROCm that is currently supported (1.6)
* the runtime will send packets that direct the CP to
* invalidate the GPUs caches. We do this automatically on
* each kernel launch in the CU, so this is safe for now.
* TODO: For now we simply tell the HSAPP to finish the packet and write a
* completion signal, if any. However, in the future proper handing may be
* required for vendor specific packets.
*
* In the version of ROCm that is currently supported the runtime will send
* packets that direct the CP to invalidate the GPU caches. We do this
* automatically on each kernel launch in the CU, so that situation is safe
* for now.
*/
void
GPUCommandProcessor::submitVendorPkt(void *raw_pkt, uint32_t queue_id,
Addr host_pkt_addr)
{
auto vendor_pkt = (_hsa_generic_vendor_pkt *)raw_pkt;
if (vendor_pkt->completion_signal) {
sendCompletionSignal(vendor_pkt->completion_signal);
}
warn("Ignoring vendor packet\n");
hsaPP->finishPkt(raw_pkt, queue_id);
}