* Deprecates the setting of FS/SE mode via the `Simulator` module.
* Moved the creation of the `Root` object from the `Simulator` to the
board.
* Moved the setting of `sim_quantum` from the `Simulator` to the
processor.
* Allows for easier development of boards which support both SE and FS
mode simulation by moving board setup function calls to occur after the
set_workload function is call which sets a boards stats `is_fs` status.
Ruby requires each machine type to have a continuous set of version
numbers starting at 0. We were hiding this from users/developers by
using a Python class variable in the stdlib. Unfortunately, with
multiple ruby systems this doesn't work anymore.
As a stop-gap this change adds "resetting" these versions to the
beginning of `incorporate_caches`. It would be better to fix this in the
C++ code (and assign these numbers in C++ probably via the RubySystem),
but that's a bigger change than is needed right now.
---------
Signed-off-by: Jason Lowe-Power <jason@lowepower.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Previously, whether the board object or the memory_system returned
the memory ports was not consistent in the cache_hierarchies
This commit makes it consistently use the board. Note: the board
is a better place so it can customize the ports (e.g., add I/O
components or other things.
This commit also makes the arm board consistent with the other
boards and removes the specialized `get_mem_ports` that was not
used.
This change allows for the `_setup_memory_range` and `_setup_board`
functions to know if the board is to run a FS or SE workload, thus
allowing for a baord to handle both cases considerably easier than
before. With this change all functions are called after FS or SE
is declared via the `_set_fullsystem` function and thus all can
accomodate for SE and FS workloads.
There are several parts to this PR to work towards #1349 .
(1) Make RubySystem::getBlockSizeBytes non-static by providing ways to
access the block size or passing the block size explicitly to classes.
The main changes are:
- DataBlocks must be explicitly allocated. A default ctor still exists
to avoid needing to heavily modify SLICC. The size can be set using a
realloc function, operator=, or copy ctor. This is handled completely
transparently meaning no protocol or config changes are required.
- WriteMask now requires block size to be set. This is also handled
transparently by modifying the SLICC parser to identify WriteMask
types and call setBlockSize().
- AbstractCacheEntry and TBE classes now require block size to be set.
This is handled transparently by modifying the SLICC parser to
identify these classes and call initBlockSize() which calls
setBlockSize() for any DataBlock or WriteMask.
- All AbstractControllers now have a pointer to RubySystem. This is
assigned in SLICC generated code and requires no changes to protocol
or configs.
- The Ruby Message class now requires block size in all constructors.
This is added to the argument list automatically by the SLICC parser.
(2) Relax dependence on common functions in
src/mem/ruby/common/Address.hh
so that RubySystem::getBlockSizeBits is no longer static. Many classes
already have a way to get block size from the previous commit, so they
simply multiple by 8 to get the number of bits. For handling SLICC and
reducing the number of changes, define makeCacheLine, getOffset, etc. in
RubyPort and AbstractController. The only protocol changes required are
to change any "RubySystem::foo()" calls with "m_ruby_system->foo()".
For classes which do not have a way to get access to block size but
still used makeLineAddress, getOffset, etc., the block size must be
passed to that class. This requires some changes to the SimObject
interface for two commonly used classes: DirectoryMemory and
RubyPrefecther, resulting in user-facing API changes
User-facing API changes:
- DirectoryMemory and RubyPrefetcher now require the cache line size as
a non-optional argument.
- RubySequencer SimObjects now require RubySystem as a non-optional
argument.
- TesterThread in the GPU ruby tester now requires the cache line size
as a non-optional argument.
(3) Removes static member variables in RubySystem which control
randomization, cooldown, and warmup. These are mostly used by the Ruby
Network. The network classes are modified to take these former static
variables as parameters which are passed to the corresponding method
(e.g., enqueue, delayHead, etc.) rather than needing a RubySystem object
at all.
Change-Id: Ia63c2ad5cf0bf9d1cbdffba5d3a679bb4d3b1220
(4) There are two major SLICC generated static methods:
getNumControllers()
on each cache controller which returns the number of controllers created
by the configs at run time and the functions which access this method,
which are MachineType_base_count and MachineType_base_number. These need
to be removed to create multiple RubySystem objects otherwise NetDest,
version value, and other objects are incorrect.
To remove the static requirement, MachineType_base_count and
MachineType_base_number are moved to RubySystem. Any class which needs
to call these methods must now have a pointer to a RubySystem. To enable
that, several changes are made:
- RubyRequest and Message now require a RubySystem pointer in the
constructor. The pointer is passed to fields in the Message class
which require a RubySystem pointer (e.g., NetDest). SLICC is modified
to do this automatically.
- SLICC structures may now optionally take an "implicit constructor"
which can be used to call a non-default constructor for locally
defined variables (e.g., temporary variables within SLICC actions). A
statement such as "NetDest bcast_dest;" in SLICC will implicitly
append a call to the NetDest constructor taking RubySystem, for
example.
- RubySystem gets passed to Ruby network objects (Network, Topology).
The setting of the `sim_quantum` parameter makes considerably more sense
to occur in the Processor. Through the `_pre_instnatiate` functions this
is now possible.
It makes much more sense for the Root Object to be create within the
board and passed where required. Creating it in the Simulator class is
not required.
For this to work the signuature of the `_pre_instantiate` function in
`AbstractBoard` has been updated to return the Root object.
THis is deprecated in favor of the board determining whether the
simulation is FS or SE. Usually this will be contingent on which
`set_workload` funciton has been called. Regardless, it is the board's
responsibility. The user should not need to explicitly declare this any
longer.
This commit changes metric units (e.g. kB, MB, and GB) to binary units
(KiB, MiB, GiB) in various files. This PR covers files that were missed
by a previous PR that also made these changes.
This change adds MADT entries to the X86Board. Previously, the kernel in
full-system mode was complaining about a `ACPI BIOS Error (bug): Invalid
table length 0x24 in RSDT/XSDT (20190816/tbutils-291)`. This patch fixes
the invalid length and initializes all the tables correctly.
Signed-off-by: Kaustav Goswami <kggoswami@ucdavis.edu>
This PR changes memory and cache sizes in various parts of the gem5
codebase to use binary units (e.g. KiB) instead of metric units (e.g.
kB). This makes the codebase more consistent, as gem5 automatically
converts memory and cache sizes that are in metric units to binary
units.
This PR also adds a warning message to let users know when an
auto-conversion from base 10 to base 2 units occurs.
There were a few places in configs and in the comments of various files
where I didn't change the metric units, as I couldn't figure out where
the parameters with those units were being used.
Previously, when passing the -re option while using multisim, the files
simerr.txt and simout.txt would be redirected into the m5out directory
instead of the correct subdirectory. They would also have a name of the
format
Spawn_gem5PoolWorker-some-integer_(simout|simerr).txt, which doesn't
indicate which simulation the files correspond to.
This commit fixes these issues by redirecting simerr.txt and simout.txt
into the correct subdirectory.
Change-Id: I0a25a9fd8dc672949f5f85fc5ca6452529301a73
This commit refactors the base 10 to base 2 error message such
that it uses the preexisting _split_suffix function instead
of a new function based off of _split_suffix. This commit also
removes the new helper function used previously.
Change-Id: I44d9ac3d8b98bcff33d6bfea7ffbdb5009272ede
**Note**: Erin needs to complete the commit by expanding this test to
properly test the behavior of this change.
To run the pyunit tests:
```sh
scons build/ALL/gem5.opt -j`nproc`
./build/ALL/gem5.opt tests/run_pyunit.py
```
Change-Id: I8cea0fe8b088e03e84072a000444953768bc3151
According to the pybind documentation, "When combining *args or **kwargs
with Keyword arguments you should not include py::arg tags for the
py::args and py::kwargs arguments."
In the current implementation of gem5, if you use the cxxMethod
decorator on a function that has *args or **kwargs, gem5 will
incorrectly add these variables to the pybind generated declaration.
I.e., def f(arg1, arg2, *args, **kwargs): -> .def("f", &f,
py::arg("arg1"), py::arg("arg2"), py::arg("*args"), py::arg("**kwargs"))
which is incorrect pybind code.
To fix this problem, we should ignore variables in the generator if they
are *args or **kwargs. This change skips these variables when creating
the pybind declaration.
Change-Id: I44a1e0eb0b5fc5c1e1d423ba145d456bff92c6b8
This commit refactors the error message added to convert.py.
A mapping between the base 10 and base 2 suffix magnitudes
(e.g. k: ki, M: Mi, etc.) and a new function that extracts the
magnitude and numerical value have been added. Also, a warning
message has been added to the toMemoryBandwidth function in
addition to the one in toMemorySize.
Change-Id: I3ae157d13c7089d38a34a6e4c35a2b58978106d0
This commit contains the rest of the base 2 vs base 10 cache/memory
size clarifications. It also changes the warning message to use
warn(). With these changes, the warning message should now no
longer show up during a fresh compilation of gem5.
Change-Id: Ia63f841bdf045b76473437f41548fab27dc19631
This commit adds a warning message for when cache or memory sizes
will be automatically converted from metric units (e.g. kB) to
binary units (e.g. KiB).
Change-Id: I4ddf199ff2f00c78bbcb147e04bd88c496fa16ae
This commit changes files in src/python/gem5 so that memory and
cache sizes use "KiB", "MiB", and "GiB" instead of "KB", etc. This
makes the codebase more consistent, as gem5 will automatically
convert memory and cache sizes that are in metric units (KB) to
binary units (KiB).
Change-Id: If5b1e908ddcff7182b71e789229a3ba1fa6ad1f1
- Updated search query so that resources that are not compatible with
the gem5 version are still downloaded and used but a warning is thrown
instead of returning an error.
In `get_default_kernel_root_val()`, now prioiritizes
the explicit disk_device passed from the user over the
default implemented by the board.
Also adjusts syntax for selecting this value in
`set_kernel_disk_workload()` for consistency.
Change-Id: Icddcf438f5b96c2288c3cc608782f191df2c394e
Added documentation for `strided_generator.py` and
`strided_generator_core.py.`
Updated clarity of documentation for `linear_generator.py`,
`linear_generator_core.py`, `random_generator.py`, and
`random_generator_core.py`.
Made `max_addr` exclusive instead of inclusive for strided and linear
traffic generation in `strided_gen.cc` and `linear_gen.cc`.
This commit moves the requires() call that checks the cache coherence
protocol above the imports. This change was made for the chi private l1,
ruby mesi three level, mesi two level, and mi example cache hierarchies.
This ensures that a clear error message about having the wrong coherence
protocol is printed, rather than a less useful message.
Change-Id: I3bac1ffcb1f8a9d94e486237f880cf248e442ba8
- gem5 was querying the full version of gem5 that is `24.0.0.0` while
searching for resources.
This was causing an error to find resources on staging branch.
This change trims the gem5 version to be just the major.minor version.
Change-Id: I30c3a1b38c631981f797ef0fd2b616e6a66ca18e
This change adds a new utility function for processing Spatter traces
into SpatterKernels under parse_kernels.
Additionally, it adds documentation for all the utility functions in
spatter_kernel.py.
Lastly, it adds an example script for running one spatter trace using
SpatterGenerator to the examples.
This change extend the AbstractMemory class to add a getter method that
allows other components to get the memory's range without interleaving.
This method will be useful if other components in the system need to
interleave the memory range different to the way the memory has
interleaved them.
This change fixes the way indices are generated in a multi generator
setup.
It changes it from all cores generating the same trace of indices for
accessing the index array to each core generating an interleaved subset
of indices.
For an example look below for traces (indices to index array) in a 2
core setup.
Before:
core_0: 0, 1, 2, 3, 4, 5, 6, 7, ...
core_1: 0, 1, 2, 3, 4, 5, 6, 7, ...
After:
core_0: 0, 1, 2, 3, 8, 9, 10, 11, ...
core_1: 4, 5, 6, 7, 12, 13, 14, 15, ...
Additionally, this change fixes the SpatterKernel class in the standard
library to comply with the change in the SpatterGen source code.
Often, you want to add another argument to the default kernel arguments.
This function allows you to do that on the `kernel_disk_workload` board
mixin.
This allows for multiple gem5 simulations to be spawned from a single
parent gem5 process, as defined in a simgle gem5 configuration. In this
design _all_ the `Simulator`s are defined in the simulation script and
then added to the mutlisim module. For example:
```py
from gem5.simulate.Simulator import Simulator
import gem5.utils.multisim as multisim
# Construct the board[0] and board[1] as you wish here...
simulator1 = Simulator(board=board[0], id="board-1")
simulator2 = Simulator(board=board[1], id="board-2")
multisim.add_simulator(simulator1)
multisim.add_simulator(simulator2)
```
This specifies that two simulations are to be run in parallel in
seperate threads: one specified by `simulator1` and another by
`simulator2`. They are then added to MultiSim via the
`multisim.add_simulator` function. The user can specify an id via the
Simulator constructor. This is used to give each process a unique id and
output directory name. Given this, the id should be a helpful name
describing the simulation being specified. If not specified one is
automatically given.
To run these simulators we use `<gem5 binary> -m gem5.utils.multisim
<script> -p <num_processes>`. Note: multisim is an executable module in
gem5. This is the same module we input into our scripts to add the
simulators. This is an intentionally modular encapsulated design. When
the module processes a script it will schedule multiple gem5 jobs and,
dependent on the number of processes specified, will create child gem5
processes to processes tjese jobs (jobs are just gem5 simulations in
this case). The `--processes` (`-p`) argument is optional and if not
specified the max number of processes which can be run concurrently will
be the number of available threads on the host system.
The id for each process is used to create a subdirectory inside the
`outputdor` (`m5out`) of that id name. E.g, in the example above the
ID's are `board-1` and `board-2`. Therefore the m5 out directory will
look as follows:
```sh
- m5out
- board-1
- stats.txt
- config.ini
- config.json
- terminal.out
- board-2
- stats.txt
- config.ini
- config.json
- terminal.out
```
Each simulations output is encapsulated inside the subdirectory of the
id name.
If the multisim configuation script is passed directly to gem5 (like a
traditional gem5 configuraiton script, i.e.: `<gem5 binary> <script>`),
the user may run a single simulation specified in that script by passing
its id as an argument. E.g. `<gem5 binary> <script> board-1` will run
the `board-1` simulation specified in `script`. If no argument is passed
an Exception is raised asking the user to either specify or use the
MultiSim module if multiprocessing is needed.
If the user desires a list of ids of the simulations specified in a
given MultiSim script, they can do so by passing the `--list` (`-l`)
parameter to the config script. I.e., `<gem5 binary> <script> --list`
will list all the IDs for all the simulations specified in`script`.
This change comes with two new example scripts found in
'configs/example/gem5_library/multsim" to demonstrate multisim in both
an SE and FS mode simulation. Tests have been added which run these
scripts as part of gem5' Daily suite of tests.
Notes
=====
* **Bug fixed**: The `NoCache` classic cache hierarchy has been modified
so the Xbar is no longet set with a `__func__` call. This interfered
with MultiProcessing as this structure is not serializable via Pickle.
This was quite bad design anyway so should be changed
* **Change**: `readfile_contents` parameter previously wrote its value
to a file called "readfile" in the output dorectory. This has been
changed to write to a file called "readfile_{hash}" with "{hash}" being
a hash of the `readfile_contents`. This ensures that, during multisim
running, this file is not overwritten by other processes.
* **Removal note**: This implementation supercedes the functionality
outlined in 'src/python/gem5/utils/multiprocessing'. As such, this code
has been removed.
Limitations/Things to Fix/Improve
=================================
* Though each Simulator process has its own output directory (a
subdirectory within m5out, with an ID set by the user unique to that
Simulator), the stdout and stderr are still output to the terminal, not
the output directory. This results in: 1. stdout and stderr data lost
and not recorded for these runs. 2. An incredibly noisy terminal output.
* Each process uses the same cached resources. While there are locks on
resources when downloading, each processes will hash the resources they
require to ensure they are valid. This is very inefficient in cases
where resources are common between processes (e.g., you may have 10
processes each using the same disk image with each processes hashing the
disk images independently to give the same result to validate the
resources).
Change-Id: Ief5a3b765070c622d1f0de53ebd545c85a3f0eee
---------
Signed-off-by: Jason Lowe-Power <jason@lowepower.com>
Co-authored-by: Jason Lowe-Power <jason@lowepower.com>
This PR adds source code for C++ implementation of SpatterGen as well as
SpatterKernel. SpatterGen uses a PyBindMethod to add kernels to the
backend code. This way the process of processing json files could be
offloaded to python. In addition it adds standard library components for
SpatterGenCore and SpatterGen. These two components follow the same
structure as AbstractCore and AbstractProcessor. In addition
spatter_kernel.py adds a definition for SpatterKernel in python to make
adding kernels to C++ easier. Also it adds utility functions for parsing
dictionaries read from json as well as partitioning traces for multicore
setups.
This change adds code for SpatterGenCore and SpatterGen as well
as SpatterKernel to the standard library. SpatterGenCore and
SpatterGen follow the same structure as AbstractCore and
AbstractProcessor. spatter_kernel.py adds utility functions
to parse dictionaries as well as partition a list into
multiple lists through interleaving to be used when setting up
a multicore SpatterGen.
Change-Id: I003553e97f901c0724f5feac0bb6e21a020bd6ad
Bigs fixed of note:
1. The 'find' method has been fixed to work. This involved making
'children' a class implemented per-subclass as required.
2. The 'get_all_stats_of_name' method has been removed. This was not
working at all correctly and is largely doing what 'find' does.
2. The functionality to get an element in a vector via an attribute call
(i.e., self.vector1 == self.vector[1]) has been implemented this
maintaining backwards compatibility with the regular Python stats.
Change-Id: I31a4ccc723937018a3038dcdf491c82629ddbbb2
The SimStat Object is nothing more than a group of other SimStats and is
therefore logically a group. With this, functionality can be shared more
easily.
Change-Id: I5dce23a02d5871e640b422654ca063e590b1429a
1. Adds newly added PyStat classes to "__init__.py", ensuring they can
all be accessed via a `m5.ext.pystats` import.
2. Simplifies the layout out "__init__.py" to just import all classes
from all files.
Change-Id: I43bfc5e7ff1aec837e661905304c6fb10b00c90e
This commit addresses Jason's comment
(https://github.com/gem5/gem5/pull/996#discussion_r1613870880) which
highlighted putting the `_m5.stats.processDumpQueue` call in the
iteration through the `root` object in `get_simstat` caused this
function be potentially called many times when it only needs to be
called once. This chance moved this call to just before this iteration
and will tehrefore only be called once (if required) per `get_simstat`
execution.
Change-Id: I16908b6dee063a0df7877a19e215883963bfb081
