The test `ruby_test_test-ALL-x86_64-opt-MatchStdout` is currently
failing because the reference file doesn't match the actual output. This
PR changes the reference file to match.
This commit is adding two python files:
* ruby_mem_test.py is the canonical gem5 configuration script,
and it is an adaptation of the existing ruby_mem_test.py [1].
The main difference is the use of the TlmController as a
cache controller, and the use of TlmGenerator instead of
the MemTest memory tester. The config is minimally setting up
the system. The extent of the testing is specified in the second
python file:
* read_shared_unit.py: "unit-test" for the CHI ReadShared request
The file should be passed to the ruby_mem_test.py as cmdline
argument:
build/ARM/gem5.opt <>/ruby_mem_test.py <>/read_shared_unit.py
This is a simple testing file. We should ideally generate separate
test files for separate transactions/scenarios.
The test file can have whatever for inside, it only needs to comply
to the minimal interface required by the ruby_mem_test.py, which is
to define the following function:
def test_all(generator):
[1]: https://github.com/gem5/gem5/blob/stable/\
configs/example/ruby_mem_test.py
Change-Id: I767ede9b8572f3eafe677c84da45fd904d77e319
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
This commit removes the `protocol=None` argument in various
gem5_verify_config()s because protocol is set to None by
default. Also, two print statements that were taken out in
previous commits were put back in with different function calls.
This commit changes the fs/linux/arm and learning_gem5 tests as
they were previously failing with the Ruby change. The
fs/linux/arm long tests require the addition of a new gem5 build,
ARM_X86, which builds the ARM and X86 ISAs with the
MESI_Two_Level cache hierarchy.
This commit changes MESIThreeLevel, MIExample, and OctopiCache
so they work with this PR. It also adds MESIThreeLevel and
OctopiCache to the testlib tests.
This commit adds trusted_stats.json files for CHIL1,
MESI_Three_Level, MIExample, and OctopiCache. These jsons are
used to verify the output of tests from test_memory_traffic_gen.py.
This adds all of the hierarchies supported in the standard library. We
can soon move to using a different build target and run all hierarchies!
Change-Id: Ic065a679ea34c3bb2f71b3b133806d240039fbb5
Signed-off-by: Jason Lowe-Power <jason@lowepower.com>
This change does many things, but they must all be atomically done.
**USER FACING CHANGE**: The Ruby protocols in Kconfig have changed names
(they are now the same case as the SLICC file names). So, after this
commit, your build configurations need to be updated. You can do so by
running `scons menuconfig <build dir>` and selecting the right ruby
options. Alternatively, if you're using a `build_opts` file, you can run
`scons defconfig build/<ISA> build_opts/<ISA>` which should update your
config correctly.
Detailed changes are described below.
Kconfig changes:
- Kconfig files in ruby now must all be declared in the ruby/Kconfig
file
- All of the protocol names are changed to match their slicc file names
including the case
- A new option is available called "Use multiple protocols" which should
be selected if multiple protocols are selected. This is only used to
set the PROTOCOL variable to "MULTIPLE" when in multiple mode.
- The PROTOCOL variable can now be "MULTIPLE" which means it will be
ignored. If it's not "MULTIPLE" then it holds the "main" protocol,
which is necessary for backwards compatibility with the Ruby.py files.
Ruby config changes:
To make this change backwards compatible with Ruby.py, this change adds
a new "protocol" config called MULTIPLE.py which is used to allow the
user to set a "--protocol" option on the command line. This is only
needed if you are using a gem5 binary with multiple protocols but need
to use Ruby.py.
stdlib changes:
- Make the coherence protocol file behave like the ISA file
- Add a function to get the coherence protocol from the `CacheHierarchy`
like we do with the ISA in the `Processor`.
- Use this function where `get_runtime_coherence_protocol` was used
- Update the requires code to work with the ne CoherenceProtocol
- Fix a typo in the AMD Hammer name and also add the missing MSI
protocol
Scons changes:
- In Ruby we now gather up all of the protocols and build them all if
there are multiple protocols
- There's some bending over backwards to tell the user if they are using
an out of date gem5.build/config file and how to update it
- Note that multiple ruby protocols adds a significant amount of time to
the build since we have to run slicc twice for each file.
build_opts:
- Update all files with new names
- Add a new NULL_All_Ruby that will be used for testing
Signed-off-by: Jason Lowe-Power <jason@lowepower.com>
At some point 'system' -> 'board' in the stdlib code the replacement
policy tests used. Due to this the output is slightly different meaning
the refs need updated.
This was causing the Daily Tests to fail.
* Deprecates the setting of FS/SE mode via the `Simulator` module.
* Moved the creation of the `Root` object from the `Simulator` to the
board.
* Moved the setting of `sim_quantum` from the `Simulator` to the
processor.
* Allows for easier development of boards which support both SE and FS
mode simulation by moving board setup function calls to occur after the
set_workload function is call which sets a boards stats `is_fs` status.
With this patch the pannotia tests now:
1. Download the resources to 'gpu-pannotia' in the
'tests/gem5/resources' directory. This is where other test resources are
store.
2. Download thr USA-road-d.NY.gr dataset from Google cloud bucket in a
decompressed state.
2. Avoid re-download the resources if they are already present on the
host machine.
The Daily tests have been failing as the learning-gem5 Ruby test now
exits at tick 9831 instead of tick 9981.
**Note**: The cause of this change is currently unknown. I'm not sure if
this is symptomatic of something bigger but for now I only observe this
bug failure and this patch at least silences the error.
It makes much more sense for the Root Object to be create within the
board and passed where required. Creating it in the Simulator class is
not required.
For this to work the signuature of the `_pre_instantiate` function in
`AbstractBoard` has been updated to return the Root object.
This commit changes metric units (e.g. kB, MB, and GB) to binary units
(KiB, MiB, GiB) in various files. This PR covers files that were missed
by a previous PR that also made these changes.
This PR changes memory and cache sizes in various parts of the gem5
codebase to use binary units (e.g. KiB) instead of metric units (e.g.
kB). This makes the codebase more consistent, as gem5 automatically
converts memory and cache sizes that are in metric units to binary
units.
This PR also adds a warning message to let users know when an
auto-conversion from base 10 to base 2 units occurs.
There were a few places in configs and in the comments of various files
where I didn't change the metric units, as I couldn't figure out where
the parameters with those units were being used.
This commit adds a test that checks that strings representing
base 10 memory sizes or base 10 memory bandwidths are correctly
converted to strings representing base 2 values.
Change-Id: Ie8cac15f06b4ceb1786484fea4e8ba2111f4e8d3
**Note**: Erin needs to complete the commit by expanding this test to
properly test the behavior of this change.
To run the pyunit tests:
```sh
scons build/ALL/gem5.opt -j`nproc`
./build/ALL/gem5.opt tests/run_pyunit.py
```
Change-Id: I8cea0fe8b088e03e84072a000444953768bc3151
This commit contains the rest of the base 2 vs base 10 cache/memory
size clarifications. It also changes the warning message to use
warn(). With these changes, the warning message should now no
longer show up during a fresh compilation of gem5.
Change-Id: Ia63f841bdf045b76473437f41548fab27dc19631
A new host tag `gcn_gpu` has been added. This allows for selection of
those GPU tests which depend upon the gcn-gpu docker image to run.
In addition to this, the square GPU tests has been moved to the CI
tests. This ensures some GPU code is compiled and run on every PR.
This allows for multiple gem5 simulations to be spawned from a single
parent gem5 process, as defined in a simgle gem5 configuration. In this
design _all_ the `Simulator`s are defined in the simulation script and
then added to the mutlisim module. For example:
```py
from gem5.simulate.Simulator import Simulator
import gem5.utils.multisim as multisim
# Construct the board[0] and board[1] as you wish here...
simulator1 = Simulator(board=board[0], id="board-1")
simulator2 = Simulator(board=board[1], id="board-2")
multisim.add_simulator(simulator1)
multisim.add_simulator(simulator2)
```
This specifies that two simulations are to be run in parallel in
seperate threads: one specified by `simulator1` and another by
`simulator2`. They are then added to MultiSim via the
`multisim.add_simulator` function. The user can specify an id via the
Simulator constructor. This is used to give each process a unique id and
output directory name. Given this, the id should be a helpful name
describing the simulation being specified. If not specified one is
automatically given.
To run these simulators we use `<gem5 binary> -m gem5.utils.multisim
<script> -p <num_processes>`. Note: multisim is an executable module in
gem5. This is the same module we input into our scripts to add the
simulators. This is an intentionally modular encapsulated design. When
the module processes a script it will schedule multiple gem5 jobs and,
dependent on the number of processes specified, will create child gem5
processes to processes tjese jobs (jobs are just gem5 simulations in
this case). The `--processes` (`-p`) argument is optional and if not
specified the max number of processes which can be run concurrently will
be the number of available threads on the host system.
The id for each process is used to create a subdirectory inside the
`outputdor` (`m5out`) of that id name. E.g, in the example above the
ID's are `board-1` and `board-2`. Therefore the m5 out directory will
look as follows:
```sh
- m5out
- board-1
- stats.txt
- config.ini
- config.json
- terminal.out
- board-2
- stats.txt
- config.ini
- config.json
- terminal.out
```
Each simulations output is encapsulated inside the subdirectory of the
id name.
If the multisim configuation script is passed directly to gem5 (like a
traditional gem5 configuraiton script, i.e.: `<gem5 binary> <script>`),
the user may run a single simulation specified in that script by passing
its id as an argument. E.g. `<gem5 binary> <script> board-1` will run
the `board-1` simulation specified in `script`. If no argument is passed
an Exception is raised asking the user to either specify or use the
MultiSim module if multiprocessing is needed.
If the user desires a list of ids of the simulations specified in a
given MultiSim script, they can do so by passing the `--list` (`-l`)
parameter to the config script. I.e., `<gem5 binary> <script> --list`
will list all the IDs for all the simulations specified in`script`.
This change comes with two new example scripts found in
'configs/example/gem5_library/multsim" to demonstrate multisim in both
an SE and FS mode simulation. Tests have been added which run these
scripts as part of gem5' Daily suite of tests.
Notes
=====
* **Bug fixed**: The `NoCache` classic cache hierarchy has been modified
so the Xbar is no longet set with a `__func__` call. This interfered
with MultiProcessing as this structure is not serializable via Pickle.
This was quite bad design anyway so should be changed
* **Change**: `readfile_contents` parameter previously wrote its value
to a file called "readfile" in the output dorectory. This has been
changed to write to a file called "readfile_{hash}" with "{hash}" being
a hash of the `readfile_contents`. This ensures that, during multisim
running, this file is not overwritten by other processes.
* **Removal note**: This implementation supercedes the functionality
outlined in 'src/python/gem5/utils/multiprocessing'. As such, this code
has been removed.
Limitations/Things to Fix/Improve
=================================
* Though each Simulator process has its own output directory (a
subdirectory within m5out, with an ID set by the user unique to that
Simulator), the stdout and stderr are still output to the terminal, not
the output directory. This results in: 1. stdout and stderr data lost
and not recorded for these runs. 2. An incredibly noisy terminal output.
* Each process uses the same cached resources. While there are locks on
resources when downloading, each processes will hash the resources they
require to ensure they are valid. This is very inefficient in cases
where resources are common between processes (e.g., you may have 10
processes each using the same disk image with each processes hashing the
disk images independently to give the same result to validate the
resources).
Change-Id: Ief5a3b765070c622d1f0de53ebd545c85a3f0eee
---------
Signed-off-by: Jason Lowe-Power <jason@lowepower.com>
Co-authored-by: Jason Lowe-Power <jason@lowepower.com>
Bigs fixed of note:
1. The 'find' method has been fixed to work. This involved making
'children' a class implemented per-subclass as required.
2. The 'get_all_stats_of_name' method has been removed. This was not
working at all correctly and is largely doing what 'find' does.
2. The functionality to get an element in a vector via an attribute call
(i.e., self.vector1 == self.vector[1]) has been implemented this
maintaining backwards compatibility with the regular Python stats.
Change-Id: I31a4ccc723937018a3038dcdf491c82629ddbbb2
1. Thests here for the Scalar tasks are named appropriately. Not just
generic "SimStats tess".
2. We remove 'simstat' terminology. The correct word is "Pystats".
Change-Id: Idebc4e750f4be7f140ad6bff9c6772f580a24861
The big thing missing from the Vector stats was that each position in
the vector could have it's own unique id (a str, float, or int) and each
position in the vector can have its own description. Therefore, to add
this the Vector is represented as a dictionary mapping the unique ID to
a Pystat Scaler (whcih can have it's own unique description.
Change-Id: I3a8634f43298f6491300cf5a4f9d25dee8101808
This reverts commit c1de2b8762. We revert
the commit as Ruby does not use get_runtime_isa anymore after [1]
[1]: https://github.com/gem5/gem5/pull/241
Change-Id: Iaac8d64194bbd53a9b1a57a796ff92f763c75a87
