The `get_simstat` function in `src/python/m5/stats/gem5stats.py` was
returning an error when a non-Root Simobject was passed:
```
AttributeError: object 'PyTrafficGen' has no attribute 'name'
At:
build/NULL_MESI_Two_Level/python/m5/SimObject.py(1430): __getattr__
build/NULL_MESI_Two_Level/python/m5/stats/gem5stats.py(279): get_simstat
```
The issue was an assumption that SimObjects have a field `name`. They
do not. To get a SimObject's name the `get_name()` function must be
used. This patch fixes this issue.
In addition to this fix, the documentation in this function has been
improved to state more clearly what can be passed and what shall be
returned. Previously it was somewhat unclear.
Change-Id: I33538120015280bb6260ccf8eba6b75ff43d280e
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/54943
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
This model is used to store and represent the "new" hierarchical stats
at the Python level. Over time these classes may be extended with
functions to ease in the analysis of gem5 stats. Though, for this
commit, such functions have been kept to a minimum.
`m5/pystats/loader.py` contains functions for translating the gem5 `_m5.stats`
statistics exposed via Pybind11 to the Python Stats model. For example:
```
import m5.pystats.gem5stats as gem5stats
simstat = gem5stats.get_simstat(root)
```
All the python Stats model classes inherit from JsonSerializable meaning
they can be translated to JSON. For example:
```
import m5.pystats.gem5stats as gem5stats
simstat = gem5stats.get_simstat(root)
with open('test.json', 'w') as f:
simstat.dump(f)
```
The stats have also been exposed via the python statistics API. Via
command line, a JSON output may be specified with the argument
`--stats-file json://<file path>`.
Change-Id: I253a869f6b6d8c0de4dbed708892ee0cc33c5665
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/38615
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This commit makes it possible to make invocations such as:
gem5.opt se.py --stats-root 'system.cpu[:].dtb' --stats-root 'system.membus'
When --stats-root is given, only stats that are under any of the root
SimObjects get dumped. E.g. the above invocation would dump stats such as:
system.cpu0.dtb.walker.pwrStateResidencyTicks::UNDEFINED
system.cpu1.dtb.walker.pwrStateResidencyTicks::UNDEFINED
system.membus.pwrStateResidencyTicks::UNDEFINED
system.membus.trans_dist::ReadReq
but not for example `system.clk_domain.clock`.
If the --stats-root is given, only new stats as defined at:
Idc8ff448b9f70a796427b4a5231e7371485130b4 get dumped, and old ones are
ignored. The commits following that one have done some initial conversion
work, but many stats are still in the old format.
Change-Id: Iadaef26edf9a678b39f774515600884fbaeec497
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/28628
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The alignment spaces in stats.txt takes up a lot of space and increases
simulation time, this commit adds the option to disable them with:
--stats-file stats.txt?spaces=False
Sample old lines with ?desc=False:
system.cpu.op_class::FloatMultAcc 0 0.00% 65.92%
system.cpu.op_class::FloatDiv 0 0.00% 65.92%
Sample new lines with ?desc=False;spaces=False:
system.cpu.op_class::FloatMultAcc 0 0.00% 65.92%
system.cpu.op_class::FloatDiv 0 0.00% 65.92%
On a 1000 dumpstats m5op loop spaces=False reduces:
* size: from 38MB to 20MB
* time: from 4.5s to 3.5s
Change-Id: Ib738b996b5646c329094cf61aaa1d977e844e759
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/28627
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Some objects need to know that we are about to dump stats to perform
prepare statistics. This is currently done by registering a callback
with the stat system. Expose this callback as a virtual method
in Stats::Group to make this pattern more convenient.
Change-Id: I5aa475b7d04c288e45f5f413ab7a1907b971dae5
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/21139
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
New-world stats didn't implement the old gem5 quirk where the index of
an element in a vector of size 1 is ignored. This affects the object
path when printed in a stat file. Implement the same quirk as the
original code to make sure names of old- and new-world stats are
compatible.
Change-Id: I12696d40baaed2d0bb1ef061e87c3b1d0f4bb4b1
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/21163
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
This changeset add support for stat dumps in the HDF5 file
format. HDF5 is a binary data format that represents data in a
file-system-like balanced tree. It has native support for
N-dimensional arrays and binary data (e.g., frame buffers).
It has the following benefits over traditional text stat files:
* Efficient storage of time series (multiple stat dumps)
* Fast lookup of stats
* Plenty of existing tooling (e.g., Python libraries and graphical
viewers)
* File format can be used to store frame buffers together with
normal stats.
Drawbacks:
* Large startup cost (single stat dump larger than text equivalent)
* Stat dumps are slower than text
Known limitations:
* Distributions and histograms aren't supported.
HDF5 stat output can be enabled using the 'h5' URL scheme when
overriding the stat file name on gem5's command line. The following
parameters are supported:
* chunking (unsigned): Number of time steps to pre-allocate
(default: 10)
* desc (bool): Output stat descriptions (default: True)
* formulas (bool): Output derived stats (default: True)
Example gem5 command line:
./build/ARM/gem5.opt \
--stats-file="h5://stats.h5?desc=False;formulas=False" \
configs/example/fs.py
Example Python stat consumer that computes IPC:
import h5py
f = h5py.File('stats.h5', 'r')
group = f['/system/cpu']
for i, c in zip(group['committedInsts'], group['numCycles']):
print i, c, i / c
Change-Id: I351c6cbff2fb7bef9012f47876ba227ed288975b
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/8121
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Ciro Santilli <ciro.santilli@arm.com>
Add support for partial stat dumps by passing an optional 'root'
keyword argument to m5.stats.dump(). Specifying root slightly changes
the semantics of the dump command. For legacy reasons, gem5 only
allows one stat dump per tick. This is likely a limitation introduced
as a hack to prevent automatic dumping at the end of simulation from
interfering with explicit dumping from a simulation script. This
restriction does not apply when specifying a root. However, these stat
dumps will still prevent an additional stat dump in the same tick with
an unspecified root.
N.B.: This new API /only/ works for new-style stats that have an
explicit hierarchy. Legacy stats will not be dumped if a root is
specified.
Change-Id: Idc8ff448b9f70a796427b4a5231e7371485130b4
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19369
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This change makes the stat system aware of the hierarchical nature of
stats. The aim is to achieve the following goals:
* Make the SimObject hierarchy explicit in the stat system (i.e.,
get rid of name() + ".foo"). This makes stat naming less fragile
and makes it possible to implement hierarchical formats like
XML/HDF5/JSON in a clean way.
* Make it more convenient to split stats into a separate
struct/class that can be bound to a SimObject. This makes the
namespace cleaner and makes stat accesses a bit more obvious.
* Make it possible to build groups of stats in C++ that can be used
in subcomponents in a SimObject (similar to what we do for
checkpoint sections). This makes it easier to structure large
components.
* Enable partial stat dumps. Some of our internal users have been
asking for this since a full stat dump can be large.
* Enable better stat access from Python.
This changeset implements solves the first three points by introducing
a class (Stats::Group) that owns statistics belonging to the same
object. SimObjects inherit from Stats::Group since they typically have
statistics.
New-style statistics need to be associated with a parent group at
instantiation time. Instantiation typically sets the name and the
description, other parameters need to be set by overriding
Group::regStats() just like with legacy stats. Simple objects with
scalar stats can typically avoid implementing regStats() altogether
since the stat name and description are both specified in the
constructor.
For convenience reasons, statistics groups can be merged into other
groups. This means that a SimObject can create a stat struct that
inherits from Stats::Group and merge it into the parent group
(SimObject). This can make the code cleaner since statistics tracking
gets grouped into a single object.
Stat visitors have a new API to expose the group structure. The
Output::beginGroup(name) method is called at the beginning of a group
and the Output::endGroup() method is called when all stats, and
sub-groups, have been visited. Flat formats (e.g., the text format)
typically need to maintain a stack to track the full path to a stat.
Legacy, flat, statistics are still supported after applying this
change. These stats don't belong to any group and stat visitors will
not see a Output::beginGroup(name) call before their corresponding
Output::visit() methods are called.
Change-Id: I9025d61dfadeabcc8ecf30813ab2060def455648
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/19368
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Python 3 has restructured some packages. Specifically, __builtin__ has
been renamed to builtins and urlparse has been included in urllib.
Change-Id: I81f8f3942471db1043006a36abbad6e5a49e0a43
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15994
Reviewed-by: Juha Jäykkä <juha.jaykka@arm.com>
Add a mechanism to configure the stat output format using a URL-like
syntax. This makes it possible to specify both an output format
(currently, only text is supported) and override default
parameters.
On the Python-side, this is implemented using a helper function
(m5.stats.addStatVisitor) that adds a visitor to the list of active
stat visitors. The helper function parses a URL-like stat
specification to determine the stat output type. Optional parameters
can be specified to change how stat visitors behave.
For example, to output stats in text format without stat descriptions:
m5.stats.addStatVisitor("text://stats.txt?desc=False")
From the command line:
gem5.opt --stats-file="text://stats.txt?desc=False"
Internally, the stat framework uses the _url_factory decorator
to wrap a Python function with the fn(path, **kwargs) signature in a
function that takes a parsed URL as its only argument. The path and
keyword arguments are automatically derived from the URL in the
wrapper function.
New output formats can be registered in the m5.stats.factories
dictionary. This dictionary contains a mapping between format names
(URL schemes) and factory methods.
To retain backwards compatibility, the code automatically assumes that
the user wants text output if no format has been specified (i.e., when
specifying a plain path).
Change-Id: Ic4dce93ab4ead07ffdf71e55a22ba0ae5a143061
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Ilias Vougioukas <ilias.vougioukas@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>
Swig wrappers for native objects currently share the _m5.internal name
space with Python code. This is undesirable if we ever want to switch
from Swig to some other framework for native binding (e.g., PyBind11
or Boost::Python). This changeset moves all of such wrappers to the
_m5 namespace, which is now reserved for native code.
Change-Id: I2d2bc12dbc05b57b7c5a75f072e08124413d77f3
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Call the stat visitor from the stat itself rather than casting stats
in Python. This reduces the number of ways visitors are called.
Change-Id: Ic4d0b7b32e3ab9897b9a34cd22d353f4da62d738
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Sascha Bischoff <sascha.bischoff@arm.com>
Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Joe Gross <joseph.gross@amd.com>
Some configuration scripts need periodic stat dumps. One of the ways
this can be achieved is by using the pariodicStatDump helper
function. This function was previously only exported in the internal
name space. Export it as a normal function in m5.stat instead.
Change-Id: Ic88bf1fd33042a62ab436d5944d8ed778264ac98
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Sascha Bischoff <sascha.bischoff@arm.com>
Add the ability to build libgem5 without embedded Python or the
ability to configure with Python.
This is a prelude to a patch to allow config.ini files to be loaded
into libgem5 using only C++ which would make embedding gem5 within
other simulation systems easier.
This adds a few registration interfaces to things which cross
between Python and C++. Namely: stats dumping and SimObject resolving
Track the point in the initialization where statistics have been registered.
After this point registering new masterIds can no longer work as some
SimObjects may have sized stats vectors based on the previous value. If someone
tries to register a masterId after this point the simulator executes fatal().
