This method depended on all of the EmbeddedPyBind objects having all
been constructed already so that it would have a complete list. This
would only be true if it was called after static intialization was
complete, which is not true if python is ready to go as soon as gem5
(in library form) is loaded.
This change makes EmbeddedPyBind able to defer initialization of a
module more generically than before, so that they can wait for either
another module to be initiailized, or the _m5 package itself.
Change-Id: I6b636524f969bd9882d8c1a7594dc36eb4e78037
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/54005
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
By separating out this utility class, we make it possible to build
embedded python modules into other binarys without dragging along lots
of other, unrelated gem5 dependencies.
Also, move the class from sim/init.hh (which is a largely unrelated
name) to python/embedded.hh which much more directly describes what that
file contains.
Change-Id: Ia83439144893ad8401a5d51003e2686d9c9b2d7b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/49418
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Put that code into a singleton class in src/proto, so that it gets
called during initialization and teardown of gem5 without cluttering up
gem5Main. This also removes the need to use #ifdefs to guard for
actualling having protobuf support.
Change-Id: I93b5d994eee478a9c159a3f3d02b3e996af02a3e
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/49416
Maintainer: Gabe Black <gabe.black@gmail.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Use pybind11 to avoid having to use the python C API directly, which is
simpler, easier to read, and less error prone. Also, use its
PYBIND11_EMBEDDED_MODULE macro to set up the _m5 module instead of a
callback which has to be proactively called from main().
Change-Id: I9c8bcebea934844d16a1fdd88f66a5e66ef0486f
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/49413
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
m5MainCommands had been a way to override the python code which would
get the python side of gem5 started, used by some "unit" tests which
were really tests of all of gem5 but focus on a particular area. Those
tests have either been converted into real unit tests or eliminated,
and so that mechanism is no longer needed.
This change eliminates that mechanism, and also uses pybind11 to
significantly simplify the code that calls m5.main().
Change-Id: I553ae17074cd5389708f1b7313630a13a6946d76
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/49412
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Previously, the importer module was built into gem5 as a compressed
bytecode blob like all the other code, and it had to be singled out and
installed manually so that it could help bring in all the other modules.
That adds some amount of complexity since it has to be identified and
treated as a special case.
Instead, this change builds it into gem5 using pybind11's
PYBIND11_EMBEDDED_MODULE macro, and a string that gets evaluated into
the new module's __dict__. This means the importer module is
automatically available just by building in that .cc, and it can just be
imported to start using it.
Theoretically all the embedded python could be handled this way, but
that would mean building it into gem5 as raw strings which wouldn't even
be compiled into byte code until run time. That would take more space in
the binary, and also delay catching simple errors.
Change-Id: Ic600bf6bce41a53289a2833484a655dd5a226e03
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/49410
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Apply the gem5 namespace to the codebase.
Some anonymous namespaces could theoretically be removed,
but since this change's main goal was to keep conflicts
at a minimum, it was decided not to modify much the
general shape of the files.
A few missing comments of the form "// namespace X" that
occurred before the newly added "} // namespace gem5"
have been added for consistency.
std out should not be included in the gem5 namespace, so
they weren't.
ProtoMessage has not been included in the gem5 namespace,
since I'm not familiar with how proto works.
Regarding the SystemC files, although they belong to gem5,
they actually perform integration between gem5 and SystemC;
therefore, it deserved its own separate namespace.
Files that are automatically generated have been included
in the gem5 namespace.
The .isa files currently are limited to a single namespace.
This limitation should be later removed to make it easier
to accomodate a better API.
Regarding the files in util, gem5:: was prepended where
suitable. Notice that this patch was tested as much as
possible given that most of these were already not
previously compiling.
Change-Id: Ia53d404ec79c46edaa98f654e23bc3b0e179fe2d
Signed-off-by: Daniel R. Carvalho <odanrc@yahoo.com.br>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/46323
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Reviewed-by: Matthew Poremba <matthew.poremba@amd.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This should help reduce warning spew when building with newer compilers.
The pybind11::module type has been renamed pybind11::module_ to avoid
conflicts with c++20 modules, according to the pybind11 changelog, so
this CL also updates gem5 source to use the new type. There is
supposedly an alias pybind11::module which is for compatibility, but we
still get linker errors without changing to pybind11::module_.
Change-Id: I0acb36215b33e3a713866baec43f5af630c356ee
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40255
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
The approach we currently use to register our native modules doesn't
work on Python 3. Convert the code to use the Python inittab instead
of the old ad-hoc method.
Change-Id: I961f8a33993c621473732faeaab955a882769a4b
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15979
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
These files aren't a collection of miscellaneous stuff, they're the
definition of the Logger interface, and a few utility macros for
calling into that interface (panic, warn, etc.).
Change-Id: I84267ac3f45896a83c0ef027f8f19c5e9a5667d1
Reviewed-on: https://gem5-review.googlesource.com/6226
Reviewed-by: Brandon Potter <Brandon.Potter@amd.com>
Maintainer: Gabe Black <gabeblack@google.com>
Use the PyBind11 wrapping infrastructure instead of SWIG to generate
wrappers for functionality that needs to be exported to Python. This
has several benefits:
* PyBind11 can be redistributed with gem5, which means that we have
full control of the version used. This avoid a large number of
hard-to-debug SWIG issues we have seen in the past.
* PyBind11 doesn't rely on a custom C++ parser, instead it relies on
wrappers being explicitly declared in C++. The leads to slightly
more boiler-plate code in manually created wrappers, but doesn't
doesn't increase the overall code size. A big benefit is that this
avoids strange compilation errors when SWIG doesn't understand
modern language features.
* Unlike SWIG, there is no risk that the wrapper code incorporates
incorrect type casts (this has happened on numerous occasions in
the past) since these will result in compile-time errors.
As a part of this change, the mechanism to define exported methods has
been redesigned slightly. New methods can be exported either by
declaring them in the SimObject declaration and decorating them with
the cxxMethod decorator or by adding an instance of
PyBindMethod/PyBindProperty to the cxx_exports class variable. The
decorator has the added benefit of making it possible to add a
docstring and naming the method's parameters.
The new wrappers have the following known issues:
* Global events can't be memory managed correctly. This was the
case in SWIG as well.
Change-Id: I88c5a95b6cf6c32fa9e1ad31dfc08b2e8199a763
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Andreas Hansson <andreas.hansson@arm.com>
Reviewed-by: Andrew Bardsley <andrew.bardsley@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/2231
Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>
Reviewed-by: Pierre-Yves Péneau <pierre-yves.peneau@lirmm.fr>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Python's header files set various compiler macros (e.g.,
_XOPEN_SOURCE) unconditionally. This triggers preprocessor warnings
that end up being treated as errors. The Python integration manual [1]
strongly recommends that Python.h is included before any system
header. The style guide used to mandate that Python.h is included
first in any file that needs it. This requirement was changed to
always include a source file's main header first, which ended up
triggering these errors.
This change updates the style checker to always include Python.h
before the main header file.
[1] https://docs.python.org/2/extending/extending.html
Change-Id: Id6a4f7fc64a336a8fd26691a0ca682abeb1d1579
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Pierre-Yves Péneau <pierre-yves.peneau@lirmm.fr>
This patch changes how the embedded swig code is loaded to ensure that
gem5 works with swig 3.0.9. For Python 2.7 and above, swig 3.0.9 now
relies on importlib, and actually looks in the appropriate packages,
even for the wrapped C code. However, the swig wrapper does not
explicitly place the module in the right package (it just calls
Py_InitModule), and we have to take explicit action to ensure that the
swig code can be loaded. This patch adds the information to the
generated wrappers and the appropriate calls to set the context as
part of the swig initialisation.
Previous versions of swig used to fall back on looking in the global
namespace for the wrappers (and still do for Python 2.6), but
technically things should not work without the functionality in this
patch.
Add the ability to build libgem5 without embedded Python or the
ability to configure with Python.
This is a prelude to a patch to allow config.ini files to be loaded
into libgem5 using only C++ which would make embedding gem5 within
other simulation systems easier.
This adds a few registration interfaces to things which cross
between Python and C++. Namely: stats dumping and SimObject resolving
There is a race between enabling asynchronous IO for a file descriptor
and IO events happening on that descriptor. A SIGIO won't normally be
delivered if an event is pending when asynchronous IO is
enabled. Instead, the signal will be raised the next time there is an
event on the FD. This changeset simulates a SIGIO by setting the
async_io flag when setting up asynchronous IO for an FD. This causes
the main event loop to poll all file descriptors to check for pending
IO. As a consequence of this, the old SIGALRM hack should no longer be
needed and is therefore removed.
The PollEvent class dynamically installs a SIGIO and SIGALRM handler
when a file handler is registered. Most signal handlers currently get
registered in the initSignals() function. This changeset moves the
SIGIO/SIGALRM handlers to initSignals() to live with the other signal
handlers. The original code installs SIGIO and SIGALRM with the
SA_RESTART option to prevent syscalls from returning EINTR. This
changeset consistently uses this flag for all signal handlers to
ensure that other signals that trigger asynchronous behavior (e.g.,
statistics dumping) do not cause undesirable EINTR returns.
This patch adds support for outputting protobuf messages through a
ProtoOutputStream which hides the internal streams used by the
library. The stream is created based on the name of an output file and
optionally includes compression using gzip.
The output stream will start by putting a magic number in the file,
and then for every message that is serialized prepend the size such
that the stream can be written and read incrementally. At this point
this merely serves as a proof of concept.
This patch addresses a number of minor issues that cause problems when
compiling with clang >= 3.0 and gcc >= 4.6. Most importantly, it
avoids using the deprecated ext/hash_map and instead uses
unordered_map (and similarly so for the hash_set). To make use of the
new STL containers, g++ and clang has to be invoked with "-std=c++0x",
and this is now added for all gcc versions >= 4.6, and for clang >=
3.0. For gcc >= 4.3 and <= 4.5 and clang <= 3.0 we use the tr1
unordered_map to avoid the deprecation warning.
The addition of c++0x in turn causes a few problems, as the
compiler is more stringent and adds a number of new warnings. Below,
the most important issues are enumerated:
1) the use of namespaces is more strict, e.g. for isnan, and all
headers opening the entire namespace std are now fixed.
2) another other issue caused by the more stringent compiler is the
narrowing of the embedded python, which used to be a char array,
and is now unsigned char since there were values larger than 128.
3) a particularly odd issue that arose with the new c++0x behaviour is
found in range.hh, where the operator< causes gcc to complain about
the template type parsing (the "<" is interpreted as the beginning
of a template argument), and the problem seems to be related to the
begin/end members introduced for the range-type iteration, which is
a new feature in c++11.
As a minor update, this patch also fixes the build flags for the clang
debug target that used to be shared with gcc and incorrectly use
"-ggdb".
Instead of putting all object files into m5/object/__init__.py, interrogate
the importer to find out what should be imported.
Instead of creating a single file that lists all of the embedded python
modules, use static object construction to put those objects onto a list.
Do something similar for embedded swig (C++) code.
If the user sets the environment variable M5_OVERRIDE_PY_SOURCE to
True, then imports that would normally find python code compiled into
the executable will instead first check in the absolute location where
the code was found during the build of the executable. This only
works for files in the src (or extras) directories, not automatically
generated files.
This is a developer feature!
Targets look like libm5_debug.so. This target can be dynamically
linked into another C++ program and provide just about all of the M5
features. Additionally, this library is a standalone module that can
be imported into python with an "import libm5_debug" type command
line.
This should allow m5 to be more easily embedded into other simulators.
The m5 binary adds a simple main function which then calls into the m5
libarary to start the simulation. In order to make this work
correctly, it was necessary embed python code directly into the
library instead of the zipfile hack. This is because you can't just
append the zipfile to the end of a library the way you can a binary.
As a result, Python files that are part of the m5 simulator are now
compile, marshalled, compressed, and then inserted into the library's
data section with a certain symbol name. Additionally, a new Importer
was needed to allow python to get at the embedded python code.
Small additional changes include:
- Get rid of the PYTHONHOME stuff since I don't think anyone ever used
it, and it just confuses things. Easy enough to add back if I'm wrong.
- Create a few new functions that are key to initializing and running
the simulator: initSignals, initM5Python, m5Main.
The original code for creating libm5 was inspired by a patch Michael
Adler, though the code here was done by me.
