The DictImporter's __del__ method calls unload, and that imports
sys.modules so that it can remove the modules that the DictImporter had
set up as the importer goes away.
Unfortunately, the importer only goes away when python is shutting down,
and at that time some aspects of the system, namely sys.meta_path, have
been cleaned up. When unload tries to import sys, that causes an
exception which scons/python reports but which doesn't do anything bad
otherwise.
In all of the examples of this older style of import object online, none
had a __del__ method, and none worried about cleaning up sys.modules
when they went away. In light of that, I've removed the __del__ method
entirely.
Another reason I think it's safe to remove __del__ is that the importer
was not actually being deleted even when it was removed from
sys.meta_path, and all the modules it had loaded where removed from
sys.modules. I think that was because the SimObject classes that it had
set up still had references (they are used later in the SConscript), and
those would, either directly or indirectly, refer back to the modules
and the importer. Those remaining references kept the importer alive,
preventing __del__ from being called before all those other objects were
cleaned up.
I think in python 2, the order things were cleaned up just so happened
to avoid trying to import sys when it was no longer possible, but in
python 3 that changed and resulted in this exception being thrown.
I've tried building gem5 with scons running under python 2 and python 3,
and with this change there is no error at shutdown. Both also produce a
gem5 binary which can run hello world without problems.
Change-Id: Ib1f5c7403df57fc420cec7ec0fef20a164a06991
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27247
Tested-by: Gem5 Cloud Project GCB service account <345032938727@cloudbuild.gserviceaccount.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
scons maintains an environment (in the shell sense) in the ENV
construction variable for use when running external programs. When we
run the "marshal" program which gathers up python objects to embed in
the gem5 binary, it's run by subprocess instead of through scons, and it
uses its own environment inherited from the host system.
Instead, this change makes the subprocess function use the scons
environment when calling "marshal". This ensures the environment is
consistent between this command and other commands scons runs.
This is usually not very important, but some tools like asan take
options set through the environment, and they may need to be adjusted
sometimes.
Change-Id: I671b447657ed8fad45fac7393cc1c09073bf3d3a
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27123
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Python 3 uses the concepts of two different types:
text and binary strings.
Those cannot be implicilty combined (as it was happening in python2) and
in order to be used together one of them must be converted to the other
type:
* Text can be encoded into Bytes via the encode() method
* Bytes can be decoded to Text using the decode() method
By default encode/decode will assume UTF-8 format
JIRA: https://gem5.atlassian.net/browse/GEM5-345
Change-Id: I1bdf7db17b49cc109239fd5f44791769370853f8
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26250
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
* dict methods dict.keys(), dict.items() and dict.values()
return "views" instead of lists
* The dict.iterkeys(), dict.iteritems() and dict.itervalues()
methods are no longer supported.
* map() and filter() return iterators.
* range() now behaves like xrange() used to behave, except it works with
values of arbitrary size. The latter no longer exists.
* zip() now returns an iterator.
Change-Id: Id480018239db88d7f5d60588c93719056de4a0c0
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26248
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Partial linking heuristically links together files in the same
directory by setting a special automatic tag. That tag needs to also
be maintained when scanning EXTRAS dirs so that they don't all get
lumped in with the last normal directory that was processed.
Change-Id: I2408ea0a1eeffcf6d9994c36415a35760b225b17
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/23300
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The current marshal binary is in $BUILD/python/ directory which makes
the dynamic linker fail to find shared libraries with rpath information
when the marshal is executing. Moving $BUILD/python/marshal to
$BUILD/marshal solves this problem.
This is because the marshal binary is built with the same flags as gem5
binary. And if one adds rpath information with add_local_rpath() in some
SConscripts, the marshal binary will also link to those shared libraries
and have the same rpath information. So it must be in the same
directory as gem5 binary such that the dynamic linker could find those
shared libraries with the rpath information.
Signed-off-by: Chun-Chen TK Hsu
Change-Id: I98d9c3e0180abefe6b2b6a78ee5a96324038eb53
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/18888
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Partial linking should be disabled on darwin; however, the script
fails to do so when force_lto is set, which results in gem5 building
with fast option fails on macOS. This fix changes disable_partial
logic, which should be True once it's True.
Signed-off-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Change-Id: I77d2a4cc4a9bf5c92c800c004eb744bb7081c42e
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/16888
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
We currently use the Python version used by scons to marshal Python
code. This doesn't work when building gem5 with Python 3 support since
scons typically runs in Python 2.7. Add a custom marshal helper that
links with the same library as gem5 to generate byte code that is
guaranteed to work in gem5's Python interpreter.
Change-Id: I665b0f2078726d4c055d74a3e668a580fc613b59
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/16422
Reviewed-by: Gabe Black <gabeblack@google.com>
Many functions that used to return lists (e.g., dict.items()) now
return iterators and their iterator counterparts (e.g.,
dict.iteritems()) have been removed. Switch calls to the Python 2.7
iterator methods to use the Python 3 equivalent and add explicit list
conversions where necessary.
Change-Id: I0c18114955af8f4932d81fb689a0adb939dafaba
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15992
Reviewed-by: Juha Jäykkä <juha.jaykka@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
When linking in a dynamic library which is in the gem5 build directory,
it's useful to set RPATH so that you don't have to set LD_LIBRARY_PATH
when you run gem5 so that the dynamic linker can find it.
Since it's tricky and not entirely obvious how to set up those paths
correctly, this change adds a small convenience function which does
that for you. It also handles situations where the same dynamic
library may be linked into different binaries in different directories
which each need a different relative RPATH. It does that by letting the
environment for each binary set a construction variable which says
how to get from that particular binary back to the build directory.
This helper method then sets RPATH to start at $ORIGIN (the binary),
to follow that relative path to the variant build directory, and then
the per-library but not per-binary path to the library's directory.
This change also adds the -z origin linker flag which makes the linker
handle $ORIGIN properly.
Change-Id: I45f4d72cd14396a73e0b963cea6a39d9bfb7f984
Reviewed-on: https://gem5-review.googlesource.com/c/16566
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
The loop accidentally used a = when it should have used a +=, meaning
only the sources from the final filter would be used.
Change-Id: Ie066a5f85696f05d9ad3cf61f928b12deb39475b
Reviewed-on: https://gem5-review.googlesource.com/c/16285
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>
Since f2bda876f7, the build system started
adding a length for generated blobs as in:
const std::size_t variable_len = 123;
There were two types of blob files, ones with a header and the ones
without.
The ones with the header, also include the header in the .cc of the blob,
which contains a declaration:
extern const std::size_t variable_len;
Therefore, the ones without header, don't have that extern declaration,
which makes them static according to the C++ standard.
clang then correctly interprets that as problematic due to
-Wunused-const-variable, while GCC does not notice this.
This patch removes the length declaration from the blob files that don't
have the header. Those files currently don't use the length.
Change-Id: I3fc61b28f887fc1015288857328ead2f3b34c6e6
Reviewed-on: https://gem5-review.googlesource.com/c/15955
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Up until Apple LLVM version 10.0.0 (clang-1000.11.45.5), partial
linked objects do not expose symbols that are marked with the hidden
visibility and consequently building gem5 on Mac OS fails. As a
workaround, we disable partial linking, however, we may want to
revisit in the future.
Change-Id: I0a26dae082bf723c2bd49d90e4497e44ecab9c41
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15235
Reviewed-by: Andrea Mondelli <andrea.mondelli@ucf.edu>
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Building gem5 binaries or regression test binaries needs to be done
from within the make_env function which builds an environment for each
flavor of build (opt, fast, debug, etc.). That makes it impossible to
add new types of binaries without modifying the central SConscript.
This change refactors how binaries are set up so that the class that
represents them handles the details of how the binary should be built.
Also, a metaclass and some lists track types of binaries and individual
instances of binaries so that they can be iterated over automatically
in make_env.
Each new executable class can define a declare_all class function which
calls declare() on individual instances. declare_all is a place to do
any processing that only has to happen once (for instance specializing
the environment) for a particular family of executables.
Change-Id: I8a6ee9438280cd67e6c0b92ca28738a53cb16950
Reviewed-on: https://gem5-review.googlesource.com/10915
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
The original implementation of UnitTest forced all the output binaries
to live in the unittest directory, effectively forcing a flat
namespace, and seperating the tests from the things they were supposed
to be testing.
This changes makes them work more like the newer GTest tests in that
they can be based out of whatever directory makes sense, although
they're currently all still in unittest for the time being.
This change also gets rid of automatically tagging the sources
associated with a test with the tests name. The first reason for that
was that this also forced a flat namespace, since the tests names
didn't have any reference to the test's path. Second, this way of
pulling in additional files wasn't necessary any more, now that the
UnitTest sources could be source filters like they can be for GTests.
Change-Id: I3d96ed766ac5170842dbd6daee39f2873bcd6c75
Reviewed-on: https://gem5-review.googlesource.com/10701
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
Starting with version 3, scons imposes using the print function instead
of the print statement in code it processes. To get things building
again, this change moves all python code within gem5 to use the
function version. Another change by another author separately made this
same change to the site_tools and site_init.py files.
Change-Id: I2de7dc3b1be756baad6f60574c47c8b7e80ea3b0
Reviewed-on: https://gem5-review.googlesource.com/8761
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>
There are some cases where scons incorrectly adds the same object
multiple times to the linker command line. This seems to be caused by
the test's source list being updated in place when determining test
framework dependencies. Fix this by explicitly copying the source list
and manipulate the copy.
Without this change, the following command fails:
scons ./build/ARM/unittests.opt/base/pixeltest.xml
Whereas this command succeeds:
scons ./build/ARM/base/pixeltest.opt
Change-Id: I642efdf9d62a5478e49ba51efe1a3a5ba453e21f
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jack Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/6641
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
The GTest framework supports result generation in XML (JUnit). Enable
this by creating unit test specific targets in the
build/${BUILD_OPTS}/unittests.${VARIANT} directory. Targets in the
directory use the following naming convention:
${SRC_PATH}/${TEST_NAME}.xml
For example, the opt version of the bitunion test built for ARM would
have this path:
build/ARM/unittests.opt/base/bituniontest.xml
Change-Id: I174dff16817734db05b08ce1d5bcf52e8697bbac
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jack Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/6624
Reviewed-by: Gabe Black <gabeblack@google.com>
SCons currently scrubs the environment variables used by GTests too
aggressively. This breaks systems where libraries are installed in
non-standard locations that need to be specified in
LD_LIBRARY_PATH. Run said tests in the gtest_env SCons environment
which white-lists the important environment variables.
Change-Id: I5fc8fb5e51f09644dc976ee97b21c78ab349bf7d
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jack Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/6623
Reviewed-by: Gabe Black <gabeblack@google.com>
This change introduces the idea of a SourceFilter which is an object
that can filter a SourceList and which can be composed with other
SourceFilters using | and & operators. This means a filter can be
constructed ahead of time, possibly before all sources have been
discovered, and then later applied to any SourceList necessary.
This change also modifies GTest so that it accepts SourceFilters in
addition to normal source files. These filters will be applied to the
final list of all sources, and the result included in the build for
that test.
By default, gtests will build in all sources tagged with 'gtest lib'.
This change also introduces the keyword argument "skip_lib" which will
exclude those files. They can then be left out entirely, or they can be
re-included as part of a more elaborate filter. That would be useful if
someone wanted to write a unit test for, for instance, the warn, etc.
macros which rely on the gtest logging support. Those classes could
be replaced by something under the control of the unit test, while
still including the rest of the gtest library.
Change-Id: I13a846dc884b86b9fdcaf809edefd57bb4168b8e
Reviewed-on: https://gem5-review.googlesource.com/6262
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Telling scons to build build/${BUILD_OPTS}/unittests.${VARIANT} will
get it to build and run all googletest based unit tests under the
${BUILD_OPTS} build options (ARM, ALPHA, X86, etc.), and compiled with
the flags, etc., for the ${VARIANT} variant (ie. opt, debug, etc.).
This will make it easy to run the unit tests without having to actually
know where they are, what tests are available, etc.
This target is called unittests* and not something based on gtest or
googletest since it's my intention for all unit tests to be based on
googletest, making the distinction unnecessary. Since the target is
essentially part of the external interface for scons, I wanted to name
it something general so it'll be less likely that we have to change it.
Change-Id: I8fdec768d821974309c92a2ce4c96dce7df24fa5
Reviewed-on: https://gem5-review.googlesource.com/6282
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Brandon Potter <Brandon.Potter@amd.com>
Maintainer: Gabe Black <gabeblack@google.com>
There were a few places where tags weren't being converted to sets
correctly which unfortunately only manifested when called in certain
ways. This would be a pretty reasonable place to add some python unit
tests...
Change-Id: I87509369b4ec6f702b7521e52bf63701a87ec436
Reviewed-on: https://gem5-review.googlesource.com/6261
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
scons gets upset if two different environments are used to set up a
particular object file. This change adds two dicts to the SourceFile
class, one for static and one for shared object files, which are keyed
off of the appropriate suffix. If a suffix hasn't been set up yet,
a new node of the appropriate type is set up and stored in the cache,
and then whatever is in the cache (new or old) is returned.
Change-Id: Ice4b4fc728b438a4d3316c3ff6667c0480d2a6d7
Reviewed-on: https://gem5-review.googlesource.com/6224
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
This type of unit test is almost the same as UnitTest, except that
it builds in the googletest library with the appropriate flags, and
it also omits all the gem5 object files by default.
Change-Id: I230db7ede563a73a9da0ebeed44aca389f724465
Reviewed-on: https://gem5-review.googlesource.com/6084
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Git rid of the unused partial_objs variable, and change append to
extend since the result of builders is a list of nodes and not
individual nodes, even if there's only one target.
Change-Id: Ie51225d3ea699eb7da18199ac3bf3e868e3994a6
Reviewed-on: https://gem5-review.googlesource.com/5985
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
The make_obj function took a boolean value which just selected which
of the two lines it in would actually do something. This change breaks
it into two lambdas, make_static and make_shared, which just do
whichever line would have been requested, making the funciton name more
self descriptive and getting rid of the generally unnamed and opaque
boolean argument.
Change-Id: I457e40034b7e7f5a3e7294a8e1f15bbd42e0720e
Reviewed-on: https://gem5-review.googlesource.com/5984
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tags are just arbitrary strings which are attached to source files
which mark them as having some property. By default, all source files
have the "gem5 lib" tag added to them which marks them as part of the
gem5 library, the primary component of the gem5 binary but also a
seperable component for use in, for example, system C.
The tags can be completely overridden by setting the "tags" parameter
on Source, etc., functions, and can be augmented by setting "add_tags"
which are tags that will be added, or alternatively additional tags.
It's possible to specify both, in which case the tags will be set to
the union of tags and add_tags. add_tags is supposed to be a way to
add extra tags to the default without actually overriding the default.
Both tags and add_tags can be a list/tuple/etc of tags, or a single
string which will be converted into a set internally.
Other existing tags include:
1. "python" for files that need or are used with python and are
excluded when the --without-python option is set
2. "main" for the file(s) which implement the gem5 binary's main
function.
3. The name of a unit test to group its files together.
4. Tags which group source files for partial linking.
By grouping the "tags" into a single parameter instead of taking all
extra parameters as tags, the extra parameters can, in the future, be
passed to the underlying scons environment. Also, the tags are either
present or not. With guards, they could be present and True, present
and False, or not present at all.
Change-Id: I6d0404211a393968df66f7eddfe019897b6573a2
Reviewed-on: https://gem5-review.googlesource.com/5822
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
A recent scons cleanup introduced a small bug when setting the variant path of
the regression test SConscript. When turned into a relative path, the
directory can be interpreted differently depending on what it's considered
relative to. When left as a scons node, there is no ambiguity.
Change-Id: I162afdc4fcef81db096ec8304f81bc98fa6bf899
Reviewed-on: https://gem5-review.googlesource.com/5221
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>
