This reimplements the previously reverted change: Always generate default
create() methods.
The pybind code should only be generated when python is enabled. This
change passes whether python is enabled into the SimObject code creation
method. Then, the params code is optionally included.
Note: Due to some problems in GCC's linker (or something else...) we
need to have a single file with all of the generated code for the
SimObject.
Change-Id: I0f93b3d787d47f26db2de6c4447730f7df87a0dc
Issue-on: https://gem5.atlassian.net/browse/GEM5-1003
Signed-off-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/46820
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Debug flags were directly declared as global objects; however, the
destruction order of global objects in different translation units are
not defined in c++, so the destructor of other objects cannot safely
utilize debug flags to output extra information.
We now define those flags as references to objects allocated on the heap
so our flags will never get destructed. Note that this won't really
introduce any memory leak, as the program is getting terminated anyway.
Change-Id: I21f84ae17ac20653636ca67d1111c0c7855c0149
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/45582
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Gabe Black <gabe.black@gmail.com>
It was quite incovenient to declare tags. tags either
had to be added to the Source declaration of included
files, or using (with_tag()) would need to be added
with all the indirect sub-tags. For example,
Assume:
- B.cc refers to symbols in A.cc
- C.cc refers to symbols in B.cc (i.e., it needs A transitively)
- D.cc refers to symbols in A.cc and C.cc (i.e., it needs B trans.)
So either their SConscript would be:
Source('A.cc', add_tags=['B', 'D'])
Source('B.cc', add_tags='B')
Source('C.cc', add_tags='C')
Source('D.cc', add_tags='D')
GTest('A.test', 'A.test.cc', 'a.cc')
GTest('B.test', 'B.test.cc', with_tag('B'))
GTest('C.test', 'C.test.cc', with_any_tags('B', 'C'))
GTest('D.test', 'D.test.cc', with_any_tags('B', 'C', 'D'))
or:
Source('A.cc', add_tags=['B', 'C', 'D'])
Source('B.cc', add_tags=['B', 'C', 'D'])
Source('C.cc', add_tags=['C', 'D'])
Source('D.cc', add_tags='D')
GTest('A.test', 'A.test.cc', 'a.cc')
GTest('B.test', 'B.test.cc', with_tag('B'))
GTest('C.test', 'C.test.cc', with_tag('C'))
GTest('D.test', 'D.test.cc', with_tag('D'))
This change makes it simpler. The tag should be added
only to the files directly included by the functionality
being tagged. Using the same example:
Source('A.cc', add_tags=['B', 'D'])
Source('B.cc', add_tags='B')
Source('C.cc', add_tags='C')
Source('D.cc', add_tags='D')
env.TagImplies('B', 'A')
env.TagImplies('C', 'B')
env.TagImplies('D', ['A', 'C'])
GTest('A.test', 'A.test.cc', 'a.cc')
GTest('B.test', 'B.test.cc', with_tag('B'))
GTest('C.test', 'C.test.cc', with_tag('C'))
GTest('D.test', 'D.test.cc', with_tag('D'))
This also means that when a file no longer refers to
symbols from other file, or when it starts refering to
symbols from another file, one only needs to change
the dependencies of the tag directly being modified,
not the tags that rely on (imply) them. That is, on
the previous example, if C stops refering to symbols
from B, then tags that imply C do not need to be
modified:
env.TagImplies('B','A')
env.TagImplies('D', ['A', 'C'])
Change-Id: I5be07b01864f8d5df83f59002dfd2f01c73d5e09
Signed-off-by: Daniel R. Carvalho <odanrc@yahoo.com.br>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/43587
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
We were originally generating default create() methods along side the
pybind definitions, but unfortunately those are only included when
python support is included. Since the SimObject Param structs are
unconditionally provided even if the thing calling their create()
methods is not, we need to also unconditionally provide the default
create() definitions. We do that by putting them in their own new .cc
files.
Change-Id: I29d1573d578794b3fe7ec2bc16ef5c8c58e56d0e
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/42589
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Earl Ou <shunhsingou@google.com>
The name of the build is opt, so it should be fully optomized. Also, the
fast build, the only one with LTO historically, is dangerous to use
since it disables many error checks. I personally run gem5 many times
while developing, iterating and trying to fix bugs, and so want it to
run quickly then too, not just the final time when collecting results.
Also, since they mirror the opt build, the perf and prof builds also
have LTO options added.
This has the nice side effect of speeding up the build time of build/X86
significantly (6:20 -> 4:27) due to parallelization of the link, and
reduces the size of the build/X86 directory (with debug compression
enabled) from 3.4GB to 2.8GB.
The size of build/X86/python/_m5 is still 1.6GB, so still more than half
of the total size of build/X86.
Change-Id: I8feabf99454693fdd100d9e1a64fdeae53362f75
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40815
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Earl Ou <shunhsingou@google.com>
That tag was intended to mark an Executable subclass as abstract, aka
only suitable for using as bases for other Executable subclasses and not
for direct instantiation. The only place it was used was the base
Executable class however, and that class is actually directly useful
when setting up a generic executable from other gem5 sources.
Change-Id: I70204b63c03bb45bf21b8c312a7b8581be5e0cab
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40616
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Add an "All" compound debug flag, which encapsulates all
debug flags.
Since this is the broadest compound flag, allowing users
to include it would imply in extremely generic includes.
Moreover, it is highly unlikely that any correct C++ code
would ever use all debug flags. Therefore, a header file
for this flag is not generated to force users to directly
include only the debug flags they need.
Change-Id: If40f2f708be1495fa2b2380266164d5d44d7cffa
Signed-off-by: Daniel R. Carvalho <odanrc@yahoo.com.br>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/39077
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Nathanael Premillieu <nathanael.premillieu@huawei.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Debug flags are flags that aid with debugging by printing
relevant information when enabled. Debug-formatting flags
define how the debug flags will print the information.
Although a viability, this patch does not support declaring
compound format flags.
As a side effect, now debug flags and debug-formatting flags
are printed in different lists, when using --debug-help.
Change-Id: Ieae68745276218cf4e9c1d37d7bf3bd1f19709ae
Signed-off-by: Daniel R. Carvalho <odanrc@yahoo.com.br>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/39076
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The SmartDict, used by buildEnv, has been added long time ago for
the following reasons: (checking its documentation)
---
The SmartDict class fixes a couple of issues with using the content
of os.environ or similar dicts of strings as Python variables:
1) Undefined variables should return False rather than raising KeyError.
2) String values of 'False', '0', etc., should evaluate to False
(not just the empty string).
---
These are valid reasons, but I believe they should be addressed in
a more standardized way by using a common dictionary.
1) We should simply rely on dict.get
if buildEnv.get('KEY', False/None):
2) We should discourage the use of stringified False or 0.
If we are using a dictionary, can't we just pass those values as
booleans?
The SmartDict is basically converting every value into a
string ("Variable") at every access (__getitem__)
The Variable is a string + some "basic" conversion methods
What is the problem of passing every dict value as a string?
The problem is the ambiguity on the boolean conversion.
If a variable is modelling a boolean, we can return true if
the value is 'yes', 'true'... and false if the value is
'no', 'false' etc. We should raise an exception if it is
something different, like a typo (e.g.) 'Fasle'.
But if the variable is not modelling a boolean, we don't know
how to handle that. How should we convert 'mystring' ?
If we decide to treat 'mystring' as True (which is basically
what a str.__bool__ would return) we will break typoes detection,
as 'Fasle' will now be converted to True, rather than raising
an exception.
Change-Id: I960fbfb1ec0f703e1e372dd752ee75f00632acac
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/37775
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
These files are used to generate stubs for calling across GRPC
protocols, an RPC mechanism which is based around the protocol buffer
system.
The support for these files is heavily based on and calls into the
existing protobuf file support, but with the extra step which generates
the additional .grpc.pb.cc and .grpc.pb.h files.
Change-Id: I89022928c08aa9f7ed024b7380ddcc54ca75b55e
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/37277
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
There are several benefits to using a Builder. First, the action we're
executing is shared between all uses of the Builder. The number of
times this particular builder is called is small, but it should still
be a little more efficient.
Second, we can use SCons's emitter mechanism to generate the .pb.cc and
.pb.h target files in a little more general way.
Also, this change adds a Scanner for .proto files which will scan them
for imports and let SCons manage those implicit dependencies properly.
The scanner is a bit simplistic as described in a comment in the
source, but should work pretty well in practice with reasonably
formatted files, and in particular some files I'm working with that
include imports.
Change-Id: Iaf2498e61133d6f713d6ccaf199422b882c5894f
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/37276
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The ProtoBuf support in src/SConscript was split into two parts, one
where the ProtoBuf sources were declared, and the other where scons was
told how to buld the .cc and .hh files and the .cc was added to the
build.
As far as I can tell, there was no real reason to have things split up
like that, at least not currently. This change moves everything into
the ProtoBuf class definition, and this should behave the same as
before but be a little easier to understand and maintain.
Change-Id: I02320f50ece53d90c14b5062bd6b1167210f46c3
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/37275
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
There were two issues with how paths were handled for these files.
1. The code in the ProtoBuf class would drop the subdirectory part of
the path name when generating the name of the .cc and .h files the
protoc compiler would output. Since protoc wouldn't generate files
where scons expected, it would fail when it tried to build the .cc.
2. protoc will use the --proto_path and --cpp_out settings to figure
out what path to use for generated files. It will remove the
--proto_path prefix it found the .proto file with from the files path,
and then add the rest to the --cpp_out prefix.
The input files should come from the build directory using symlinks
set up by scons, and the output files should end up alongside them.
That means the --proto_path setting should be the build directory, and
so should --cpp_out. That's fortunately simpler than what was there
before, since it doesn't depend on what the source or targets are.
Change-Id: I69692d2fe3813011982f0c1c9824589a132f93ed
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/37218
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Compound flags are currently constructed using a constructor with a
finite set of arguments that default to nullptr that refer to child
flags. C++11 introduces two cleaner ways to achieve the same thing,
variadic templates and initializer_list. Use an initializer list to
pass dependent flags.
Change-Id: Iadcd04986ab20efccfae9b92b26c079b9612262e
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/34115
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
If the cxx_type value changes, then the way the parameter is declared
in the param struct will also change, and the header needs to be
updated. scons would miss this sort of change before because it was only
checking the module the SimObject's source came from. The python names
and types of the parameters could stay the same, but the C++
representation might have changed because of edits somewhere else.
This CL assumes that cxx_type is the only thing that will change and
transparently affect the params struct. I tried making scons sensitive
to the entire ptype which would capture other parameters, but that
didn't work for some reason. This should be pretty safe in general, but
not 100% safe
Issue-on: https://gem5.atlassian.net/browse/GEM5-753
Change-Id: I06774889e60b987f727799f55d7ea2a775b6a319
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/33695
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
We already provide to the user the CCFLAGS_EXTRA, LDFLAGS_EXTRA
variables to pass flags to scons when compiling/linking gem5.
Those variables are not passed to the marshal object.
We add an extra pair:
MARSHAL_CCFLAGS_EXTRA, MARSHAL_LDFLAGS_EXTRA
to add flag injection capabilities to the marshal object.
The patch is also renaming base_py_env to marshal_env.
This happens for 2 reasons:
1) At the moment the marshal compilation is the only task
making use of the base python environment.
2) Consistency with the EXTRA variable names added with this patch.
I could have named them as BASE_XXFLAGS_EXTRA, but it seems too much
generic and users might be confused by that, as they might think
the BASE_XXFLAGS_EXTRA is a subset of the XXFLAGS_EXTRA so that
setting it will affect gem5 compilation as well.
Change-Id: I3e420caa897059455ff8f35462db2b38da050e93
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/30016
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Setting disable_partial part way through the checks for various build
targets is incorrect and will affect targets based on the order they're
checked.
This change moves the check earlier, makes it consistent across all
builds whether fast is included or not, and stops passing it in as an
option to makeEnv since it now applies universally.
By disabling partial linking consistently, we avoid missing bugs where
only the "fast" version of gem5 doesn't build correctly because of the
multitude of g++ bugs having to do with combining LTO and partial
linking.
This also simplifies the logic in the SConscript by having fewer
independently moving parts.
Change-Id: Iff69f39868e948d3b9a5b11ea80bbfed19419b59
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/29303
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
The DictImporter's __del__ method calls unload, and that imports
sys.modules so that it can remove the modules that the DictImporter had
set up as the importer goes away.
Unfortunately, the importer only goes away when python is shutting down,
and at that time some aspects of the system, namely sys.meta_path, have
been cleaned up. When unload tries to import sys, that causes an
exception which scons/python reports but which doesn't do anything bad
otherwise.
In all of the examples of this older style of import object online, none
had a __del__ method, and none worried about cleaning up sys.modules
when they went away. In light of that, I've removed the __del__ method
entirely.
Another reason I think it's safe to remove __del__ is that the importer
was not actually being deleted even when it was removed from
sys.meta_path, and all the modules it had loaded where removed from
sys.modules. I think that was because the SimObject classes that it had
set up still had references (they are used later in the SConscript), and
those would, either directly or indirectly, refer back to the modules
and the importer. Those remaining references kept the importer alive,
preventing __del__ from being called before all those other objects were
cleaned up.
I think in python 2, the order things were cleaned up just so happened
to avoid trying to import sys when it was no longer possible, but in
python 3 that changed and resulted in this exception being thrown.
I've tried building gem5 with scons running under python 2 and python 3,
and with this change there is no error at shutdown. Both also produce a
gem5 binary which can run hello world without problems.
Change-Id: Ib1f5c7403df57fc420cec7ec0fef20a164a06991
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27247
Tested-by: Gem5 Cloud Project GCB service account <345032938727@cloudbuild.gserviceaccount.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
scons maintains an environment (in the shell sense) in the ENV
construction variable for use when running external programs. When we
run the "marshal" program which gathers up python objects to embed in
the gem5 binary, it's run by subprocess instead of through scons, and it
uses its own environment inherited from the host system.
Instead, this change makes the subprocess function use the scons
environment when calling "marshal". This ensures the environment is
consistent between this command and other commands scons runs.
This is usually not very important, but some tools like asan take
options set through the environment, and they may need to be adjusted
sometimes.
Change-Id: I671b447657ed8fad45fac7393cc1c09073bf3d3a
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27123
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Python 3 uses the concepts of two different types:
text and binary strings.
Those cannot be implicilty combined (as it was happening in python2) and
in order to be used together one of them must be converted to the other
type:
* Text can be encoded into Bytes via the encode() method
* Bytes can be decoded to Text using the decode() method
By default encode/decode will assume UTF-8 format
JIRA: https://gem5.atlassian.net/browse/GEM5-345
Change-Id: I1bdf7db17b49cc109239fd5f44791769370853f8
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26250
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
* dict methods dict.keys(), dict.items() and dict.values()
return "views" instead of lists
* The dict.iterkeys(), dict.iteritems() and dict.itervalues()
methods are no longer supported.
* map() and filter() return iterators.
* range() now behaves like xrange() used to behave, except it works with
values of arbitrary size. The latter no longer exists.
* zip() now returns an iterator.
Change-Id: Id480018239db88d7f5d60588c93719056de4a0c0
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26248
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Partial linking heuristically links together files in the same
directory by setting a special automatic tag. That tag needs to also
be maintained when scanning EXTRAS dirs so that they don't all get
lumped in with the last normal directory that was processed.
Change-Id: I2408ea0a1eeffcf6d9994c36415a35760b225b17
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/23300
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The current marshal binary is in $BUILD/python/ directory which makes
the dynamic linker fail to find shared libraries with rpath information
when the marshal is executing. Moving $BUILD/python/marshal to
$BUILD/marshal solves this problem.
This is because the marshal binary is built with the same flags as gem5
binary. And if one adds rpath information with add_local_rpath() in some
SConscripts, the marshal binary will also link to those shared libraries
and have the same rpath information. So it must be in the same
directory as gem5 binary such that the dynamic linker could find those
shared libraries with the rpath information.
Signed-off-by: Chun-Chen TK Hsu
Change-Id: I98d9c3e0180abefe6b2b6a78ee5a96324038eb53
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/18888
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Partial linking should be disabled on darwin; however, the script
fails to do so when force_lto is set, which results in gem5 building
with fast option fails on macOS. This fix changes disable_partial
logic, which should be True once it's True.
Signed-off-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Change-Id: I77d2a4cc4a9bf5c92c800c004eb744bb7081c42e
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/16888
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
We currently use the Python version used by scons to marshal Python
code. This doesn't work when building gem5 with Python 3 support since
scons typically runs in Python 2.7. Add a custom marshal helper that
links with the same library as gem5 to generate byte code that is
guaranteed to work in gem5's Python interpreter.
Change-Id: I665b0f2078726d4c055d74a3e668a580fc613b59
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/16422
Reviewed-by: Gabe Black <gabeblack@google.com>
