Commit Graph

249 Commits

Author SHA1 Message Date
Giacomo Travaglini
d574381081 scons, python: Remove SmartDict from python utilities
The SmartDict, used by buildEnv, has been added long time ago for
the following reasons: (checking its documentation)

---
The SmartDict class fixes a couple of issues with using the content
of os.environ or similar dicts of strings as Python variables:

1) Undefined variables should return False rather than raising KeyError.

2) String values of 'False', '0', etc., should evaluate to False
   (not just the empty string).
---

These are valid reasons, but I believe they should be addressed in
a more standardized way by using a common dictionary.

1) We should simply rely on dict.get

if buildEnv.get('KEY', False/None):

2) We should discourage the use of stringified False or 0.
If we are using a dictionary, can't we just pass those values as
booleans?
The SmartDict is basically converting every value into a
string ("Variable") at every access (__getitem__)
The Variable is a string + some "basic" conversion methods
What is the problem of passing every dict value as a string?

The problem is the ambiguity on the boolean conversion.

If a variable is modelling a boolean, we can return true if
the value is 'yes', 'true'... and false if the value is
'no', 'false' etc. We should raise an exception if it is
something different, like a typo (e.g.) 'Fasle'.
But if the variable is not modelling a boolean, we don't know
how to handle that. How should we convert 'mystring' ?

If we decide to treat 'mystring' as True (which is basically
what a str.__bool__ would return) we will break typoes detection,
as 'Fasle' will now be converted to True, rather than raising
an exception.

Change-Id: I960fbfb1ec0f703e1e372dd752ee75f00632acac
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/37775
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-11-20 16:54:31 +00:00
Gabe Black
8e34e0b920 scons: Add support for GRPC protobuf files.
These files are used to generate stubs for calling across GRPC
protocols, an RPC mechanism which is based around the protocol buffer
system.

The support for these files is heavily based on and calls into the
existing protobuf file support, but with the extra step which generates
the additional .grpc.pb.cc and .grpc.pb.h files.

Change-Id: I89022928c08aa9f7ed024b7380ddcc54ca75b55e
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/37277
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-11-12 22:08:42 +00:00
Gabe Black
6bbdfa370c scons: Convert ProtoBuf to use a scons Builder and Scanner.
There are several benefits to using a Builder. First, the action we're
executing is shared between all uses of the Builder. The number of
times this particular builder is called is small, but it should still
be a little more efficient.

Second, we can use SCons's emitter mechanism to generate the .pb.cc and
.pb.h target files in a little more general way.

Also, this change adds a Scanner for .proto files which will scan them
for imports and let SCons manage those implicit dependencies properly.
The scanner is a bit simplistic as described in a comment in the
source, but should work pretty well in practice with reasonably
formatted files, and in particular some files I'm working with that
include imports.

Change-Id: Iaf2498e61133d6f713d6ccaf199422b882c5894f
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/37276
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-11-12 22:08:21 +00:00
Gabe Black
0957ab118c scons: Consolidate the ProtoBuf code.
The ProtoBuf support in src/SConscript was split into two parts, one
where the ProtoBuf sources were declared, and the other where scons was
told how to buld the .cc and .hh files and the .cc was added to the
build.

As far as I can tell, there was no real reason to have things split up
like that, at least not currently. This change moves everything into
the ProtoBuf class definition, and this should behave the same as
before but be a little easier to understand and maintain.

Change-Id: I02320f50ece53d90c14b5062bd6b1167210f46c3
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/37275
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-11-12 22:08:02 +00:00
Gabe Black
598f15cf24 scons: Fix how directories are handled for protobuf files.
There were two issues with how paths were handled for these files.

1. The code in the ProtoBuf class would drop the subdirectory part of
the path name when generating the name of the .cc and .h files the
protoc compiler would output. Since protoc wouldn't generate files
where scons expected, it would fail when it tried to build the .cc.

2. protoc will use the --proto_path and --cpp_out settings to figure
out what path to use for generated files. It will remove the
--proto_path prefix it found the .proto file with from the files path,
and then add the rest to the --cpp_out prefix.

The input files should come from the build directory using symlinks
set up by scons, and the output files should end up alongside them.
That means the --proto_path setting should be the build directory, and
so should --cpp_out. That's fortunately simpler than what was there
before, since it doesn't depend on what the source or targets are.

Change-Id: I69692d2fe3813011982f0c1c9824589a132f93ed
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/37218
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-11-12 22:07:42 +00:00
Andreas Sandberg
133e84ff5c scons: Simplify arch enum generation
C++ allows a trailing comma after the last item in an enum, so there
is no need for a special case.

Change-Id: I6ead36b4a8562b4a7a5aec88e4f6390182eedf56
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/34116
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-09-07 08:42:07 +00:00
Andreas Sandberg
e9075be4f9 base: Cleanup Debug::CompoundFlag
Compound flags are currently constructed using a constructor with a
finite set of arguments that default to nullptr that refer to child
flags. C++11 introduces two cleaner ways to achieve the same thing,
variadic templates and initializer_list. Use an initializer list to
pass dependent flags.

Change-Id: Iadcd04986ab20efccfae9b92b26c079b9612262e
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/34115
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-09-07 08:42:07 +00:00
Gabe Black
f0e63088ab scons: Make scons aware of changes to cxx_type in param types.
If the cxx_type value changes, then the way the parameter is declared
in the param struct will also change, and the header needs to be
updated. scons would miss this sort of change before because it was only
checking the module the SimObject's source came from. The python names
and types of the parameters could stay the same, but the C++
representation might have changed because of edits somewhere else.

This CL assumes that cxx_type is the only thing that will change and
transparently affect the params struct. I tried making scons sensitive
to the entire ptype which would capture other parameters, but that
didn't work for some reason. This should be pretty safe in general, but
not 100% safe

Issue-on: https://gem5.atlassian.net/browse/GEM5-753
Change-Id: I06774889e60b987f727799f55d7ea2a775b6a319
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/33695
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-09-01 21:55:50 +00:00
Gabe Black
9d31dde29c scons: Remove the plumbing for running regression tests from scons.
All of these tests have been migrated to the new framework, so there's
no reason to leave the old plumbing lying around.

Change-Id: Iaa5412864354d5754a68a9f53f30aa42f07ec2eb
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/32117
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
2020-08-05 09:39:17 +00:00
Giacomo Travaglini
a1b6471fe5 scons: Add MARSHAL_XXFLAGS_EXTRA for the marshal object
We already provide to the user the CCFLAGS_EXTRA, LDFLAGS_EXTRA
variables to pass flags to scons when compiling/linking gem5.
Those variables are not passed to the marshal object.
We add an extra pair:

MARSHAL_CCFLAGS_EXTRA, MARSHAL_LDFLAGS_EXTRA

to add flag injection capabilities to the marshal object.

The patch is also renaming base_py_env to marshal_env.
This happens for 2 reasons:

1) At the moment the marshal compilation is the only task
making use of the base python environment.

2) Consistency with the EXTRA variable names added with this patch.
I could have named them as BASE_XXFLAGS_EXTRA, but it seems too much
generic and users might be confused by that, as they might think
the BASE_XXFLAGS_EXTRA is a subset of the XXFLAGS_EXTRA so that
setting it will affect gem5 compilation as well.

Change-Id: I3e420caa897059455ff8f35462db2b38da050e93
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/30016
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-06-22 09:08:42 +00:00
Gabe Black
d45c4b7523 scons: Fix how partial linking is disabled.
Setting disable_partial part way through the checks for various build
targets is incorrect and will affect targets based on the order they're
checked.

This change moves the check earlier, makes it consistent across all
builds whether fast is included or not, and stops passing it in as an
option to makeEnv since it now applies universally.

By disabling partial linking consistently, we avoid missing bugs where
only the "fast" version of gem5 doesn't build correctly because of the
multitude of g++ bugs having to do with combining LTO and partial
linking.

This also simplifies the logic in the SConscript by having fewer
independently moving parts.

Change-Id: Iff69f39868e948d3b9a5b11ea80bbfed19419b59
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/29303
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
2020-06-12 08:48:14 +00:00
Bobby R. Bruce
77482d59a5 scons,python: Update makeTheGPUISA to Python3
This function was causing an error to occur when trying to compile HSAIL
and GCN in a Python3 environment. It has now been upgraded to work in
both Python2 and Python3.

Change-Id: If8d6ee1e08c47d5a36182afc10cf86a8e905bda0
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/29410
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-05-26 09:14:09 +00:00
Bobby R. Bruce
fc3112d4bd misc: Fixed HSAIL_X86 compilation errors
HSAIL_X86 fail to compile. This patch enables compilation.

Issue-on: https://gem5.atlassian.net/browse/GEM5-556

Change-Id: I663e529622ed90254eaf8be01e23991ed8271b5b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/29293
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-05-23 01:08:55 +00:00
Nikos Nikoleris
a0414b585b scons: Build the marshal binary in a bare minimum environment
This change adds an additional bare minimum environment that includes
python only and changes the marshal binary to compile using it.

Change-Id: Id5d1ee6899796d746d8dc1a004cfe4795f040c55
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/28428
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-05-01 23:23:52 +00:00
Bobby R. Bruce
571ab4ad70 base,misc: Added version to code
This version ID string can be accessed from elsewhere in the CPP
codebase using `extern const char *gem5Version;`. The python variable
"gem5Version" may be accessed in by importing the generated
"defines.py". When gem5 now runs it will output the version
currently being used to the CLI.

Change-Id: I4feb6b7317cd2922c56eb8ce3e532b0bfa0e6ed3
Issue-on: https://gem5.atlassian.net/browse/GEM5-411
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27787
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-04-22 19:50:39 +00:00
Gabe Black
7e303da76f scons: Fix an exception in the DictImporter on scons shutdown.
The DictImporter's __del__ method calls unload, and that imports
sys.modules so that it can remove the modules that the DictImporter had
set up as the importer goes away.

Unfortunately, the importer only goes away when python is shutting down,
and at that time some aspects of the system, namely sys.meta_path, have
been cleaned up. When unload tries to import sys, that causes an
exception which scons/python reports but which doesn't do anything bad
otherwise.

In all of the examples of this older style of import object online, none
had a __del__ method, and none worried about cleaning up sys.modules
when they went away. In light of that, I've removed the __del__ method
entirely.

Another reason I think it's safe to remove __del__ is that the importer
was not actually being deleted even when it was removed from
sys.meta_path, and all the modules it had loaded where removed from
sys.modules. I think that was because the SimObject classes that it had
set up still had references (they are used later in the SConscript), and
those would, either directly or indirectly, refer back to the modules
and the importer. Those remaining references kept the importer alive,
preventing __del__ from being called before all those other objects were
cleaned up.

I think in python 2, the order things were cleaned up just so happened
to avoid trying to import sys when it was no longer possible, but in
python 3 that changed and resulted in this exception being thrown.

I've tried building gem5 with scons running under python 2 and python 3,
and with this change there is no error at shutdown. Both also produce a
gem5 binary which can run hello world without problems.

Change-Id: Ib1f5c7403df57fc420cec7ec0fef20a164a06991
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27247
Tested-by: Gem5 Cloud Project GCB service account <345032938727@cloudbuild.gserviceaccount.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
2020-04-02 22:25:07 +00:00
Gabe Black
bb3bc1fdd0 scons: Use the scons environment when marshalling.
scons maintains an environment (in the shell sense) in the ENV
construction variable for use when running external programs. When we
run the "marshal" program which gathers up python objects to embed in
the gem5 binary, it's run by subprocess instead of through scons, and it
uses its own environment inherited from the host system.

Instead, this change makes the subprocess function use the scons
environment when calling "marshal". This ensures the environment is
consistent between this command and other commands scons runs.

This is usually not very important, but some tools like asan take
options set through the environment, and they may need to be adjusted
sometimes.

Change-Id: I671b447657ed8fad45fac7393cc1c09073bf3d3a
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27123
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-03-26 21:44:36 +00:00
Giacomo Travaglini
1bf2821ae7 misc: Make exception handling python3 compliant
Change-Id: I37d0e97e9762e21c7a0ad315cf7684a19119b5b4
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26251
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
2020-03-13 14:24:07 +00:00
Giacomo Travaglini
e06ec8c423 misc: Text vs Byte string in python3
Python 3 uses the concepts of two different types:
text and binary strings.
Those cannot be implicilty combined (as it was happening in python2) and
in order to be used together one of them must be converted to the other
type:

* Text can be encoded into Bytes via the encode() method
* Bytes can be decoded to Text using the decode() method

By default encode/decode will assume UTF-8 format

JIRA: https://gem5.atlassian.net/browse/GEM5-345

Change-Id: I1bdf7db17b49cc109239fd5f44791769370853f8
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26250
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
2020-03-13 14:24:07 +00:00
Giacomo Travaglini
bb95aa14c6 scons: file removed from python3
Change-Id: I95794fec5cc9c6e72c9ae53f586052be71436c87
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26249
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
2020-03-13 14:24:07 +00:00
Giacomo Travaglini
10b4842407 misc: Views and Iterators instead of Lists in python3
* dict methods dict.keys(), dict.items() and dict.values()
return "views" instead of lists

* The dict.iterkeys(), dict.iteritems() and dict.itervalues()
methods are no longer supported.

* map() and filter() return iterators.

* range() now behaves like xrange() used to behave, except it works with
values of arbitrary size. The latter no longer exists.

* zip() now returns an iterator.

Change-Id: Id480018239db88d7f5d60588c93719056de4a0c0
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26248
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-03-13 14:24:07 +00:00
Giacomo Travaglini
656b18a414 misc: Replace basestring with six.string_types
Change-Id: I914adb545fe82efed386085d2f191ec54efccbf3
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26246
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-03-10 09:35:56 +00:00
Giacomo Travaglini
3c1f87c81c misc: Replace __metaclass__ with six.add_metaclass
Change-Id: I45f73b71266c1edf9c4ec521d58db779b16bd591
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26245
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-03-10 09:35:56 +00:00
Gabe Black
9164511ef1 scons: Remove authors from the main SConsctruct/SConscript files.
Change-Id: I48987ea4d829e722caf16126dd82f2c580e7836a
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/25412
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
2020-02-17 10:05:36 +00:00
Gabe Black
dc26205529 scons: Disable spurious "array-bounds" warnings for protobuf cc files.
These files are generated and so, even if they're wrong, there isn't
anything we can do about it.

Change-Id: I933057a04f09dd1c22b525a102278bbdc5fbc22b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/24924
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-02-01 09:25:15 +00:00
Gabe Black
017ffc268a scons: Add a mechanism to append flags when building particular files.
This could be used to tweak settings for a particular file if it needed
special treatment. I plan to use this for protobuf files which generate
code that produce a warning in gcc 9 which turns into an error.

Change-Id: I53e2dac48cd68f0cc8ad9031d8484c5036d6e4a6
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/24923
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2020-02-01 09:25:07 +00:00
Gabe Black
3b9395cd02 scons: Set the partial linking group for EXTRAS dirs.
Partial linking heuristically links together files in the same
directory by setting a special automatic tag. That tag needs to also
be maintained when scanning EXTRAS dirs so that they don't all get
lumped in with the last normal directory that was processed.

Change-Id: I2408ea0a1eeffcf6d9994c36415a35760b225b17
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/23300
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2019-12-07 00:04:40 +00:00
Gabe Black
80283488ff scons: Use the new error() and warning() methods.
Also clean up some error messages which were missing capitalization,
etc.

Change-Id: Iaef6b4343a693d30b579e72218cbb7723ebf7d48
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/22886
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2019-11-21 23:37:03 +00:00
Gabe Black
a434c6cc45 scons: Use HAVE_PROTOC when building protobuf files.
Also wrap PROTOC in {}s which is better form.

Change-Id: I3f80c260593a1d5b7fb5394fe4b71cb774e652ce
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/22884
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2019-11-21 23:37:03 +00:00
Chun-Chen TK Hsu
874bb1a760 scons: Move the marshal binary to the build directory
The current marshal binary is in $BUILD/python/ directory which makes
the dynamic linker fail to find shared libraries with rpath information
when the marshal is executing. Moving $BUILD/python/marshal to
$BUILD/marshal solves this problem.

This is because the marshal binary is built with the same flags as gem5
binary. And if one adds rpath information with add_local_rpath() in some
SConscripts, the marshal binary will also link to those shared libraries
and have the same rpath information.  So it must be in the same
directory as gem5 binary such that the dynamic linker could find those
shared libraries with the rpath information.

Signed-off-by: Chun-Chen TK Hsu
Change-Id: I98d9c3e0180abefe6b2b6a78ee5a96324038eb53
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/18888
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
2019-05-21 15:46:50 +00:00
Hoa Nguyen
2a28a4f8d2 scons: fix disable_partial logic for fast binary
Partial linking should be disabled on darwin; however, the script
fails to do so when force_lto is set, which results in gem5 building
with fast option fails on macOS. This fix changes disable_partial
logic, which should be True once it's True.

Signed-off-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Change-Id: I77d2a4cc4a9bf5c92c800c004eb744bb7081c42e
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/16888
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
2019-03-18 22:22:25 +00:00
Andreas Sandberg
c38a6523ab scons: Marshal Python sources using the same Python as gem5
We currently use the Python version used by scons to marshal Python
code. This doesn't work when building gem5 with Python 3 support since
scons typically runs in Python 2.7. Add a custom marshal helper that
links with the same library as gem5 to generate byte code that is
guaranteed to work in gem5's Python interpreter.

Change-Id: I665b0f2078726d4c055d74a3e668a580fc613b59
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/16422
Reviewed-by: Gabe Black <gabeblack@google.com>
2019-02-26 10:27:07 +00:00
Andreas Sandberg
7d71f6641f python: Make iterator handling Python 3 compatible
Many functions that used to return lists (e.g., dict.items()) now
return iterators and their iterator counterparts (e.g.,
dict.iteritems()) have been removed. Switch calls to the Python 2.7
iterator methods to use the Python 3 equivalent and add explicit list
conversions where necessary.

Change-Id: I0c18114955af8f4932d81fb689a0adb939dafaba
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15992
Reviewed-by: Juha Jäykkä <juha.jaykka@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
2019-02-22 10:47:36 +00:00
Gabe Black
0a7372633a scons: Add a convenience method to set RPATH for local libraries.
When linking in a dynamic library which is in the gem5 build directory,
it's useful to set RPATH so that you don't have to set LD_LIBRARY_PATH
when you run gem5 so that the dynamic linker can find it.

Since it's tricky and not entirely obvious how to set up those paths
correctly, this change adds a small convenience function which does
that for you. It also handles situations where the same dynamic
library may be linked into different binaries in different directories
which each need a different relative RPATH. It does that by letting the
environment for each binary set a construction variable which says
how to get from that particular binary back to the build directory.
This helper method then sets RPATH to start at $ORIGIN (the binary),
to follow that relative path to the variant build directory, and then
the per-library but not per-binary path to the library's directory.

This change also adds the -z origin linker flag which makes the linker
handle $ORIGIN properly.

Change-Id: I45f4d72cd14396a73e0b963cea6a39d9bfb7f984
Reviewed-on: https://gem5-review.googlesource.com/c/16566
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
2019-02-21 22:37:21 +00:00
Andreas Sandberg
5cd4248672 python: Replace dict.has_key with 'key in dict'
Python 3 has removed dict.has_key in favour of 'key in dict'.

Change-Id: I9852a5f57d672bea815308eb647a0ce45624fad5
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15987
Reviewed-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
2019-02-12 17:36:12 +00:00
Gabe Black
8174d7c87b scons: Change an = to a += when accumulating sources from filters.
The loop accidentally used a = when it should have used a +=, meaning
only the sources from the final filter would be used.

Change-Id: Ie066a5f85696f05d9ad3cf61f928b12deb39475b
Reviewed-on: https://gem5-review.googlesource.com/c/16285
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>
2019-02-11 07:33:50 +00:00
Ciro Santilli
ff5ad434d9 scons: fix unused auto-generated blob variable in clang
Since f2bda876f7, the build system started
adding a length for generated blobs as in:

const std::size_t variable_len = 123;

There were two types of blob files, ones with a header and the ones
without.

The ones with the header, also include the header in the .cc of the blob,
which contains a declaration:

extern const std::size_t variable_len;

Therefore, the ones without header, don't have that extern declaration,
which makes them static according to the C++ standard.

clang then correctly interprets that as problematic due to
-Wunused-const-variable, while GCC does not notice this.

This patch removes the length declaration from the blob files that don't
have the header. Those files currently don't use the length.

Change-Id: I3fc61b28f887fc1015288857328ead2f3b34c6e6
Reviewed-on: https://gem5-review.googlesource.com/c/15955
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
2019-02-06 16:11:24 +00:00
Ciro Santilli
9712a635a4 scons: add helpers to access GDB XML description files
Change-Id: Ic3b18887544b7710ed07a86d28dc62d8441b3476
Reviewed-on: https://gem5-review.googlesource.com/c/15255
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
2019-01-22 11:35:12 +00:00
Ciro Santilli
f2bda876f7 scons: allow embedding arbitrary blobs into the gem5 executable
The initial motivation for this is to embed the GDB XML target
description files into the executable.

Change-Id: I721e8dd37119d8e6eb376d7e9050b1094282bacc
Reviewed-on: https://gem5-review.googlesource.com/c/15136
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
2019-01-22 11:35:12 +00:00
Nikos Nikoleris
dcc15d6d07 scons: Disable partial linking on Mac OS
Up until Apple LLVM version 10.0.0 (clang-1000.11.45.5), partial
linked objects do not expose symbols that are marked with the hidden
visibility and consequently building gem5 on Mac OS fails. As a
workaround, we disable partial linking, however, we may want to
revisit in the future.

Change-Id: I0a26dae082bf723c2bd49d90e4497e44ecab9c41
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/15235
Reviewed-by: Andrea Mondelli <andrea.mondelli@ucf.edu>
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
2019-01-07 14:52:00 +00:00
Gabe Black
6df90da9fb scons: Generalize building binaries.
Building gem5 binaries or regression test binaries needs to be done
from within the make_env function which builds an environment for each
flavor of build (opt, fast, debug, etc.). That makes it impossible to
add new types of binaries without modifying the central SConscript.

This change refactors how binaries are set up so that the class that
represents them handles the details of how the binary should be built.
Also, a metaclass and some lists track types of binaries and individual
instances of binaries so that they can be iterated over automatically
in make_env.

Each new executable class can define a declare_all class function which
calls declare() on individual instances. declare_all is a place to do
any processing that only has to happen once (for instance specializing
the environment) for a particular family of executables.

Change-Id: I8a6ee9438280cd67e6c0b92ca28738a53cb16950
Reviewed-on: https://gem5-review.googlesource.com/10915
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
2018-06-26 00:15:12 +00:00
Gabe Black
31791727d3 tests: Make "UnitTest"s more like GTest so they can be in other dirs.
The original implementation of UnitTest forced all the output binaries
to live in the unittest directory, effectively forcing a flat
namespace, and seperating the tests from the things they were supposed
to be testing.

This changes makes them work more like the newer GTest tests in that
they can be based out of whatever directory makes sense, although
they're currently all still in unittest for the time being.

This change also gets rid of automatically tagging the sources
associated with a test with the tests name. The first reason for that
was that this also forced a flat namespace, since the tests names
didn't have any reference to the test's path. Second, this way of
pulling in additional files wasn't necessary any more, now that the
UnitTest sources could be source filters like they can be for GTests.

Change-Id: I3d96ed766ac5170842dbd6daee39f2873bcd6c75
Reviewed-on: https://gem5-review.googlesource.com/10701
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
2018-06-13 23:02:03 +00:00
Gabe Black
0bb50e6745 scons: Switch from the print statement to the print function.
Starting with version 3, scons imposes using the print function instead
of the print statement in code it processes. To get things building
again, this change moves all python code within gem5 to use the
function version. Another change by another author separately made this
same change to the site_tools and site_init.py files.

Change-Id: I2de7dc3b1be756baad6f60574c47c8b7e80ea3b0
Reviewed-on: https://gem5-review.googlesource.com/8761
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Gabe Black <gabeblack@google.com>
2018-03-06 23:39:01 +00:00
Andreas Sandberg
a45289dee8 scons, tests: Fix occasional linking error
There are some cases where scons incorrectly adds the same object
multiple times to the linker command line. This seems to be caused by
the test's source list being updated in place when determining test
framework dependencies. Fix this by explicitly copying the source list
and manipulate the copy.

Without this change, the following command fails:
scons ./build/ARM/unittests.opt/base/pixeltest.xml

Whereas this command succeeds:
scons ./build/ARM/base/pixeltest.opt

Change-Id: I642efdf9d62a5478e49ba51efe1a3a5ba453e21f
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jack Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/6641
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
2017-12-13 23:57:08 +00:00
Andreas Sandberg
ed371e0fec scons, tests: Add support for GTest XML generation
The GTest framework supports result generation in XML (JUnit). Enable
this by creating unit test specific targets in the
build/${BUILD_OPTS}/unittests.${VARIANT} directory. Targets in the
directory use the following naming convention:

${SRC_PATH}/${TEST_NAME}.xml

For example, the opt version of the bitunion test built for ARM would
have this path:

build/ARM/unittests.opt/base/bituniontest.xml

Change-Id: I174dff16817734db05b08ce1d5bcf52e8697bbac
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jack Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/6624
Reviewed-by: Gabe Black <gabeblack@google.com>
2017-12-13 23:57:08 +00:00
Andreas Sandberg
bace877821 scons: Make sure GTests have the right environment variables
SCons currently scrubs the environment variables used by GTests too
aggressively. This breaks systems where libraries are installed in
non-standard locations that need to be specified in
LD_LIBRARY_PATH. Run said tests in the gtest_env SCons environment
which white-lists the important environment variables.

Change-Id: I5fc8fb5e51f09644dc976ee97b21c78ab349bf7d
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jack Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/6623
Reviewed-by: Gabe Black <gabeblack@google.com>
2017-12-13 23:57:08 +00:00
Gabe Black
66a55ce3ff tests: Accept SourceFilters as sources for GTest.
This change introduces the idea of a SourceFilter which is an object
that can filter a SourceList and which can be composed with other
SourceFilters using | and & operators. This means a filter can be
constructed ahead of time, possibly before all sources have been
discovered, and then later applied to any SourceList necessary.

This change also modifies GTest so that it accepts SourceFilters in
addition to normal source files. These filters will be applied to the
final list of all sources, and the result included in the build for
that test.

By default, gtests will build in all sources tagged with 'gtest lib'.
This change also introduces the keyword argument "skip_lib" which will
exclude those files. They can then be left out entirely, or they can be
re-included as part of a more elaborate filter. That would be useful if
someone wanted to write a unit test for, for instance, the warn, etc.
macros which rely on the gtest logging support. Those classes could
be replaced by something under the control of the unit test, while
still including the rest of the gtest library.

Change-Id: I13a846dc884b86b9fdcaf809edefd57bb4168b8e
Reviewed-on: https://gem5-review.googlesource.com/6262
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
2017-12-07 10:07:53 +00:00
Gabe Black
08fc1d5ee3 tests: Add a pseudo target to run all the unit tests for build/variant.
Telling scons to build build/${BUILD_OPTS}/unittests.${VARIANT} will
get it to build and run all googletest based unit tests under the
${BUILD_OPTS} build options (ARM, ALPHA, X86, etc.), and compiled with
the flags, etc., for the ${VARIANT} variant (ie. opt, debug, etc.).

This will make it easy to run the unit tests without having to actually
know where they are, what tests are available, etc.

This target is called unittests* and not something based on gtest or
googletest since it's my intention for all unit tests to be based on
googletest, making the distinction unnecessary. Since the target is
essentially part of the external interface for scons, I wanted to name
it something general so it'll be less likely that we have to change it.

Change-Id: I8fdec768d821974309c92a2ce4c96dce7df24fa5
Reviewed-on: https://gem5-review.googlesource.com/6282
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Brandon Potter <Brandon.Potter@amd.com>
Maintainer: Gabe Black <gabeblack@google.com>
2017-12-07 06:02:08 +00:00
Gabe Black
71accb5f78 scons: Several fixes having to do with tags and sets.
There were a few places where tags weren't being converted to sets
correctly which unfortunately only manifested when called in certain
ways. This would be a pretty reasonable place to add some python unit
tests...

Change-Id: I87509369b4ec6f702b7521e52bf63701a87ec436
Reviewed-on: https://gem5-review.googlesource.com/6261
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
2017-12-06 01:07:03 +00:00
Gabe Black
0b11c2ea6d scons: Track and reuse object nodes for a given source file.
scons gets upset if two different environments are used to set up a
particular object file. This change adds two dicts to the SourceFile
class, one for static and one for shared object files, which are keyed
off of the appropriate suffix. If a suffix hasn't been set up yet,
a new node of the appropriate type is set up and stored in the cache,
and then whatever is in the cache (new or old) is returned.

Change-Id: Ice4b4fc728b438a4d3316c3ff6667c0480d2a6d7
Reviewed-on: https://gem5-review.googlesource.com/6224
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
2017-12-06 01:02:20 +00:00