A comment in that section of SConstruct talked about how ParseConfig
couldn't understand -Xlinker and -export-dynamic, and how that meant
they needed to parse the output of python-config manually. Some
searching around turned up this:
https://bugs.python.org/issue36508
which seems to suggest that exactly those two options weren't actually
supposed to be returned by python-config in the first place, and are
really for python's own internal build.
This change switches over to using ParseConfig, but to avoid bad options
sneaking through, it uses an option argument to ParseConfig to intervene
before the options are consumed and filter out everything except
required libraries, and library and header search directories.
Change-Id: I1a50eb030019a447b919381c32168fb3bb27b89b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40971
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Instead, create a new tool called EnvDefaults. This tool now needs
to be explicitly listed in the set of tools available through an
Environment, but that's easy to do in on place (all environments should
stem from main). Hijacking default like we were doing was (as far as I
can tell) not how that tool was intended to be used, and doing things
this way is a bit less hacky.
Also, we can split more Builders, etc, out of SConstruct, and these will
need to attach to main in the same way. We're going to need a list of
tools when main is constructed one way or the other, so we might as well
follow the rules.
Change-Id: I392e1639cb69d373c64970dccf45258000498cc3
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40965
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
This has two purposes. First, SCons assumes that once you call
Configure, you won't set up the environment the Configure is based on
until after you get the environment back from it again with
conf.Finish(). We get away with this when the cache mode for config
tests is not "force", since Configure just reuses the environment we
pass in, and any changes we make are immediately communicated between
the two.
If the cache mode *is* "force" though, SCons modifies the decider so
that everything the conf environment goes to build looks like it's out
of date. It does that by cloning the original environment, and then
using that clone to do its tests. That causes a problem because we have
a long lived "conf" object and make further changes to main, and since
the two environments are now separate the one in conf doesn't see those
updates.
Second, and more subtly, we export our "main" and "env" environments so
that other SConsopts and SConscript files can use them and define things
in them. The way Configure is designed, if the config caching mode is
"force", then it will create a new environment, and then that
environment will replace what the, for instance, "main" variable points
to when "main = conf.Finish()" is executed.
Unfortunately, if we've already Export()-ed main, we've exported what
the "main" variable pointed to at that time. Our view of "main" will
track with the value that conf.Finish() returned, but since that
construction environment is mearly derived from the main we Exported and
not actually the same thing, they have diverged at that point and will
behave independently.
To solve both of these problems, this change modifies the
gem5_scons.Configure() method so that it's a context manager instead of
a regular function. As before, it will call Configure for us and create
a configuration context, which it will yield as the "with" value. When
the context exits, all the variables in the context Finish() returns
will be shoved back into the original context with Replace(). This isn't
perfect since variables which were deleted in the environment (probably
very rare in practice) will not exist and so will not overwrite the
still existent variable in the original dict.
This has several advantages. The environment never splits into two
copies which continue on independently. It makes the lifetime of a
configuration context short, which is good because behavior during that
time is tricky and unintuitive. It also makes the scope of the context
very clear, so that you won't miss the fact that you're in a special
setting and need to pay attention to what environment you're modifying.
Also, this keeps the conceptual overhead of configuration localized to
where the configuration is happening. In parts of the SConscripts which
are not doing anything with conf, etc, they don't have to modify their
behavior since no configuration context is active.
This change is based on this change from Hanhwi Jang who identified this
problem and proposed an initial solution:
https://gem5-review.googlesource.com/c/public/gem5/+/44265
Change-Id: Iae0a292d6b375c5da98619f31392ca1de6216fcd
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/44389
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Hanhwi Jang <jang.hanhwi@gmail.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
This option would set the -msse2 compiler flag to avoid using x87
floating point, but every 64 bit x86 CPU supports sse2, and so sse2 is
enabled by default on compilers targeting it. 32 bit x86 CPUs are
obscure enough that it's not worth preserving this marginally meaningful
option which would only matter for them.
Change-Id: Ib425c261b27f1f74300e714619787d04970065ce
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40879
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This flag was necessary because of self assignments in the ISA parser
where self assignments are often hints to the parser itself, and in one
case because a pybind-ism used to attach the -= operator looked like a
self assignment.
This change narrows the scope of the flag that disables this warning to
only files generated by the ISA parser, and the single file in the
systemc code which uses that operator overload.
Change-Id: Ib64fc72e46f894cba9064afcdbdcc5859c30e745
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40952
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Use SConsopts files local to individual domains to pull
non-foundational build code out of SConstruct. This greatly simplifies
SConstruct, and also makes it easier to find build configuration having
to do with particular pieces of gem5.
This change also converts some python level variables, all_protocols,
protocol_dirs, and slicc_includes, into the environment where the timing
of their initialization is more flexible.
Change-Id: Ie61ceb75ae9e5557cc400603c972a9582e99c1ea
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40872
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Gabe Black <gabe.black@gmail.com>
The default behavior already is to add color to the output. The option
had an action of 'store_true' which would normally set the default value
to False, but the --no-colors option immediately after it had the action
'store_false' which apparently set the common dest 'use_colors' to True
by default. The net effect was that the --colors option did nothing.
Change-Id: I6d6473f85921f90bb629ecb32a0f0f81c02d7914
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40875
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
The BUILD_GPU setting was being set by adding a -D to the command line
at the top level SConstruct. Instead, add BUILD_GPU to export_vars so
that it ends up in a config/build_gpu.hh header. Also switch it from the
fairly dangerous #ifdef style to the safer #if.
Change-Id: Ic93d37c9d7671023a6978842dbb2750c3e92f8d8
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40873
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
If these tools aren't enabled and CXX isn't set, scons will look for
generically named compiler aliases like cc and c++. These will generally
work, but if scons knows that the compiler is specifically clang, it
will set the CXXCONFIG variable we can use to do compiler version
checking.
Because scons blindly forces a tool into use if you specify it, we need
to use the FindTool method which will check a list of tools and add the
first one that it actually finds.
Change-Id: Ie6bebb8eab531989575c878bee07189541756d2c
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41673
Reviewed-by: Earl Ou <shunhsingou@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
gcc uses "make" to parallelize LTO. If we're using gcc and make isn't
found, we have to use single threaded LTO instead. A warning will let
the user know what's happening and that they might want to correct the
situation.
Technically gcc can use the MAKE environment variable to override the
program it uses, although I assume it still has to be "make" compatible.
Given the fairly low likelihood that someone will need that override and
the fact that scons won't pipe that variable through unless we plumb it
up, we'll just ignore that for now.
Change-Id: I891b213ece2a75bd8a915ee91f4130458dab397b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41773
Reviewed-by: Earl Ou <shunhsingou@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The mismatches were from places where Params structs had been declared
as classes instead of structs, and ruby's MachineID struct.
A comment describing why the warning had been disabled said that it was
because of libstdc++ version 4.8. As far as I can tell, that version is
old enough to be outside the window we support, and so that should no
longer be a problem. It looks like the oldest version of gcc we
support, 5.0, corresponds with approximately libstdc++ version 6.0.21.
https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html#abi.versioning
Change-Id: I75ad92f3723a1883bd47e3919c5572a353344047
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40953
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The name of the build is opt, so it should be fully optomized. Also, the
fast build, the only one with LTO historically, is dangerous to use
since it disables many error checks. I personally run gem5 many times
while developing, iterating and trying to fix bugs, and so want it to
run quickly then too, not just the final time when collecting results.
Also, since they mirror the opt build, the perf and prof builds also
have LTO options added.
This has the nice side effect of speeding up the build time of build/X86
significantly (6:20 -> 4:27) due to parallelization of the link, and
reduces the size of the build/X86 directory (with debug compression
enabled) from 3.4GB to 2.8GB.
The size of build/X86/python/_m5 is still 1.6GB, so still more than half
of the total size of build/X86.
Change-Id: I8feabf99454693fdd100d9e1a64fdeae53362f75
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40815
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Earl Ou <shunhsingou@google.com>
In the background, scons invokes commands using what is essentially
'sh -c ${CMD}'. It's a little more sophisticated than that, but the net
effect is that all of the arguments of a given command are collected
into a single argument for sh. While Linux has a very generous limit
on the number of command line arguments you can have, and a relatively
generous limit on the total size of the arguments and environment
variables which can be read with "getconf ARG_MAX" (about 2MB only my
system), the limit on an individual argument, defined in the little
documented constant MAX_ARG_STRLEN, is much smaller, 32 pages or
131072 bytes from what I've read.
Unfortunately, when building gem5, especially without partial linking,
the command line for the final linking step can be very long, and with
some extra files brought in with EXTRAS can relatively easily exceed
this limit.
To work around this problem, this change adds a new scons "tool" which
replaces the SPAWN and PSPAWN construction environment variables (not
the shell environment, scon's env, aka main) with wrappes. These
wrappers create a temporary file, write the command into it, and then
tell the original SPAWN/PSPAWN implementation to run that file as the
actual command.
This works, but it has two potential drawbacks. First, it creates
extra temporary files which may add some overhead to the build. The
overhead is likely small, but is non-zero.
Second, while this restores the original breaks between command line
arguments, it doesn't completely avoid the total argument/environment
size limit which would be harder to hit, but does still exist.
Another option would be to use the same sort of technique, but to use
gcc's (and I assume clang's) ability to read options from a file using
an @file argument, where file is replaced with the name of the
arguments.
The upside of this approach is that it avoids all of these limits
since gcc is just reading a file, and the only limits are ones it self
imposes. Also, this would only apply to invocations of gcc, and so
would be worth plumbing in to only affect, say, the linking step.
There would be no hypothetical overhead from creating temp files.
One downside though, is that this is not a universal approach and
would break anything which doesn't understand @file style arguments.
Also, when intercepting calls to the shell, the arguments we're
getting have been made appropriate for that environment (escaping,
quoting, etc). The rules for escaping arguments in an @file may not be
quite the same as for the shell, and so there may be weird cases where
a command line gets garbled this way.
Given the tradeoffs, I think always putting the commands into temp
files and running them as scripts is the way to go.
Change-Id: I0a5288aed745a432ed72ffd990ceded2b9422585
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41273
Reviewed-by: Earl Ou <shunhsingou@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
The original implementation had a function we'd call back into later
which checked if a given ISA has KVM support on the current host.
Instead, this change reverse that and statically figures out which
single target ISA could possibly run under KVM on this host. Then later,
we can just check if the ISA we're trying to build is that one.
Change-Id: I3e7e06180983dfcc4611181718eb5d6210844d36
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40870
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Collapse nesting, and constrain the try/except to only the line in that
section which might actually be expected to throw an exception when
things are working normally.
This makes the code easier to read, and won't absorb and discard
exceptions which are thrown for unexpected reasons.
Change-Id: Id3ac417ab5545bef35ed8a87d05211c8ef2f5a50
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40867
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Collapse the check_hdf5 to the only place it was called, to set a simple
flag variable. Pull the invariant have_pkg_config check out of
check_hdf5_pkg, since if we don't have pkg-config there's no reason to
even try to use to set up hdf5. Turn the one-off, non-standard "Warning"
print into an actual warning().
Change-Id: I04793ae862aeaf1605467d205837b9dd744f93cb
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40865
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
There are really only two options current, glibc or none. If there's a
working implementation there's no real reason to select none, and if
there isn't there's no other option but to select none.
Instead of building up a list, having a default, and making what option
on the list is selected configurable, boil it down to either using glibc
if that implementation is detected, or warning and using none. Also
merge the "normal" and *BSD versions of the checks to reduce redundancy.
The complexity can be added back in if/when there are other
implementations to choose from.
Change-Id: I27c77996a00018302f4daea40924cf059d5a4323
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40864
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
