In the background, scons invokes commands using what is essentially
'sh -c ${CMD}'. It's a little more sophisticated than that, but the net
effect is that all of the arguments of a given command are collected
into a single argument for sh. While Linux has a very generous limit
on the number of command line arguments you can have, and a relatively
generous limit on the total size of the arguments and environment
variables which can be read with "getconf ARG_MAX" (about 2MB only my
system), the limit on an individual argument, defined in the little
documented constant MAX_ARG_STRLEN, is much smaller, 32 pages or
131072 bytes from what I've read.
Unfortunately, when building gem5, especially without partial linking,
the command line for the final linking step can be very long, and with
some extra files brought in with EXTRAS can relatively easily exceed
this limit.
To work around this problem, this change adds a new scons "tool" which
replaces the SPAWN and PSPAWN construction environment variables (not
the shell environment, scon's env, aka main) with wrappes. These
wrappers create a temporary file, write the command into it, and then
tell the original SPAWN/PSPAWN implementation to run that file as the
actual command.
This works, but it has two potential drawbacks. First, it creates
extra temporary files which may add some overhead to the build. The
overhead is likely small, but is non-zero.
Second, while this restores the original breaks between command line
arguments, it doesn't completely avoid the total argument/environment
size limit which would be harder to hit, but does still exist.
Another option would be to use the same sort of technique, but to use
gcc's (and I assume clang's) ability to read options from a file using
an @file argument, where file is replaced with the name of the
arguments.
The upside of this approach is that it avoids all of these limits
since gcc is just reading a file, and the only limits are ones it self
imposes. Also, this would only apply to invocations of gcc, and so
would be worth plumbing in to only affect, say, the linking step.
There would be no hypothetical overhead from creating temp files.
One downside though, is that this is not a universal approach and
would break anything which doesn't understand @file style arguments.
Also, when intercepting calls to the shell, the arguments we're
getting have been made appropriate for that environment (escaping,
quoting, etc). The rules for escaping arguments in an @file may not be
quite the same as for the shell, and so there may be weird cases where
a command line gets garbled this way.
Given the tradeoffs, I think always putting the commands into temp
files and running them as scripts is the way to go.
Change-Id: I0a5288aed745a432ed72ffd990ceded2b9422585
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41273
Reviewed-by: Earl Ou <shunhsingou@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
When calling TryCompile with an empty string for the source, useful
when just testing a build flag, some versions of SCons will apparently
create some sort of equivalence between Value('') and Value(0). That
shows itself when creating config file headers, where Value(0) is
switched with Value(''), and the header defines a macro with which
expands to nothing rather than a macro which expands to 0. Later uses
of the macro of the form:
#if CONFIG_VARIABLE
fail because CONFIG_VARIABLE expands to nothing. If it expanded to 0
like it's supposed to, then the guarded block of code would be excluded
correctly.
Change-Id: Ie324ec5f8dd307c65745b9326a11230e10caa0bd
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41213
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Gabe Black <gabe.black@gmail.com>
These functions where correctly returning whether a flag had existed,
and also correctly not installing it if asked not to. Unfortunately if
they *were* asked to install the flag, they ignored whether or not it
had actually existed to begin with.
Change-Id: I2dca0e1a0ddbc182576d48237aeea5452a02c51b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41159
Maintainer: Gabe Black <gabe.black@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
SCons has a system of "tools", which basically detect versions of build
tools (compilers, linkers, etc) and set up an environment with the
appropriate build variable substitutions for that tool to be used.
For instance, there would be a "tool" for gcc, and it would detect if
gcc is present on the system, and if so would set the "CC" variable to
"gcc". An actually tool as defined by SCons would be a lot more
sophisticated than that and set more variables, but that's the basic
idea.
To help modularize the gem5 SConstruct file, I moved code which would
set up git commit hooks into a "tool" which helped modularize it and
reduce the size of SConstruct.
This isn't quite right since, while the code does detect if git was
used to check out the source (if there is a .git file at the root), it
doesn't really modify the environment at all. It will also be invoked
every time any environment is set up, although right now that will only
be the DefaultEnvironment, what's used when loose functions like
Builder or Command are called with, and the "main" environment which
all the others are Clone-d from.
Normally, when SCons sets up a new environment, either implicitly or
when Environment() is called, it sets up a bunch of built in tools
which are fixed within SCons itself. If you want, you can add a "tools"
argument to Environment (or to the DefaultEnvironment() function) which
will replace that list of tools. That can be used to make an
environment use the new "git" tool, but it isn't automatic.
SCons also lets you override default tools by creating your own with
the same name as the default. To make loading the git tool automatic,
I added an override "default" tool which, in addition to setting some
defaults which should apply to all environments, also pulled in other
tools, at that time "git" and "mercurial" (RIP).
Unfortunately, that meant that today, apparently particularly with
SCons version 4, *any* Environment would pull in "git", and all of
"git"'s dependencies, even if SCons wasn't set up enough for those to
work.
To break that dependency, this change stops the default tool from
automatically loading the git tool, although it does continue to set
other defaults which have very minimal external dependencies. When
creating the "main" Environment in the SConstruct, the "git" tool is
now added in explicitly. Since the list of tools replaces the original
and doesn't extend it, we have to add in "default" explicitly as well.
Really, the "git" tool should be converted from the tool interface into
something more appropriate, like perhaps a small module under
site_scons which site_init.py can import and call. When that happens,
main can be declared like normal again.
While making this change, I also got rid of a few nonstandard additions
to the main environment that were little used and not really necessary.
When reading the SConstruct, it wasn't very obvious where those extra
values were coming from, and they didn't really add any value.
Change-Id: I574db42fc2196bf62fc13d6754357c753ceb9117
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/38616
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Yu-hsin Wang <yuhsingw@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
As gem5 has started to use Python2 incompatible features, compiling
gem5 in a Python2 environment results in an error.
This commit addresses this issue by raising an Exception when scons
is run in a Python2 environment, and adding a few pointers on how to
install Python3 and on how to use scons in a Python3 environment. The
solution works in a system where both Python2 and Python3 are
installed.
JIRA: https://gem5.atlassian.net/browse/GEM5-797
Change-Id: I98d4a39f586f39d9253ab2517b77e86c5ed19466
Signed-off-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/36157
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
We saw some strange behavior when building scons without an interactive
TTY. This seems be caused by the control signal set from
curses.initscr() and endwin(). To avoid issues, we should avoid those
operation when running in non interactive situation.
Change-Id: I9cf8e48a786d47d567ba193f0b069f638e8db647
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/35595
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
It appears that scons bugs are not on tigris.org any more and are now
on github, although fortunately old bugs seem to have been ported over
and have the same numbering.
This CL updates URLs which were in comments in the gem5 source,
specifically in scons scripts, to point to the corresponding github
version.
I also checked to see if these bugs were still open, or if we could
remove our workarounds for them.
1. 2356 is still open, and has been fairly recently assigned.
2. 2611 is marked as fixed. We might be able to implement the
workaround in its last comment from August of 2019.
3. 2811 has been marked fixed, and as best I can tell the fix first
appeared in around version 3.0 of scons. If/when that is our
minimum version, we can remove the workaround in
site_scons/site_tools/default.py. That is mostly fixing an annoying
spurious rebuild by scons which does not affect correctness, so even
if we remove that workaround we shouldn't break earlier versions,
although it would be obnoxious for people that are affected by it
and best avoided.
Change-Id: I0d74820f399044c6f80148bf3022d07d7bf6f4e5
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/32114
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This will still be technically possible with the right converters, but
this removes the tags, ignore file, and style checking hooks related to
mercurial. We no longer maintain a mercurial mirror of the main git
repository, and this support adds clutter and could diverge from the git
style hooks, etc, over time.
Change-Id: Icf4833c4f0fda51ea98989d1d741432ae3ddc6dd
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/31174
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
When building gem5, it's possible for warnings printed early in the
build to be quickly wisked away in a see of compile lines, never to be
seen again (or driven off the end of the scrollback buffer).
To avoid those messages getting lost or ignored, this change adds a
mechanism to aggregate them into a list so that they can be summarized
at the end of the build, successful or not.
Change-Id: Ie13320717698fcbcd3a8f8d1c062467e8d6d2914
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27129
Tested-by: Gem5 Cloud Project GCB service account <345032938727@cloudbuild.gserviceaccount.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Gabe Black <gabeblack@google.com>
Otherwise the error and warning messages get chopped off and wrapped by
the terminal wherever they happened to end. That's ugly and hard to
read.
This mechanism attempts to wrap the text using the console width which
it attempts to determine in two ways, first with shutil which should
work in python 3.3 and above, and then with the curses python module. If
neither of those works, it just falls back to 80 columns which is not
ideal but is reasonable.
Change-Id: I961936295505f93f5f36eb6d9cebc5073b5f793b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27128
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
* dict methods dict.keys(), dict.items() and dict.values()
return "views" instead of lists
* The dict.iterkeys(), dict.iteritems() and dict.itervalues()
methods are no longer supported.
* map() and filter() return iterators.
* range() now behaves like xrange() used to behave, except it works with
values of arbitrary size. The latter no longer exists.
* zip() now returns an iterator.
Change-Id: Id480018239db88d7f5d60588c93719056de4a0c0
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26248
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
These methods will make reporting errors less verbose and more
consistent, since they'll handle some formating, setting colors,
prefixing with an appropriate "Warning:" or "Error:" tag, and exiting
in the case of an error.
Change-Id: Iddea5bf342a4fc4b26002d8e98292f9dc57fa8cc
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/22885
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Scons on Fedora 27 imports print_function from the future[1] as a
result of which a gem5 build errors out with a syntax error. Make all
the scons scripts that use the print statement import the print_function
from future and replace the statements with print function calls.
[1] 34cf3bdb17
Change-Id: I67b7ef978fd7567f94d3cd9a904f8a0c1af07ffb
Signed-off-by: Siddhesh Poyarekar <siddhesh.poyarekar@gmail.com>
Reviewed-on: https://gem5-review.googlesource.com/8321
Reviewed-by: Gabe Black <gabeblack@google.com>
Reviewed-by: Anthony Gutierrez <anthony.gutierrez@amd.com>
Maintainer: Gabe Black <gabeblack@google.com>
One of the latest commits affecting the build system introduced a change
in the environment setting: rather than using the ENV dictionary,
environment key-val pairs were saved directly into the class. This
patch reverts that behaviour.
Change-Id: If206ba0a34ac8c35695a493cb90654b7ec81971b
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/5941
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Make site_init.py manage sys.path, and the "default" tool set
PYTHONPATH on any environment that's created. The paths to add are
tracked in a common gem5_python_paths.py.
Change-Id: I3387d4394d47a2f9c83322644cfd05909c6890fa
Reviewed-on: https://gem5-review.googlesource.com/5564
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
This imports various environment variables into the scons environment,
and sets some general properties on it. These are basically just copied
directly from the SConstruct and have the same behavior here.
gem5_tool_list will be used later on to add scons "tools" which should
be automatically added to new Environment objects.
Change-Id: Ib255955090c7b1e1cb80c703c18a9c867fcf1c9e
Reviewed-on: https://gem5-review.googlesource.com/5562
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
In an effort to shrink and modularize the main scons files, this
change pulls the scons and python version checking code out of the
main file and into site_init.py which runs before the SConstruct
starts. This will be a place to put really generic code which has to
do with the very fundemental aspects of getting scons to work. Other
checks, like checks for particular tools or particular versions of
tools, will happen in other more specialized files.
Change-Id: Icd00ecadbe1141aef4dbadcf42d6ddef1f3a701f
Reviewed-on: https://gem5-review.googlesource.com/5561
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
