If gem5.build already exists within a directory, then that build
directory can be used without having to worry about variants.
If it doesn't exist and we find a build/${VARIANT} style path, then we
use that as the anchor.
In either case, the variant name is the final component of the build
path. The parse_build_path function had been separating that out, but it
was just put back onto the path again anyway by the only caller, and
then split out again when that path was consumed. We save a step by not
splitting it out in parse_build_path.
Change-Id: I8705b3dbb7664748f5525869cb188df70319d403
The code generation in gem5's build system requires the use of Regular
Expression flags when defining the regular expressions used for
tokenization. However, the Python Lex-Yacc (PLY) [1] library used by
gem5 does not allow the user sufficient control of the flags for RE
compilation.
Previously, gem5 used inline RE flags to control RE compilation.
However, from Python 3.11, inline RE flags must be at the start of the
RE string. Because PLY wraps the user supplied RE strings before
compilation, there is no way for the user to supply a RE string with
the inline flag at the start. This makes gem5 incompatible with Python
3.11 when using PLY.
This change modifies gem5's build files to patch `re.compile` with a
wrapped version that can handle embedded flags anywhere in the RE
string, for all current versions of Python. The patched version
re-formats the user supplied RE string to convert inline RE flags to
explicit RE flags.
This patch is intended as a temporary stop-gap until PLY can be fixed
upstream.
See the gem5 Issue Tracker [2] for more details.
[1] https://github.com/dabeaz/ply
[2] https://gem5.atlassian.net/browse/GEM5-1321
Change-Id: I3ab371f2e5cf267c0a89caaf8a2bacfed78545ef
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/70237
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Reviewed-by: Boris Shingarov <shingarov@labware.com>
Tested-by: kokoro <noreply+kokoro@google.com>
These nodes are really for working with text, and they do strange and
broken things otherwise. This change adds a pair of convenience
functions which uses pickle to serialize an object for the Value to
hold, and to unpack the Value later.
Change-Id: Id48822ce3de283b003d4cc7ef6b563a70e321ca6
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/48371
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Instead, create a new tool called EnvDefaults. This tool now needs
to be explicitly listed in the set of tools available through an
Environment, but that's easy to do in on place (all environments should
stem from main). Hijacking default like we were doing was (as far as I
can tell) not how that tool was intended to be used, and doing things
this way is a bit less hacky.
Also, we can split more Builders, etc, out of SConstruct, and these will
need to attach to main in the same way. We're going to need a list of
tools when main is constructed one way or the other, so we might as well
follow the rules.
Change-Id: I392e1639cb69d373c64970dccf45258000498cc3
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40965
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
In the background, scons invokes commands using what is essentially
'sh -c ${CMD}'. It's a little more sophisticated than that, but the net
effect is that all of the arguments of a given command are collected
into a single argument for sh. While Linux has a very generous limit
on the number of command line arguments you can have, and a relatively
generous limit on the total size of the arguments and environment
variables which can be read with "getconf ARG_MAX" (about 2MB only my
system), the limit on an individual argument, defined in the little
documented constant MAX_ARG_STRLEN, is much smaller, 32 pages or
131072 bytes from what I've read.
Unfortunately, when building gem5, especially without partial linking,
the command line for the final linking step can be very long, and with
some extra files brought in with EXTRAS can relatively easily exceed
this limit.
To work around this problem, this change adds a new scons "tool" which
replaces the SPAWN and PSPAWN construction environment variables (not
the shell environment, scon's env, aka main) with wrappes. These
wrappers create a temporary file, write the command into it, and then
tell the original SPAWN/PSPAWN implementation to run that file as the
actual command.
This works, but it has two potential drawbacks. First, it creates
extra temporary files which may add some overhead to the build. The
overhead is likely small, but is non-zero.
Second, while this restores the original breaks between command line
arguments, it doesn't completely avoid the total argument/environment
size limit which would be harder to hit, but does still exist.
Another option would be to use the same sort of technique, but to use
gcc's (and I assume clang's) ability to read options from a file using
an @file argument, where file is replaced with the name of the
arguments.
The upside of this approach is that it avoids all of these limits
since gcc is just reading a file, and the only limits are ones it self
imposes. Also, this would only apply to invocations of gcc, and so
would be worth plumbing in to only affect, say, the linking step.
There would be no hypothetical overhead from creating temp files.
One downside though, is that this is not a universal approach and
would break anything which doesn't understand @file style arguments.
Also, when intercepting calls to the shell, the arguments we're
getting have been made appropriate for that environment (escaping,
quoting, etc). The rules for escaping arguments in an @file may not be
quite the same as for the shell, and so there may be weird cases where
a command line gets garbled this way.
Given the tradeoffs, I think always putting the commands into temp
files and running them as scripts is the way to go.
Change-Id: I0a5288aed745a432ed72ffd990ceded2b9422585
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41273
Reviewed-by: Earl Ou <shunhsingou@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
We saw some strange behavior when building scons without an interactive
TTY. This seems be caused by the control signal set from
curses.initscr() and endwin(). To avoid issues, we should avoid those
operation when running in non interactive situation.
Change-Id: I9cf8e48a786d47d567ba193f0b069f638e8db647
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/35595
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
When building gem5, it's possible for warnings printed early in the
build to be quickly wisked away in a see of compile lines, never to be
seen again (or driven off the end of the scrollback buffer).
To avoid those messages getting lost or ignored, this change adds a
mechanism to aggregate them into a list so that they can be summarized
at the end of the build, successful or not.
Change-Id: Ie13320717698fcbcd3a8f8d1c062467e8d6d2914
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27129
Tested-by: Gem5 Cloud Project GCB service account <345032938727@cloudbuild.gserviceaccount.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Gabe Black <gabeblack@google.com>
Otherwise the error and warning messages get chopped off and wrapped by
the terminal wherever they happened to end. That's ugly and hard to
read.
This mechanism attempts to wrap the text using the console width which
it attempts to determine in two ways, first with shutil which should
work in python 3.3 and above, and then with the curses python module. If
neither of those works, it just falls back to 80 columns which is not
ideal but is reasonable.
Change-Id: I961936295505f93f5f36eb6d9cebc5073b5f793b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27128
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
* dict methods dict.keys(), dict.items() and dict.values()
return "views" instead of lists
* The dict.iterkeys(), dict.iteritems() and dict.itervalues()
methods are no longer supported.
* map() and filter() return iterators.
* range() now behaves like xrange() used to behave, except it works with
values of arbitrary size. The latter no longer exists.
* zip() now returns an iterator.
Change-Id: Id480018239db88d7f5d60588c93719056de4a0c0
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/26248
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
These methods will make reporting errors less verbose and more
consistent, since they'll handle some formating, setting colors,
prefixing with an appropriate "Warning:" or "Error:" tag, and exiting
in the case of an error.
Change-Id: Iddea5bf342a4fc4b26002d8e98292f9dc57fa8cc
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/22885
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>
Tested-by: kokoro <noreply+kokoro@google.com>
