This path fix build failed when gem5's path contains Non-ASCII
characters.
build_tools/marshal.py embed "abspath" and "modpath" into the
generated c++ file. However, if either of them contains Non-ASCII
characters, marshal.py will throw an exception.
This can be fixed if we set a suitable $LANG for gem5py. For
example, we can simply set $LANG=en_US.UTF-8 in src/SConscript
to fix this.
This patch fixes this bug by passing the host's $LANG environment
variable to gem5py. Because for users who may use Non-ASCII
characters, their $LANG is generally set to their native language
(e.g. zh_CN.UTF-8, en_US.UTF-8, etc.), which contains Non-ASCII
characters. Also, it is more reasonable to use the host $LANG than
to directly hard-code the $LANG used by gem5py into a language
that supports non-ASCII characters.
Change-Id: Ia0343cbd6995af3a9e43bf217ad3f186d5633fed
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/58369
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This makes what are configuration and what are internal SCons variables
explicit and separate, and makes it unnecessary to call out what
variables to export to C++.
These variables will also be plumbed into and out of kconfiglib in later
changes.
Change-Id: Iaf5e098d7404af06285c421dbdf8ef4171b3f001
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/56892
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Sometimes a library is needed to support particular functionality in
gem5, and that functionality is only used (or even desirable) in
certain binaries SCons can build. We can currently filter sources to
include in a particular executable using tags, but libraries have been
added to the environment globally using the LIBS variable which applies
to all Executables.
This change adds a SourceLib() mechanism which is a new category of
source which represents libraries. This is independent from classes
which inherit from SourceFile which represent actual files, as opposed
to more abstract libraries.
When gem5 builds an executable, the filters it provides are used to
select both Source()-es, aka c/c++ files, and libraries. If something
like a unit test does not need all the libraries gem5 proper does,
then those won't be picked up by its filter, and it won't include them.
Change-Id: I003e029eb82f7800a7ecff698c260e2d18ea2900
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/58069
Reviewed-by: Yu-hsin Wang <yuhsingw@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
These variables will always be imported from the environment, and never
from previous builds. For SCons to actually use these values, they need
to not only be in the environment variables external commands SCons runs
sees, they also have to be promoted to actual SCons construction
variables.
Change-Id: I54035442d70972396f1788bd0773877222d7a7c5
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/56386
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
When checking whether a given option is supported, we try to compile or
link a small test program and see if that succeeded or not. There are
some options which are not really supported, but if you use them the
compiler might produce an error instead of failing to build. An example
of that is clang (version 9 at least) which will warn if you try to
disabled a warning it doesn't know about.
On the develop branch, this is fine since we have werror enabled, and
even these warnings will prevent the options from being enabled. On the
release branches where we disable werror for better future
compatibility, this causes a problem because these options still pass
the test and end up being enabled. This doesn't break the build, but
can cause a bunch of annoying and unnecessary warnings which clutter
up the build output.
To remove this inconsistency, we can enable werror just for the test
compiles or links which decide whether or not an option is supported.
That won't necessarily differentiate *why* a warning was generated, so
if through some strange combination of circumstances something else
causes a warning pervasively through the build, then this may think
an option is to blame and not enable it even though it's fine. This
seems unlikely though, and not worth worrying about without a specific
example showing that it can/will actually happen.
Change-Id: I6a1e3273c0b646a5636dc9986c70dcd5332f6f64
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/54624
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
The tags for source files need to be resolved as well as the tags for
the filters themselves.
For example, the 'x86 isa' tag could imply 'gem5 lib'. If we filter for
all files which include the tag 'gem5 lib', we would also want to get
all the files that used the tag 'x86 isa' by itself, since that would
imply 'gem5 lib' as well.
Change-Id: Ie513778cc5b8ac1f1893728b14e84cabdaabf0e6
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/51828
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
On a clean build, the git tool will wait for input() before installing
git hooks. Allow bypassing this via a command-line flag, making it
possible to perform a clean build non-interactively.
Bug: 199780674
Test: build_gem5 --install-hooks
Change-Id: I48be2c1a7c2335a2f4f6359adf582ca8b0ae5939
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/50410
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
These nodes are really for working with text, and they do strange and
broken things otherwise. This change adds a pair of convenience
functions which uses pickle to serialize an object for the Value to
hold, and to unpack the Value later.
Change-Id: Id48822ce3de283b003d4cc7ef6b563a70e321ca6
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/48371
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Instead, create a new tool called EnvDefaults. This tool now needs
to be explicitly listed in the set of tools available through an
Environment, but that's easy to do in on place (all environments should
stem from main). Hijacking default like we were doing was (as far as I
can tell) not how that tool was intended to be used, and doing things
this way is a bit less hacky.
Also, we can split more Builders, etc, out of SConstruct, and these will
need to attach to main in the same way. We're going to need a list of
tools when main is constructed one way or the other, so we might as well
follow the rules.
Change-Id: I392e1639cb69d373c64970dccf45258000498cc3
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40965
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
This has two purposes. First, SCons assumes that once you call
Configure, you won't set up the environment the Configure is based on
until after you get the environment back from it again with
conf.Finish(). We get away with this when the cache mode for config
tests is not "force", since Configure just reuses the environment we
pass in, and any changes we make are immediately communicated between
the two.
If the cache mode *is* "force" though, SCons modifies the decider so
that everything the conf environment goes to build looks like it's out
of date. It does that by cloning the original environment, and then
using that clone to do its tests. That causes a problem because we have
a long lived "conf" object and make further changes to main, and since
the two environments are now separate the one in conf doesn't see those
updates.
Second, and more subtly, we export our "main" and "env" environments so
that other SConsopts and SConscript files can use them and define things
in them. The way Configure is designed, if the config caching mode is
"force", then it will create a new environment, and then that
environment will replace what the, for instance, "main" variable points
to when "main = conf.Finish()" is executed.
Unfortunately, if we've already Export()-ed main, we've exported what
the "main" variable pointed to at that time. Our view of "main" will
track with the value that conf.Finish() returned, but since that
construction environment is mearly derived from the main we Exported and
not actually the same thing, they have diverged at that point and will
behave independently.
To solve both of these problems, this change modifies the
gem5_scons.Configure() method so that it's a context manager instead of
a regular function. As before, it will call Configure for us and create
a configuration context, which it will yield as the "with" value. When
the context exits, all the variables in the context Finish() returns
will be shoved back into the original context with Replace(). This isn't
perfect since variables which were deleted in the environment (probably
very rare in practice) will not exist and so will not overwrite the
still existent variable in the original dict.
This has several advantages. The environment never splits into two
copies which continue on independently. It makes the lifetime of a
configuration context short, which is good because behavior during that
time is tricky and unintuitive. It also makes the scope of the context
very clear, so that you won't miss the fact that you're in a special
setting and need to pay attention to what environment you're modifying.
Also, this keeps the conceptual overhead of configuration localized to
where the configuration is happening. In parts of the SConscripts which
are not doing anything with conf, etc, they don't have to modify their
behavior since no configuration context is active.
This change is based on this change from Hanhwi Jang who identified this
problem and proposed an initial solution:
https://gem5-review.googlesource.com/c/public/gem5/+/44265
Change-Id: Iae0a292d6b375c5da98619f31392ca1de6216fcd
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/44389
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Hanhwi Jang <jang.hanhwi@gmail.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
In the background, scons invokes commands using what is essentially
'sh -c ${CMD}'. It's a little more sophisticated than that, but the net
effect is that all of the arguments of a given command are collected
into a single argument for sh. While Linux has a very generous limit
on the number of command line arguments you can have, and a relatively
generous limit on the total size of the arguments and environment
variables which can be read with "getconf ARG_MAX" (about 2MB only my
system), the limit on an individual argument, defined in the little
documented constant MAX_ARG_STRLEN, is much smaller, 32 pages or
131072 bytes from what I've read.
Unfortunately, when building gem5, especially without partial linking,
the command line for the final linking step can be very long, and with
some extra files brought in with EXTRAS can relatively easily exceed
this limit.
To work around this problem, this change adds a new scons "tool" which
replaces the SPAWN and PSPAWN construction environment variables (not
the shell environment, scon's env, aka main) with wrappes. These
wrappers create a temporary file, write the command into it, and then
tell the original SPAWN/PSPAWN implementation to run that file as the
actual command.
This works, but it has two potential drawbacks. First, it creates
extra temporary files which may add some overhead to the build. The
overhead is likely small, but is non-zero.
Second, while this restores the original breaks between command line
arguments, it doesn't completely avoid the total argument/environment
size limit which would be harder to hit, but does still exist.
Another option would be to use the same sort of technique, but to use
gcc's (and I assume clang's) ability to read options from a file using
an @file argument, where file is replaced with the name of the
arguments.
The upside of this approach is that it avoids all of these limits
since gcc is just reading a file, and the only limits are ones it self
imposes. Also, this would only apply to invocations of gcc, and so
would be worth plumbing in to only affect, say, the linking step.
There would be no hypothetical overhead from creating temp files.
One downside though, is that this is not a universal approach and
would break anything which doesn't understand @file style arguments.
Also, when intercepting calls to the shell, the arguments we're
getting have been made appropriate for that environment (escaping,
quoting, etc). The rules for escaping arguments in an @file may not be
quite the same as for the shell, and so there may be weird cases where
a command line gets garbled this way.
Given the tradeoffs, I think always putting the commands into temp
files and running them as scripts is the way to go.
Change-Id: I0a5288aed745a432ed72ffd990ceded2b9422585
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41273
Reviewed-by: Earl Ou <shunhsingou@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
When calling TryCompile with an empty string for the source, useful
when just testing a build flag, some versions of SCons will apparently
create some sort of equivalence between Value('') and Value(0). That
shows itself when creating config file headers, where Value(0) is
switched with Value(''), and the header defines a macro with which
expands to nothing rather than a macro which expands to 0. Later uses
of the macro of the form:
#if CONFIG_VARIABLE
fail because CONFIG_VARIABLE expands to nothing. If it expanded to 0
like it's supposed to, then the guarded block of code would be excluded
correctly.
Change-Id: Ie324ec5f8dd307c65745b9326a11230e10caa0bd
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41213
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Gabe Black <gabe.black@gmail.com>
These functions where correctly returning whether a flag had existed,
and also correctly not installing it if asked not to. Unfortunately if
they *were* asked to install the flag, they ignored whether or not it
had actually existed to begin with.
Change-Id: I2dca0e1a0ddbc182576d48237aeea5452a02c51b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41159
Maintainer: Gabe Black <gabe.black@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
SCons has a system of "tools", which basically detect versions of build
tools (compilers, linkers, etc) and set up an environment with the
appropriate build variable substitutions for that tool to be used.
For instance, there would be a "tool" for gcc, and it would detect if
gcc is present on the system, and if so would set the "CC" variable to
"gcc". An actually tool as defined by SCons would be a lot more
sophisticated than that and set more variables, but that's the basic
idea.
To help modularize the gem5 SConstruct file, I moved code which would
set up git commit hooks into a "tool" which helped modularize it and
reduce the size of SConstruct.
This isn't quite right since, while the code does detect if git was
used to check out the source (if there is a .git file at the root), it
doesn't really modify the environment at all. It will also be invoked
every time any environment is set up, although right now that will only
be the DefaultEnvironment, what's used when loose functions like
Builder or Command are called with, and the "main" environment which
all the others are Clone-d from.
Normally, when SCons sets up a new environment, either implicitly or
when Environment() is called, it sets up a bunch of built in tools
which are fixed within SCons itself. If you want, you can add a "tools"
argument to Environment (or to the DefaultEnvironment() function) which
will replace that list of tools. That can be used to make an
environment use the new "git" tool, but it isn't automatic.
SCons also lets you override default tools by creating your own with
the same name as the default. To make loading the git tool automatic,
I added an override "default" tool which, in addition to setting some
defaults which should apply to all environments, also pulled in other
tools, at that time "git" and "mercurial" (RIP).
Unfortunately, that meant that today, apparently particularly with
SCons version 4, *any* Environment would pull in "git", and all of
"git"'s dependencies, even if SCons wasn't set up enough for those to
work.
To break that dependency, this change stops the default tool from
automatically loading the git tool, although it does continue to set
other defaults which have very minimal external dependencies. When
creating the "main" Environment in the SConstruct, the "git" tool is
now added in explicitly. Since the list of tools replaces the original
and doesn't extend it, we have to add in "default" explicitly as well.
Really, the "git" tool should be converted from the tool interface into
something more appropriate, like perhaps a small module under
site_scons which site_init.py can import and call. When that happens,
main can be declared like normal again.
While making this change, I also got rid of a few nonstandard additions
to the main environment that were little used and not really necessary.
When reading the SConstruct, it wasn't very obvious where those extra
values were coming from, and they didn't really add any value.
Change-Id: I574db42fc2196bf62fc13d6754357c753ceb9117
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/38616
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Yu-hsin Wang <yuhsingw@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
As gem5 has started to use Python2 incompatible features, compiling
gem5 in a Python2 environment results in an error.
This commit addresses this issue by raising an Exception when scons
is run in a Python2 environment, and adding a few pointers on how to
install Python3 and on how to use scons in a Python3 environment. The
solution works in a system where both Python2 and Python3 are
installed.
JIRA: https://gem5.atlassian.net/browse/GEM5-797
Change-Id: I98d4a39f586f39d9253ab2517b77e86c5ed19466
Signed-off-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/36157
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
We saw some strange behavior when building scons without an interactive
TTY. This seems be caused by the control signal set from
curses.initscr() and endwin(). To avoid issues, we should avoid those
operation when running in non interactive situation.
Change-Id: I9cf8e48a786d47d567ba193f0b069f638e8db647
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/35595
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
It appears that scons bugs are not on tigris.org any more and are now
on github, although fortunately old bugs seem to have been ported over
and have the same numbering.
This CL updates URLs which were in comments in the gem5 source,
specifically in scons scripts, to point to the corresponding github
version.
I also checked to see if these bugs were still open, or if we could
remove our workarounds for them.
1. 2356 is still open, and has been fairly recently assigned.
2. 2611 is marked as fixed. We might be able to implement the
workaround in its last comment from August of 2019.
3. 2811 has been marked fixed, and as best I can tell the fix first
appeared in around version 3.0 of scons. If/when that is our
minimum version, we can remove the workaround in
site_scons/site_tools/default.py. That is mostly fixing an annoying
spurious rebuild by scons which does not affect correctness, so even
if we remove that workaround we shouldn't break earlier versions,
although it would be obnoxious for people that are affected by it
and best avoided.
Change-Id: I0d74820f399044c6f80148bf3022d07d7bf6f4e5
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/32114
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This will still be technically possible with the right converters, but
this removes the tags, ignore file, and style checking hooks related to
mercurial. We no longer maintain a mercurial mirror of the main git
repository, and this support adds clutter and could diverge from the git
style hooks, etc, over time.
Change-Id: Icf4833c4f0fda51ea98989d1d741432ae3ddc6dd
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/31174
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabeblack@google.com>
Maintainer: Andreas Sandberg <andreas.sandberg@arm.com>
Tested-by: kokoro <noreply+kokoro@google.com>
When building gem5, it's possible for warnings printed early in the
build to be quickly wisked away in a see of compile lines, never to be
seen again (or driven off the end of the scrollback buffer).
To avoid those messages getting lost or ignored, this change adds a
mechanism to aggregate them into a list so that they can be summarized
at the end of the build, successful or not.
Change-Id: Ie13320717698fcbcd3a8f8d1c062467e8d6d2914
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27129
Tested-by: Gem5 Cloud Project GCB service account <345032938727@cloudbuild.gserviceaccount.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Gabe Black <gabeblack@google.com>
Otherwise the error and warning messages get chopped off and wrapped by
the terminal wherever they happened to end. That's ugly and hard to
read.
This mechanism attempts to wrap the text using the console width which
it attempts to determine in two ways, first with shutil which should
work in python 3.3 and above, and then with the curses python module. If
neither of those works, it just falls back to 80 columns which is not
ideal but is reasonable.
Change-Id: I961936295505f93f5f36eb6d9cebc5073b5f793b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/27128
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
