This patch addresses the issue of gem5 failing to build when
the build directory path contains non-ASCII characters.
The previous patches[1] that attempted to fix this issue
became ineffective after the upgrade of Python and pybind11
to new versions. This new patch manually sets the locale in
marshal.py based on the `LC_CTYPE` environment variable,
providing a comprehensive solution that works with Non-ASCII
build directory paths.
[1] https://gem5-review.googlesource.com/c/public/gem5/+/58369
Change-Id: I3ad28b6ee52fd347d2fe71f279baab629e88d12c
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/70818
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
The code generation in gem5's build system requires the use of Regular
Expression flags when defining the regular expressions used for
tokenization. However, the Python Lex-Yacc (PLY) [1] library used by
gem5 does not allow the user sufficient control of the flags for RE
compilation.
Previously, gem5 used inline RE flags to control RE compilation.
However, from Python 3.11, inline RE flags must be at the start of the
RE string. Because PLY wraps the user supplied RE strings before
compilation, there is no way for the user to supply a RE string with
the inline flag at the start. This makes gem5 incompatible with Python
3.11 when using PLY.
This change modifies gem5's build files to patch `re.compile` with a
wrapped version that can handle embedded flags anywhere in the RE
string, for all current versions of Python. The patched version
re-formats the user supplied RE string to convert inline RE flags to
explicit RE flags.
This patch is intended as a temporary stop-gap until PLY can be fixed
upstream.
See the gem5 Issue Tracker [2] for more details.
[1] https://github.com/dabeaz/ply
[2] https://gem5.atlassian.net/browse/GEM5-1321
Change-Id: I3ab371f2e5cf267c0a89caaf8a2bacfed78545ef
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/70237
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Reviewed-by: Boris Shingarov <shingarov@labware.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This adds a new scons flag --no-duplicate-sources to build without
linking source files to the build directory.
I find this very helpful when using CLion, since I can now generate a
compilation database using
`bear scons build/ALL/gem5.debug --no-duplicate-sources` and CLion will
now correctly semantically analyze all the files inside src/.
It also ensures that clicking on a build warning/error now opens the
real source file rather than a symlink.
This is not enabled by default since it's possible that certain use
cases are not working correctly, but the basic testing I've done so
far appears to work just fine.
It appears that with this change the `<root>/src` directory is no longer
added to `PYTHONPATH` when running `tests/main.py`, so this change
depends on https://gem5-review.git.corp.google.com/c/public/gem5/+/68757
Change-Id: Iddc9bf9c8211e68e5432c0a07f5c95f427c1ca16
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/68518
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
During gem5 build, the compiler may produce some large intermediate
files. The default path is /tmp, but in some usecase, it's under a small
file system, and we may want to change the storage path to a sufficient
large file system. This CL captures TMPDIR environment variables, to
allow users change the default temporary directory.
Change-Id: Ib3fad301f36df9f3f08eb9b6cfeb4df1b7df5d1a
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/64873
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
If the pre-commit could not be installed the compilation would continue
as the exit code from running the pre-commit install script was not
read or processed. This commit adds a check. If the install is
unsuccessful the users is asked whether they want to continue the
compilation or not.
This check can be ignored with the '--ignore-style'. The tests have been
updated to include this flag in all cases we compile gem5 to ensure
tests remain automated and uninterrupted on Kokoro/Jenkins.
Change-Id: Iaf4db71300883b828b00d77784c9bb46b2698f89
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/63012
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Apparently, sometimes a SCons Value node created with an integer value
0 will have a string representation which is the empty string, which
is obviously problematic. To work around this bug, convert the value
to a string explicitly before creating the Value node, so that we get
a sensible translation instead.
Change-Id: I4a09b2fb366aaf2d4d1c7e6f8cc84224b2d010c4
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/59472
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Yu-hsin Wang <yuhsingw@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
This path fix build failed when gem5's path contains Non-ASCII
characters.
build_tools/marshal.py embed "abspath" and "modpath" into the
generated c++ file. However, if either of them contains Non-ASCII
characters, marshal.py will throw an exception.
This can be fixed if we set a suitable $LANG for gem5py. For
example, we can simply set $LANG=en_US.UTF-8 in src/SConscript
to fix this.
This patch fixes this bug by passing the host's $LANG environment
variable to gem5py. Because for users who may use Non-ASCII
characters, their $LANG is generally set to their native language
(e.g. zh_CN.UTF-8, en_US.UTF-8, etc.), which contains Non-ASCII
characters. Also, it is more reasonable to use the host $LANG than
to directly hard-code the $LANG used by gem5py into a language
that supports non-ASCII characters.
Change-Id: Ia0343cbd6995af3a9e43bf217ad3f186d5633fed
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/58369
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This makes what are configuration and what are internal SCons variables
explicit and separate, and makes it unnecessary to call out what
variables to export to C++.
These variables will also be plumbed into and out of kconfiglib in later
changes.
Change-Id: Iaf5e098d7404af06285c421dbdf8ef4171b3f001
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/56892
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Sometimes a library is needed to support particular functionality in
gem5, and that functionality is only used (or even desirable) in
certain binaries SCons can build. We can currently filter sources to
include in a particular executable using tags, but libraries have been
added to the environment globally using the LIBS variable which applies
to all Executables.
This change adds a SourceLib() mechanism which is a new category of
source which represents libraries. This is independent from classes
which inherit from SourceFile which represent actual files, as opposed
to more abstract libraries.
When gem5 builds an executable, the filters it provides are used to
select both Source()-es, aka c/c++ files, and libraries. If something
like a unit test does not need all the libraries gem5 proper does,
then those won't be picked up by its filter, and it won't include them.
Change-Id: I003e029eb82f7800a7ecff698c260e2d18ea2900
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/58069
Reviewed-by: Yu-hsin Wang <yuhsingw@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
These variables will always be imported from the environment, and never
from previous builds. For SCons to actually use these values, they need
to not only be in the environment variables external commands SCons runs
sees, they also have to be promoted to actual SCons construction
variables.
Change-Id: I54035442d70972396f1788bd0773877222d7a7c5
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/56386
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
When checking whether a given option is supported, we try to compile or
link a small test program and see if that succeeded or not. There are
some options which are not really supported, but if you use them the
compiler might produce an error instead of failing to build. An example
of that is clang (version 9 at least) which will warn if you try to
disabled a warning it doesn't know about.
On the develop branch, this is fine since we have werror enabled, and
even these warnings will prevent the options from being enabled. On the
release branches where we disable werror for better future
compatibility, this causes a problem because these options still pass
the test and end up being enabled. This doesn't break the build, but
can cause a bunch of annoying and unnecessary warnings which clutter
up the build output.
To remove this inconsistency, we can enable werror just for the test
compiles or links which decide whether or not an option is supported.
That won't necessarily differentiate *why* a warning was generated, so
if through some strange combination of circumstances something else
causes a warning pervasively through the build, then this may think
an option is to blame and not enable it even though it's fine. This
seems unlikely though, and not worth worrying about without a specific
example showing that it can/will actually happen.
Change-Id: I6a1e3273c0b646a5636dc9986c70dcd5332f6f64
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/54624
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
The tags for source files need to be resolved as well as the tags for
the filters themselves.
For example, the 'x86 isa' tag could imply 'gem5 lib'. If we filter for
all files which include the tag 'gem5 lib', we would also want to get
all the files that used the tag 'x86 isa' by itself, since that would
imply 'gem5 lib' as well.
Change-Id: Ie513778cc5b8ac1f1893728b14e84cabdaabf0e6
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/51828
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
On a clean build, the git tool will wait for input() before installing
git hooks. Allow bypassing this via a command-line flag, making it
possible to perform a clean build non-interactively.
Bug: 199780674
Test: build_gem5 --install-hooks
Change-Id: I48be2c1a7c2335a2f4f6359adf582ca8b0ae5939
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/50410
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
These nodes are really for working with text, and they do strange and
broken things otherwise. This change adds a pair of convenience
functions which uses pickle to serialize an object for the Value to
hold, and to unpack the Value later.
Change-Id: Id48822ce3de283b003d4cc7ef6b563a70e321ca6
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/48371
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Instead, create a new tool called EnvDefaults. This tool now needs
to be explicitly listed in the set of tools available through an
Environment, but that's easy to do in on place (all environments should
stem from main). Hijacking default like we were doing was (as far as I
can tell) not how that tool was intended to be used, and doing things
this way is a bit less hacky.
Also, we can split more Builders, etc, out of SConstruct, and these will
need to attach to main in the same way. We're going to need a list of
tools when main is constructed one way or the other, so we might as well
follow the rules.
Change-Id: I392e1639cb69d373c64970dccf45258000498cc3
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40965
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
This has two purposes. First, SCons assumes that once you call
Configure, you won't set up the environment the Configure is based on
until after you get the environment back from it again with
conf.Finish(). We get away with this when the cache mode for config
tests is not "force", since Configure just reuses the environment we
pass in, and any changes we make are immediately communicated between
the two.
If the cache mode *is* "force" though, SCons modifies the decider so
that everything the conf environment goes to build looks like it's out
of date. It does that by cloning the original environment, and then
using that clone to do its tests. That causes a problem because we have
a long lived "conf" object and make further changes to main, and since
the two environments are now separate the one in conf doesn't see those
updates.
Second, and more subtly, we export our "main" and "env" environments so
that other SConsopts and SConscript files can use them and define things
in them. The way Configure is designed, if the config caching mode is
"force", then it will create a new environment, and then that
environment will replace what the, for instance, "main" variable points
to when "main = conf.Finish()" is executed.
Unfortunately, if we've already Export()-ed main, we've exported what
the "main" variable pointed to at that time. Our view of "main" will
track with the value that conf.Finish() returned, but since that
construction environment is mearly derived from the main we Exported and
not actually the same thing, they have diverged at that point and will
behave independently.
To solve both of these problems, this change modifies the
gem5_scons.Configure() method so that it's a context manager instead of
a regular function. As before, it will call Configure for us and create
a configuration context, which it will yield as the "with" value. When
the context exits, all the variables in the context Finish() returns
will be shoved back into the original context with Replace(). This isn't
perfect since variables which were deleted in the environment (probably
very rare in practice) will not exist and so will not overwrite the
still existent variable in the original dict.
This has several advantages. The environment never splits into two
copies which continue on independently. It makes the lifetime of a
configuration context short, which is good because behavior during that
time is tricky and unintuitive. It also makes the scope of the context
very clear, so that you won't miss the fact that you're in a special
setting and need to pay attention to what environment you're modifying.
Also, this keeps the conceptual overhead of configuration localized to
where the configuration is happening. In parts of the SConscripts which
are not doing anything with conf, etc, they don't have to modify their
behavior since no configuration context is active.
This change is based on this change from Hanhwi Jang who identified this
problem and proposed an initial solution:
https://gem5-review.googlesource.com/c/public/gem5/+/44265
Change-Id: Iae0a292d6b375c5da98619f31392ca1de6216fcd
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/44389
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Hanhwi Jang <jang.hanhwi@gmail.com>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
In the background, scons invokes commands using what is essentially
'sh -c ${CMD}'. It's a little more sophisticated than that, but the net
effect is that all of the arguments of a given command are collected
into a single argument for sh. While Linux has a very generous limit
on the number of command line arguments you can have, and a relatively
generous limit on the total size of the arguments and environment
variables which can be read with "getconf ARG_MAX" (about 2MB only my
system), the limit on an individual argument, defined in the little
documented constant MAX_ARG_STRLEN, is much smaller, 32 pages or
131072 bytes from what I've read.
Unfortunately, when building gem5, especially without partial linking,
the command line for the final linking step can be very long, and with
some extra files brought in with EXTRAS can relatively easily exceed
this limit.
To work around this problem, this change adds a new scons "tool" which
replaces the SPAWN and PSPAWN construction environment variables (not
the shell environment, scon's env, aka main) with wrappes. These
wrappers create a temporary file, write the command into it, and then
tell the original SPAWN/PSPAWN implementation to run that file as the
actual command.
This works, but it has two potential drawbacks. First, it creates
extra temporary files which may add some overhead to the build. The
overhead is likely small, but is non-zero.
Second, while this restores the original breaks between command line
arguments, it doesn't completely avoid the total argument/environment
size limit which would be harder to hit, but does still exist.
Another option would be to use the same sort of technique, but to use
gcc's (and I assume clang's) ability to read options from a file using
an @file argument, where file is replaced with the name of the
arguments.
The upside of this approach is that it avoids all of these limits
since gcc is just reading a file, and the only limits are ones it self
imposes. Also, this would only apply to invocations of gcc, and so
would be worth plumbing in to only affect, say, the linking step.
There would be no hypothetical overhead from creating temp files.
One downside though, is that this is not a universal approach and
would break anything which doesn't understand @file style arguments.
Also, when intercepting calls to the shell, the arguments we're
getting have been made appropriate for that environment (escaping,
quoting, etc). The rules for escaping arguments in an @file may not be
quite the same as for the shell, and so there may be weird cases where
a command line gets garbled this way.
Given the tradeoffs, I think always putting the commands into temp
files and running them as scripts is the way to go.
Change-Id: I0a5288aed745a432ed72ffd990ceded2b9422585
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41273
Reviewed-by: Earl Ou <shunhsingou@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
When calling TryCompile with an empty string for the source, useful
when just testing a build flag, some versions of SCons will apparently
create some sort of equivalence between Value('') and Value(0). That
shows itself when creating config file headers, where Value(0) is
switched with Value(''), and the header defines a macro with which
expands to nothing rather than a macro which expands to 0. Later uses
of the macro of the form:
#if CONFIG_VARIABLE
fail because CONFIG_VARIABLE expands to nothing. If it expanded to 0
like it's supposed to, then the guarded block of code would be excluded
correctly.
Change-Id: Ie324ec5f8dd307c65745b9326a11230e10caa0bd
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41213
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Gabe Black <gabe.black@gmail.com>
These functions where correctly returning whether a flag had existed,
and also correctly not installing it if asked not to. Unfortunately if
they *were* asked to install the flag, they ignored whether or not it
had actually existed to begin with.
Change-Id: I2dca0e1a0ddbc182576d48237aeea5452a02c51b
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/41159
Maintainer: Gabe Black <gabe.black@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
