There appears to have been an assumption here that `Popen` would raise
an exception if the command run returned non-zero. This is not the case.
This commit fixes this by obtaining the return code and throwing an
exception if it is non-zero.
This bug caused some minor issues as Exception handling code to handle
the non-zero case elsewhere in Scons was never executed.
this adds an option --with-libcxx, that adds the -stdlib=libc++ flag to
link against libc++ instead of libstdc++ on Linux. Currently this is
only possible with clang and may not work with all build configurations
(e.g. protobuf linked against libstdc++), so this needs to be opt-in
rather than being on by default for clang whenever libc++ is detected.
Change-Id: Ib4022a58bb2dbd32417c58f01c7443a02ff710fe
Previously, the `subprocess` module was used to execute shell command
installing precommit hook. However, after #431 [1], the import of the
`subprocess` module was overriden by `asyncio.subprocess`, which has a
different API to execute the shell command. This change removes the
`asyncio.subprocess` import.
[1] https://github.com/gem5/gem5/pull/431
Change-Id: I9a7d51f85518089d258ab57c5d849a36dcf128e9
Signed-off-by: Hoa Nguyen <hn@hnpl.org>
These two utilities help update an old config to add settings for new
config options. The difference between them is that oldconfig asks what
new settings you want to use, while olddefconfig automatically picks the
defaults.
Change-Id: Icd3e57f834684e620705beb884faa5b6e2cc7baa
This helper utility lets you save the defconfig which would give rise to
a given config. For instance, you could use menuconfig to set up a
config how you want it with the options you cared about configured, and
then use savedefconfig to save a defconfig of that somewhere to the
side, in the gem5 defconfig directory, etc. Then later, you could use
that defconfig to set up a new build directory with that same config,
even if the kconfig options have changed a little bit since then.
A saved defconfig like that can also be a good way to visually see what
options have been set to something interesting, and an easier way to
pass a config to someone else to use, to put in bug reports, etc.
Change-Id: Ifd344278638c59b48c261b36058832034c009c78
This helper lists config options which are new in the Kconfig and which
are not currently set in the config file.
Change-Id: I0c426d85c0cf0d2bdbac599845669165285a82a0
This little utility lets you set particular values in an existing config
without having to open up the whole menuconfig interface.
Also reorganize things in kconfig.py a little to help share code between
wrappers.
Change-Id: I7cba0c0ef8d318d6c39e49c779ebb2bbdc3d94c8
If you call scons with the fist argument set to menuconfig, that means
to run menuconfig on the path following it. Or in other words, if
you ran this command:
scons menuconfig build/foo/bar
That would tell SCons to set up a build directory at the path
build/foo/bar, and then invoke menuconfig so you can set up its
configuration.
In addition to using this mechanism to set up a new build directory, you
can also use it to reconfigure an existing directory.
This supplements and does not replace the existing mechanism of using
"build/${VARIANT}" to select a config with defconfig.
Change-Id: Ief8e8c2ee6477799455c2004bef06c64be5cc1db
These are not yet consumed by anything, but convert all the settings
from SCons variables to Kconfig variables.
If you have existing SConsopts files which need to be converted, you
should take a look at KCONFIG.md to learn about how kconfig is used in
gem5. You should decide if any variables need to be available to C++ or
kconfig itself, and whether those are options which should be detected
automatically, or should be up to the user. Options which should be
measured automatically should still be in SConsopts files, while user
facing options should be added to new or existing Kconfig files.
Generally, make sure you're storing c++/kconfig visible options in
env['CONF'][...]. Also remove references to sticky_vars since persistent
options should now be handled with kconfig, and export_vars since
everything in env['CONF'] is now exported automatically.
Switch SCons/gem5 to use Kconfig for configuration, except EXTRAS which
is still a sticky SCons variable. This is necessary because EXTRAS also
controls what config options exist. If it came from Kconfig itself, then
there would be a circular dependency. This dependency could
theoretically be handled by reparsing the Kconfig when EXTRAS
directories were added or removed, but that would be complicated, and
isn't supported by kconfiglib. It wouldn't be worth the significant
effort it would take to add it, just to use Kconfig more purely.
Change-Id: I29ab1940b2d7b0e6635a490452d05befe5b4a2c9
If gem5.build already exists within a directory, then that build
directory can be used without having to worry about variants.
If it doesn't exist and we find a build/${VARIANT} style path, then we
use that as the anchor.
In either case, the variant name is the final component of the build
path. The parse_build_path function had been separating that out, but it
was just put back onto the path again anyway by the only caller, and
then split out again when that path was consumed. We save a step by not
splitting it out in parse_build_path.
Change-Id: I8705b3dbb7664748f5525869cb188df70319d403
This patch addresses the issue of gem5 failing to build when
the build directory path contains non-ASCII characters.
The previous patches[1] that attempted to fix this issue
became ineffective after the upgrade of Python and pybind11
to new versions. This new patch manually sets the locale in
marshal.py based on the `LC_CTYPE` environment variable,
providing a comprehensive solution that works with Non-ASCII
build directory paths.
[1] https://gem5-review.googlesource.com/c/public/gem5/+/58369
Change-Id: I3ad28b6ee52fd347d2fe71f279baab629e88d12c
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/70818
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
The code generation in gem5's build system requires the use of Regular
Expression flags when defining the regular expressions used for
tokenization. However, the Python Lex-Yacc (PLY) [1] library used by
gem5 does not allow the user sufficient control of the flags for RE
compilation.
Previously, gem5 used inline RE flags to control RE compilation.
However, from Python 3.11, inline RE flags must be at the start of the
RE string. Because PLY wraps the user supplied RE strings before
compilation, there is no way for the user to supply a RE string with
the inline flag at the start. This makes gem5 incompatible with Python
3.11 when using PLY.
This change modifies gem5's build files to patch `re.compile` with a
wrapped version that can handle embedded flags anywhere in the RE
string, for all current versions of Python. The patched version
re-formats the user supplied RE string to convert inline RE flags to
explicit RE flags.
This patch is intended as a temporary stop-gap until PLY can be fixed
upstream.
See the gem5 Issue Tracker [2] for more details.
[1] https://github.com/dabeaz/ply
[2] https://gem5.atlassian.net/browse/GEM5-1321
Change-Id: I3ab371f2e5cf267c0a89caaf8a2bacfed78545ef
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/70237
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Reviewed-by: Boris Shingarov <shingarov@labware.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This adds a new scons flag --no-duplicate-sources to build without
linking source files to the build directory.
I find this very helpful when using CLion, since I can now generate a
compilation database using
`bear scons build/ALL/gem5.debug --no-duplicate-sources` and CLion will
now correctly semantically analyze all the files inside src/.
It also ensures that clicking on a build warning/error now opens the
real source file rather than a symlink.
This is not enabled by default since it's possible that certain use
cases are not working correctly, but the basic testing I've done so
far appears to work just fine.
It appears that with this change the `<root>/src` directory is no longer
added to `PYTHONPATH` when running `tests/main.py`, so this change
depends on https://gem5-review.git.corp.google.com/c/public/gem5/+/68757
Change-Id: Iddc9bf9c8211e68e5432c0a07f5c95f427c1ca16
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/68518
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
During gem5 build, the compiler may produce some large intermediate
files. The default path is /tmp, but in some usecase, it's under a small
file system, and we may want to change the storage path to a sufficient
large file system. This CL captures TMPDIR environment variables, to
allow users change the default temporary directory.
Change-Id: Ib3fad301f36df9f3f08eb9b6cfeb4df1b7df5d1a
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/64873
Reviewed-by: Gabe Black <gabeblack@google.com>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
If the pre-commit could not be installed the compilation would continue
as the exit code from running the pre-commit install script was not
read or processed. This commit adds a check. If the install is
unsuccessful the users is asked whether they want to continue the
compilation or not.
This check can be ignored with the '--ignore-style'. The tests have been
updated to include this flag in all cases we compile gem5 to ensure
tests remain automated and uninterrupted on Kokoro/Jenkins.
Change-Id: Iaf4db71300883b828b00d77784c9bb46b2698f89
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/63012
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Apparently, sometimes a SCons Value node created with an integer value
0 will have a string representation which is the empty string, which
is obviously problematic. To work around this bug, convert the value
to a string explicitly before creating the Value node, so that we get
a sensible translation instead.
Change-Id: I4a09b2fb366aaf2d4d1c7e6f8cc84224b2d010c4
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/59472
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Yu-hsin Wang <yuhsingw@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
This path fix build failed when gem5's path contains Non-ASCII
characters.
build_tools/marshal.py embed "abspath" and "modpath" into the
generated c++ file. However, if either of them contains Non-ASCII
characters, marshal.py will throw an exception.
This can be fixed if we set a suitable $LANG for gem5py. For
example, we can simply set $LANG=en_US.UTF-8 in src/SConscript
to fix this.
This patch fixes this bug by passing the host's $LANG environment
variable to gem5py. Because for users who may use Non-ASCII
characters, their $LANG is generally set to their native language
(e.g. zh_CN.UTF-8, en_US.UTF-8, etc.), which contains Non-ASCII
characters. Also, it is more reasonable to use the host $LANG than
to directly hard-code the $LANG used by gem5py into a language
that supports non-ASCII characters.
Change-Id: Ia0343cbd6995af3a9e43bf217ad3f186d5633fed
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/58369
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This makes what are configuration and what are internal SCons variables
explicit and separate, and makes it unnecessary to call out what
variables to export to C++.
These variables will also be plumbed into and out of kconfiglib in later
changes.
Change-Id: Iaf5e098d7404af06285c421dbdf8ef4171b3f001
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/56892
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Sometimes a library is needed to support particular functionality in
gem5, and that functionality is only used (or even desirable) in
certain binaries SCons can build. We can currently filter sources to
include in a particular executable using tags, but libraries have been
added to the environment globally using the LIBS variable which applies
to all Executables.
This change adds a SourceLib() mechanism which is a new category of
source which represents libraries. This is independent from classes
which inherit from SourceFile which represent actual files, as opposed
to more abstract libraries.
When gem5 builds an executable, the filters it provides are used to
select both Source()-es, aka c/c++ files, and libraries. If something
like a unit test does not need all the libraries gem5 proper does,
then those won't be picked up by its filter, and it won't include them.
Change-Id: I003e029eb82f7800a7ecff698c260e2d18ea2900
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/58069
Reviewed-by: Yu-hsin Wang <yuhsingw@google.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
These variables will always be imported from the environment, and never
from previous builds. For SCons to actually use these values, they need
to not only be in the environment variables external commands SCons runs
sees, they also have to be promoted to actual SCons construction
variables.
Change-Id: I54035442d70972396f1788bd0773877222d7a7c5
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/56386
Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Jason Lowe-Power <power.jg@gmail.com>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
When checking whether a given option is supported, we try to compile or
link a small test program and see if that succeeded or not. There are
some options which are not really supported, but if you use them the
compiler might produce an error instead of failing to build. An example
of that is clang (version 9 at least) which will warn if you try to
disabled a warning it doesn't know about.
On the develop branch, this is fine since we have werror enabled, and
even these warnings will prevent the options from being enabled. On the
release branches where we disable werror for better future
compatibility, this causes a problem because these options still pass
the test and end up being enabled. This doesn't break the build, but
can cause a bunch of annoying and unnecessary warnings which clutter
up the build output.
To remove this inconsistency, we can enable werror just for the test
compiles or links which decide whether or not an option is supported.
That won't necessarily differentiate *why* a warning was generated, so
if through some strange combination of circumstances something else
causes a warning pervasively through the build, then this may think
an option is to blame and not enable it even though it's fine. This
seems unlikely though, and not worth worrying about without a specific
example showing that it can/will actually happen.
Change-Id: I6a1e3273c0b646a5636dc9986c70dcd5332f6f64
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/54624
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
The tags for source files need to be resolved as well as the tags for
the filters themselves.
For example, the 'x86 isa' tag could imply 'gem5 lib'. If we filter for
all files which include the tag 'gem5 lib', we would also want to get
all the files that used the tag 'x86 isa' by itself, since that would
imply 'gem5 lib' as well.
Change-Id: Ie513778cc5b8ac1f1893728b14e84cabdaabf0e6
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/51828
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
On a clean build, the git tool will wait for input() before installing
git hooks. Allow bypassing this via a command-line flag, making it
possible to perform a clean build non-interactively.
Bug: 199780674
Test: build_gem5 --install-hooks
Change-Id: I48be2c1a7c2335a2f4f6359adf582ca8b0ae5939
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/50410
Reviewed-by: Gabe Black <gabe.black@gmail.com>
Maintainer: Gabe Black <gabe.black@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
These nodes are really for working with text, and they do strange and
broken things otherwise. This change adds a pair of convenience
functions which uses pickle to serialize an object for the Value to
hold, and to unpack the Value later.
Change-Id: Id48822ce3de283b003d4cc7ef6b563a70e321ca6
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/48371
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Hoa Nguyen <hoanguyen@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
