scons: Fix gem5 Python3.11 build.

The code generation in gem5's build system requires the use of Regular
Expression flags when defining the regular expressions used for
tokenization. However, the Python Lex-Yacc (PLY) [1] library used by
gem5 does not allow the user sufficient control of the flags for RE
compilation.

Previously, gem5 used inline RE flags to control RE compilation.
However, from Python 3.11, inline RE flags must be at the start of the
RE string. Because PLY wraps the user supplied RE strings before
compilation, there is no way for the user to supply a RE string with
the inline flag at the start. This makes gem5 incompatible with Python
3.11 when using PLY.

This change modifies gem5's build files to patch `re.compile` with a
wrapped version that can handle embedded flags anywhere in the RE
string, for all current versions of Python. The patched version
re-formats the user supplied RE string to convert inline RE flags to
explicit RE flags.

This patch is intended as a temporary stop-gap until PLY can be fixed
upstream.

See the gem5 Issue Tracker [2] for more details.

[1] https://github.com/dabeaz/ply
[2] https://gem5.atlassian.net/browse/GEM5-1321

Change-Id: I3ab371f2e5cf267c0a89caaf8a2bacfed78545ef
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/70237
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Reviewed-by: Boris Shingarov <shingarov@labware.com>
Tested-by: kokoro <noreply+kokoro@google.com>

SConstruct

@@ -1,6 +1,6 @@
 # -*- mode:python -*-
 
-# Copyright (c) 2013, 2015-2020 ARM Limited
+# Copyright (c) 2013, 2015-2020, 2023 ARM Limited
 # All rights reserved.
 #
 # The license below extends only to copyright in the software and shall
@@ -171,6 +171,10 @@ SetOption('warn', 'no-duplicate-environment')
 
 Export('MakeAction')
 
+# Patch re.compile to support inline flags anywhere within a RE
+# string. Required to use PLY with Python 3.11+.
+gem5_scons.patch_re_compile_for_inline_flags()
+
 ########################################################################
 #
 # Set up the main build environment.

site_scons/gem5_scons/__init__.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2013, 2015-2017 ARM Limited
+# Copyright (c) 2013, 2015-2017, 2023 ARM Limited
 # All rights reserved.
 #
 # The license below extends only to copyright in the software and shall
@@ -302,6 +302,48 @@ def FromValue(node):
     return pickle.loads(node.read())
 
 
+def patch_re_compile_for_inline_flags():
+    """Patch `re.compile` with a version that can handle RE strings with
+    inline flags anywhere within them. This is required to use PLY
+    with Python 3.11+.
+
+    """
+
+    import re
+    from functools import partial
+
+    def _inline_flag_aware_re_compile(re_compile, re_str, flags=0x0):
+        """Provide an alternative implementation of `re.compile` that allows
+        inline flags that are not at the start of the regular
+        expression string.
+
+        From Python 3.11, the `re` module only supports inline flags
+        at the start of the RE string. This makes it impossible to add
+        flags to the Lexer strings when using PLY, because PLY embeds
+        the user supplied token REs, and does not provide sufficient
+        control of the `flags` argument.
+
+        """
+        _flags_map = {
+            ("(?a)", b"(?a)"): re.ASCII,
+            ("(?i)", b"(?i)"): re.IGNORECASE,
+            ("(?L)", b"(?L)"): re.LOCALE,
+            ("(?m)", b"(?m)"): re.MULTILINE,
+            ("(?s)", b"(?s)"): re.DOTALL,
+            ("(?x)", b"(?x)"): re.VERBOSE,
+        }
+        for (pattern_s, pattern_b), flag in _flags_map.items():
+            pattern = pattern_b if isinstance(re_str, bytes) else pattern_s
+            replacement = b"" if isinstance(re_str, bytes) else ""
+            if pattern in re_str:
+                flags |= flag
+                re_str = re_str.replace(pattern, replacement)
+        return re_compile(re_str, flags)
+
+    # Patch the default `re.compile`
+    re.compile = partial(_inline_flag_aware_re_compile, re.compile)
+
+
 __all__ = [
     "Configure",
     "EnvDefaults",
@@ -312,4 +354,5 @@ __all__ = [
     "MakeActionTool",
     "ToValue",
     "FromValue",
+    "patch_re_compile_for_inline_flags",
 ]