derek/gem5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

stdlib: Refactor gem5 Vision/gem5-resources code

Browse Source 
This patch includes several changes to the gem5 tools interface to the
gem5-resources infrastructure. These are:

* The old download and JSON query functions have been removed from the
  downloader module. These functions were used for directly downloading
  and inspecting the resource JSON file, hosted at
  https://resources.gem5.org/resources. This information is now obtained
  via `gem5.client`. If a resources JSON file is specified as a client,
  it should conform to the new schema:
  https//resources.gem5.org/gem5-resources-schema.json. The old schema
  (pre-v23.0) is no longer valid. Tests have been updated to reflect
  this change. Those which tested these old functions have been removed.
* Unused imports have been removed.
* For the resource query functions, and those tasked with obtaining the
  resources, the parameter `gem5_version` has been added. In all cases
  it does the same thing:
    * It will filter results based on compatibility to the
      `gem5_version` specified. If no resources are compatible the
      latest version of that resource is chosen (though a warning is
      thrown).
    * By default it is set to the current gem5 version.
    * It is optional. If `None`, this filtering functionality is not
      carried out.
    * Tests have been updated to fix the version to “develop” so the
      they do not break between versions.
* The `gem5_version` parameters will filter using a logic which will
  base compatibility on the specificity of the gem5-version specified in
  a resource’s data. If a resource has a compatible gem5-version of
  “v18.4” it will be compatible with any minor/hotfix version within the
  v18.4 release (this can be seen as matching on “v18.4.*.*”.) Likewise,
  if a resource has a compatible gem5-version of “v18.4.1” then it’s
  only compatible with the v18.4.1 release but any of it’s hot fix
  releases (“v18.4.1.*”).
* The ‘list_resources’ function has been updated to use the
  “gem5.client” APIs to get resource information from the clients
  (MongoDB or a JSON file). This has been designed to remain backwards
  compatible to as much as is possible, though, due to schema changes,
  the function does search across all versions of gem5.
* `get_resources` function was added to the `AbstractClient`. This is a
   more general function than `get_resource_by_id`. It was
  primarily created to handle the `list_resources` update but is a
  useful update to the API. The `get_resource_by_id` function has been
  altered to function as a wrapped to the `get_resources` function.
* Removed “GEM5_RESOURCE_JSON” code has been removed. This is no longer
  used.
* Tests have been cleaned up a little bit to be easier to read.
* Some docstrings have been updated.

Things that are left TODO with this code:

* The client_wrapper/client/abstract_client abstractions are rather
  pointless. In particular the client_wrapper and client classes could
  be merged.
* The downloader module no longer does much and should have its
  functions merged into other modules.
* With the addition of the `get_resources` function, much of the code in
  the `AbstractClient` could be simplified.

Change-Id: I0ce48e88b93a2b9db53d4749861fa0b5f9472053
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/71506
Reviewed-by: Kunal Pai <kunpai@ucdavis.edu>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This commit is contained in:
Bobby R. Bruce
2023-06-12 14:09:15 -07:00
committed by Bobby Bruce
parent 328aaa626f
commit 82587ce71b
21 changed files with 445 additions and 603 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
src/python/gem5/resources/client.py
View File

@@ -31,6 +31,7 @@ from typing import Optional, Dict, List
from .client_api.client_wrapper import ClientWrapper
from gem5.gem5_default_config import config
from m5.util import inform
from _m5 import core
def getFileContent(file_path: Path) -> Dict:
@@ -49,17 +50,7 @@ def getFileContent(file_path: Path) -> Dict:
clientwrapper = None
def get_resource_json_obj(
resource_id,
resource_version: Optional[str] = None,
clients: Optional[List[str]] = None,
) -> Dict:
"""
Get the resource json object from the clients wrapper
:param resource_id: The resource id
:param resource_version: The resource version
:param clients: The list of clients to query
"""
def _get_clientwrapper():
global clientwrapper
if clientwrapper is None:
# First check if the config file path is provided in the environment variable
@@ -78,7 +69,42 @@ def get_resource_json_obj(
gem5_config = config
inform("Using default config")
clientwrapper = ClientWrapper(gem5_config)
return clientwrapper
return clientwrapper.get_resource_json_obj_from_client(
resource_id, resource_version, clients
def list_resources(
clients: Optional[List[str]] = None,
gem5_version: Optional[str] = core.gem5Version,
) -> Dict[str, List[str]]:
"""
List all the resources available
:param clients: The list of clients to query
:param gem5_version: The gem5 version of the resource to get. By default,
it is the gem5 version of the current build. If set to none, it will return
all gem5 versions of the resource.
:return: A Python Dict where the key is the resource id and the value is
a list of all the supported resource versions.
"""
return _get_clientwrapper().list_resources(clients, gem5_version)
def get_resource_json_obj(
resource_id,
resource_version: Optional[str] = None,
clients: Optional[List[str]] = None,
gem5_version: Optional[str] = core.gem5Version,
) -> Dict:
"""
Get the resource json object from the clients wrapper
:param resource_id: The resource id
:param resource_version: The resource version
:param clients: The list of clients to query
:param gem5_version: The gem5 versions to filter the resources based on
compatibility. By default, it is the gem5 version of the current build.
If None, filtering based on compatibility is not performed.
"""
return _get_clientwrapper().get_resource_json_obj_from_client(
resource_id, resource_version, clients, gem5_version
)

56
src/python/gem5/resources/client_api/abstract_client.py
View File

@@ -25,7 +25,7 @@
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
from abc import ABC, abstractmethod
from typing import Any, Dict, List
from typing import Any, Dict, List, Optional
import urllib.parse
@@ -63,9 +63,61 @@ class AbstractClient(ABC):
return False
@abstractmethod
def get_resources(
self,
resource_id: Optional[str] = None,
resource_version: Optional[str] = None,
gem5_version: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""
:param resource_id: The ID of the Resource. Optional, if not set, all
resources will be returned.
:param resource_version: The version of the Resource. Optional, if
not set, all resource versions will be returned. Note: If `resource_id`
is not set, this parameter will be ignored.
:param gem5_version: The version of gem5. Optional, if not set, all
versions will be returned.
:return: A list of all the Resources with the given ID.
"""
raise NotImplementedError
def filter_incompatible_resources(
self,
resources_to_filter: List[Dict[str, Any]],
gem5_version: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""Returns a filtered list resources based on gem5 version
compatibility.
Note: This function assumes if the minor component of
a resource's gem5_version is not specified, the resource is compatible
with all minor versions of the same major version.
Likewise, if no hot-fix component is specified, it is assumed that
the resource is compatible with all hot-fix versions of the same
minor version.
* '20.1' would be compatible with gem5 '20.1.1.0' and '20.1.2.0'.
* '21.5.2' would be compatible with gem5 '21.5.2.0' and '21.5.2.0'.
* '22.3.2.4' would only be compatible with gem5 '22.3.2.4'.
:param resources_to_filter: The list of resources to filter.
:param gem5_version: The gem5 version in which the filtered resources
should be compatible. If None, no filtering will be done.
:
"""
if not gem5_version:
return resources_to_filter
filtered_resources = []
for resource in resources_to_filter:
for version in resource["gem5_versions"]:
if gem5_version.startswith(version):
filtered_resources.append(resource)
return filtered_resources
def get_resources_by_id(self, resource_id: str) -> List[Dict[str, Any]]:
"""
:param resource_id: The ID of the Resource.
:return: A list of all the Resources with the given ID.
"""
raise NotImplementedError
return self.get_resources(resource_id=resource_id)

22
src/python/gem5/resources/client_api/atlasclient.py
View File

@@ -64,14 +64,26 @@ class AtlasClient(AbstractClient):
token = result["access_token"]
return token
def get_resources_by_id(self, resource_id: str) -> List[Dict[str, Any]]:
def get_resources(
self,
resource_id: Optional[str] = None,
resource_version: Optional[str] = None,
gem5_version: Optional[str] = None,
) -> List[Dict[str, Any]]:
url = f"{self.url}/action/find"
data = {
"dataSource": self.dataSource,
"collection": self.collection,
"database": self.database,
"filter": {"id": resource_id},
}
filter = {}
if resource_id:
filter["id"] = resource_id
if resource_version is not None:
filter["resource_version"] = resource_version
if filter:
data["filter"] = filter
data = json.dumps(data).encode("utf-8")
headers = {
@@ -88,4 +100,8 @@ class AtlasClient(AbstractClient):
result = json.loads(response.read().decode("utf-8"))
resources = result["documents"]
return resources
# I do this as a lazy post-processing step because I can't figure out
# how to do this via an Atlas query, which may be more efficient.
return self.filter_incompatible_resources(
resources_to_filter=resources, gem5_version=gem5_version
)

81
src/python/gem5/resources/client_api/client_wrapper.py
View File

@@ -59,6 +59,38 @@ class ClientWrapper:
warn(f"Error creating client {client}: {str(e)}")
return clients
def list_resources(
self,
clients: Optional[List[str]] = None,
gem5_version: Optional[str] = core.gem5Version,
) -> Dict[str, List[str]]:
clients_to_search = (
list(self.clients.keys()) if clients is None else clients
)
# There's some duplications of functionality here (similar code in
# `get_all_resources_by_id`. This code could be refactored to avoid
# this).
resources = []
for client in clients_to_search:
if client not in self.clients:
raise Exception(f"Client: {client} does not exist")
try:
resources.extend(
self.clients[client].get_resources(
gem5_version=gem5_version
)
)
except Exception as e:
warn(f"Error getting resources from client {client}: {str(e)}")
to_return = {}
for resource in resources:
if resource["id"] not in to_return:
to_return[resource["id"]] = []
to_return[resource["id"]].append(resource["resource_version"])
return to_return
def get_all_resources_by_id(
self,
resource_id: str,
@@ -98,6 +130,7 @@ class ClientWrapper:
resource_id: str,
resource_version: Optional[str] = None,
clients: Optional[List[str]] = None,
gem5_version: Optional[str] = core.gem5Version,
) -> Dict:
"""
This function returns the resource object from the client with the
@@ -106,6 +139,9 @@ class ClientWrapper:
:param resource_version: The version of the resource to search for.
:param clients: A list of clients to search through. If None, all
clients are searched.
:param gem5_version: The gem5 version to check compatibility with. If
None, no compatibility check is performed. By default, is the current
version of gem5.
:return: The resource object as a Python dictionary if found.
If not found, exception is thrown.
"""
@@ -124,7 +160,9 @@ class ClientWrapper:
else:
compatible_resources = (
self._get_resources_compatible_with_gem5_version(resources)
self._get_resources_compatible_with_gem5_version(
resources, gem5_version=gem5_version
)
)
if len(compatible_resources) == 0:
resource_to_return = self._sort_resources(resources)[0]
@@ -133,7 +171,10 @@ class ClientWrapper:
compatible_resources
)[0]
self._check_resource_version_compatibility(resource_to_return)
if gem5_version:
self._check_resource_version_compatibility(
resource_to_return, gem5_version=gem5_version
)
return resource_to_return
@@ -172,16 +213,31 @@ class ClientWrapper:
) -> List:
"""
Returns a list of compatible resources with the current gem5 version.
Note: This function assumes if the minor component of
a resource's gem5_version is not specified, it that the
resource is compatible all minor versions of the same major version.
Likewise, if no hot-fix component is specified, it is assumed that
the resource is compatible with all hot-fix versions of the same
minor version.
* '20.1' would be compatible with gem5 '20.1.1.0' and '20.1.2.0'.
* '21.5.2' would be compatible with gem5 '21.5.2.0' and '21.5.2.0'.
* '22.3.2.4' would only be compatible with gem5 '22.3.2.4'.
:param resources: A list of resources to filter.
:return: A list of compatible resources as Python dictionaries.
If no compatible resources are found, the original list of resources
is returned.
**Note**: This is a big duplication of code. This functionality already
exists in the `AbstractClient` class. This code should be refactored
to avoid this duplication.
"""
compatible_resources = [
resource
for resource in resources
if gem5_version in resource["gem5_versions"]
]
compatible_resources = []
for resource in resources:
for version in resource["gem5_versions"]:
if gem5_version.startswith(version):
compatible_resources.append(resource)
return compatible_resources
def _sort_resources(self, resources: List) -> List:
@@ -213,7 +269,12 @@ class ClientWrapper:
"""
if not resource:
return False
if gem5_version not in resource["gem5_versions"]:
if (
gem5_version
and not self._get_resources_compatible_with_gem5_version(
[resource], gem5_version=gem5_version
)
):
warn(
f"Resource {resource['id']} with version "
f"{resource['resource_version']} is not known to be compatible"

38
src/python/gem5/resources/client_api/jsonclient.py
View File

@@ -58,13 +58,31 @@ class JSONClient(AbstractClient):
)
self.resources = json.loads(response.read().decode("utf-8"))
def get_resources_by_id(self, resource_id: str) -> List[Dict[str, Any]]:
"""
:param resource_id: The ID of the Resource.
:return: A list of all the Resources with the given ID.
"""
return [
resource
for resource in self.resources
if resource["id"] == resource_id
]
def get_resources_json(self) -> List[Dict[str, Any]]:
"""Returns a JSON representation of the resources."""
return self.resources
def get_resources(
self,
resource_id: Optional[str] = None,
resource_version: Optional[str] = None,
gem5_version: Optional[str] = None,
) -> List[Dict[str, Any]]:
filter = self.resources # Unfiltered.
if resource_id:
filter = [ # Filter by resource_id.
resource
for resource in filter
if resource["id"] == resource_id
]
if resource_version:
filter = [ # Filter by resource_version.
resource
for resource in filter
if resource["resource_version"] == resource_version
]
# Filter by gem5_version.
return self.filter_incompatible_resources(
resources_to_filter=filter, gem5_version=gem5_version
)

271
src/python/gem5/resources/downloader.py
View File

@@ -24,24 +24,24 @@
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
import json
import urllib.request
import urllib.parse
import hashlib
import os
import shutil
import gzip
import hashlib
import base64
import time
import random
from pathlib import Path
import tarfile
from tempfile import gettempdir
from urllib.error import HTTPError
from typing import List, Dict, Set, Optional
from typing import List, Optional, Dict
from .client import get_resource_json_obj
from _m5 import core
from .client import (
get_resource_json_obj,
list_resources as client_list_resources,
)
from .md5_utils import md5_file, md5_dir
from ..utils.progress_bar import tqdm, progress_hook
@@ -53,188 +53,6 @@ information about resources from resources.gem5.org.
"""
def _resources_json_version_required() -> str:
"""
Specifies the version of resources.json to obtain.
"""
return "develop"
def _get_resources_json_uri() -> str:
return "https://resources.gem5.org/resources.json"
def _url_validator(url):
try:
result = urllib.parse.urlparse(url)
return all([result.scheme, result.netloc, result.path])
except:
return False
def _get_resources_json_at_path(path: str, use_caching: bool = True) -> Dict:
"""
Returns a resource JSON, in the form of a Python Dict. The location
of the JSON must be specified.
If `use_caching` is True, and a URL is passed, a copy of the JSON will be
cached locally, and used for up to an hour after retrieval.
:param path: The URL or local path of the JSON file.
:param use_caching: True if a cached file is to be used (up to an hour),
otherwise the file will be retrieved from the URL regardless. True by
default. Only valid in cases where a URL is passed.
"""
# If a local valid path is passed, just load it.
if Path(path).is_file():
return json.load(open(path))
# If it's not a local path, it should be a URL. We check this here and
# raise an Exception if it's not.
if not _url_validator(path):
raise Exception(
f"Resources location '{path}' is not a valid path or URL."
)
download_path = os.path.join(
gettempdir(),
f"gem5-resources-{hashlib.md5(path.encode()).hexdigest()}"
f"-{str(os.getuid())}.json",
)
# We apply a lock on the resources file for when it's downloaded, or
# re-downloaded, and read. This stops a corner-case from occuring where
# the file is re-downloaded while being read by another gem5 thread.
# Note the timeout is 120 so the `_download` function is given time to run
# its Truncated Exponential Backoff algorithm
# (maximum of roughly 1 minute). Typically this code will run quickly.
with FileLock(f"{download_path}.lock", timeout=120):
# The resources.json file can change at any time, but to avoid
# excessive retrieval we cache a version locally and use it for up to
# an hour before obtaining a fresh copy.
#
# `time.time()` and `os.path.getmtime(..)` both return an unix epoch
# time in seconds. Therefore, the value of "3600" here represents an
# hour difference between the two values. `time.time()` gets the
# current time, and `os.path.getmtime(<file>)` gets the modification
# time of the file. This is the most portable solution as other ideas,
# like "file creation time", are not always the same concept between
# operating systems.
if (
not use_caching
or not os.path.exists(download_path)
or (time.time() - os.path.getmtime(download_path)) > 3600
):
_download(path, download_path)
with open(download_path) as f:
file_contents = f.read()
try:
to_return = json.loads(file_contents)
except json.JSONDecodeError:
# This is a bit of a hack. If the URL specified exists in a Google
# Source repo (which is the case when on the gem5 develop branch) we
# retrieve the JSON in base64 format. This cannot be loaded directly as
# text. Conversion is therefore needed.
to_return = json.loads(base64.b64decode(file_contents).decode("utf-8"))
return to_return
def _get_resources_json() -> Dict:
"""
Gets the Resources JSON.
:returns: The Resources JSON (as a Python Dictionary).
"""
path = os.getenv("GEM5_RESOURCE_JSON", _get_resources_json_uri())
to_return = _get_resources_json_at_path(path=path)
# If the current version pulled is not correct, look up the
# "previous-versions" field to find the correct one.
# If the resource JSON file does not have a "version" field or it's
# null/None, then we will use this resource JSON file (this is usefull for
# testing purposes).
version = _resources_json_version_required()
json_version = None if "version" not in to_return else to_return["version"]
if json_version and json_version != version:
if version in to_return["previous-versions"].keys():
to_return = _get_resources_json_at_path(
path=to_return["previous-versions"][version]
)
else:
# This should never happen, but we thrown an exception to explain
# that we can't find the version.
raise Exception(
f"Version '{version}' of resources.json cannot be found."
)
return to_return
def _get_url_base() -> str:
"""
Obtains the "url_base" string from the resources.json file.
:returns: The "url_base" string value from the resources.json file.
"""
json = _get_resources_json()
if "url_base" in json.keys():
return json["url_base"]
return ""
def _get_resources(
valid_types: Set[str], resources_group: Optional[Dict] = None
) -> Dict[str, Dict]:
"""
A recursive function to get all the workload/resource of the specified type
in the resources.json file.
:param valid_types: The type to return (i.e., "resource" or "workload).
:param resource_group: Used for recursion: The current resource group being
iterated through.
:returns: A dictionary of artifact names to the resource JSON objects.
"""
if resources_group is None:
resources_group = _get_resources_json()["resources"]
to_return = {}
for resource in resources_group:
if resource["type"] in valid_types:
# If the type is valid then we add it directly to the map
# after a check that the name is unique.
if resource["name"] in to_return.keys():
raise Exception(
f"Error: Duplicate resource with name '{resource['name']}'."
)
to_return[resource["name"]] = resource
elif resource["type"] == "group":
# If it's a group we get recursive. We then check to see if there
# are any duplication of keys.
new_map = _get_resources(
valid_types=valid_types, resources_group=resource["contents"]
)
intersection = set(new_map.keys()).intersection(to_return.keys())
if len(intersection) > 0:
# Note: if this error is received it's likely an error with
# the resources.json file. The resources names need to be
# unique keyes.
raise Exception(
f"Error: Duplicate resources with names: {str(intersection)}."
)
to_return.update(new_map)
return to_return
def _download(url: str, download_to: str, max_attempts: int = 6) -> None:
"""
Downloads a file.
@@ -336,61 +154,26 @@ def _download(url: str, download_to: str, max_attempts: int = 6) -> None:
)
def list_resources() -> List[str]:
def list_resources(
clients: Optional[List] = None, gem5_version: Optional[str] = None
) -> Dict[str, List[str]]:
"""
Lists all available resources by name.
Lists all available resources. Returns a dictionary where the key is the
id of the resources and the value is a list of that resource's versions.
:param clients: A list of clients to use when listing resources. If None,
all clients will be used. None by default.
:param gem5_version: The gem5 version to which all resources should be
compatible with. If None, compatibility of resources is not considered and
all resources will be returned.
**Note**: This function is here for legacy reasons. The `list_resources`
function was originally stored here. In order to remain backwards
compatible, this function will call the `client_list_resources` function
:returns: A list of resources by name.
"""
from .resource import _get_resource_json_type_map
return _get_resources(
valid_types=_get_resource_json_type_map.keys()
).keys()
def get_workload_json_obj(workload_name: str) -> Dict:
"""
Get a JSON object of a specified workload.
:param workload_name: The name of the workload.
:raises Exception: An exception is raised if the specified workload does
not exit.
"""
workload_map = _get_resources(valid_types={"workload"})
if workload_name not in workload_map:
raise Exception(
f"Error: Workload with name {workload_name} does not exist"
)
return workload_map[workload_name]
def get_resources_json_obj(resource_name: str) -> Dict:
"""
Get a JSON object of a specified resource.
:param resource_name: The name of the resource.
:returns: The JSON object (in the form of a dictionary).
:raises Exception: An exception is raised if the specified resources does
not exist.
"""
from .resource import _get_resource_json_type_map
resource_map = _get_resources(
valid_types=_get_resource_json_type_map.keys()
)
if resource_name not in resource_map:
raise Exception(
f"Error: Resource with name '{resource_name}' does not exist"
)
return resource_map[resource_name]
return client_list_resources(clients=clients, gem5_version=gem5_version)
def get_resource(
@@ -401,6 +184,7 @@ def get_resource(
download_md5_mismatch: bool = True,
resource_version: Optional[str] = None,
clients: Optional[List] = None,
gem5_version: Optional[str] = core.gem5Version,
) -> None:
"""
Obtains a gem5 resource and stored it to a specified location. If the
@@ -429,6 +213,10 @@ def get_resource(
:param clients: A list of clients to use when obtaining the resource. If
None, all clients will be used. None by default.
:param gem5_version: The gem5 version to use when obtaining the resource.
By default, the version of gem5 being used is used. This is used primarily
for testing purposes.
:raises Exception: An exception is thrown if a file is already present at
`to_path` but it does not have the correct md5 sum. An exception will also
be thrown is a directory is present at `to_path`
@@ -444,6 +232,7 @@ def get_resource(
resource_name,
resource_version=resource_version,
clients=clients,
gem5_version=gem5_version,
)
if os.path.exists(to_path):

18
src/python/gem5/resources/resource.py
View File

@@ -28,6 +28,7 @@ from abc import ABCMeta
import os
from pathlib import Path
from m5.util import warn, fatal
from _m5 import core
from .downloader import get_resource
@@ -559,17 +560,15 @@ def obtain_resource(
download_md5_mismatch: bool = True,
resource_version: Optional[str] = None,
clients: Optional[List] = None,
gem5_version=core.gem5Version,
) -> AbstractResource:
"""
This function primarily serves as a factory for resources. It will return
the correct `AbstractResource` implementation based on the resource
requested, by referencing the "resource.json" file (by default, that hosted
at https://resources.gem5.org/resources.json). In addition to this, this
function will download the resource if not detected in the
`resource_directory`.
requested.
:param resource_name: The name of the gem5 resource as it appears under the
"name" field in the `resource.json` file.
"id" field in the `resource.json` file.
:param resource_directory: The location of the directory in which the
resource is to be stored. If this parameter is not set, it will set to
the environment variable `GEM5_RESOURCE_DIR`. If the environment is not
@@ -582,11 +581,17 @@ def obtain_resource(
Not a required parameter. None by default.
:param clients: A list of clients to search for the resource. If this
parameter is not set, it will default search all clients.
:param gem5_version: The gem5 version to use to filter incompatible
resource versions. By default set to the current gem5 version. If None,
this filtering is not performed.
"""
# Obtain the resource object entry for this resource
resource_json = get_resource_json_obj(
resource_id, resource_version=resource_version, clients=clients
resource_id,
resource_version=resource_version,
clients=clients,
gem5_version=gem5_version,
)
to_path = None
@@ -629,6 +634,7 @@ def obtain_resource(
download_md5_mismatch=download_md5_mismatch,
resource_version=resource_version,
clients=clients,
gem5_version=gem5_version,
)
# Obtain the type from the JSON. From this we will determine what subclass

12
src/python/gem5/resources/workload.py
View File

@@ -27,6 +27,8 @@
from .resource import obtain_resource
from .client import get_resource_json_obj
from _m5 import core
from typing import Dict, Any, List, Optional
@@ -160,6 +162,7 @@ class Workload(AbstractWorkload):
resource_directory: Optional[str] = None,
resource_version: Optional[str] = None,
clients: Optional[List] = None,
gem5_version: Optional[str] = core.gem5Version,
) -> None:
"""
This constructor will load the workload details from the workload with
@@ -201,12 +204,17 @@ class Workload(AbstractWorkload):
:param resource_directory: An optional parameter that specifies where
any resources should be download and accessed from. If None, a default
location will be used. None by default.
:param gem5_version: The gem5 version for the Workload to be loaded.
By default, the current gem5 version is used. This will filter
resources which are incompatible with the current gem5 version. If
None, no filtering will be done.
"""
workload_json = get_resource_json_obj(
workload_name,
resource_version=resource_version,
clients=clients,
gem5_version=gem5_version,
)
func = workload_json["function"]
@@ -219,7 +227,9 @@ class Workload(AbstractWorkload):
value = workload_json["resources"][key]
assert isinstance(value, str)
params[key] = obtain_resource(
value, resource_directory=resource_directory
value,
resource_directory=resource_directory,
gem5_version=gem5_version,
)
if "additional_params" in workload_json: