derek/gem5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

stdlib, tests, configs: Introduce gem5 Vision to resources

Browse Source 
This patch makes changes to the stdlib based on the gem5 Vision project.
Firstly, a MongoDB database is supported.
A JSON database's support is continued.
The JSON can either be a local path or a raw GitHub link.

The data for these databases is stored in src/python
under "gem5-config.json".
This will be used by default.
However, the configuration can be overridden:
- by providing a path using the GEM5_CONFIG env variable.
- by placing a gem5-config.json file in the current working directory.

An AbstractClient is an abstract class that implements
searching and sorting relevant to the databases.

Clients is an optional list that can be passed
while defining any Resource class and obtain_resource.
These databases can be defined in the config JSON.

Resources now have versions. This allows for a
single version, e.g., 'x86-ubuntu-boot', to have
multiple versions. As such, the key of a resource is
its ID and Version (e.g., 'x86-ubuntu-boot/v2.1.0').
Different versions of a resource might be compatible
with different versions of gem5.

By default, it picks the latest version compatible with the gem5 Version
of the user.

A gem5 resource schema now has additional fields.
These are:
- source_url: Stores URL of GitHub Source of the resource.
- license: License information of the resource.
- tags: Words to identify a resource better, like hello for hello-world
- example_usage: How to use the resource in a simulation.
- gem5_versions: List of gem5 versions that resource is compatible with.
- resource_version: The version of the resource itself.
- size: The download size of the resource, if it exists.
- code_examples: List of objects.
These objects contain the path to where a resource is
used in gem5 example config scripts,
and if the resource itself is used in tests or not.
- category: Category of the resource, as defined by classes in
src/python/gem5/resources/resource.py.

Some fields have been renamed:
- "name" is changed to "id"
- "documentation" is changed to "description"

Besides these, the schema also supports resource specialization.
It adds fields relevant to a specific resource as specified in
src/python/gem5/resources/resource.py
These changes have been made to better present
information on the new gem5 Resources website.

But, they do not affect the way resources are used by a gem5 user.
This patch is also backwards compatible.
Existing code doesn't break with this new infrastructure.

Also, refs in the tests have been changed to match this new schema.
Tests have been changed to work with the two clients.

Change-Id: Ia9bf47f7900763827fd5e873bcd663cc3ecdba40
Co-authored-by: Kunal Pai <kunpai@ucdavis.edu>
Co-authored-by: Parth Shah <helloparthshah@gmail.com>
Co-authored-by: Harshil Patel <harshilp2107@gmail.com>
Co-authored-by: aarsli <arsli@ucdavis.edu>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/71278
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
This commit is contained in:
Kunal Pai
2023-05-21 23:35:40 -07:00
committed by Bobby Bruce
parent cf2d5b68a9
commit 70ec55ce2a
26 changed files with 2413 additions and 342 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
src/python/SConscript
View File

@@ -264,12 +264,24 @@ PySource('gem5.prebuilt.riscvmatched',
PySource('gem5.prebuilt.riscvmatched',
'gem5/prebuilt/riscvmatched/riscvmatched_core.py')
PySource('gem5.resources', 'gem5/resources/__init__.py')
PySource('gem5.resources', 'gem5/resources/client.py')
PySource('gem5.resources', 'gem5/resources/downloader.py')
PySource('gem5.resources', 'gem5/resources/md5_utils.py')
PySource('gem5.resources', 'gem5/resources/resource.py')
PySource('gem5.resources', 'gem5/resources/workload.py')
PySource('gem5.resources', 'gem5/resources/looppoint.py')
PySource('gem5.resources', 'gem5/resources/elfie.py')
PySource('gem5.resources.client_api',
'gem5/resources/client_api/__init__.py')
PySource('gem5.resources.client_api',
'gem5/resources/client_api/jsonclient.py')
PySource('gem5.resources.client_api',
'gem5/resources/client_api/atlasclient.py')
PySource('gem5.resources.client_api',
'gem5/resources/client_api/client_wrapper.py')
PySource('gem5.resources.client_api',
'gem5/resources/client_api/abstract_client.py')
PySource('gem5', 'gem5_default_config.py')
PySource('gem5.utils', 'gem5/utils/__init__.py')
PySource('gem5.utils', 'gem5/utils/filelock.py')
PySource('gem5.utils', 'gem5/utils/override.py')

84
src/python/gem5/resources/client.py Normal file
View File

@@ -0,0 +1,84 @@
# Copyright (c) 2023 The Regents of the University of California
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met: redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer;
# redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution;
# neither the name of the copyright holders nor the names of its
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
import json
from pathlib import Path
import os
from typing import Optional, Dict, List
from .client_api.client_wrapper import ClientWrapper
from gem5.gem5_default_config import config
from m5.util import inform
def getFileContent(file_path: Path) -> Dict:
"""
Get the content of the file at the given path
:param file_path: The path of the file
:return: The content of the file
"""
if file_path.exists():
with open(file_path, "r") as file:
return json.load(file)
else:
raise Exception(f"File not found at {file_path}")
clientwrapper = None
def get_resource_json_obj(
resource_id,
resource_version: Optional[str] = None,
clients: Optional[List[str]] = None,
) -> Dict:
"""
Get the resource json object from the clients wrapper
:param resource_id: The resource id
:param resource_version: The resource version
:param clients: The list of clients to query
"""
global clientwrapper
if clientwrapper is None:
# First check if the config file path is provided in the environment variable
if "GEM5_CONFIG" in os.environ:
config_file_path = Path(os.environ["GEM5_CONFIG"])
gem5_config = getFileContent(config_file_path)
inform("Using config file specified by $GEM5_CONFIG")
inform(f"Using config file at {os.environ['GEM5_CONFIG']}")
# If not, check if the config file is present in the current directory
elif (Path().cwd().resolve() / "gem5-config.json").exists():
config_file_path = Path().resolve() / "gem5-config.json"
gem5_config = getFileContent(config_file_path)
inform(f"Using config file at {config_file_path}")
# If not, use the default config in the build directory
else:
gem5_config = config
inform("Using default config")
clientwrapper = ClientWrapper(gem5_config)
return clientwrapper.get_resource_json_obj_from_client(
resource_id, resource_version, clients
)

0
src/python/gem5/resources/client_api/__init__.py Normal file
View File

71
src/python/gem5/resources/client_api/abstract_client.py Normal file
View File

@@ -0,0 +1,71 @@
# Copyright (c) 2023 The Regents of the University of California
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met: redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer;
# redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution;
# neither the name of the copyright holders nor the names of its
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
from abc import ABC, abstractmethod
from typing import Any, Dict, List
import urllib.parse
class AbstractClient(ABC):
def verify_status_code(self, status_code: int) -> None:
"""
Verifies that the status code is 200.
:param status_code: The status code to verify.
"""
if status_code == 200:
return
if status_code == 429:
raise Exception("Panic: Too many requests")
if status_code == 401:
raise Exception("Panic: Unauthorized")
if status_code == 404:
raise Exception("Panic: Not found")
if status_code == 400:
raise Exception("Panic: Bad request")
if status_code == 500:
raise Exception("Panic: Internal server error")
raise Exception(f"Panic: Unknown status code {status_code}")
def _url_validator(self, url: str) -> bool:
"""
Validates the provided URL.
:param url: The URL to be validated.
:return: True if the URL is valid, False otherwise.
"""
try:
result = urllib.parse.urlparse(url)
return all([result.scheme, result.netloc, result.path])
except:
return False
@abstractmethod
def get_resources_by_id(self, resource_id: str) -> List[Dict[str, Any]]:
"""
:param resource_id: The ID of the Resource.
:return: A list of all the Resources with the given ID.
"""
raise NotImplementedError

91
src/python/gem5/resources/client_api/atlasclient.py Normal file
View File

@@ -0,0 +1,91 @@
# Copyright (c) 2023 The Regents of the University of California
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met: redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer;
# redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution;
# neither the name of the copyright holders nor the names of its
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
from urllib import request, parse
from urllib.error import HTTPError, URLError
from typing import Optional, Dict, Union, Type, Tuple, List, Any
import json
from .abstract_client import AbstractClient
class AtlasClient(AbstractClient):
def __init__(self, config: Dict[str, str]):
"""
Initializes a connection to a MongoDB Atlas database.
:param uri: The URI for connecting to the MongoDB server.
:param db: The name of the database to connect to.
:param collection: The name of the collection within the database.
"""
self.apiKey = config["apiKey"]
self.url = config["url"]
self.collection = config["collection"]
self.database = config["database"]
self.dataSource = config["dataSource"]
self.authUrl = config["authUrl"]
def get_token(self):
data = {"key": self.apiKey}
data = json.dumps(data).encode("utf-8")
req = request.Request(
self.authUrl,
data=data,
headers={"Content-Type": "application/json"},
)
try:
response = request.urlopen(req)
except HTTPError as e:
self.verify_status_code(e.status)
return None
result = json.loads(response.read().decode("utf-8"))
token = result["access_token"]
return token
def get_resources_by_id(self, resource_id: str) -> List[Dict[str, Any]]:
url = f"{self.url}/action/find"
data = {
"dataSource": self.dataSource,
"collection": self.collection,
"database": self.database,
"filter": {"id": resource_id},
}
data = json.dumps(data).encode("utf-8")
headers = {
"Authorization": f"Bearer {self.get_token()}",
"Content-Type": "application/json",
}
req = request.Request(url, data=data, headers=headers)
try:
response = request.urlopen(req)
except HTTPError as e:
self.verify_status_code(e.status)
return None
result = json.loads(response.read().decode("utf-8"))
resources = result["documents"]
return resources

228
src/python/gem5/resources/client_api/client_wrapper.py Normal file
View File

@@ -0,0 +1,228 @@
# Copyright (c) 2023 The Regents of the University of California
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met: redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer;
# redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution;
# neither the name of the copyright holders nor the names of its
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
from .jsonclient import JSONClient
from .atlasclient import AtlasClient
from _m5 import core
from typing import Optional, Dict, List
from distutils.version import StrictVersion
import itertools
from m5.util import warn
class ClientWrapper:
def __init__(self, config):
self.clients = self.create_clients(config)
def create_clients(
self,
config: Dict,
) -> Dict:
"""
This function creates respective client object for each source in the
config file according to the type of source.
Params: config: config file containing the source information
Returns: clients: dictionary of clients for each source
"""
clients = {}
for client in config["sources"]:
client_source = config["sources"][client]
try:
if client_source["isMongo"]:
clients[client] = AtlasClient(client_source)
else:
clients[client] = JSONClient(client_source["url"])
except Exception as e:
warn(f"Error creating client {client}: {str(e)}")
return clients
def get_all_resources_by_id(
self,
resource_id: str,
clients: Optional[List[str]] = None,
) -> List[Dict]:
"""
This function returns all the resources with the given id from all the
sources.
:param resource_id: The id of the resource to search for.
:param clients: A list of clients to search through. If None, all
clients are searched.
:return: A list of resources as Python dictionaries.
"""
resources = []
if not clients:
clients = list(self.clients.keys())
for client in clients:
if client not in self.clients:
raise Exception(f"Client: {client} does not exist")
try:
resources.extend(
self.clients[client].get_resources_by_id(resource_id)
)
except Exception as e:
warn(f"Error getting resources from client {client}: {str(e)}")
# check if no 2 resources have the same id and version
for res1, res2 in itertools.combinations(resources, 2):
if res1["resource_version"] == res2["resource_version"]:
raise Exception(
f"Resource {resource_id} has multiple resources with "
f"the same version: {res1['resource_version']}"
)
return resources
def get_resource_json_obj_from_client(
self,
resource_id: str,
resource_version: Optional[str] = None,
clients: Optional[List[str]] = None,
) -> Dict:
"""
This function returns the resource object from the client with the
given id and version.
:param resource_id: The id of the resource to search for.
:param resource_version: The version of the resource to search for.
:param clients: A list of clients to search through. If None, all
clients are searched.
:return: The resource object as a Python dictionary if found.
If not found, exception is thrown.
"""
# getting all the resources with the given id from the dictionary
resources = self.get_all_resources_by_id(resource_id, clients)
# if no resource with the given id is found, return None
if len(resources) == 0:
raise Exception(f"Resource with ID '{resource_id}' not found.")
resource_to_return = None
if resource_version:
resource_to_return = self._search_version_in_resources(
resources, resource_id, resource_version
)
else:
compatible_resources = (
self._get_resources_compatible_with_gem5_version(resources)
)
if len(compatible_resources) == 0:
resource_to_return = self._sort_resources(resources)[0]
else:
resource_to_return = self._sort_resources(
compatible_resources
)[0]
self._check_resource_version_compatibility(resource_to_return)
return resource_to_return
def _search_version_in_resources(
self, resources: List, resource_id: str, resource_version: str
) -> Dict:
"""
Searches for the resource with the given version. If the resource is
not found, an exception is thrown.
:param resources: A list of resources to search through.
:param resource_version: The version of the resource to search for.
:return: The resource object as a Python dictionary if found.
If not found, None is returned.
"""
return_resource = next(
iter(
[
resource
for resource in resources
if resource["resource_version"] == resource_version
]
),
None,
)
if not return_resource:
raise Exception(
f"Resource {resource_id} with version '{resource_version}'"
" not found.\nResource versions can be found at: "
"https://resources.gem5.org/"
f"resources/{resource_id}/versions"
)
return return_resource
def _get_resources_compatible_with_gem5_version(
self, resources: List, gem5_version: str = core.gem5Version
) -> List:
"""
Returns a list of compatible resources with the current gem5 version.
:param resources: A list of resources to filter.
:return: A list of compatible resources as Python dictionaries.
If no compatible resources are found, the original list of resources
is returned.
"""
compatible_resources = [
resource
for resource in resources
if gem5_version in resource["gem5_versions"]
]
return compatible_resources
def _sort_resources(self, resources: List) -> List:
"""
Sorts the resources by ID.
If the IDs are the same, the resources are sorted by version.
:param resources: A list of resources to sort.
:return: A list of sorted resources.
"""
return sorted(
resources,
key=lambda resource: (
resource["id"].lower(),
StrictVersion(resource["resource_version"]),
),
reverse=True,
)
def _check_resource_version_compatibility(
self, resource: dict, gem5_version: Optional[str] = core.gem5Version
) -> bool:
"""
Checks if the resource is compatible with the gem5 version.
Prints a warning if the resource is not compatible.
:param resource: The resource to check.
:optional param gem5_version: The gem5 version to check
compatibility with.
:return: True if the resource is compatible, False otherwise.
"""
if not resource:
return False
if gem5_version not in resource["gem5_versions"]:
warn(
f"Resource {resource['id']} with version "
f"{resource['resource_version']} is not known to be compatible"
f" with gem5 version {gem5_version}. "
"This may cause problems with your simulation. "
"This resource's compatibility "
"with different gem5 versions can be found here: "
"https://resources.gem5.org"
f"/resources/{resource['id']}/versions"
)
return False
return True

70
src/python/gem5/resources/client_api/jsonclient.py Normal file
View File

@@ -0,0 +1,70 @@
# Copyright (c) 2023 The Regents of the University of California
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met: redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer;
# redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution;
# neither the name of the copyright holders nor the names of its
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
import json
from pathlib import Path
from urllib import request
from typing import Optional, Dict, Union, Type, Tuple, List, Any
from .abstract_client import AbstractClient
from urllib.error import URLError
from m5.util import warn
class JSONClient(AbstractClient):
def __init__(self, path: str):
"""
Initializes a JSON client.
:param path: The path to the Resource, either URL or local.
"""
self.path = path
self.resources = []
if Path(self.path).is_file():
self.resources = json.load(open(self.path))
elif not self._url_validator(self.path):
raise Exception(
f"Resources location '{self.path}' is not a valid path or URL."
)
else:
req = request.Request(self.path)
try:
response = request.urlopen(req)
except URLError as e:
raise Exception(
f"Unable to open Resources location '{self.path}': {e}"
)
self.resources = json.loads(response.read().decode("utf-8"))
def get_resources_by_id(self, resource_id: str) -> List[Dict[str, Any]]:
"""
:param resource_id: The ID of the Resource.
:return: A list of all the Resources with the given ID.
"""
return [
resource
for resource in self.resources
if resource["id"] == resource_id
]

23
src/python/gem5/resources/downloader.py
View File

@@ -41,6 +41,7 @@ from tempfile import gettempdir
from urllib.error import HTTPError
from typing import List, Dict, Set, Optional
from .client import get_resource_json_obj
from .md5_utils import md5_file, md5_dir
from ..utils.progress_bar import tqdm, progress_hook
@@ -398,6 +399,8 @@ def get_resource(
unzip: bool = True,
untar: bool = True,
download_md5_mismatch: bool = True,
resource_version: Optional[str] = None,
clients: Optional[List] = None,
) -> None:
"""
Obtains a gem5 resource and stored it to a specified location. If the
@@ -419,6 +422,13 @@ def get_resource(
will delete this local resource and re-download it if this parameter is
True. True by default.
:param resource_version: The version of the resource to be obtained. If
None, the latest version of the resource compatible with the working
directory's gem5 version will be obtained. None by default.
:param clients: A list of clients to use when obtaining the resource. If
None, all clients will be used. None by default.
:raises Exception: An exception is thrown if a file is already present at
`to_path` but it does not have the correct md5 sum. An exception will also
be thrown is a directory is present at `to_path`
@@ -430,11 +440,13 @@ def get_resource(
# minutes.Most resources should be downloaded and decompressed in this
# timeframe, even on the most constrained of systems.
with FileLock(f"{to_path}.lock", timeout=900):
resource_json = get_resources_json_obj(resource_name)
resource_json = get_resource_json_obj(
resource_name,
resource_version=resource_version,
clients=clients,
)
if os.path.exists(to_path):
if os.path.isfile(to_path):
md5 = md5_file(Path(to_path))
else:
@@ -495,9 +507,8 @@ def get_resource(
)
)
# Get the URL. The URL may contain '{url_base}' which needs replaced
# with the correct value.
url = resource_json["url"].format(url_base=_get_url_base())
# Get the URL.
url = resource_json["url"]
_download(url=url, download_to=download_dest)
print(f"Finished downloading resource '{resource_name}'.")

155
src/python/gem5/resources/resource.py
View File

@@ -29,13 +29,15 @@ import os
from pathlib import Path
from m5.util import warn, fatal
from .downloader import get_resource, get_resources_json_obj
from .downloader import get_resource
from .looppoint import LooppointCsvLoader, LooppointJsonLoader
from ..isas import ISA, get_isa_from_str
from typing import Optional, Dict, Union, Type, Tuple, List
from .client import get_resource_json_obj
"""
Resources are items needed to run a simulation, such as a disk image, kernel,
or binary. The gem5 project provides pre-built resources, with sources, at
@@ -67,18 +69,20 @@ class AbstractResource:
def __init__(
self,
resource_version: Optional[str] = None,
local_path: Optional[str] = None,
documentation: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
):
"""
:param local_path: The path on the host system where this resource is
located.
:param documentation: Documentation describing this resource. Not a
:param description: Description describing this resource. Not a
required parameter. By default is None.
:param source: The source (as in "source code") for this resource. This
string should navigate users to where the source for this resource
may be found. Not a required parameter. By default is None.
:param resource_version: Version of the resource itself.
"""
if local_path and not os.path.exists(local_path):
@@ -88,16 +92,21 @@ class AbstractResource:
)
self._local_path = local_path
self._documentation = documentation
self._description = description
self._source = source
self._version = resource_version
def get_resource_version(self) -> str:
"""Returns the version of the resource."""
return self._version
def get_local_path(self) -> Optional[str]:
"""Returns the local path of the resource."""
return self._local_path
def get_documentation(self) -> Optional[str]:
"""Returns documentation associated with this resource."""
return self._documentation
def get_description(self) -> Optional[str]:
"""Returns description associated with this resource."""
return self._description
def get_source(self) -> Optional[str]:
"""Returns information as to where the source for this resource may be
@@ -112,7 +121,8 @@ class FileResource(AbstractResource):
def __init__(
self,
local_path: str,
documentation: Optional[str] = None,
resource_version: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
**kwargs,
):
@@ -123,8 +133,9 @@ class FileResource(AbstractResource):
super().__init__(
local_path=local_path,
documentation=documentation,
description=description,
source=source,
resource_version=resource_version,
)
@@ -134,11 +145,11 @@ class DirectoryResource(AbstractResource):
def __init__(
self,
local_path: str,
documentation: Optional[str] = None,
resource_version: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
**kwargs,
):
if not os.path.isdir(local_path):
raise Exception(
f"DirectoryResource path specified, {local_path}, is not a "
@@ -147,8 +158,9 @@ class DirectoryResource(AbstractResource):
super().__init__(
local_path=local_path,
documentation=documentation,
description=description,
source=source,
resource_version=resource_version,
)
@@ -158,15 +170,17 @@ class DiskImageResource(FileResource):
def __init__(
self,
local_path: str,
documentation: Optional[str] = None,
resource_version: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
root_partition: Optional[str] = None,
**kwargs,
):
super().__init__(
local_path=local_path,
documentation=documentation,
description=description,
source=source,
resource_version=resource_version,
)
self._root_partition = root_partition
@@ -181,15 +195,17 @@ class BinaryResource(FileResource):
def __init__(
self,
local_path: str,
documentation: Optional[str] = None,
resource_version: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
architecture: Optional[Union[ISA, str]] = None,
**kwargs,
):
super().__init__(
local_path=local_path,
documentation=documentation,
description=description,
source=source,
resource_version=resource_version,
)
self._architecture = None
@@ -210,16 +226,18 @@ class BootloaderResource(BinaryResource):
def __init__(
self,
local_path: str,
documentation: Optional[str] = None,
resource_version: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
architecture: Optional[Union[ISA, str]] = None,
**kwargs,
):
super().__init__(
local_path=local_path,
documentation=documentation,
description=description,
architecture=architecture,
source=source,
resource_version=resource_version,
)
@@ -229,14 +247,16 @@ class GitResource(DirectoryResource):
def __init__(
self,
local_path: str,
documentation: Optional[str] = None,
resource_version: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
**kwargs,
):
super().__init__(
local_path=local_path,
documentation=documentation,
description=description,
source=source,
resource_version=resource_version,
)
@@ -246,16 +266,18 @@ class KernelResource(BinaryResource):
def __init__(
self,
local_path: str,
documentation: Optional[str] = None,
resource_version: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
architecture: Optional[Union[ISA, str]] = None,
**kwargs,
):
super().__init__(
local_path=local_path,
documentation=documentation,
description=description,
source=source,
architecture=architecture,
resource_version=resource_version,
)
@@ -270,14 +292,16 @@ class CheckpointResource(DirectoryResource):
def __init__(
self,
local_path: str,
documentation: Optional[str] = None,
resource_version: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
**kwargs,
):
super().__init__(
local_path=local_path,
documentation=documentation,
description=description,
source=source,
resource_version=resource_version,
)
@@ -290,12 +314,13 @@ class SimpointResource(AbstractResource):
def __init__(
self,
resource_version: Optional[str] = None,
simpoint_interval: int = None,
simpoint_list: List[int] = None,
weight_list: List[float] = None,
warmup_interval: int = 0,
workload_name: Optional[str] = None,
documentation: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
local_path: Optional[str] = None,
**kwargs,
@@ -314,8 +339,9 @@ class SimpointResource(AbstractResource):
super().__init__(
local_path=local_path,
documentation=documentation,
description=description,
source=source,
resource_version=resource_version,
)
self._weight_list = weight_list
@@ -402,15 +428,17 @@ class LooppointCsvResource(FileResource, LooppointCsvLoader):
def __init__(
self,
local_path: str,
documentation: Optional[str] = None,
resource_version: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
**kwargs,
):
FileResource.__init__(
self,
local_path=local_path,
documentation=documentation,
description=description,
source=source,
resource_version=resource_version,
)
LooppointCsvLoader.__init__(self, pinpoints_file=Path(local_path))
@@ -419,16 +447,18 @@ class LooppointJsonResource(FileResource, LooppointJsonLoader):
def __init__(
self,
local_path: str,
resource_version: Optional[str] = None,
region_id: Optional[Union[str, int]] = None,
documentation: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
**kwargs,
):
FileResource.__init__(
self,
local_path=local_path,
documentation=documentation,
description=description,
source=source,
resource_version=resource_version,
)
LooppointJsonLoader.__init__(
self, looppoint_file=local_path, region_id=region_id
@@ -446,8 +476,9 @@ class SimpointDirectoryResource(SimpointResource):
weight_file: str,
simpoint_interval: int,
warmup_interval: int,
resource_version: Optional[str] = None,
workload_name: Optional[str] = None,
documentation: Optional[str] = None,
description: Optional[str] = None,
source: Optional[str] = None,
**kwargs,
):
@@ -478,8 +509,9 @@ class SimpointDirectoryResource(SimpointResource):
warmup_interval=warmup_interval,
workload_name=workload_name,
local_path=local_path,
documentation=documentation,
description=description,
source=source,
resource_version=resource_version,
)
def get_simpoint_file(self) -> Path:
@@ -522,9 +554,11 @@ class SimpointDirectoryResource(SimpointResource):
def obtain_resource(
resource_name: str,
resource_id: str,
resource_directory: Optional[str] = None,
download_md5_mismatch: bool = True,
resource_version: Optional[str] = None,
clients: Optional[List] = None,
) -> AbstractResource:
"""
This function primarily serves as a factory for resources. It will return
@@ -544,10 +578,16 @@ def obtain_resource(
have the correct md5 value, the resoruce will be deleted and
re-downloaded if this value is True. Otherwise an exception will be
thrown. True by default.
:param resource_version: Version of the resource itself.
Not a required parameter. None by default.
:param clients: A list of clients to search for the resource. If this
parameter is not set, it will default search all clients.
"""
# Obtain the JSON resource entry for this resource
resource_json = get_resources_json_obj(resource_name)
# Obtain the resource object entry for this resource
resource_json = get_resource_json_obj(
resource_id, resource_version=resource_version, clients=clients
)
to_path = None
# If the "url" field is specified, the resoruce must be downloaded.
@@ -580,38 +620,37 @@ def obtain_resource(
os.makedirs(resource_directory, exist_ok=True)
# This is the path to which the resource is to be stored.
to_path = os.path.join(resource_directory, resource_name)
to_path = os.path.join(resource_directory, resource_id)
# Download the resource if it does not already exist.
get_resource(
resource_name=resource_name,
to_path=os.path.join(resource_directory, resource_name),
resource_name=resource_id,
to_path=os.path.join(resource_directory, resource_id),
download_md5_mismatch=download_md5_mismatch,
resource_version=resource_version,
clients=clients,
)
# Obtain the type from the JSON. From this we will determine what subclass
# of `AbstractResource` we are to create and return.
resources_type = resource_json["type"]
resources_category = resource_json["category"]
if resources_type == "resource":
if resources_category == "resource":
# This is a stop-gap measure to ensure to work with older versions of
# the "resource.json" file. These should be replaced with their
# respective specializations ASAP and this case removed.
if (
"additional_metadata" in resource_json
and "root_partition" in resource_json["additional_metadata"]
):
if "root_partition" in resource_json:
# In this case we should return a DiskImageResource.
root_partition = resource_json["additional_metadata"][
"root_partition"
]
root_partition = resource_json["root_partition"]
return DiskImageResource(
local_path=to_path, root_partition=root_partition
local_path=to_path,
root_partition=root_partition,
**resource_json,
)
return CustomResource(local_path=to_path)
assert resources_type in _get_resource_json_type_map
resource_class = _get_resource_json_type_map[resources_type]
assert resources_category in _get_resource_json_type_map
resource_class = _get_resource_json_type_map[resources_category]
# Once we know what AbstractResource subclass we are using, we create it.
# The fields in the JSON object are assumed to map like-for-like to the
@@ -694,6 +733,7 @@ class CustomDiskImageResource(DiskImageResource):
def __init__(
self,
local_path: str,
resource_version: Optional[str] = None,
root_partition: Optional[str] = None,
metadata: Dict = {},
):
@@ -702,6 +742,7 @@ class CustomDiskImageResource(DiskImageResource):
:param root_partition: The root disk partition to use.
:param metadata: Metadata for the resource. **Warning:** As of "
"v22.1.1, this parameter is not used.
:param resource_version: Version of the resource itself.
"""
warn(
"The `CustomDiskImageResource` class is deprecated. Please use "
@@ -713,13 +754,19 @@ class CustomDiskImageResource(DiskImageResource):
"`CustomDiskImageResource` constructor. This parameter is not "
"used."
)
super().__init__(local_path=local_path, root_partition=root_partition)
super().__init__(
local_path=local_path,
root_partition=root_partition,
resource_version=resource_version,
)
def Resource(
resource_name: str,
resource_id: str,
resource_directory: Optional[str] = None,
download_md5_mismatch: bool = True,
resource_version: Optional[str] = None,
clients: Optional[List[str]] = None,
) -> AbstractResource:
"""
This function was created to maintain backwards compability for v21.1.0
@@ -737,9 +784,11 @@ def Resource(
)
return obtain_resource(
resource_name=resource_name,
resource_id=resource_id,
resource_directory=resource_directory,
download_md5_mismatch=download_md5_mismatch,
resource_version=resource_version,
clients=clients,
)

27
src/python/gem5/resources/workload.py
View File

@@ -24,10 +24,10 @@
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
from .downloader import get_workload_json_obj
from .resource import obtain_resource
from .client import get_resource_json_obj
from typing import Dict, Any, Optional
from typing import Dict, Any, List, Optional
class AbstractWorkload:
@@ -155,7 +155,11 @@ class Workload(AbstractWorkload):
"""
def __init__(
self, workload_name: str, resource_directory: Optional[str] = None
self,
workload_name: str,
resource_directory: Optional[str] = None,
resource_version: Optional[str] = None,
clients: Optional[List] = None,
) -> None:
"""
This constructor will load the workload details from the workload with
@@ -167,13 +171,13 @@ class Workload(AbstractWorkload):
```json
{
"type" : "workload",
"name" : "x86-ubuntu-18.04-echo-hello",
"documentation" : "Description of workload here",
"category" : "workload",
"id" : "x86-ubuntu-18.04-echo-hello",
"description" : "Description of workload here",
"function" : "set_kernel_disk_workload",
"resources" : {
"kernel" : "x86-linux-kernel-5.4.49",
"disk_image" : "x86-ubuntu-18.04-img"
"disk-image" : "x86-ubuntu-18.04-img"
},
"additional_params" : {
"readfile_contents" : "m5_exit; echo 'hello'; m5_exit"
@@ -187,7 +191,7 @@ class Workload(AbstractWorkload):
```python
board.set_kernel_disk_workload(
kernel = Resource("x86-linux-kernel-5.4.49"),
disk_image = Resource("x86-ubuntu-18.04-img"),
disk-image = Resource("x86-ubuntu-18.04-img"),
readfile_contents = "m5_exit; echo 'hello'; m5_exit",
)
```
@@ -198,7 +202,12 @@ class Workload(AbstractWorkload):
any resources should be download and accessed from. If None, a default
location will be used. None by default.
"""
workload_json = get_workload_json_obj(workload_name=workload_name)
workload_json = get_resource_json_obj(
workload_name,
resource_version=resource_version,
clients=clients,
)
func = workload_json["function"]
assert isinstance(func, str)

12
src/python/gem5/utils/simpoint.py
View File

@@ -83,15 +83,11 @@ class SimPoint:
simpoint_file_path = simpoint_directory.get_simpoint_file()
weight_file_path = simpoint_resource.get_weight_file()
simpoint_interval = (
simpoint_resource.get_metadata()
.get("additional_metadata")
.get("simpoint_interval")
simpoint_interval = simpoint_resource.get_metadata().get(
"simpoint_interval"
)
warmup_interval = (
simpoint_resource.get_metadata()
.get("additional_metadata")
.get("warmup_interval")
warmup_interval = simpoint_resource.get_metadata().get(
"warmup_interval"
)
self._simpoint_interval = simpoint_interval

39
src/python/gem5_default_config.py Normal file
View File

@@ -0,0 +1,39 @@
# Copyright (c) 2023 The Regents of the University of California
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met: redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer;
# redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution;
# neither the name of the copyright holders nor the names of its
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
config = {
"sources": {
"gem5-resources": {
"dataSource": "gem5-vision",
"database": "gem5-vision",
"collection": "versions_test",
"url": "https://data.mongodb-api.com/app/data-ejhjf/endpoint/data/v1",
"authUrl": "https://realm.mongodb.com/api/client/v2.0/app/data-ejhjf/auth/providers/api-key/login",
"apiKey": "OIi5bAP7xxIGK782t8ZoiD2BkBGEzMdX3upChf9zdCxHSnMoiTnjI22Yw5kOSgy9",
"isMongo": True,
}
}
}