derek/gem5
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

stdlib: Refactor gem5 Vision/gem5-resources code

Browse Source 
This patch includes several changes to the gem5 tools interface to the
gem5-resources infrastructure. These are:

* The old download and JSON query functions have been removed from the
  downloader module. These functions were used for directly downloading
  and inspecting the resource JSON file, hosted at
  https://resources.gem5.org/resources. This information is now obtained
  via `gem5.client`. If a resources JSON file is specified as a client,
  it should conform to the new schema:
  https//resources.gem5.org/gem5-resources-schema.json. The old schema
  (pre-v23.0) is no longer valid. Tests have been updated to reflect
  this change. Those which tested these old functions have been removed.
* Unused imports have been removed.
* For the resource query functions, and those tasked with obtaining the
  resources, the parameter `gem5_version` has been added. In all cases
  it does the same thing:
    * It will filter results based on compatibility to the
      `gem5_version` specified. If no resources are compatible the
      latest version of that resource is chosen (though a warning is
      thrown).
    * By default it is set to the current gem5 version.
    * It is optional. If `None`, this filtering functionality is not
      carried out.
    * Tests have been updated to fix the version to “develop” so the
      they do not break between versions.
* The `gem5_version` parameters will filter using a logic which will
  base compatibility on the specificity of the gem5-version specified in
  a resource’s data. If a resource has a compatible gem5-version of
  “v18.4” it will be compatible with any minor/hotfix version within the
  v18.4 release (this can be seen as matching on “v18.4.*.*”.) Likewise,
  if a resource has a compatible gem5-version of “v18.4.1” then it’s
  only compatible with the v18.4.1 release but any of it’s hot fix
  releases (“v18.4.1.*”).
* The ‘list_resources’ function has been updated to use the
  “gem5.client” APIs to get resource information from the clients
  (MongoDB or a JSON file). This has been designed to remain backwards
  compatible to as much as is possible, though, due to schema changes,
  the function does search across all versions of gem5.
* `get_resources` function was added to the `AbstractClient`. This is a
   more general function than `get_resource_by_id`. It was
  primarily created to handle the `list_resources` update but is a
  useful update to the API. The `get_resource_by_id` function has been
  altered to function as a wrapped to the `get_resources` function.
* Removed “GEM5_RESOURCE_JSON” code has been removed. This is no longer
  used.
* Tests have been cleaned up a little bit to be easier to read.
* Some docstrings have been updated.

Things that are left TODO with this code:

* The client_wrapper/client/abstract_client abstractions are rather
  pointless. In particular the client_wrapper and client classes could
  be merged.
* The downloader module no longer does much and should have its
  functions merged into other modules.
* With the addition of the `get_resources` function, much of the code in
  the `AbstractClient` could be simplified.

Change-Id: I0ce48e88b93a2b9db53d4749861fa0b5f9472053
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/71506
Reviewed-by: Kunal Pai <kunpai@ucdavis.edu>
Maintainer: Jason Lowe-Power <power.jg@gmail.com>
Tested-by: kokoro <noreply+kokoro@google.com>
(cherry picked from commit 82587ce71b)
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/71739
Reviewed-by: Bobby Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby Bruce <bbruce@ucdavis.edu>
This commit is contained in:
Bobby R. Bruce
2023-06-12 14:09:15 -07:00
committed by Bobby Bruce
parent b22afde3be
commit aff1e7b64c
21 changed files with 445 additions and 603 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
tests/gem5/configs/download_check.py
View File

@@ -26,10 +26,11 @@
from gem5.resources.downloader import (
list_resources,
get_resources_json_obj,
get_resource,
)
from gem5.resources.client import get_resource_json_obj
from gem5.resources.md5_utils import md5
import os
@@ -51,6 +52,15 @@ parser.add_argument(
"checked",
)
parser.add_argument(
"--gem5-version",
type=str,
required=False,
help="The gem5 version to check the resources against. Resources not "
"compatible with this version will be ignored. If not set, no "
"compatibility tests are performed.",
)
parser.add_argument(
"--download-directory",
type=str,
@@ -67,39 +77,59 @@ if not Path(args.download_directory).exists():
ids = args.ids
resource_list = list_resources(gem5_version=args.gem5_version)
if len(ids) == 0:
ids = list_resources()
ids = resource_list
# We log all the errors as they occur then dump them at the end. This means we
# can be aware of all download errors in a single failure.
errors = str()
for id in ids:
if id not in list_resources():
if id not in resource_list:
errors += (
f"Resource with ID '{id}' not found in "
+ f"`list_resources()`.{os.linesep}"
)
continue
resource_json = get_resources_json_obj(id)
download_path = os.path.join(args.download_directory, id)
try:
get_resource(resource_name=id, to_path=download_path)
except Exception as e:
errors += f"Failure to download resource '{id}'.{os.linesep}"
errors += f"Exception message:{os.linesep}{str(e)}"
errors += f"{os.linesep}{os.linesep}"
continue
for resource_version in ids[id]:
if md5(Path(download_path)) != resource_json["md5sum"]:
errors += (
f"Downloaded resource '{id}' md5 "
+ f"({md5(Path(download_path))}) differs to that in the "
+ f"JSON ({resource_json['md5sum']}).{os.linesep}"
resource_json = get_resource_json_obj(
resource_id=id,
resource_version=resource_version,
gem5_version=args.gem5_version,
)
if resource_json["category"] == "workload":
# Workloads are not downloaded as part of this test.
continue
download_path = os.path.join(
args.download_directory, f"{id}-v{resource_version}"
)
try:
get_resource(
resource_name=id,
resource_version=resource_version,
gem5_version=args.gem5_version,
to_path=download_path,
)
except Exception as e:
errors += (
f"Failure to download resource '{id}', "
+ f"v{resource_version}.{os.linesep}"
)
errors += f"Exception message:{os.linesep}{str(e)}"
errors += f"{os.linesep}{os.linesep}"
continue
# Remove the downloaded resource.
if md5(Path(download_path)) != resource_json["md5sum"]:
errors += (
f"Downloaded resource '{id}' md5 "
+ f"({md5(Path(download_path))}) differs to that recorded in "
+ f" gem5-resources ({resource_json['md5sum']}).{os.linesep}"
)
# Remove the downloaded resource.
shutil.rmtree(download_path, ignore_errors=True)
# If errors exist, raise an exception highlighting them.

109
tests/pyunit/stdlib/resources/pyunit_client_wrapper_checks.py
View File

@@ -25,13 +25,9 @@
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
import unittest
from gem5.isas import ISA
from gem5.resources.client import get_resource_json_obj
import gem5.resources.client
from gem5.resources.client_api.client_wrapper import ClientWrapper
from typing import Dict
from unittest.mock import patch
from unittest import mock
import json
from urllib.error import HTTPError
import io
@@ -62,23 +58,8 @@ mock_config_mongo = {
},
}
mock_config_combined = {
"sources": {
"gem5-resources": {
"dataSource": "gem5-vision",
"database": "gem5-vision",
"collection": "versions_test",
"url": "https://data.mongodb-api.com/app/data-ejhjf/endpoint/data/v1",
"authUrl": "https://realm.mongodb.com/api/client/v2.0/app/data-ejhjf/auth/providers/api-key/login",
"apiKey": "OIi5bAP7xxIGK782t8ZoiD2BkBGEzMdX3upChf9zdCxHSnMoiTnjI22Yw5kOSgy9",
"isMongo": True,
},
"baba": {
"url": mock_json_path,
"isMongo": False,
},
},
}
mock_config_combined = mock_config_mongo
mock_config_combined["sources"]["baba"] = mock_config_json["sources"]["baba"]
mock_json = {}
@@ -145,12 +126,12 @@ class ClientWrapperTestSuite(unittest.TestCase):
def test_get_resource_json_obj(self):
# Test that the resource object is correctly returned
resource = "this-is-a-test-resource"
resource = get_resource_json_obj(resource)
resource = get_resource_json_obj(resource, gem5_version="develop")
self.assertEqual(resource["id"], "this-is-a-test-resource")
self.assertEqual(resource["resource_version"], "2.0.0")
self.assertEqual(resource["resource_version"], "1.1.0")
self.assertEqual(resource["category"], "binary")
self.assertEqual(
resource["description"], "This is a test resource but double newer"
resource["description"], "This is a test resource but newer"
)
self.assertEqual(
resource["source_url"],
@@ -167,7 +148,9 @@ class ClientWrapperTestSuite(unittest.TestCase):
resource_id = "test-id"
client = "invalid"
with self.assertRaises(Exception) as context:
get_resource_json_obj(resource_id, clients=[client])
get_resource_json_obj(
resource_id, clients=[client], gem5_version="develop"
)
self.assertTrue(
f"Client: {client} does not exist" in str(context.exception)
)
@@ -181,7 +164,9 @@ class ClientWrapperTestSuite(unittest.TestCase):
resource_id = "this-is-a-test-resource"
resource_version = "1.0.0"
resource = get_resource_json_obj(
resource_id, resource_version=resource_version
resource_id,
resource_version=resource_version,
gem5_version="develop",
)
self.assertEqual(resource["id"], "this-is-a-test-resource")
self.assertEqual(resource["resource_version"], "1.0.0")
@@ -200,17 +185,18 @@ class ClientWrapperTestSuite(unittest.TestCase):
@patch("urllib.request.urlopen", side_effect=mocked_requests_post)
def test_get_resource_json_obj_1(self, mock_get):
resource = "x86-ubuntu-18.04-img"
resource = get_resource_json_obj(resource)
resource = get_resource_json_obj(resource, gem5_version="develop")
self.assertEqual(resource["id"], "x86-ubuntu-18.04-img")
self.assertEqual(resource["resource_version"], "1.1.0")
self.assertEqual(resource["resource_version"], "2.0.0")
self.assertEqual(resource["category"], "disk-image")
self.assertEqual(
resource["description"],
"A disk image containing Ubuntu 18.04 for x86. This image will run an `m5 readfile` instruction after booting. If no script file is specified an `m5 exit` instruction will be executed.",
"This is a test resource",
)
self.assertEqual(
resource["source_url"],
"https://github.com/gem5/gem5-resources/tree/develop/src/x86-ubuntu",
"https://github.com/gem5/gem5-resources/tree/develop/"
"src/x86-ubuntu",
)
self.assertEqual(resource["architecture"], "X86")
@@ -227,6 +213,7 @@ class ClientWrapperTestSuite(unittest.TestCase):
resource_id,
resource_version=resource_version,
clients=["gem5-resources"],
gem5_version="develop",
)
self.assertEqual(resource["id"], "x86-ubuntu-18.04-img")
self.assertEqual(resource["resource_version"], "1.0.0")
@@ -246,7 +233,9 @@ class ClientWrapperTestSuite(unittest.TestCase):
def test_get_resource_json_obj_with_id_invalid_mongodb(self, mock_get):
resource_id = "invalid-id"
with self.assertRaises(Exception) as context:
get_resource_json_obj(resource_id, clients=["gem5-resources"])
get_resource_json_obj(
resource_id, clients=["gem5-resources"], gem5_version="develop"
)
self.assertTrue(
"Resource with ID 'invalid-id' not found."
in str(context.exception)
@@ -267,12 +256,13 @@ class ClientWrapperTestSuite(unittest.TestCase):
resource_id,
resource_version=resource_version,
clients=["gem5-resources"],
gem5_version="develop",
)
self.assertTrue(
f"Resource x86-ubuntu-18.04-img with version '2.5.0'"
" not found.\nResource versions can be found at: "
f"https://resources.gem5.org/resources/x86-ubuntu-18.04-img/versions"
in str(context.exception)
"https://resources.gem5.org/resources/x86-ubuntu-18.04-img/"
"versions" in str(context.exception)
)
@patch(
@@ -286,12 +276,13 @@ class ClientWrapperTestSuite(unittest.TestCase):
get_resource_json_obj(
resource_id,
resource_version=resource_version,
gem5_version="develop",
)
self.assertTrue(
f"Resource this-is-a-test-resource with version '2.5.0'"
"Resource this-is-a-test-resource with version '2.5.0'"
" not found.\nResource versions can be found at: "
f"https://resources.gem5.org/resources/this-is-a-test-resource/versions"
in str(context.exception)
"https://resources.gem5.org/resources/this-is-a-test-resource/"
"versions" in str(context.exception)
)
@patch(
@@ -308,11 +299,13 @@ class ClientWrapperTestSuite(unittest.TestCase):
resource_id_mongo,
resource_version=resource_version_mongo,
clients=["gem5-resources"],
gem5_version="develop",
)
resource_json = get_resource_json_obj(
resource_id_json,
resource_version=resource_version_json,
clients=["baba"],
gem5_version="develop",
)
self.assertEqual(resource_mongo["id"], "x86-ubuntu-18.04-img")
self.assertEqual(resource_mongo["resource_version"], "1.0.0")
@@ -322,7 +315,8 @@ class ClientWrapperTestSuite(unittest.TestCase):
)
self.assertEqual(
resource_mongo["source_url"],
"https://github.com/gem5/gem5-resources/tree/develop/src/x86-ubuntu",
"https://github.com/gem5/gem5-resources/tree/develop/src/"
"x86-ubuntu",
)
self.assertEqual(resource_mongo["architecture"], "X86")
@@ -347,6 +341,7 @@ class ClientWrapperTestSuite(unittest.TestCase):
resource_id = "simpoint-resource"
resource = get_resource_json_obj(
resource_id,
gem5_version="develop",
)
self.assertEqual(resource["id"], resource_id)
self.assertEqual(resource["resource_version"], "0.2.0")
@@ -371,6 +366,7 @@ class ClientWrapperTestSuite(unittest.TestCase):
resource_id = "x86-ubuntu-18.04-img"
resource_json = get_resource_json_obj(
resource_id,
gem5_version="develop",
)
self.assertEqual(resource_json["id"], "x86-ubuntu-18.04-img")
@@ -378,8 +374,7 @@ class ClientWrapperTestSuite(unittest.TestCase):
self.assertEqual(resource_json["category"], "disk-image")
resource_json = get_resource_json_obj(
resource_id,
resource_version="1.0.0",
resource_id, resource_version="1.0.0", gem5_version="develop"
)
self.assertEqual(resource_json["id"], "x86-ubuntu-18.04-img")
@@ -396,6 +391,7 @@ class ClientWrapperTestSuite(unittest.TestCase):
with self.assertRaises(Exception) as context:
get_resource_json_obj(
resource_id,
gem5_version="develop",
)
self.assertTrue(
f"Resource {resource_id} has multiple resources with"
@@ -428,6 +424,7 @@ class ClientWrapperTestSuite(unittest.TestCase):
with contextlib.redirect_stderr(f):
get_resource_json_obj(
resource_id,
gem5_version="develop",
)
self.assertTrue(
"Error getting resources from client gem5-resources:"
@@ -440,21 +437,7 @@ class ClientWrapperTestSuite(unittest.TestCase):
@patch(
"gem5.resources.client.clientwrapper",
ClientWrapper(
{
"sources": {
"gem5-resources": {
"dataSource": "gem5-vision",
"database": "gem5-vision",
"collection": "versions_test",
"url": "https://data.mongodb-api.com/app/data-ejhjf/endpoint/data/v",
"authUrl": "https://realm.mongodb.com/api/client/v2.0/app/data-ejhjf/auth/providers/api-key/login",
"apiKey": "OIi5bAP7xxIGK782t8ZoiD2BkBGEzMdX3upChf9zdCxHSnMoiTnjI22Yw5kOSgy9",
"isMongo": True,
}
},
}
),
ClientWrapper(mock_config_mongo),
)
@patch("urllib.request.urlopen", side_effect=mocked_requests_post)
def test_invalid_url(self, mock_get):
@@ -464,6 +447,7 @@ class ClientWrapperTestSuite(unittest.TestCase):
with contextlib.redirect_stderr(f):
get_resource_json_obj(
resource_id,
gem5_version="develop",
)
self.assertTrue(
"Error getting resources from client gem5-resources:"
@@ -476,21 +460,7 @@ class ClientWrapperTestSuite(unittest.TestCase):
@patch(
"gem5.resources.client.clientwrapper",
ClientWrapper(
{
"sources": {
"gem5-resources": {
"dataSource": "gem5-vision",
"database": "gem5-vision",
"collection": "versions_test",
"url": "https://data.mongodb-api.com/app/data-ejhjf/endpoint/data/v1",
"authUrl": "https://realm.mongodb.com/api/client/v2.0/app/data-ejhjf/auth/providers/api-key/login",
"apiKey": "OIi5bAP7xxIGK782t8ZoiD2BkBGEzMdX3upChf9zdCxHSnMoiTnjI22Yw5kOSgy9",
"isMongo": True,
}
},
}
),
ClientWrapper(mock_config_mongo),
)
@patch("urllib.request.urlopen", side_effect=mocked_requests_post)
def test_invalid_url(self, mock_get):
@@ -500,6 +470,7 @@ class ClientWrapperTestSuite(unittest.TestCase):
with contextlib.redirect_stderr(f):
get_resource_json_obj(
resource_id,
gem5_version="develop",
)
self.assertTrue(
"Error getting resources from client gem5-resources:"

37
tests/pyunit/stdlib/resources/pyunit_downloader_checks.py → tests/pyunit/stdlib/resources/pyunit_json_client_checks.py
View File

@@ -30,15 +30,11 @@ import os
from typing import Dict
import json
from gem5.resources.downloader import (
_get_resources_json_at_path,
_get_resources_json,
_resources_json_version_required,
)
from gem5.resources.client_api.jsonclient import JSONClient
class ResourceDownloaderTestSuite(unittest.TestCase):
"""Test cases for gem5.resources.downloader"""
class JSONClientTestSuite(unittest.TestCase):
"""Test cases for gem5.resources.client_api.jsonclient"""
@classmethod
def setUpClass(cls) -> str:
@@ -142,12 +138,9 @@ class ResourceDownloaderTestSuite(unittest.TestCase):
file.close()
cls.file_path = file.name
os.environ["GEM5_RESOURCE_JSON"] = cls.file_path
@classmethod
def tearDownClass(cls) -> None:
os.remove(cls.file_path)
del os.environ["GEM5_RESOURCE_JSON"]
def verify_json(self, json: Dict) -> None:
"""
@@ -167,32 +160,22 @@ class ResourceDownloaderTestSuite(unittest.TestCase):
self.assertEquals("test-version", json[3]["id"])
def test_get_resources_json_at_path(self) -> None:
# Tests the gem5.resources.downloader._get_resources_json_at_path()
# function.
# Tests JSONClient.get_resources_json()
json = _get_resources_json_at_path(path=self.file_path)
self.verify_json(json=json)
def test_get_resources_json(self) -> None:
# Tests the gem5.resources.downloader._get_resources_json() function.
json = _get_resources_json()
client = JSONClient(path=self.file_path)
json = client.get_resources_json()
self.verify_json(json=json)
def test_get_resources_json_invalid_url(self) -> None:
# Tests the gem5.resources.downloader._get_resources_json() function in
# case where an invalid url is passed as the URL/PATH of the
# resources.json file.
# Tests the JSONClient.get_resources_json() function in case where an
# invalid url is passed as the URL/PATH of the resources JSON file.
path = "NotAURLorFilePath"
os.environ["GEM5_RESOURCE_JSON"] = path
with self.assertRaises(Exception) as context:
_get_resources_json()
client = JSONClient(path=path)
json = client.get_resources_json()
self.assertTrue(
f"Resources location '{path}' is not a valid path or URL."
in str(context.exception)
)
# Set back to the old path
os.environ["GEM5_RESOURCE_JSON"] = self.file_path

47
tests/pyunit/stdlib/resources/pyunit_obtain_resources_check.py
View File

@@ -30,12 +30,7 @@ import io
import contextlib
from pathlib import Path
from gem5.resources.resource import *
from gem5.resources.looppoint import (
LooppointCsvLoader,
LooppointJsonLoader,
)
from gem5.resources.resource import obtain_resource, BinaryResource
from gem5.isas import ISA
@@ -61,24 +56,6 @@ mock_config_json = {
new=ClientWrapper(mock_config_json),
)
class TestObtainResourcesCheck(unittest.TestCase):
@classmethod
def setUpClass(cls):
"""Prior to running the suite we set the resource directory to
"ref/resource-specialization.json"
"""
os.environ["GEM5_RESOURCE_JSON"] = os.path.join(
os.path.realpath(os.path.dirname(__file__)),
"refs",
"obtain-resource.json",
)
@classmethod
def tearDownClass(cls) -> None:
"""After running the suite we unset the gem5-resource JSON file, as to
not interfere with others tests.
"""
del os.environ["GEM5_RESOURCE_JSON"]
def get_resource_dir(cls) -> str:
"""To ensure the resources are cached to the same directory as all
other tests, this function returns the location of the testing
@@ -99,26 +76,27 @@ class TestObtainResourcesCheck(unittest.TestCase):
resource = obtain_resource(
resource_id="test-binary-resource",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertEquals("2.5.0", resource.get_resource_version())
self.assertEquals("1.7.0", resource.get_resource_version())
self.assertIsInstance(resource, BinaryResource)
# self.assertIn(gem5Version, resource.get_gem5_versions())
self.assertEquals("test description", resource.get_description())
self.assertEquals(
"test description v1.7.0", resource.get_description()
)
self.assertEquals("src/test-source", resource.get_source())
self.assertEquals(ISA.ARM, resource.get_architecture())
def test_obtain_resources_with_version_compatible(self):
gem5Version = core.gem5Version
resource = obtain_resource(
resource_id="test-binary-resource",
resource_directory=self.get_resource_dir(),
resource_version="1.7.0",
resource_version="1.5.0",
gem5_version="develop",
)
self.assertEquals("1.7.0", resource.get_resource_version())
self.assertEquals("1.5.0", resource.get_resource_version())
self.assertIsInstance(resource, BinaryResource)
# self.assertIn(gem5Version, resource.get_gem5_versions())
self.assertEquals(
"test description v1.7.0", resource.get_description()
"test description for 1.5.0", resource.get_description()
)
self.assertEquals("src/test-source", resource.get_source())
self.assertEquals(ISA.ARM, resource.get_architecture())
@@ -143,6 +121,7 @@ class TestObtainResourcesCheck(unittest.TestCase):
resource_id="test-binary-resource",
resource_directory=self.get_resource_dir(),
resource_version="1.5.0",
gem5_version="develop",
)
self.assertEquals("1.5.0", resource.get_resource_version())
self.assertIsInstance(resource, BinaryResource)
@@ -157,6 +136,7 @@ class TestObtainResourcesCheck(unittest.TestCase):
obtain_resource(
resource_id="invalid-id",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertTrue(
"Resource with ID 'invalid-id' not found."
@@ -169,6 +149,7 @@ class TestObtainResourcesCheck(unittest.TestCase):
resource_id="invalid-id",
resource_directory=self.get_resource_dir(),
resource_version="1.7.0",
gem5_version="develop",
)
self.assertTrue(
"Resource with ID 'invalid-id' not found."
@@ -182,8 +163,6 @@ class TestObtainResourcesCheck(unittest.TestCase):
resource_directory=self.get_resource_dir(),
resource_version="3.0.0",
)
print("context.exception: ", context.exception)
print(str(context.exception))
self.assertTrue(
f"Resource test-binary-resource with version '3.0.0'"
" not found.\nResource versions can be found at: "

72
tests/pyunit/stdlib/resources/pyunit_resource_download_checks.py
View File

@@ -1,72 +0,0 @@
# Copyright (c) 2023 The Regents of the University of California
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met: redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer;
# redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution;
# neither the name of the copyright holders nor the names of its
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
import unittest
import tempfile
import os
from typing import Dict
from gem5.resources.downloader import (
get_resources_json_obj,
)
class ResourceDownloadTestSuite(unittest.TestCase):
"""Test cases for gem5.resources.downloader"""
@classmethod
def setUpClass(cls) -> str:
pass
def get_resource_json_by_id(self) -> None:
"""Get a resource by its id"""
resources = get_resources_json_obj("test-version")
self.assertEqual(resources["id"], "test-version")
self.assertEqual(resources["resource_version"], "2.0.0")
def get_resource_json_invalid_id(self) -> None:
"""Should throw an exception when trying to get a resource that doesn't exist"""
with self.assertRaises(Exception) as context:
get_resources_json_obj("this-resource-doesnt-exist")
self.assertTrue(
f"Error: Resource with name 'this-resource-doesnt-exist' does not exist"
in str(context.exception)
)
def get_resource_json_by_id_and_version(self) -> None:
"""Get a resource by its id and version"""
resources = get_resources_json_obj("test-version", "1.0.0")
self.assertEqual(resources["id"], "test-version")
self.assertEqual(resources["resource_version"], "1.0.0")
def get_resource_json_by_id_and_invalid_version(self) -> None:
"""Get a resource by its id and an invalid version (does not exist)"""
with self.assertRaises(Exception) as context:
get_resources_json_obj("test-version", "3.0.0")
self.assertTrue(
f"Specified Version 3.0.0 does not exist for the resource 'test-version'."
in str(context.exception)
)

29
tests/pyunit/stdlib/resources/pyunit_resource_specialization.py
View File

@@ -62,24 +62,6 @@ class ResourceSpecializationSuite(unittest.TestCase):
function.
"""
@classmethod
def setUpClass(cls):
"""Prior to running the suite we set the resource directory to
"ref/resource-specialization.json"
"""
os.environ["GEM5_RESOURCE_JSON"] = os.path.join(
os.path.realpath(os.path.dirname(__file__)),
"refs",
"resource-specialization.json",
)
@classmethod
def tearDownClass(cls) -> None:
"""After running the suite we unset the gem5-resource JSON file, as to
not interfere with others tests.
"""
del os.environ["GEM5_RESOURCE_JSON"]
def get_resource_dir(cls) -> str:
"""To ensure the resources are cached to the same directory as all
other tests, this function returns the location of the testing
@@ -99,6 +81,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource = obtain_resource(
resource_id="binary-example",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertIsInstance(resource, BinaryResource)
@@ -114,6 +97,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource = obtain_resource(
resource_id="kernel-example",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertIsInstance(resource, KernelResource)
@@ -129,6 +113,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource = obtain_resource(
resource_id="bootloader-example",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertIsInstance(resource, BootloaderResource)
@@ -144,6 +129,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource = obtain_resource(
resource_id="disk-image-example",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertIsInstance(resource, DiskImageResource)
@@ -159,6 +145,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource = obtain_resource(
resource_id="checkpoint-example",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertIsInstance(resource, CheckpointResource)
@@ -173,6 +160,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource = obtain_resource(
resource_id="git-example",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertIsInstance(resource, GitResource)
@@ -185,6 +173,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource = obtain_resource(
resource_id="simpoint-directory-example",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertIsInstance(resource, SimpointDirectoryResource)
@@ -219,6 +208,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource = obtain_resource(
resource_id="simpoint-example",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertIsInstance(resource, SimpointResource)
@@ -240,6 +230,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource_id="file-example",
resource_directory=self.get_resource_dir(),
resource_version="1.0.0",
gem5_version="develop",
)
self.assertIsInstance(resource, FileResource)
@@ -268,6 +259,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource = obtain_resource(
resource_id="looppoint-pinpoint-csv-resource",
resource_directory=self.get_resource_dir(),
gem5_version="develop",
)
self.assertIsInstance(resource, LooppointCsvResource)
@@ -289,6 +281,7 @@ class ResourceSpecializationSuite(unittest.TestCase):
resource_id="looppoint-json-restore-resource-region-1",
resource_directory=self.get_resource_dir(),
resource_version="1.0.0",
gem5_version="develop",
)
self.assertIsInstance(resource, LooppointJsonResource)

48
tests/pyunit/stdlib/resources/pyunit_workload_checks.py
View File

@@ -40,17 +40,7 @@ from gem5.resources.client_api.client_wrapper import ClientWrapper
from unittest.mock import patch
from pathlib import Path
mock_config_json1 = {
"sources": {
"baba": {
"url": Path(__file__).parent
/ "refs/workload-checks-custom-workload.json",
"isMongo": False,
}
},
}
mock_config_json2 = {
mock_config_json = {
"sources": {
"baba": {
"url": Path(__file__).parent / "refs/workload-checks.json",
@@ -68,29 +58,19 @@ class CustomWorkloadTestSuite(unittest.TestCase):
@classmethod
@patch(
"gem5.resources.client.clientwrapper",
new=ClientWrapper(mock_config_json1),
new=ClientWrapper(mock_config_json),
)
def setUpClass(cls) -> None:
os.environ["GEM5_RESOURCE_JSON"] = os.path.join(
os.path.realpath(os.path.dirname(__file__)),
"refs",
"workload-checks-custom-workload.json",
)
cls.custom_workload = CustomWorkload(
function="set_se_binary_workload",
parameters={
"binary": obtain_resource("x86-hello64-static"),
"binary": obtain_resource(
"x86-hello64-static", gem5_version="develop"
),
"arguments": ["hello", 6],
},
)
@classmethod
def tearDownClass(cls):
# Unset the environment variable so this test does not interfere with
# others.
os.environ["GEM5_RESOURCE_JSON"]
def test_get_function_str(self) -> None:
# Tests `CustomResource.get_function_str`
@@ -140,7 +120,8 @@ class CustomWorkloadTestSuite(unittest.TestCase):
"test", self.custom_workload.get_parameters()["binary"]
)
# We set the overridden parameter back to it's old valu self.custom_workload.set_parameter("binary", old_value)
# We set the overridden parameter back to it's old value
self.custom_workload.set_parameter("binary", old_value)
class WorkloadTestSuite(unittest.TestCase):
@@ -151,21 +132,10 @@ class WorkloadTestSuite(unittest.TestCase):
@classmethod
@patch(
"gem5.resources.client.clientwrapper",
ClientWrapper(mock_config_json2),
ClientWrapper(mock_config_json),
)
def setUpClass(cls):
os.environ["GEM5_RESOURCE_JSON"] = os.path.join(
os.path.realpath(os.path.dirname(__file__)),
"refs",
"workload-checks.json",
)
cls.workload = Workload("simple-boot")
@classmethod
def tearDownClass(cls):
# Unset the environment variable so this test does not interfere with
# others.
os.environ["GEM5_RESOURCE_JSON"]
cls.workload = Workload("simple-boot", gem5_version="develop")
def test_get_function_str(self) -> None:
# Tests `Resource.get_function_str`

4
tests/pyunit/stdlib/resources/refs/mongo-mock.json
View File

@@ -22,7 +22,7 @@
"source_url": "https://github.com/gem5/gem5-resources/tree/develop/src/x86-ubuntu",
"resource_version": "1.0.0",
"gem5_versions": [
"23.0"
"develop"
],
"example_usage": "get_resource(resource_name=\"x86-ubuntu-18.04-img\")"
},
@@ -49,7 +49,7 @@
"source_url": "https://github.com/gem5/gem5-resources/tree/develop/src/x86-ubuntu",
"resource_version": "1.1.0",
"gem5_versions": [
"23.0"
"develop"
],
"example_usage": "get_resource(resource_name=\"x86-ubuntu-18.04-img\")"
}

8
tests/pyunit/stdlib/resources/refs/obtain-resource.json
View File

@@ -24,7 +24,7 @@
"source": "src/test-source",
"resource_version": "2.0.0",
"gem5_versions": [
"develop"
"23.0"
]
},
{
@@ -38,7 +38,8 @@
"source": "src/test-source",
"resource_version": "1.7.0",
"gem5_versions": [
"develop"
"develop",
"develop-2"
]
},
{
@@ -52,8 +53,7 @@
"source": "src/test-source",
"resource_version": "1.5.0",
"gem5_versions": [
"21.1",
"22.1"
"develop"
]
}
]

18
tests/pyunit/stdlib/resources/refs/resource-specialization.json
View File

@@ -10,6 +10,7 @@
"source": "src/linux-kernel",
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -25,6 +26,7 @@
"root_partition": "1",
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -39,6 +41,7 @@
"source": "src/simple",
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -51,6 +54,7 @@
"url": "http://dist.gem5.org/dist/develop/test-progs/hello/bin/arm/linux/hello64-static",
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -66,6 +70,7 @@
"url": "http://dist.gem5.org/dist/develop/checkpoints/riscv-hello-example-checkpoint.tar",
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -75,10 +80,11 @@
"description": null,
"is_zipped": false,
"is_tar_archive": true,
"md5sum": "71b2cb004fe2cda4556f0b1a38638af6",
"md5sum": "3a57c1bb1077176c4587b8a3bf4f8ace",
"url": "http://dist.gem5.org/dist/develop/checkpoints/riscv-hello-example-checkpoint.tar",
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -87,11 +93,12 @@
"id": "file-example",
"description": null,
"is_zipped": false,
"md5sum": "71b2cb004fe2cda4556f0b1a38638af6",
"md5sum": "2efd144c11829ab18d54eae6371e120a",
"url": "http://dist.gem5.org/dist/develop/checkpoints/riscv-hello-example-checkpoint.tar",
"source": null,
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -106,6 +113,7 @@
"url": "http://dist.gem5.org/dist/develop/checkpoints/riscv-hello-example-checkpoint.tar",
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -125,6 +133,7 @@
"workload_name": "Example Workload",
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -148,6 +157,7 @@
],
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -161,6 +171,7 @@
"source": null,
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
},
@@ -170,11 +181,12 @@
"description": "A looppoint json file resource.",
"is_zipped": false,
"region_id": "1",
"md5sum": "a71ed64908b082ea619b26b940a643c1",
"md5sum": "efb85ebdf90c5cee655bf2e05ae7692a",
"url": "http://dist.gem5.org/dist/develop/looppoints/x86-matrix-multiply-omp-100-8-looppoint-json-20230128",
"source": null,
"resource_version": "1.0.0",
"gem5_versions": [
"develop",
"23.0"
]
}

24
tests/pyunit/stdlib/resources/refs/resources.json
View File

@@ -21,7 +21,7 @@
"source_url": "https://github.com/gem5/gem5-resources/tree/develop/src/asmtest",
"resource_version": "1.0.0",
"gem5_versions": [
"23.0"
"develop"
],
"example_usage": "get_resource(resource_name=\"rv64mi-p-sbreak\")"
},
@@ -48,7 +48,7 @@
"source_url": "https://github.com/gem5/gem5-resources/tree/develop/src/asmtest",
"resource_version": "1.1.0",
"gem5_versions": [
"23.0"
"develop"
],
"example_usage": "get_resource(resource_name=\"rv64mi-p-sbreak\")"
},
@@ -71,7 +71,7 @@
"source_url": "https://github.com/gem5/gem5-resources/tree/develop/src/asmtest",
"resource_version": "2.0.0",
"gem5_versions": [
"23.1"
"999.1"
],
"example_usage": "get_resource(resource_name=\"rv64mi-p-sbreak\")"
},
@@ -94,7 +94,7 @@
"source_url": "",
"resource_version": "1.0.0",
"gem5_versions": [
"23.0"
"develop"
],
"workload_name": "x86-print-this-15000-with-simpoints",
"example_usage": "get_resource(resource_name=\"x86-print-this-1500-simpoints\")",
@@ -122,7 +122,7 @@
"source_url": "",
"resource_version": "0.2.0",
"gem5_versions": [
"23.0"
"develop"
],
"workload_name": "x86-print-this-15000-with-simpoints",
"example_usage": "get_resource(resource_name=\"x86-print-this-1500-simpoints\")",
@@ -150,7 +150,7 @@
"source_url": "",
"resource_version": "0.2.0",
"gem5_versions": [
"23.0"
"develop"
],
"workload_name": "x86-print-this-15000-with-simpoints",
"example_usage": "get_resource(resource_name=\"x86-print-this-1500-simpoints\")",
@@ -178,7 +178,7 @@
"source_url": "",
"resource_version": "0.2.0",
"gem5_versions": [
"23.0"
"develop"
],
"workload_name": "x86-print-this-15000-with-simpoints",
"example_usage": "get_resource(resource_name=\"x86-print-this-1500-simpoints\")",
@@ -206,7 +206,7 @@
"source_url": "",
"resource_version": "0.2.0",
"gem5_versions": [
"23.0"
"develop"
],
"workload_name": "x86-print-this-15000-with-simpoints",
"example_usage": "get_resource(resource_name=\"x86-print-this-1500-simpoints\")",
@@ -234,7 +234,7 @@
"source_url": "",
"resource_version": "0.2.0",
"gem5_versions": [
"23.0"
"develop"
],
"workload_name": "x86-print-this-15000-with-simpoints",
"example_usage": "get_resource(resource_name=\"x86-print-this-1500-simpoints\")",
@@ -262,7 +262,7 @@
"source_url": "",
"resource_version": "0.2.0",
"gem5_versions": [
"23.0"
"develop"
],
"workload_name": "x86-print-this-15000-with-simpoints",
"example_usage": "get_resource(resource_name=\"x86-print-this-1500-simpoints\")",
@@ -290,7 +290,7 @@
"source_url": "",
"resource_version": "0.2.0",
"gem5_versions": [
"23.0"
"develop"
],
"workload_name": "x86-print-this-15000-with-simpoints",
"example_usage": "get_resource(resource_name=\"x86-print-this-1500-simpoints\")",
@@ -322,7 +322,7 @@
"source_url": "https://github.com/gem5/gem5-resources/tree/develop/src/x86-ubuntu",
"resource_version": "2.0.0",
"gem5_versions": [
"23.0"
"develop"
],
"example_usage": "get_resource(resource_name=\"x86-ubuntu-18.04-img\")"
}

16
tests/pyunit/stdlib/resources/refs/workload-checks-custom-workload.json
View File

@@ -1,16 +0,0 @@
[
{
"category": "binary",
"id": "x86-hello64-static",
"description": "A 'Hello World!' binary.",
"architecture": "X86",
"is_zipped": false,
"md5sum": "dbf120338b37153e3334603970cebd8c",
"url": "http://dist.gem5.org/dist/develop/test-progs/hello/bin/x86/linux/hello64-static",
"source": "src/simple",
"resource_version": "1.0.0",
"gem5_versions": [
"23.0"
]
}
]

20
tests/pyunit/stdlib/resources/refs/workload-checks.json
View File

@@ -10,7 +10,7 @@
"source": "src/linux-kernel",
"resource_version": "1.0.0",
"gem5_versions": [
"23.0"
"develop"
]
},
{
@@ -25,7 +25,7 @@
"root_partition": "1",
"resource_version": "1.0.0",
"gem5_versions": [
"23.0"
"develop"
]
},
{
@@ -42,7 +42,21 @@
},
"resource_version": "1.0.0",
"gem5_versions": [
"23.0"
"develop"
]
},
{
"category": "binary",
"id": "x86-hello64-static",
"description": "A 'Hello World!' binary.",
"architecture": "X86",
"is_zipped": false,
"md5sum": "dbf120338b37153e3334603970cebd8c",
"url": "http://dist.gem5.org/dist/develop/test-progs/hello/bin/x86/linux/hello64-static",
"source": "src/simple",
"resource_version": "1.0.0",
"gem5_versions": [
"develop"
]
}
]