7cdf8c00f8
gem5art is a utility to help manage the artifacts used in gem5 experiments, the output from those experiments, and running the experiments in parallel (artifacts, run, and tasks packages respectively). The current documentation can be found on readthedocs [1], but we are planning on migrating this to the gem5 website very soon [2]. More information on the motivation and design was discussed at the gem5 workshop last summer. See the blog post [3] for more details. The current version (v1.3.1) is already deployed on PyPI, and you can install it with `pip install gem5art-artifact gem5art-run gem5art-tasks` Once this is merged, we will update the PyPI version to match the version in gem5 (v1.4.0). The only differences are mostly documentation based (pointers to the documentation and source), but we have also updated the style to strictly match PEP8 with black [4]. gem5art is a *utility* to use with gem5. So, we expect that the versioning and release schedule will not necessarily match gem5's (hence a separate versioning structure and separate RELEASE-NOTES, etc.). [1]: https://gem5art.readthedocs.io/en/latest/ [2]: https://www.gem5.org/documentation/gem5art [3]: http://www.gem5.org/2020/05/26/gem5art.html [4]: https://github.com/psf/black Change-Id: Ic8af63edf0cb7df4693a46413f7278a3e8ac6846 Signed-off-by: Jason Lowe-Power <jason@lowepower.com> Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/42121 Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br> Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu> Reviewed-by: Ayaz Akram <yazakram@ucdavis.edu> Reviewed-by: Jason Lowe-Power <power.jg@gmail.com> Maintainer: Jason Lowe-Power <power.jg@gmail.com> Tested-by: kokoro <noreply+kokoro@google.com>
2.7 KiB
gem5art tasks package
This package contains two parallel task libraries for running gem5 experiments.
he actual gem5 experiment can be executed with the help of Python multiprocessing support, Celery or even without using any job manager (a job can be directly launched by calling run() function of gem5Run object).
This package implicitly depends on the gem5art run package.
Please cite the gem5art paper when using the gem5art packages. This documentation can be found on the gem5 website
Use of Python Multiprocessing
This is a simple way to run gem5 jobs using Python multiprocessing library. You can use the following function in your job launch script to execute gem5art run objects:
run_job_pool([a list containing all run objects you want to execute], num_parallel_jobs = [Number of parallel jobs you want to run])
Use of Celery
Celery server can run many gem5 tasks asynchronously.
Once a user creates a gem5Run object (discussed previously) while using gem5art, this object needs to be passed to a method run_gem5_instance() registered with Celery app, which is responsible for starting a Celery task to run gem5. The other argument needed by the run_gem5_instance() is the current working directory.
Celery server can be started with the following command:
celery -E -A gem5art.tasks.celery worker --autoscale=[number of workers],0
This will start a server with events enabled that will accept gem5 tasks as defined in gem5art. It will autoscale from 0 to desired number of workers.
Celery relies on a message broker RabbitMQ for communication between the client and workers.
If not already installed, you need to install RabbitMQ on your system (before running celery) using:
apt-get install rabbitmq-server
Monitoring Celery
Celery does not explicitly show the status of the runs by default. flower, a Python package, is a web-based tool for monitoring and administrating Celery.
To install the flower package,
pip install flower
You can monitor the celery cluster doing the following:
flower -A gem5art.tasks.celery --port=5555
This will start a webserver on port 5555.
Removing all tasks
celery -A gem5art.tasks.celery purge
Viewing state of all jobs in celery
celery -A gem5art.tasks.celery events
Tasks API Documentation
Task
----
.. automodule:: gem5art.tasks.tasks
:members:
:undoc-members:
.. automodule:: gem5art.tasks.celery
:members:
:undoc-members: