From d1f9f987472a0b28754180c2a9b58c55b5551d46 Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Wed, 4 Oct 2023 15:07:16 -0700
Subject: [PATCH 01/11] util: Make all runners the same type

Having two types of GitHub Action Runners has not yielded much benefit
and caused confusion and inefficiencies. This change simplifies things
to having just one runner with 8-cores and 16GB of memory. It is
sufficient to build gem5 and run most simulations.

Change-Id: Ic49ae5e98b02086f153f4ae2a4eedd8a535786c8
---
 util/github-runners-vagrant/README.md         | 25 +++-----
 .../{Vagrantfile-runner => Vagrantfile}       | 12 +---
 .../Vagrantfile-builder                       | 57 -------------------
 util/github-runners-vagrant/vm_manager.sh     | 28 +++------
 4 files changed, 17 insertions(+), 105 deletions(-)
 rename util/github-runners-vagrant/{Vagrantfile-runner => Vagrantfile} (80%)
 delete mode 100644 util/github-runners-vagrant/Vagrantfile-builder

diff --git a/util/github-runners-vagrant/README.md b/util/github-runners-vagrant/README.md
index 7d0f116260..2ea5b3e588 100644
--- a/util/github-runners-vagrant/README.md
+++ b/util/github-runners-vagrant/README.md
@@ -21,10 +21,10 @@ sudo apt-get install qemu libvirt-daemon-system libvirt-clients ebtables dnsmasq
 sudo apt purge vagrant-libvirt
 ```
 
-## Set up the Vagrantfiles for the GitHub repository
+## Set up the Vagrant for the GitHub repository
 
 First, generate a Personal Access Token, which you can create [here](https://github.com/settings/tokens)
-Make sure to set admin permissions on this token, then replace instances of `<PERSONAL ACCESS TOKEN>` in the Vagrantfiles ("Vagrantfile-builder" and "Vagrant-runner") with your token.
+Make sure to set admin permissions on this token, then replace the of `<PERSONAL ACCESS TOKEN>` in the Vagrantfile  with your token.
 
 Next, replace instances of `<GITHUB REPO>` with your GitHub account name and the repository name, separated by a forward slash.
 For example, if your GitHub account name is `example` and your repository name is `example-repo`, you would replace `<GITHUB REPO>` with `example/example-repo`.
@@ -46,28 +46,19 @@ vagrant plugin install vagrant-libvirt
 vagrant plugin install vagrant-reload
 ```
 
-## The "builder" and "runner" VMs
-
-The number of CPUs and the memory size differs between the "Vagrantfile-builder" and "Vagrantfile-runner".
-
-In our work we have two types of machines "runners" and "builders".
-Runners are single core machines with 8GB of memory, and builders are 4 core machines with 16GB of memory.
-The latter is used for building gem5 binaries while the former is used for running instances of gem5.
-You can expect each machine to take up approximately 60GB of disk space though VMs will consume the disk space they require.
-
-The "Vagrantfile-builder" file is set to create a runner machine and the "Vagrantfile-builder" file is set to create a builder machine.
-
-Specifying which Vagrantfile to use is done by setting the `VAGRANT_VAGRANTFILE` environment variable.
-
 ## Creating the virtual machine
 
+The Vagrantfile in this directory defines a VM that can be used to create a GitHub Actions runner.
+It has 4-cores, 16GB of RAM, and 60GB of disk space.
+This is sufficient to both compile gem5 and run most simulations.
+
 Each VM on your host system must have a unique name.
 Give the VM to be created a unique name by setting the `<VM NAME>` variables in the Vagrantfile you wish to utilize.
 
 Then run:
 
 ```sh
-VAGRANT_VAGRANTFILE=<VAGRANTFILE> vagrant up --provider=libvirt
+vagrant up --provider=libvirt
 ```
 
 This should automatically create your machine, as well as configure and start up a Github Actions runner.
@@ -80,7 +71,7 @@ If you wish to create more than one runner you must edit the `<VM NAME>` in the
 ## Helper scripts
 
 The "vm_manager" script can be used to set up multiple builder and runner VMs.
-To use this script simply modify the `NUM_RUNNERS`, `NUM_BUILDERS`, `RUNNER_PREFIX`, and `BUILDER_PREFIX` variables to the desired values.
+To use this script simply modify the `NUM_RUNNERS` and `RUNNER_PREFIX` variables to the desired values.
 Then run the script with:
 
 ```sh
diff --git a/util/github-runners-vagrant/Vagrantfile-runner b/util/github-runners-vagrant/Vagrantfile
similarity index 80%
rename from util/github-runners-vagrant/Vagrantfile-runner
rename to util/github-runners-vagrant/Vagrantfile
index eb4054ed16..4bc0b85fd7 100644
--- a/util/github-runners-vagrant/Vagrantfile-runner
+++ b/util/github-runners-vagrant/Vagrantfile
@@ -13,23 +13,15 @@ Vagrant.configure("2") do |config|
 
   config.vm.provider "libvirt" do |vb|
     # Customize the amount of cpus and memory on the VM:
-    vb.cpus = "1".to_i
-    vb.memory = "8192".to_i
+    vb.cpus = "4".to_i
+    vb.memory = "16384".to_i
   end
 
   # sets up vm
   config.vm.provision :shell, path: "provision_root.sh"
   config.vm.provision :shell, privileged: false, path: "provision_nonroot.sh"
-  # To ensure we don't run out of memory, we enable dynamic Swap Space. This is
-  # done via the "swapspace" daemon: https://pqxx.org/development/swapspace/
-  config.vm.provision :shell, inline: "sudo apt install swapspace -y"
   # The provision_root.sh adds the vagrant user to the docker group, so we need to reload the VM.
   config.vm.provision :reload
-  config.vm.provision :shell, run: 'always', inline: <<-SHELL
-    # When running gem5 in SE mode we must overcommit memory.
-    # This is run on every startup of the VM.
-    /sbin/sysctl vm.overcommit_memory=1
-  SHELL
   config.vm.provision :shell, privileged: false, run: 'always', inline: <<-SHELL
     if [ -d ~/actions-runner ]; then
         # This will be run everytime the VM is run (once created).
diff --git a/util/github-runners-vagrant/Vagrantfile-builder b/util/github-runners-vagrant/Vagrantfile-builder
deleted file mode 100644
index c0c2380b95..0000000000
--- a/util/github-runners-vagrant/Vagrantfile-builder
+++ /dev/null
@@ -1,57 +0,0 @@
-# -*- mode: ruby -*-
-# vi: set ft=ruby :
-
-Vagrant.configure("2") do |config|
-  config.vm.box = "generic/ubuntu2204"
-  config.vm.box_check_update = true
-  config.vm.define "<VM NAME>"
-  config.vm.hostname = "<VM NAME>"
-  # allows us to ssh into the machine, addressing the problem below
-  # https://www.reddit.com/r/vagrant/comments/sb7hfl/new_to_vagrant_getting_efault_warning/
-  config.ssh.username = "vagrant"
-  config.ssh.password = "vagrant"
-
-  config.vm.provider "libvirt" do |vb|
-    # Customize the amount of cpus and memory on the VM:
-    vb.cpus = "4".to_i
-    vb.memory = "16384".to_i
-  end
-
-  # sets up vm
-  config.vm.provision :shell, path: "provision_root.sh"
-  config.vm.provision :shell, privileged: false, path: "provision_nonroot.sh"
-  # To ensure we don't run out of memory, we enable dynamic Swap Space. This is
-  # done via the "swapspace" daemon: https://pqxx.org/development/swapspace/
-  config.vm.provision :shell, inline: "sudo apt install swapspace -y"
-  # The provision_root.sh adds the vagrant user to the docker group, so we need to reload the VM.
-  config.vm.provision :reload
-  config.vm.provision :shell, run: 'always', inline: <<-SHELL
-    # When running gem5 in SE mode we must overcommit memory.
-    # This is run on every startup of the VM.
-    /sbin/sysctl vm.overcommit_memory=1
-  SHELL
-  config.vm.provision :shell, privileged: false, run: 'always', inline: <<-SHELL
-    if [ -d ~/actions-runner ]; then
-        # This will be run everytime the VM is run (once created).
-        cd actions-runner
-        nohup ./run.sh &
-    else
-        # This will be run the first time the VM is created.
-        mkdir ~/actions-runner && cd ~/actions-runner
-        curl -so actions-runner-linux-x64-2.304.0.tar.gz -L \
-            https://github.com/actions/runner/releases/download/v2.304.0/actions-runner-linux-x64-2.304.0.tar.gz
-        tar xzf ./actions-runner-linux-x64-2.304.0.tar.gz
-
-        # configure the runner
-        # echo automatically sets the name of the runner, and the tags
-        # create a personal access token with admin permission and copy it into the curl command
-        echo -ne '\n\nbuild\n\n' | ./config.sh --url https://github.com/<GITHUB REPO> --token $(curl -L \
-            -X POST   -H "Accept: application/vnd.github+json"   -H "Authorization: Bearer <PERSONAL ACCESS TOKEN>" \
-            -H "X-GitHub-Api-Version: 2022-11-28" \
-           https://api.github.com/repos/<GITHUB REPO>/actions/runners/registration-token | jq -r '.token')
-        # start the runner
-        nohup ./run.sh &
-    fi
-  SHELL
-
-end
diff --git a/util/github-runners-vagrant/vm_manager.sh b/util/github-runners-vagrant/vm_manager.sh
index ce46eb3fe7..da07217121 100755
--- a/util/github-runners-vagrant/vm_manager.sh
+++ b/util/github-runners-vagrant/vm_manager.sh
@@ -1,9 +1,7 @@
 #!/bin/bash
 
-NUM_RUNNERS=20
-NUM_BUILDERS=3
-RUNNER_PREFIX="$(hostname)-runner-"
-BUILDER_PREFIX="$(hostname)-builder-"
+NUM_RUNNERS=3
+RUNNER_PREFIX_PREFIX="$(hostname)"
 
 SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 export VAGRANT_HOME=${SCRIPT_DIR}
@@ -19,25 +17,13 @@ fi
 
 
 for (( i=1; i<=NUM_RUNNERS; i++ )); do
-    sed -i "s/  config.vm.define.*/  config.vm.define \"${RUNNER_PREFIX}${i}\"/g" Vagrantfile-runner
-    sed -i "s/  config.vm.hostname.*/  config.vm.hostname = \"${RUNNER_PREFIX}${i}\"/g" Vagrantfile-runner
+    sed -i "s/  config.vm.define.*/  config.vm.define \"${RUNNER_PREFIX}-${i}\"/g" Vagrantfile
+    sed -i "s/  config.vm.hostname.*/  config.vm.hostname = \"${RUNNER_PREFIX}-${i}\"/g" Vagrantfile
     if [[ "${param}" == "destroy" ]]; then
-        VAGRANT_VAGRANTFILE=Vagrantfile-runner vagrant destroy -f
+        VAGRANT_VAGRANTFILE=Vagrantfile vagrant destroy -f
     elif [[ "${param}" == "shutdown" ]]; then
-        VAGRANT_VAGRANTFILE=Vagrantfile-runner vagrant halt -f
+        VAGRANT_VAGRANTFILE=Vagrantfile vagrant halt -f
     else
-        VAGRANT_VAGRANTFILE=Vagrantfile-runner vagrant up --provider=libvirt
-    fi
-done
-
-for (( i=1; i<=NUM_BUILDERS; i++ )); do
-    sed -i "s/  config.vm.define.*/  config.vm.define \"${BUILDER_PREFIX}${i}\"/g" Vagrantfile-builder
-    sed -i "s/  config.vm.hostname.*/  config.vm.hostname = \"${BUILDER_PREFIX}${i}\"/g" Vagrantfile-builder
-    if [[ "${param}" == "destroy" ]]; then
-        VAGRANT_VAGRANTFILE=Vagrantfile-builder vagrant destroy -f
-    elif [[ "${param}" == "shutdown" ]]; then
-        VAGRANT_VAGRANTFILE=Vagrantfile-builder vagrant halt -f
-    else
-        VAGRANT_VAGRANTFILE=Vagrantfile-builder vagrant up --provider=libvirt
+        VAGRANT_VAGRANTFILE=Vagrantfile vagrant up --provider=libvirt
     fi
 done

From 3e1c0b0714bddd973670d9a23d033cc077102487 Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Wed, 4 Oct 2023 15:18:45 -0700
Subject: [PATCH 02/11] util: Move runners from gem5 repository to gem5 org

There are two places self-hosted runners can exist on GitHub:

1. At the level of the repository: In this case the runners can only be
used by that repository and runners can only be distinguished from one
another by labels.
2. At the level of the organization: In this case the runners can be
used by any repository in the organization, thus increasing their
versatility. In addition to labels, runners in the level of the
organization can be organized into groups.

While we do not use our self-hosted runners on other repositories, there
may be future use for this, so we might as well enable it now.

Change-Id: Id5e113194314336221dcdc8c2858b352afcbaf6e
---
 util/github-runners-vagrant/README.md   | 5 ++---
 util/github-runners-vagrant/Vagrantfile | 4 ++--
 2 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/util/github-runners-vagrant/README.md b/util/github-runners-vagrant/README.md
index 2ea5b3e588..0b93ff6693 100644
--- a/util/github-runners-vagrant/README.md
+++ b/util/github-runners-vagrant/README.md
@@ -26,8 +26,7 @@ sudo apt purge vagrant-libvirt
 First, generate a Personal Access Token, which you can create [here](https://github.com/settings/tokens)
 Make sure to set admin permissions on this token, then replace the of `<PERSONAL ACCESS TOKEN>` in the Vagrantfile  with your token.
 
-Next, replace instances of `<GITHUB REPO>` with your GitHub account name and the repository name, separated by a forward slash.
-For example, if your GitHub account name is `example` and your repository name is `example-repo`, you would replace `<GITHUB REPO>` with `example/example-repo`.
+Next, replace instances of `<GITHUB_ORG>` with your GitHub organization you wish to add this runner.
 
 ## Install Vagrant Plugins
 
@@ -62,7 +61,7 @@ vagrant up --provider=libvirt
 ```
 
 This should automatically create your machine, as well as configure and start up a Github Actions runner.
-You can check the status of the runner here: https://github.com/<account>/<repo>/settings/actions/runners
+You can check the status of the runner here: https://github.com/organizations/{GITHUB_ORG}/settings/actions/runners
 
 If the runner ever shows as offline, you can rerun the `vagrant up --provider=libvirt` command to make sure everything is working properly.
 
diff --git a/util/github-runners-vagrant/Vagrantfile b/util/github-runners-vagrant/Vagrantfile
index 4bc0b85fd7..0f1eb9ee49 100644
--- a/util/github-runners-vagrant/Vagrantfile
+++ b/util/github-runners-vagrant/Vagrantfile
@@ -37,10 +37,10 @@ Vagrant.configure("2") do |config|
         # configure the runner
         # echo automatically sets the name of the runner, and the tags
         # create a personal access token with admin permission and copy it into the curl command
-        echo -ne '\n\nrun\n\n' | ./config.sh --url https://github.com/<GITHUB REPO> --token $(curl -L \
+        echo -ne '\n\nrun\n\n' | ./config.sh --url https://github.com/<GITHUB_ORG> --token $(curl -L \
             -X POST   -H "Accept: application/vnd.github+json"   -H "Authorization: Bearer <PERSONAL ACCESS TOKEN>" \
             -H "X-GitHub-Api-Version: 2022-11-28" \
-           https://api.github.com/repos/<GITHUB REPO>/actions/runners/registration-token | jq -r '.token')
+            https://api.github.com/orgs/${GITHUB_ORG}/actions/runners/registration-token | jq -r '.token')
         # start the runner
         nohup ./run.sh &
     fi

From a924fa3bdcde86894ab24a434cb035a9362bfabf Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Wed, 4 Oct 2023 17:21:13 -0700
Subject: [PATCH 03/11] util: Add action-run.sh to run Action Runners

The "action-run.sh" action replaces inline scripting in the Vagrantfile.

The major improvement is this script runs an infinite loop and
configures the runners to be ephemeral. This means they cease to exist
after a job is complete. The script then cleans the VM workspace and the
loop restarts by configuring and setting up another runner. This means
our VMs no longer accumulate files that eventually lead to the VM
running out of space.

Change-Id: Iba6dc9a480f5805042602f120fc84bdc47a96d55
---
 util/github-runners-vagrant/Vagrantfile   | 30 +++------
 util/github-runners-vagrant/action-run.sh | 81 +++++++++++++++++++++++
 2 files changed, 89 insertions(+), 22 deletions(-)
 create mode 100755 util/github-runners-vagrant/action-run.sh

diff --git a/util/github-runners-vagrant/Vagrantfile b/util/github-runners-vagrant/Vagrantfile
index 0f1eb9ee49..b51a8c0a9c 100644
--- a/util/github-runners-vagrant/Vagrantfile
+++ b/util/github-runners-vagrant/Vagrantfile
@@ -1,6 +1,9 @@
 # -*- mode: ruby -*-
 # vi: set ft=ruby :
 
+PERSONAL_ACCESS_TOKEN="<PERSONAL ACCESS TOKEN>"
+GITHUB_ORG="<GITHUB_ORG>"
+
 Vagrant.configure("2") do |config|
   config.vm.box = "generic/ubuntu2204"
   config.vm.box_check_update = true
@@ -22,27 +25,10 @@ Vagrant.configure("2") do |config|
   config.vm.provision :shell, privileged: false, path: "provision_nonroot.sh"
   # The provision_root.sh adds the vagrant user to the docker group, so we need to reload the VM.
   config.vm.provision :reload
-  config.vm.provision :shell, privileged: false, run: 'always', inline: <<-SHELL
-    if [ -d ~/actions-runner ]; then
-        # This will be run everytime the VM is run (once created).
-        cd actions-runner
-        nohup ./run.sh &
-    else
-        # This will be run the first time the VM is created.
-        mkdir ~/actions-runner && cd ~/actions-runner
-        curl -so actions-runner-linux-x64-2.304.0.tar.gz -L \
-            https://github.com/actions/runner/releases/download/v2.304.0/actions-runner-linux-x64-2.304.0.tar.gz
-        tar xzf ./actions-runner-linux-x64-2.304.0.tar.gz
+  # Copy the "action-run.sh" script from the host to the VM.
+  builder.vm.provision "file", source: "./action-run.sh", destination: "/tmp/action-run.sh"
+  builder.vm.provision :shell, privileged: false,  inline: "cp /tmp/action-run.sh ."
+  # Execute the actions-run.sh script on every boot. This configures the and starts the runner.
+  config.vm.provision :shell, privileged: false, run: 'always', inline: "./action-run.sh #{PERSONAL_ACCESS_TOKEN} #{GITHUB_ORG} >> action-run.log 2>&1 &"
 
-        # configure the runner
-        # echo automatically sets the name of the runner, and the tags
-        # create a personal access token with admin permission and copy it into the curl command
-        echo -ne '\n\nrun\n\n' | ./config.sh --url https://github.com/<GITHUB_ORG> --token $(curl -L \
-            -X POST   -H "Accept: application/vnd.github+json"   -H "Authorization: Bearer <PERSONAL ACCESS TOKEN>" \
-            -H "X-GitHub-Api-Version: 2022-11-28" \
-            https://api.github.com/orgs/${GITHUB_ORG}/actions/runners/registration-token | jq -r '.token')
-        # start the runner
-        nohup ./run.sh &
-    fi
-  SHELL
 end
diff --git a/util/github-runners-vagrant/action-run.sh b/util/github-runners-vagrant/action-run.sh
new file mode 100755
index 0000000000..b802a7c739
--- /dev/null
+++ b/util/github-runners-vagrant/action-run.sh
@@ -0,0 +1,81 @@
+#!/bin/bash
+
+# Copyright (c) 2023 The Regents of the University of California
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met: redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer;
+# redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution;
+# neither the name of the copyright holders nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+set -x
+
+# No argument checking here, this is run directly in the Vagrantfile.
+PERSONAL_ACCESS_TOKEN="$1"
+GITHUB_ORG="$2"
+LABELS="$3"
+WORK_DIR="_work"
+
+# This checks there isn't another instance of this script running.
+# If this script is run twice then more than one runner can be active in the
+# VM and this causes problems.
+if [[ `pgrep -f $0` != "$$" ]]; then
+    echo "Another instance of shell already exist! Exiting"
+    exit
+fi
+
+# If the tarball isn't here then download it and extract it.
+# Note: we don't delete the tarball, we use it to check if we've already
+# downloaded it and extracted it.
+if [ ! -f "actions-runner-linux-x64-2.304.0.tar.gz" ]; then
+    wget https://github.com/actions/runner/releases/download/v2.304.0/actions-runner-linux-x64-2.304.0.tar.gz
+    tar xzf ./actions-runner-linux-x64-2.304.0.tar.gz
+fi
+
+# An infinite loop to re-configure and re-run the runner after each job.
+while true; do
+    # 1. Obtain the registration token.
+    token_curl=$(curl -L \
+    		      -X POST \
+    		      -H "Accept: application/vnd.github+json" \
+    		      -H "Authorization: Bearer ${PERSONAL_ACCESS_TOKEN}" \
+    		      -H "X-GitHub-Api-Version: 2022-11-28" \
+    		      https://api.github.com/orgs/${GITHUB_ORG}/actions/runners/registration-token)
+
+    token=$(echo ${token_curl} | jq -r '.token')
+
+    # 2. Configure the runner.
+    ./config.sh --unattended \
+                --url https://github.com/${GITHUB_ORG} \
+                --ephemeral \
+                --replace \
+                --work "${WORK_DIR}" \
+                --name "$(hostname)" \
+                --labels "${LABELS}" \
+                --token ${token}
+
+    # 3. Run the runner.
+    ./run.sh # This will complete with the runner being destroyed
+
+    # 4. Cleanup the machine
+    rm -rf "${WORK_DIR}"
+    docker system prune --force --volumes
+done

From 529423f47acfe53f7fea1df9d7e39c130109d059 Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Wed, 4 Oct 2023 17:37:11 -0700
Subject: [PATCH 04/11] util: Remove note about ssh use

This is confusing and setting the ssh username and password is normal.

Change-Id: Ic925e92ade47f455c86a461a267b8cad7aa6d7ba
---
 util/github-runners-vagrant/Vagrantfile | 2 --
 1 file changed, 2 deletions(-)

diff --git a/util/github-runners-vagrant/Vagrantfile b/util/github-runners-vagrant/Vagrantfile
index b51a8c0a9c..504e8e35da 100644
--- a/util/github-runners-vagrant/Vagrantfile
+++ b/util/github-runners-vagrant/Vagrantfile
@@ -9,8 +9,6 @@ Vagrant.configure("2") do |config|
   config.vm.box_check_update = true
   config.vm.define "<VM NAME>"
   config.vm.hostname = "<VM NAME>"
-  # allows us to ssh into the machine, addressing the problem below
-  # https://www.reddit.com/r/vagrant/comments/sb7hfl/new_to_vagrant_getting_efault_warning/
   config.ssh.username = "vagrant"
   config.ssh.password = "vagrant"
 

From c0cb16ba89212ce658ebe0b12b7fbe08d1cccd57 Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Wed, 4 Oct 2023 17:42:21 -0700
Subject: [PATCH 05/11] util: Create HOSTNAME variable

Change-Id: Ia68f1bef2bb9e4e5e18476b6100be80f8cf1c799
---
 util/github-runners-vagrant/Vagrantfile   | 5 +++--
 util/github-runners-vagrant/vm_manager.sh | 3 +--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/util/github-runners-vagrant/Vagrantfile b/util/github-runners-vagrant/Vagrantfile
index 504e8e35da..f81d53e145 100644
--- a/util/github-runners-vagrant/Vagrantfile
+++ b/util/github-runners-vagrant/Vagrantfile
@@ -3,12 +3,13 @@
 
 PERSONAL_ACCESS_TOKEN="<PERSONAL ACCESS TOKEN>"
 GITHUB_ORG="<GITHUB_ORG>"
+HOSTNAME="<VM NAME>"
 
 Vagrant.configure("2") do |config|
   config.vm.box = "generic/ubuntu2204"
   config.vm.box_check_update = true
-  config.vm.define "<VM NAME>"
-  config.vm.hostname = "<VM NAME>"
+  config.vm.define "#{HOSTNAME}"
+  config.vm.hostname = "#{HOSTNAME}"
   config.ssh.username = "vagrant"
   config.ssh.password = "vagrant"
 
diff --git a/util/github-runners-vagrant/vm_manager.sh b/util/github-runners-vagrant/vm_manager.sh
index da07217121..4854f5a06f 100755
--- a/util/github-runners-vagrant/vm_manager.sh
+++ b/util/github-runners-vagrant/vm_manager.sh
@@ -17,8 +17,7 @@ fi
 
 
 for (( i=1; i<=NUM_RUNNERS; i++ )); do
-    sed -i "s/  config.vm.define.*/  config.vm.define \"${RUNNER_PREFIX}-${i}\"/g" Vagrantfile
-    sed -i "s/  config.vm.hostname.*/  config.vm.hostname = \"${RUNNER_PREFIX}-${i}\"/g" Vagrantfile
+    sed -i "s/HOSTNAME=.*/HOTNAME=\"${RUNNER_PREFIX}-${i}\"/g" Vagrantfile
     if [[ "${param}" == "destroy" ]]; then
         VAGRANT_VAGRANTFILE=Vagrantfile vagrant destroy -f
     elif [[ "${param}" == "shutdown" ]]; then

From 8c2d41422387da3d5d99c09e9865bcedf22abafe Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Wed, 4 Oct 2023 17:53:08 -0700
Subject: [PATCH 06/11] util: Cleanup the provision_root.sh

Change-Id: I58215dddc34476695c7aedc77b55d338e0304198
---
 util/github-runners-vagrant/provision_root.sh | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/util/github-runners-vagrant/provision_root.sh b/util/github-runners-vagrant/provision_root.sh
index 660064f8ca..74b4facdda 100644
--- a/util/github-runners-vagrant/provision_root.sh
+++ b/util/github-runners-vagrant/provision_root.sh
@@ -5,9 +5,9 @@ set -eu -o pipefail # -x: is for debugging
 
 apt-get update
 apt-get upgrade -y
-apt-get install -y software-properties-common
 add-apt-repository --yes --update ppa:git-core/ppa
 apt-get install -y \
+  software-properties-common \
   bash \
   build-essential \
   clang-format \
@@ -25,8 +25,12 @@ apt-get install -y \
   tree \
   wget \
   yamllint \
-  zstd
-snap install jq
+  zstd \
+  jq \
+  apt-transport-https ca-certificates \
+  curl \
+  gnupg \
+  lsb-release
 
 # Install docker
 apt-get install -y apt-transport-https ca-certificates curl gnupg lsb-release
@@ -34,10 +38,11 @@ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/
 echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
 apt-get update -y
 apt-get install -y docker-ce docker-ce-cli containerd.io
-groupadd docker || true
-gpasswd -a vagrant docker
-newgrp docker
-systemctl restart docker
+
+# Add the Vagrant user to the docker group.
+# Note: The VM needs rebooted for this to take effect. `newgrp docker` doesn't
+# work.
+usermod -aG docker vagrant
 
 # Cleanup
 apt-get autoremove -y

From f36449be983b08c6518402e0c0cf64e4267521ad Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Wed, 4 Oct 2023 17:55:45 -0700
Subject: [PATCH 07/11] util: Add missing copyright notices

Change-Id: I243046c17264eb5c522285096ecf9c7e5e968322
---
 util/github-runners-vagrant/Vagrantfile       | 26 +++++++++++++++++++
 .../provision_nonroot.sh                      | 26 +++++++++++++++++++
 util/github-runners-vagrant/provision_root.sh | 26 +++++++++++++++++++
 util/github-runners-vagrant/vm_manager.sh     | 26 +++++++++++++++++++
 4 files changed, 104 insertions(+)

diff --git a/util/github-runners-vagrant/Vagrantfile b/util/github-runners-vagrant/Vagrantfile
index f81d53e145..ad294a3b38 100644
--- a/util/github-runners-vagrant/Vagrantfile
+++ b/util/github-runners-vagrant/Vagrantfile
@@ -1,6 +1,32 @@
 # -*- mode: ruby -*-
 # vi: set ft=ruby :
 
+# Copyright (c) 2023 The Regents of the University of California
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met: redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer;
+# redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution;
+# neither the name of the copyright holders nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 PERSONAL_ACCESS_TOKEN="<PERSONAL ACCESS TOKEN>"
 GITHUB_ORG="<GITHUB_ORG>"
 HOSTNAME="<VM NAME>"
diff --git a/util/github-runners-vagrant/provision_nonroot.sh b/util/github-runners-vagrant/provision_nonroot.sh
index 4465b5a192..d817959285 100644
--- a/util/github-runners-vagrant/provision_nonroot.sh
+++ b/util/github-runners-vagrant/provision_nonroot.sh
@@ -1,5 +1,31 @@
 #!/usr/bin/env bash
 
+# Copyright (c) 2023 The Regents of the University of California
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met: redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer;
+# redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution;
+# neither the name of the copyright holders nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 # fail on unset variables and command errors
 set -eu -o pipefail # -x: is for debugging
 
diff --git a/util/github-runners-vagrant/provision_root.sh b/util/github-runners-vagrant/provision_root.sh
index 74b4facdda..5a42556779 100644
--- a/util/github-runners-vagrant/provision_root.sh
+++ b/util/github-runners-vagrant/provision_root.sh
@@ -1,5 +1,31 @@
 #!/usr/bin/env bash
 
+# Copyright (c) 2023 The Regents of the University of California
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met: redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer;
+# redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution;
+# neither the name of the copyright holders nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 # fail on unset variables and command errors
 set -eu -o pipefail # -x: is for debugging
 
diff --git a/util/github-runners-vagrant/vm_manager.sh b/util/github-runners-vagrant/vm_manager.sh
index 4854f5a06f..280ee431bc 100755
--- a/util/github-runners-vagrant/vm_manager.sh
+++ b/util/github-runners-vagrant/vm_manager.sh
@@ -1,5 +1,31 @@
 #!/bin/bash
 
+# Copyright (c) 2023 The Regents of the University of California
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met: redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer;
+# redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution;
+# neither the name of the copyright holders nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
 NUM_RUNNERS=3
 RUNNER_PREFIX_PREFIX="$(hostname)"
 

From 0e5c6d9f501933ebf19affeb47480cfa8ca5a88c Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Wed, 4 Oct 2023 18:06:58 -0700
Subject: [PATCH 08/11] util: Resize VM root partition max size to ~128GB

Prior to this change we were limited to a root partition with only 60GB
of space which caused issues when running larger simulations (see:
https://github.com/gem5/gem5/issues/165).

There are two factors in this issue which this patch resolves:

1. The root partition in the VM was capped at 60GB despite the virtual
machines size being capped at 128GB. This resulted in libvirt giving the
VM free space it couldn't use. To fix this `lvextend` was added to the
"provision_root.sh" script to resize the root partition to fill the
available space.
2. The virtual machine size can be set via the `machine_virtual_size`
parameter. The minimum and default value is 128GB. This wasn't exposed
previously. Now, if we required, we can increase the size of the VM/Root
partition if we require (though I believe 128GB is more than sufficient
for now).

Fixes: https://github.com/gem5/gem5/issues/165
Change-Id: I82dd500d8807ee0164f92d91515729d5fbd598e3
---
 util/github-runners-vagrant/Vagrantfile       | 3 ++-
 util/github-runners-vagrant/provision_root.sh | 3 +++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/util/github-runners-vagrant/Vagrantfile b/util/github-runners-vagrant/Vagrantfile
index ad294a3b38..2a891b9688 100644
--- a/util/github-runners-vagrant/Vagrantfile
+++ b/util/github-runners-vagrant/Vagrantfile
@@ -40,9 +40,10 @@ Vagrant.configure("2") do |config|
   config.ssh.password = "vagrant"
 
   config.vm.provider "libvirt" do |vb|
-    # Customize the amount of cpus and memory on the VM:
+    # Customize the amount of cpus, memory, and storage on the VM:
     vb.cpus = "4".to_i
     vb.memory = "16384".to_i
+    vb.machine_virtual_size = 128 # 128G is the minimum.
   end
 
   # sets up vm
diff --git a/util/github-runners-vagrant/provision_root.sh b/util/github-runners-vagrant/provision_root.sh
index 5a42556779..f18a170c95 100644
--- a/util/github-runners-vagrant/provision_root.sh
+++ b/util/github-runners-vagrant/provision_root.sh
@@ -72,3 +72,6 @@ usermod -aG docker vagrant
 
 # Cleanup
 apt-get autoremove -y
+
+# Resize the root partition to fill up all the free size on the disk
+lvextend -l +100%FREE $(df / --output=source | sed 1d)

From 6571a54a65a1e65f9118d0a217c61db7c9611dad Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Wed, 4 Oct 2023 18:45:53 -0700
Subject: [PATCH 09/11] util: Use a Multi-Machine Vagrantfile

This patch removed the bespoke "vm_manager.sh" script in favor of a
Multi-Machine Vagrantfile.

With this the users needs to only change the variables in Vagrantfile
then use the standard `vagrant` commands to launch the VMs/Runners.

Change-Id: Ida5d2701319fd844c6a5b6fa7baf0c48b67db975
---
 util/github-runners-vagrant/README.md     | 68 ++++++-----------------
 util/github-runners-vagrant/Vagrantfile   | 47 +++++++++-------
 util/github-runners-vagrant/vm_manager.sh | 54 ------------------
 3 files changed, 43 insertions(+), 126 deletions(-)
 delete mode 100755 util/github-runners-vagrant/vm_manager.sh

diff --git a/util/github-runners-vagrant/README.md b/util/github-runners-vagrant/README.md
index 0b93ff6693..ea7fd16957 100644
--- a/util/github-runners-vagrant/README.md
+++ b/util/github-runners-vagrant/README.md
@@ -1,6 +1,6 @@
 # Setting up a Github Actions Runner with Vagrant
 
-This directory provides a way to setup a Github Actions runner using Vagrant to host the runner in a Virtual machine.
+This directory provides a way to setup Github Actions runners using Vagrant to host them in Virtual machines.
 
 This tutorial has been written with the assumption of running on a machine with Ubuntu 22.04.
 Setting up a runner on a different OS may require some changes.
@@ -21,13 +21,6 @@ sudo apt-get install qemu libvirt-daemon-system libvirt-clients ebtables dnsmasq
 sudo apt purge vagrant-libvirt
 ```
 
-## Set up the Vagrant for the GitHub repository
-
-First, generate a Personal Access Token, which you can create [here](https://github.com/settings/tokens)
-Make sure to set admin permissions on this token, then replace the of `<PERSONAL ACCESS TOKEN>` in the Vagrantfile  with your token.
-
-Next, replace instances of `<GITHUB_ORG>` with your GitHub organization you wish to add this runner.
-
 ## Install Vagrant Plugins
 
 Once everything is set properly, set the `VAGRANT_HOME` environment variable to the directory in which the Vagrant files and other scripts are stored (i.e., the CWD).
@@ -45,62 +38,35 @@ vagrant plugin install vagrant-libvirt
 vagrant plugin install vagrant-reload
 ```
 
-## Creating the virtual machine
+## Creating the virtual machines
 
-The Vagrantfile in this directory defines a VM that can be used to create a GitHub Actions runner.
-It has 4-cores, 16GB of RAM, and 60GB of disk space.
+The Vagrantfile in this directory defines the VMs that can be built and used to create a GitHub Actions runner.
+This standard VM has 4-cores, 16GB of RAM, and 60GB of disk space.
 This is sufficient to both compile gem5 and run most simulations.
 
-Each VM on your host system must have a unique name.
-Give the VM to be created a unique name by setting the `<VM NAME>` variables in the Vagrantfile you wish to utilize.
+At the top of the Vagrantfile, there are a few variables that must be set prior to creating the VMs.
 
-Then run:
+* `NUM_RUNNERS`: The number of runners to create.
+* `PERSONAL_ACCESS_TOKEN`: The GitHub personal access token to use.
+You can generate a Personal Access Token [here](https://github.com/settings/tokens)
+Make sure to set admin permissions on this token.
+* `GITHUB_ORG`: The GitHub organization to add the runners to.
+E.g., if the URL to your organization is https://github.com/orgs/gem5, then the variable should be set to "gem5".
+* `HOSTNAME` : The hostname of the VM to be created (note, this will be appended with a number to create a unique hostname for each VM).
+E.g., if set to `my-machine` and the number of runners set to `2`, two VMs will be created.
+One called `my-machine-1` and the other `my-machine-2`.
+
+When set simply run:
 
 ```sh
 vagrant up --provider=libvirt
 ```
 
-This should automatically create your machine, as well as configure and start up a Github Actions runner.
+This should automatically create your machines then configure and start up a Github Actions runner in each.
 You can check the status of the runner here: https://github.com/organizations/{GITHUB_ORG}/settings/actions/runners
 
 If the runner ever shows as offline, you can rerun the `vagrant up --provider=libvirt` command to make sure everything is working properly.
 
-If you wish to create more than one runner you must edit the `<VM NAME>` in the Vagrant file.
-
-## Helper scripts
-
-The "vm_manager" script can be used to set up multiple builder and runner VMs.
-To use this script simply modify the `NUM_RUNNERS` and `RUNNER_PREFIX` variables to the desired values.
-Then run the script with:
-
-```sh
-./vm_manager.sh
-```
-
-This script will create any VMs that don't already exist and ensure those that do exists are running.
-
-If you wish to destroy all the VMs you can run:
-
-```sh
-./vm_manager.sh destroy
-```
-
-**Note:** This script assumes "VAGRANT_HOME" is set to the CWD.
-
-## Improving stability
-
-Occasionally GitHub runner services, or VMs, go down. This is often silent and
-usually only noticable from going to the GitHub repo page "settings" -> "actions" -> "runners" and observing the status.
-When the VMs or the service stop working they need restarted.
-To do so you can sun `./vm_manager.sh`. This will cycle through the VMs and execute a `vagrant up` command.
-This does one of three things depending on the state of the VM:
-
-1. If the VM is down this will bring the VM back online and start the GitHub runner service.
-2. If the VM is up but the GitHub runner service is down, this will start the GitHub runner service.
-3. If the VM is up and the GitHub runner service is running (i.e., everything is fine) then this does nothing.
-
-Given there is no harm in running this command frequently, we recommend setting up a cron job to automatically execute `./vm_manager.sh` every few hours.
-
 ## Troubleshooting
 
 ### The default libvirt disk image storage pool is on the wrong drive
diff --git a/util/github-runners-vagrant/Vagrantfile b/util/github-runners-vagrant/Vagrantfile
index 2a891b9688..3446f703dd 100644
--- a/util/github-runners-vagrant/Vagrantfile
+++ b/util/github-runners-vagrant/Vagrantfile
@@ -27,34 +27,39 @@
 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
+NUM_RUNNERS=0 # Set this to the desired number of runners.
 PERSONAL_ACCESS_TOKEN="<PERSONAL ACCESS TOKEN>"
 GITHUB_ORG="<GITHUB_ORG>"
 HOSTNAME="<VM NAME>"
 
 Vagrant.configure("2") do |config|
-  config.vm.box = "generic/ubuntu2204"
-  config.vm.box_check_update = true
-  config.vm.define "#{HOSTNAME}"
-  config.vm.hostname = "#{HOSTNAME}"
   config.ssh.username = "vagrant"
   config.ssh.password = "vagrant"
 
-  config.vm.provider "libvirt" do |vb|
-    # Customize the amount of cpus, memory, and storage on the VM:
-    vb.cpus = "4".to_i
-    vb.memory = "16384".to_i
-    vb.machine_virtual_size = 128 # 128G is the minimum.
+  (1..NUM_RUNNERS).each do |i|
+
+    config.vm.define "#{HOSTNAME}-#{i}" do |runner|
+      runner.vm.hostname = "#{HOSTNAME}-#{i}"
+      runner.vm.box = "generic/ubuntu2204"
+      runner.vm.box_check_update = true
+
+      runner.vm.provider "libvirt" do |vb|
+        # Customize the amount of cpus, memory, and storage on the VM:
+        vb.cpus = "4".to_i
+        vb.memory = "16384".to_i
+        vb.machine_virtual_size = 128 # 128G is the minimum.
+      end
+
+      # sets up vm
+      runner.vm.provision :shell, path: "provision_root.sh"
+      runner.vm.provision :shell, privileged: false, path: "provision_nonroot.sh"
+      # The provision_root.sh adds the vagrant user to the docker group, so we need to reload the VM.
+      runner.vm.provision :reload
+      # Copy the "action-run.sh" script from the host to the VM.
+      runner.vm.provision "file", source: "./action-run.sh", destination: "/tmp/action-run.sh"
+      runner.vm.provision :shell, privileged: false,  inline: "cp /tmp/action-run.sh ."
+      # Execute the actions-run.sh script on every boot. This configures the and starts the runner.
+      runner.vm.provision :shell, privileged: false, run: 'always', inline: "./action-run.sh #{PERSONAL_ACCESS_TOKEN} #{GITHUB_ORG} >> action-run.log 2>&1 &"
+    end
   end
-
-  # sets up vm
-  config.vm.provision :shell, path: "provision_root.sh"
-  config.vm.provision :shell, privileged: false, path: "provision_nonroot.sh"
-  # The provision_root.sh adds the vagrant user to the docker group, so we need to reload the VM.
-  config.vm.provision :reload
-  # Copy the "action-run.sh" script from the host to the VM.
-  builder.vm.provision "file", source: "./action-run.sh", destination: "/tmp/action-run.sh"
-  builder.vm.provision :shell, privileged: false,  inline: "cp /tmp/action-run.sh ."
-  # Execute the actions-run.sh script on every boot. This configures the and starts the runner.
-  config.vm.provision :shell, privileged: false, run: 'always', inline: "./action-run.sh #{PERSONAL_ACCESS_TOKEN} #{GITHUB_ORG} >> action-run.log 2>&1 &"
-
 end
diff --git a/util/github-runners-vagrant/vm_manager.sh b/util/github-runners-vagrant/vm_manager.sh
deleted file mode 100755
index 280ee431bc..0000000000
--- a/util/github-runners-vagrant/vm_manager.sh
+++ /dev/null
@@ -1,54 +0,0 @@
-#!/bin/bash
-
-# Copyright (c) 2023 The Regents of the University of California
-# All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met: redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer;
-# redistributions in binary form must reproduce the above copyright
-# notice, this list of conditions and the following disclaimer in the
-# documentation and/or other materials provided with the distribution;
-# neither the name of the copyright holders nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-NUM_RUNNERS=3
-RUNNER_PREFIX_PREFIX="$(hostname)"
-
-SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
-export VAGRANT_HOME=${SCRIPT_DIR}
-
-param="up"
-if [[ $# -ge 1 ]]; then
-    param=$1
-    if [[ "${param}" != "destroy" ]] && [[ "${param}" != "shutdown" ]]; then
-        echo "Only valid parameters are 'destroy' and 'shutdown' to destroy all VMs or shutdown all VMs"
-        exit 1
-    fi
-fi
-
-
-for (( i=1; i<=NUM_RUNNERS; i++ )); do
-    sed -i "s/HOSTNAME=.*/HOTNAME=\"${RUNNER_PREFIX}-${i}\"/g" Vagrantfile
-    if [[ "${param}" == "destroy" ]]; then
-        VAGRANT_VAGRANTFILE=Vagrantfile vagrant destroy -f
-    elif [[ "${param}" == "shutdown" ]]; then
-        VAGRANT_VAGRANTFILE=Vagrantfile vagrant halt -f
-    else
-        VAGRANT_VAGRANTFILE=Vagrantfile vagrant up --provider=libvirt
-    fi
-done

From 53219bf82751db700bf66087b889444126ca187f Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Wed, 4 Oct 2023 18:48:55 -0700
Subject: [PATCH 10/11] util: Add Troubleshooting for "Vagrant failed..."

Change-Id: I01e637f09084acb6c5fbd7800b3e578a43487849
---
 util/github-runners-vagrant/README.md | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/util/github-runners-vagrant/README.md b/util/github-runners-vagrant/README.md
index ea7fd16957..ca504e3c63 100644
--- a/util/github-runners-vagrant/README.md
+++ b/util/github-runners-vagrant/README.md
@@ -86,3 +86,15 @@ virsh pool-define default-pool.xml # From here we re-add the default.
 virsh pool-start default
 virsh pool-autostart default
 ```
+
+### Error: "Vagrant failed to initialize at a very early stage"
+
+W set the `VAGRANT_HOME` environment variable to the CWD.
+It's likely this has become unset The solution is simple.
+Within the directory containing "Vagrantfile":
+
+```sh
+VAGRANT_HOME=`pwd` vagrant <command>
+```
+
+You may want to set `VAGRANT_HOME` in your .bashrc or .zshrc.

From df3bcaf143cfd256399d3f7f6b7e6e664d03a23c Mon Sep 17 00:00:00 2001
From: "Bobby R. Bruce" <bbruce@ucdavis.edu>
Date: Fri, 6 Oct 2023 00:27:23 -0700
Subject: [PATCH 11/11] util: Make all runs "build" and "run"

Change-Id: If9ecf467efa5c7118d34166953630e6c436c55a4
---
 util/github-runners-vagrant/Vagrantfile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/util/github-runners-vagrant/Vagrantfile b/util/github-runners-vagrant/Vagrantfile
index 3446f703dd..7743c3e6a4 100644
--- a/util/github-runners-vagrant/Vagrantfile
+++ b/util/github-runners-vagrant/Vagrantfile
@@ -59,7 +59,7 @@ Vagrant.configure("2") do |config|
       runner.vm.provision "file", source: "./action-run.sh", destination: "/tmp/action-run.sh"
       runner.vm.provision :shell, privileged: false,  inline: "cp /tmp/action-run.sh ."
       # Execute the actions-run.sh script on every boot. This configures the and starts the runner.
-      runner.vm.provision :shell, privileged: false, run: 'always', inline: "./action-run.sh #{PERSONAL_ACCESS_TOKEN} #{GITHUB_ORG} >> action-run.log 2>&1 &"
+      runner.vm.provision :shell, privileged: false, run: 'always', inline: "./action-run.sh #{PERSONAL_ACCESS_TOKEN} #{GITHUB_ORG} "run,build" >> action-run.log 2>&1 &"
     end
   end
 end