Advanced Gumby Experiments and Scenarios¶
In the previous tutorial, we have shown how to run a simple experiment with Gumby. Gumby spawned two instances that run independently from each other. Most of the time, however, you want more control over the actions performed during your experiment. In this tutorial, we will setup a more advanced experiment. Our experiment will spawn five instances that are time-synchronized with each other. After five seconds, each instance will write its ID to a file. When the experiment ends, we will sum up all these instance IDs and write it to a file.
Configuration file¶
Our configuration file will look as follows:
experiment_name = synchronized_instances
experiment_time = 20
instances_to_run = 5
local_instance_cmd = "process_guard.py -c launch_scenario.py -n $INSTANCES_TO_RUN -t $EXPERIMENT_TIME -m $OUTPUT_DIR -o $OUTPUT_DIR "
post_process_cmd = post_process_write_ids.sh
# The scenario file to run after an instance has spawned.
scenario_file = write_ids.scenario
# The host and port of the synchronization server.
sync_host = localhost
sync_port = __unique_port__
Most of the configuration options should be familiar at this point. We annotated two new configuration options, namely
sync_port
and scenario_file
. We further explain the effect of these configuration options.
Scenario Files¶
Note that we specify a scenario file in our configuration file. A scenario file is a file that contains all events during our experiment. Each event includes a time when the event should fire, and a reference to a Python method. The scenario file is a very convenient way of building an experiment, and can quickly be reused. For example, a scenario file can contain a particular workload that the system should process. The scenario file used in our experiment looks as follows:
&module docs.tutorials.simple_module.SimpleModule
@0:5 write_peer_id
@0:9 stop
The first two lines specify which modules to include. We will explain experiment modules later in this tutorial. For now, we will focus on the two events that are specified in this scenario file:
@0:5 write_peer_id
@0:9 stop
The above two events are scheduled to fire after five and nine seconds after experiment start, respectively.
The first event calls the write_peer_id
method that simply writes the ID of the instance, or peer, to a file.
Specifically, each Gumby instance is assigned a unique ID, starting from 1.
The write_peer_id
method is defined in the SimpleModule
class in the simple_module.py
file.
This Python code is imported in the first line of our scenario file.
The stop
method is implemented in the ExperimentModule
class, which is the superclass of SimpleModule
.
By default, an event will be executed by all peers. One can restrict which instances run a particular event. For example, the line below specifies that only peer 2 writes its peer ID:
@0:5 write_peer_id {2}
Experiment Modules¶
Gumby enables experiment designers to provide their functionality in separate modules.
In the experiment described in this tutorial, we import the SimpleModule
class in our experiment, which has the
following content:
from gumby.experiment import experiment_callback
from gumby.modules.experiment_module import ExperimentModule
class SimpleModule(ExperimentModule):
"""
A very simple experiment module that has a single callback.
"""
@experiment_callback
def write_peer_id(self):
"""
Simply write my peer ID to a file.
"""
with open("id.txt", "w") as id_file:
id_file.write("%d" % self.my_id)
This file contains the definition of the SimpleModule
class.
The class contains a single method, namely write_peer_id
.
This method simply writes the ID of the peer to a file named id.txt
.
Note that the name of this method corresponds to the event specified in our scenario file, and this method is invoked
when the event fires. The method is annotated with a experiment_callback
decorator. This is required to correctly
connect the event in the scenario file and the logic in the module.
An experiment can also import multiple modules. Gumby automatically imports these module on runtime and registers the events to the available callbacks. This allows re-use of experiment logic across different experiments.
Instance Synchronization¶
Each spawned instance independently executes the scenario file.
This makes it important that each instance starts roughly at the same time.
Gumby includes a synchronization server that prepares peers for the experiment, assigns IDs, and makes sure that peers
start execution of the scenario file roughly at the same time.
This synchronization process is coordinated by Gumby automatically.
Recall that, in contrast to the previous experiment, we do not set the experiment_server_cmd
configuration option
to blank.
This ensures that Gumby will spawn a synchronization server.
The sync_host
option specifies the IP address or host name of the machine running the synchronization server.
In this experiment we set it to localhost.
The sync_port
option in the configuration file specifies the port of the synchronization server, and indicates to
which port the spawned clients should connect. A value of __unique_port__
indicates that Gumby will pick a random
free port on runtime.
Post-experiment Data Aggregation¶
We now focus on the post-experiment script.
This post-experiment script is executed after the scenario file is finished and reads all written peer IDs and sums
them.
Note that in the configuration file, we specify to run the post_process_write_ids.sh
bash script, which looks
as follows:
#!/usr/bin/env bash
gumby/docs/tutorials/post_process_write_ids.py .
graph_process_guard_data.sh
This simple script first executes the post_process_write_ids.py
file and then calls the
graph_process_guard_data.sh
script to plot the graphs. The content of the post_process_write_ids.py
file is
as follows:
#!/usr/bin/env python3
import os
import sys
from gumby.statsparser import StatisticsParser
class IDStatisticsParser(StatisticsParser):
"""
Simply read all the id.txt files created by instances and sum up the numbers inside them.
"""
def aggregate_peer_ids(self):
peer_id_sum = 0
for _, filename, _ in self.yield_files('id.txt'):
with open(filename, "r") as peer_id_file:
read_peer_id = int(peer_id_file.read())
peer_id_sum += read_peer_id
with open("sum_id.txt", "w") as sum_id_file:
sum_id_file.write("%d" % peer_id_sum)
def run(self):
self.aggregate_peer_ids()
# cd to the output directory
os.chdir(os.environ['OUTPUT_DIR'])
parser = IDStatisticsParser(sys.argv[1])
parser.run()
This Python file defines the IDStatisticsParser
class which is a subclass of StatisticsParser
.
The latter class provides basic functionality to quickly aggregate data generated by peers.
Of particular interest is the yield_files
method that returns an iterator with files created by experiment peers
that match a particular pattern.
In the aggregate_peer_ids
method we iterate through all the files named id.txt
, read them, and aggregate the
integer value included in these files.
Then the result is written to the sum_id.txt
file.
Running the Experiment¶
You can run the experiment with the following command:
$ gumby/run.py gumby/docs/tutorials/synchronized_instances.conf
This will execute the experiment described above. You should see something similar to the log lines below:
INFO:ProcessRunner:[] ERR: 2021-07-17 17:55:23,644:INFO:1 of 5 expected subscribers connected.
INFO:ProcessRunner:[] ERR: 2021-07-17 17:55:26,639:INFO:2 of 5 expected subscribers connected.
INFO:ProcessRunner:[] ERR: 2021-07-17 17:55:27,645:INFO:4 of 5 expected subscribers connected.
INFO:ProcessRunner:[] ERR: 2021-07-17 17:55:27,646:INFO:All subscribers connected!
INFO:ProcessRunner:[] ERR: 2021-07-17 17:55:27,646:INFO:1 of 5 expected subscribers ready.
INFO:ProcessRunner:[] ERR: 2021-07-17 17:55:27,649:INFO:All subscribers are ready, pushing data!
INFO:ProcessRunner:[] ERR: 2021-07-17 17:55:27,649:INFO:Pushing a 359 bytes long json doc.
INFO:ProcessRunner:[] ERR: 2021-07-17 17:55:27,649:INFO:1 of 5 expected subscribers received the data.
INFO:ProcessRunner:[] ERR: 2021-07-17 17:55:27,649:INFO:Data sent to all subscribers, giving the go signal in 1.0 secs.
INFO:ProcessRunner:[] ERR: 2021-07-17 17:55:27,650:INFO:Starting the experiment!
These log lines indicate that the spawned instances are connecting with the synchronization server and that the experiment starts only after all instances have an ID assigned and are synchronized.
A file named sum_id.txt
should have been created in the experiment output directory. This file should contain the
value 15, which corresponds to the sum of the IDs of all participating peers (1+2+3+4+5). Note that Gumby also creates
sub-directories to store particular files created by individual peers. The directory name corresponds with the peer ID.
These sub-directories should contain the id.txt
file, created by the write_peer_id
method.
This tutorial covere more advanced concepts of Gumby, and shows how one can setup advanced experiments using scenario files and experiment modules. In the next tutorial, we show how to deploy and execute the above experiment on the DAS5 supercomputer.