# Experimenting

This document focuses on Modality's functionality for running experiments. It assumes familiarity with the underlying functionality to instrument your system, collect data, and analyze it seen in the previous guides. For a higher level overview of concepts see the Introduction. For terminology see the Glossary.

While Modality can be used in many different ways to help you understand and improve your system, at its core it is a tool for automated experimentation. Experimentation lets you ask questions about your system to learn about its behavior, and automating experiments enables you to test vastly more combinations of parameters than would otherwise be possible. The learnings derived from your experiments can help you find faults in system design, fix bugs before users find them, or optimize parameters for system performance, among other things.

A diagram of Modality's process, introducing mutations into your system, observing many conditions, and returning sophisticated analysis

# Designing Experiments

The first step to learning about your system is to design your experiment. Typically an experiment will start with a hypothesis, test various combinations of independent variable values, and, for each combination, measure some resulting values of dependent variables to see whether the hypothesis holds. Modality's building blocks neatly fit this model: mutations enable you to set values for your independent variables, and the query language lets you define dependent variables to measure, however simple or complex.

In Modality, experimental design is encoded in objectives. Objectives are defined in a TOML configuration file which is then used to create the objective in the CLI—for more details, see the reference. Besides specifying a name, objective configuration files are broken down into three main parts which map to the experimental process: mutation constraints, measurements, and stopping conditions.

# Mutation Constraints

Mutation constraints guide how Modality will generate values for your independent variables. You can specify global constraints for the maximum number of mutations per mutation epoch, preconditions for when mutations should be triggered, and tags to limit which mutators can be used in this experiment. You can also constrain individual mutators' probe, precondition, and parameter values.

TIP

A mutation epoch is a region of the trace delimited by when mutations occur and can be thought of as a single run of the experiment for some set of values.

In the following example, we specify some basic global constraints and limit the parameter values for a specific mutator:

# Global mutation constraints
[mutation_constraints]
# Allow up to 2 mutation attempts per mutation-epoch
max_concurrent_mutations = 2

# Only consider mutators which have at least one of the specified tags
mutator_tags = ['drone', 'environmental-forces', 'simulator']

# Don't generate any mutations until we have seen the ready event
mutation_precondition = 'MATCH (name = "SYSTEM_STATE_READY") AS ReadyState'


# Mutator-specific constraints
[[mutator]]
name = 'simulation-environment-mutator'

# Restrict mutations to the specific probe SIM_MUTATOR_PLUGIN
probe = 'SIM_MUTATOR_PLUGIN'

    # Constrain the impact force to [0.1, 30.0] Newtons
    [[mutator.param_constraints]]
    name = 'impact-force-magnitude'
    range = [0.1, 30.0]

    # Constrain the impact force link to either ROTOR_0 or ROTOR_1
    [[mutator.param_constraints]]
    name = 'impact-force-location-link'
    range = ['ROTOR_0', 'ROTOR_1']

# Measurements

Measurements represent the dependent variables in our experiment—the values we expect to be affected by our independent variables. At least one measurement is required per objective. In Modality, measurements are trace queries, and can therefore represent arbitrarily complex (or simple) sequences of events. For more information about trace queries, see the guide or the reference.

A basic measurement could simply check that some event occurred. In this case, we check for successful startup:

# This measurement checks for successful startup
[[measurement]]
name = 'Successful startup'

# The trace query expression to check
check = 'MATCH (name = "STARTUP_SUCCEEDED" AND probe = "COMMANDER") AS SuccessfulStartup'

Measurements also include fields to indicate whether they are expected to pass and to specify conditions indicating that the experiment should be stopped based on this measurement. In this case we specify should_pass = false, indicating that we expect the check to fail, i.e. we want there to not be any critical failure events. We also say failing_measurements = 0, meaning that if anything causes a critical failure the experiment should be stopped so we can investigate.

# This measurement simply fails if any failure-like events with severity greater than 8 occur
[[measurement]]
name = 'No critical events occur'

# The check is expected to fail
should_pass = false

# Measurement-specific stopping conditions, stop immediately if this check doesn't pass
failing_measurements = 0

# The trace query expression to check
check = 'MATCH (severity > 8) AS AnyCriticalFailures'

The above examples used extremely simple queries. Here we demonstrate a more complex measurement using the query language's filtering and aggregation capabilities:

# This measurement checks that the IMU gyroscope instability metric is nominal
[[measurement]]
name = 'IMU gyroscope instability is acceptable'

check = '''
MATCH
    (name = "IMU_GYRO_INSTABILITY" AND probe = "DRONE_IMU") AS Metric

FILTER
    Metric.payload != 0.0

AGGREGATE
    max(Metric.payload) < 0.45,
    mean(Metric.payload) < 0.2,
    stddev(Metric.payload) < 0.1
'''

All measurements defined in an objective will be recorded for each mutation epoch, i.e. each run of the experiment. Then, when analyzing the results of your experiment, you can see which mutation values led to interesting measurements.

# Stopping Conditions

The final part of defining an automated experiment is simply saying when to stop. You can set Modality loose on your system to see the results of hundreds or thousands of different mutations, but eventually the experimentation must end so you can analyze the results and gain insight into your system's behavior. The stopping_conditions section of an objective configuration lets you specify a maximum number of times to run the experiment, a time limit, and maximum numbers of times when measurements either passed or failed. In addition, as we saw above, you can create stopping conditions for any individual measurement based on the number of times it passed or failed.

# General stopping conditions, OR'd together
[stopping_conditions]
# Maximum number of related mutation-epoch attempts, i.e. runs of the experiment
attempts = 50

# Total wall-clock time to run
time = "2h 15min"

# Maximum number of times all the measurements passed
passing_measurements = 40

# Maximum number of times not all the measurements passed
failing_measurements = 20

# Managing Objectives

Once you have recorded your experiment design in an objective configuration file you must create the objective in the Modality CLI to begin interacting with it. You can also use the modality objective command to list existing objectives, inspect their definitions, and eventually see mutations and measurements associated with them.


$ modality objective create ./sample_objective.TOML

$ modality objective list
NAME               SESSIONS  SUTS
example-objective  1         1
sample objective   1         1

$ modality objective instances 'example-objective'
NAME               INSTANCE  SESSION               SUT          STATUS
example-objective  1         2021-04-16T06-51-16Z  example-sut  STOPPED

$ modality objective inspect 'example-objective'
Name: example-objective
Created at: 2021-04-16 13:51:16 UTC
Created by: example-user
SUTS: [example-sut]
Instances By Session
  2021-04-16T06-51-16Z
    [1]
Stopping Conditions
  Attempts: 20
  Time: 2min
  Passing Measurements: 15
  Failing Measurements: 5
Mutation Constraints
  Max Concurrent Mutations: 1
  Mutation Precondition: 
  Mutator Tags: []
Mutator Constraints: 2
  heartbeat-delay-mutator
    Probe: CONSUMER_PROBE
    Mutation Precondition: 
    Parameter Constraints: 1
      Parameter: heartbeat-delay
      Range: [500, 2000]
  sample-data-mutator
    Probe: PRODUCER_PROBE
    Mutation Precondition: MATCH MEASUREMENT_SAMPLED @ PRODUCER_PROBE
    Parameter Constraints: 1
      Parameter: sample
      Range: [-100, 100]
Measurements: 7
  Consumer lifecycle
    Should Pass: true
    Stopping Conditions
      Passing Measurements: 
      Failing Measurements: 
    Check:
      MATCH
          (INIT @ CONSUMER_PROBE) AS Init,
          (RECVD_MEASUREMENT_MSG @ CONSUMER_PROBE) AS RecvdMsg,
          (MEASUREMENT_SAMPLE_CHECK @ CONSUMER_PROBE AND outcome = PASS) AS SampleChecked,
          (DEINIT @ CONSUMER_PROBE) AS Fini
      WHEN
          Init -> RecvdMsg -> SampleChecked -> Fini

...

# Running Experiments

Once you have created an objective in Modality there are two main ways to run your experiment. The first is to handle all mutations and measurements manually. To do this, you must first open a session. Then, you can either call modality mutate to generate an automatic mutation or modality objective sample to take measurements for a given objective. Calling either of these commands will create an objective instance associated with this session or, if an open instance of that objective already exists, will associate the resulting mutation or measurement with that instance.

In the below example we show the simplest case of a manually managed experiment, with a single call to modality mutate followed by taking measurements over the entire session. Note that modality objective sample optionally accepts a region expression so that you can do repeated mutations and limit measurements to the relevant regions of the trace.

// Open the session for this experiment
$ modality session open simple-experiment example-sut
If you would like to use this session as your default run 'modality session use "auto-experiment"'

// Set the newly opened session as the default
$ modality session use simple-experiment

// Run a script to get the SUT to a ready state
$ ./setup-sut.sh

// Inject an automation mutation guided by our objective
$ modality mutate --objective-name example-objective

// Stimulate the SUT to see behavior resulting from mutations
$ ./stimulus.sh

// Take measurements for our objective over the entire default session
$ modality objective sample example-objective

// Close the session and its associated objective instance
$ modality session close simple-experiment

With these commands you get fine grained control over how exactly you want to run your experiment, and by calling them from a script you can automate the entire process. In addition, the modality execute command makes it easier to run experiments with the most common structure. This command takes a user-provided script which stimulates the SUT and injects mutations in whatever way is relevant to the experiment and calls that script in a loop. Measurements are recorded for each set of mutations, and this process continues until one of the objective stopping conditions is met. modality execute also accepts optional flags to specify stopping conditions to use instead of the ones in the objective and/or to control what output to print.

Here is an example script to be used with modality execute. It opens a scope, starts up the SUT processes, injects an automatic mutation, waits for resulting behavior, then closes the scope and shuts down.

#!/usr/bin/bash

set -e

PRODUCER_APP_PATH="$PWD/build/producer"
[ ! -x "$PRODUCER_APP_PATH" ] && echo "producer application is missing, did you run ./scripts/build-example?" && exit 1
CONSUMER_APP_PATH="$PWD/build/consumer"
[ ! -x "$CONSUMER_APP_PATH" ] && echo "consumer application is missing, did you run ./scripts/build-example?" && exit 1
MONITOR_APP_PATH="$PWD/build/monitor"
[ ! -x "$MONITOR_APP_PATH" ] && echo "monitor application is missing, did you run ./scripts/build-example?" && exit 1

SCOPE_NAME="StimulusScriptRunning"
modality session scope open "$SCOPE_NAME"

# Start the applications in the background, should run for ~5 seconds
"$CONSUMER_APP_PATH" &
CONSUMER_APP_PID=$!
"$PRODUCER_APP_PATH" &
PRODUCER_APP_PID=$!
"$MONITOR_APP_PATH" &
MONITOR_APP_PID=$!

sleep 1

# Generate automatic mutations based on objective configuration
modality mutate \
    --objective-name "example-objective"

sleep 5

wait $CONSUMER_APP_PID
wait $PRODUCER_APP_PID
wait $MONITOR_APP_PID

modality session scope close "$SCOPE_NAME"

exit 0

Now, to let modality execute automatically run our experiment until an objective stopping condition is met, we can do the following:

// Open a session for our experiment
$ modality session open auto-experiment example-sut
If you would like to use this session as your default run 'modality session use "auto-experiment"'

// Set the newly opened session as the default
$ modality session use auto-experiment

// Let Modality run the experiment until stopping conditions are reached
$ modality execute --objective-name example-objective --disable-script-io-inherit ./stimulus.sh
┌─ Objective Status ───────────────────────────────────────────────────────────┐
│                                                                              │
│  Objective:   example-objective                                              │
│  Instance:    1                                                              │
│  Created At:  2021-04-21 21:55:24 UTC                                        │
│  Created By:  example-user                                                   │
│  Stopped At:                                                                 │
│  Cycle Count: 0                                                              │
│                                                                              │
├─ Stopping Conditions Progress ───────────────────────────────────────────────┤
│                                                                              │
│  Time:                 00:00:00 / 00:02:00   [                        ] 0%   │
│  Passing Measurements: 0 / 15                [                        ] 0%   │
│  Failing Measurements: 0 / 5                 [                        ] 0%   │
│                                                                              │
└──────────────────────────────────────────────────────────────────────────────┘

Execute will run until a stopping condition is met, updating progress in the terminal as it goes. When it exits you can close the open session and move on to analyzing your experimental results.

# Analyzing Results

Once an invocation of modality execute has completed all that's left is to look at the results to gain a better understanding of your system's behavior. Objectives can be associated with more than one session: each of these associations is an objective instance. Thus, to get the results of a particular experiment, you must inspect the corresponding objective instance. The default behavior when inspecting an objective instance is to print some high level summary information:


$ modality objective list
NAME               SESSIONS  SUTS
example-objective  1         1
sample objective   1         1

$ modality objective instances 'example-objective'
NAME               INSTANCE  SESSION               SUT          STATUS
example-objective  1         2021-04-16T06-51-16Z  example-sut  STOPPED

$ modality objective inspect 'example-objective' 1
Name: example-objective
SUT: example-sut
Session: 2021-04-16T06-51-16Z
Created at: 2021-04-16 13:51:16 UTC
Created by: example-user
Stopped at: 2021-04-16 13:53:01 UTC
Stopping Conditions Reached: true
Stopping Conditions Progress
  Time: 00:01:45
  Passing Measurements: 15
  Failing Measurements: 0
Passing Measurements: 7
  Consumer lifecycle: 15
  Consumer to monitor heartbeat: 15
  Monitor lifecycle: 15
  Passing expectation never preceded by out-of-bounds value: 15
  Producer lifecycle: 15
  Producer to consumer communications: 15
  Producer to monitor heartbeat: 15
Failing Measurements: 0

To get details on each mutation epoch and its corresponding measurements, use the -v flag:


$ modality objective inspect 'example-objective' 1 -v
Name: example-objective
SUT: example-sut
Session: 2021-04-16T06-51-16Z
Created at: 2021-04-16 13:51:16 UTC
Created by: example-user
Stopped at: 2021-04-16 13:53:01 UTC
Stopping Conditions Reached: true
Stopping Conditions Progress
  Time: 00:01:45
  Passing Measurements: 15
  Failing Measurements: 0
Passing Measurements: 7
  Consumer lifecycle: 15
  Consumer to monitor heartbeat: 15
  Monitor lifecycle: 15
  Passing expectation never preceded by out-of-bounds value: 15
  Producer lifecycle: 15
  Producer to consumer communications: 15
  Producer to monitor heartbeat: 15
Failing Measurements: 0
Measurement Outcomes
  Execution Run: 0
    Consumer lifecycle
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 2
      Executed at: 2021-04-16 13:51:35 UTC
      Outcome: PASS
    Consumer to monitor heartbeat
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 2
      Executed at: 2021-04-16 13:51:35 UTC
      Outcome: PASS
    Monitor lifecycle
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 2
      Executed at: 2021-04-16 13:51:35 UTC
      Outcome: PASS
    Passing expectation never preceded by out-of-bounds value
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 2
      Executed at: 2021-04-16 13:51:35 UTC
      Outcome: PASS
    Producer lifecycle
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 2
      Executed at: 2021-04-16 13:51:35 UTC
      Outcome: PASS
    Producer to consumer communications
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 2
      Executed at: 2021-04-16 13:51:35 UTC
      Outcome: PASS
    Producer to monitor heartbeat
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 2
      Executed at: 2021-04-16 13:51:35 UTC
      Outcome: PASS
  Execution Run: 1
    Consumer lifecycle
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 4
      Executed at: 2021-04-16 13:51:41 UTC
      Outcome: PASS
    Consumer to monitor heartbeat
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 4
      Executed at: 2021-04-16 13:51:41 UTC
      Outcome: PASS
    Monitor lifecycle
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 4
      Executed at: 2021-04-16 13:51:41 UTC
      Outcome: PASS
    Passing expectation never preceded by out-of-bounds value
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 4
      Executed at: 2021-04-16 13:51:41 UTC
      Outcome: PASS
    Producer lifecycle
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 4
      Executed at: 2021-04-16 13:51:41 UTC
      Outcome: PASS
    Producer to consumer communications
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 4
      Executed at: 2021-04-16 13:51:41 UTC
      Outcome: PASS
    Producer to monitor heartbeat
      Region: session = '2021-04-16T06-51-16Z' AND mutation_epoch = 4
      Executed at: 2021-04-16 13:51:41 UTC
      Outcome: PASS
...

With this output you can see which mutation values led to measurements of interest. You can then query the trace or use the log command to gain an understanding of why your system behaves as it does and to eventually fix problems before they occur.