How to Run Generative Tests

# Summary

Modality generative testing creates a vast number of distinct test conditions, runs them on your system, and checks how your system behaves. Simply write a script to exercise your system and use modality execute with an objective.

Pro Generative testing is a feature of the full version of Modality.

# Use Cases

Executing a Modality objective allows you to automatically generate a vast variety of changes to system state and measure their impacts in each test case. Generative testing is at the core of many powerful Modality use cases, such as finding undiscovered risk, reproducing bugs, and troubleshooting the cause of bugs effectively.

A diagram of Modality's generative testing process, introducing mutations into your system, observing many conditions including possible bugs, and returning sophisticated analysis

# Confirm Your Objective

This guide assumes you have at least one objective present in Modality. If you're not sure what objectives are available, call modality objective list.

To view the details of an objective, including its mutation constraints, measurements, and stopping conditions, call modality objective inspect <objective-name>. See the CLI reference for a full example.

If you do not have an objective that meets your testing needs, see How to Create an Objective.

# Example modality objective list

$ modality objective list
NAME               SESSIONS  SUTS
example-objective  1         1

$ modality objective inspect example-objective
Name: example-objective
Created at: 2021-07-06 09:36:11 UTC
Created by: example-user
...

# Write a Per-Test-Case Script

Now that you have identified an objective in Modality, write a script that handles your system under test's behavior for each of the many test cases that will be run. In your tool of choice, write a script that does the following:

  1. Perform per-test-case setup: get your system to a ready state for a test case.

  2. Next, call modality mutate --objective-name <objective-name>.

    • If you want some of your test cases to pass outside the predefined safety ranges of your mutators, pass the flag --allow-outside-safety-range.
  3. Exercise your system: induce whatever behavior you would like to see in each test case. These are normal commands to your system, not modality mutations.

  4. Add per-test-case tear down: perform whatever cleanup is necessary to get to a ready state for this script to be run again.

Save your per-test-case script. We'll call it <per-test-case-script> below.

# Example Per-Test-Case Script

# per-test-case-script.sh
#!/usr/bin/env bash

set -e

PRODUCER_APP_PATH="$PWD/bin/producer"
CONSUMER_APP_PATH="$PWD/bin/consumer"
MONITOR_APP_PATH="$PWD/bin/monitor"

# Setup
# Start the applications in the background, should run for ~5 seconds
"$CONSUMER_APP_PATH" &
CONSUMER_APP_PID=$!
"$PRODUCER_APP_PATH" &
PRODUCER_APP_PID=$!
"$MONITOR_APP_PATH" &
MONITOR_APP_PID=$!

SCOPE_NAME="StimulusScriptRunning"
modality session scope open "$SCOPE_NAME"

sleep 1

# Generate automatic mutations based on objective configuration
modality mutate --objective-name "example-objective"

sleep 4

modality session scope close "$SCOPE_NAME"

sleep 1

# Tear down
wait $CONSUMER_APP_PID
wait $PRODUCER_APP_PID
wait $MONITOR_APP_PID

exit 0

# Write a Test Execution Script

Next, you'll write a script to run all of the generative tests on your live system. In your tool of choice, write a test execution script to automate the following steps:

  1. Spin up your system so that it is ready for your per-test-case script.

  2. Create a Modality session by calling modality session open <session name> <system under test name>

  3. Start generative testing by calling modality execute <per-test-case-script> --objective-name <objective-name>.

  4. Your generative tests will run until one of the stopping conditions you specified, like a time limit, is hit. If you don't specify any stopping conditions, Modality will keep generating test cases indefinitely until you stop it manually.

  5. After the tests are complete, close the session by calling modality session close <session name>

    • All of these steps can be done manually through the command line, but the benefit of using a script is that your modality observation session closes right after all of your tests are complete.

# Example Test Execution Script

# test-execution-script.sh

#!/usr/bin/env bash

modality session open "example-session" "example-sut"

modality execute \
    --objective-name example-objective \
    ./scripts/per-test-case-script

modality session close "example-session"

exit 0

# Execute Tests

Execute your test execution script. Modality will generate as many diverse conditions of system state as it can fit into your objective's constraints, run them through your system, and record the results for each test case. In the final step, you'll analyze the session, with Modality highlighting issues in the measurements you cared about.

$ ./test-execution-script.sh

# Analyze your Results

Modality has run countless tests over more distinct conditions than any human could write tests for. You can now query the results for meaningful test cases.

  1. Call modality objective instances <objective-name> to learn the <objective-instance-id> of the test session you ran.

  2. For a high-altitude view of measurement results in the generative testing session, call modality objective inspect <objective-name> <objective-instance-id>.

  3. The Failing Measurements section shows the name of each failing measurement and the number of times it failed.

    • Note that when your objective measurements accurately describe required system behaviors, every generated test case that forces a measurement to fail is pointing to a potential bug.
  4. If you'd like additional detail before forwarding the results to troubleshooters, add verbosity flags. modality objective inspect <objective-name> <objective-instance-id> -vv will give a breakdown of all mutations that were injected and all measurement results per execution run.

  5. Equipped with these failing measurements, a troubleshooter can dig into the uncovered system behavior as part of investigating the cause of bugs with Modality.

# Example: modality objective instances <objective-name>

$ modality objective instances example-objective
NAME               INSTANCE  SESSION               SUT          STATUS
example-objective  1         2021-04-16T06-51-16Z  example-sut  STOPPED

# Example: modality objective inspect <objective-name> <objective-instance-id> -vv

$ modality objective inspect example-objective 1 -vv
Name: example-objective
SUT: example-sut
Session: example-session
Created at: 2021-08-13 09:38:22 UTC
Created by: example-user
Stopped at: 2021-08-13 09:40:09 UTC
Stopping Conditions Reached: true
Stopping Conditions Progress:
  Time: 00:01:47
  Passing Measurements: 10
  Failing Measurements: 5
Passing Measurements: 8
  Consumer heartbeat timing: 10
  Consumer to monitor heartbeat: 15
  Monitor checks consumer heartbeats: 15
  Monitor checks producer heartbeats: 15
  Monitor heartbeat timeouts: 10
  Producer lifecycle: 15
  Producer to consumer communications: 15
  Producer to monitor heartbeat: 15
Failing Measurements: 2
  Consumer heartbeat timing: 5
  Monitor heartbeat timeouts: 5
Mutations: 17
  Instance: 1
    Mutator: sample-data-mutator
    Probe: PRODUCER
    Parameters: [sample=11]
  Instance: 2
    Mutator: sample-data-mutator
    Probe: PRODUCER
    Parameters: [sample=72]
  ...
Measurement Outcomes:
  Execution Run: 0
    Region: session = 'example-session' AND mutation_epoch = 2
    Outcomes:
      Measurement Name: Consumer heartbeat timing
        Executed at: 2021-08-13 09:38:40 UTC
        Outcome: PASS
      Measurement Name: Consumer to monitor heartbeat
        Executed at: 2021-08-13 09:38:40 UTC
        Outcome: PASS
      Measurement Name: Monitor checks consumer heartbeats
        Executed at: 2021-08-13 09:38:40 UTC
        Outcome: PASS
      Measurement Name: Monitor checks producer heartbeats
        Executed at: 2021-08-13 09:38:40 UTC
        Outcome: PASS
      Measurement Name: Monitor heartbeat timeouts
        Executed at: 2021-08-13 09:38:40 UTC
        Outcome: PASS
      Measurement Name: Producer lifecycle
        Executed at: 2021-08-13 09:38:40 UTC
        Outcome: PASS
      Measurement Name: Producer to consumer communications
        Executed at: 2021-08-13 09:38:40 UTC
        Outcome: PASS
      Measurement Name: Producer to monitor heartbeat
        Executed at: 2021-08-13 09:38:40 UTC
        Outcome: PASS
  Execution Run: 1
    Region: session = 'example-session' AND mutation_epoch = 4
    Outcomes:
      Measurement Name: Consumer heartbeat timing
        Executed at: 2021-08-13 09:38:47 UTC
        Outcome: PASS
      Measurement Name: Consumer to monitor heartbeat
        Executed at: 2021-08-13 09:38:47 UTC
        Outcome: PASS
      Measurement Name: Monitor checks consumer heartbeats
        Executed at: 2021-08-13 09:38:47 UTC
        Outcome: PASS
      Measurement Name: Monitor checks producer heartbeats
        Executed at: 2021-08-13 09:38:47 UTC
        Outcome: PASS
      Measurement Name: Monitor heartbeat timeouts
        Executed at: 2021-08-13 09:38:47 UTC
        Outcome: PASS
      Measurement Name: Producer lifecycle
        Executed at: 2021-08-13 09:38:47 UTC
        Outcome: PASS
      Measurement Name: Producer to consumer communications
        Executed at: 2021-08-13 09:38:47 UTC
        Outcome: PASS
      Measurement Name: Producer to monitor heartbeat
        Executed at: 2021-08-13 09:38:47 UTC
        Outcome: PASS
  ...

# Next Steps

  1. If at first your generative tests don't cause any measurement failures:

  2. Once a potential bug has been identified, developers and other troubleshooters can use these same Modality tools to dig deeper and investigate the cause of bugs with Modality.

  3. If you add new features to your system, simply apply instrumentation to any new components and execute the same generative tests again. Modality will test, trace, and report new interactions that suggest potential bugs.

  4. Once your generative tests include measurements for all of your system requirements, passing all of the tests provides confidence that your system performs to specification under a vast array of circumstances.