# CLI Reference

# Table of Contents

  1. modalitycommands
  2. Region Expressions syntax
  3. modality-probe commands

# modality

modality is the main command for interacting with Modality. This command uses a subcommand structure to expose most of Modality's functionality. The Modality CLI provides a mechanism for managing a “context” containing information about the user’s identity and the default session and SUT. When these are set, the session and SUT subcommands will automatically target their respective default. If you wish to override the default session or SUT for a particular invocation of a command you can do so with the --using option.

modality commands
modality check
modality execute
modality log
modality metrics
modality mutate
modality objective
modality query
modality session
modality status
modality sut
modality umutate
modality metrics subcommands
modality metrics complexity
modality metrics coupling
modality metrics coverage
modality metrics falsifiability
modality metrics fault-localization
modality metrics loops
modality metrics pattern-frequency
modality metrics similarity
modality metrics summary
modality metrics topology
modality session subcommands
modality session
modality session component
modality session event
modality session mutation
modality session mutator
modality session probe
modality session scope
modality sut subcommands
modality sut
modality sut component
modality sut event
modality sut mutator
modality sut probe

# modality status

# Usage

modality status
  [-f, --format <text | json>]

# Description

Display current status information, including the active user, the default SUT and session, and active session info, if applicable.

# Options

-f, --format <text | json>: The printed output format. Defaults to text.

# Examples

$ modality status
┌─ Modality Status ────────────────────────────────────────────────────────────┐
│                                                                              │
│  Default User:    jon@auxon.io                                               │
│  Default SUT:     drone                                                      │
│  Default Session: test-session                                               │
│                                                                              │
├─ Active Session Info ────────────────────────────────────────────────────────┤
│                                                                              │
│  Name: test-session                                                          │
│  Collector Connections:                                                      │
│    udp://127.0.0.1:2718                                                      │
│    http://127.0.0.1:2719                                                     │
│  Control Connections:                                                        │
|    udp://127.0.0.1:34350                                                     |
│                                                                              │
└──────────────────────────────────────────────────────────────────────────────┘

# modality sut

# Usage

modality sut
  list [-v] [--color <auto | always | never>]
  inspect <name>
  use <name> [--force]
  create <sut bundle path>
  delete <name> [--force]

# Description

List, add, or remove SUT definitions.

# Subcommands

  • list [-v] [--color <auto | always | never>]: List all known SUTs.
    • -v: Provide (more) verbose output (-v, -vv).
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name>: Print detailed information and metadata about the given SUT.
    • <name>: Name of the SUT to inspect.
  • use <name> [--force]: Set the default SUT.
    • <name>: Name of the SUT to use as default.
    • --force: Use the given name as the default SUT even if it doesn't currently exist.
  • create <sut bundle path>: Tell Modality about a new SUT by giving it a path to a directory containing a SUT.toml file.
    • <sut bundle path>: Path to a directory containing a SUT.toml file.
  • delete <name> [--force]: Remove the given SUT.
    • <name>: Name of the SUT to delete.
    • --force: Bypass the user input confirmation check.

# Examples


$ modality sut list
NAME      COMPONENTS  SESSIONS
drone     7           1
test-sut  2           2

$ modality sut list -v
NAME      COMPONENTS  SESSIONS  CREATED AT               CREATED BY
drone     7           1         2021-03-30 14:07:39 UTC  
test-sut  2           2         2021-04-05 04:45:44 UTC  test-user


$ modality sut inspect example-sut
Name: example-sut
Created at: 2021-05-11 09:57:04 UTC
Created by: example-user
Components:
  consumer-component
  producer-component
  monitor-component
Probes:
  CONSUMER_PROBE
  MONITOR_PROBE
  PRODUCER_PROBE
Events:
  INIT
  DEINIT
  RECVD_HEARTBEAT_MSG
  HEARTBEAT_SENDER_ID_CHECK
  PRODUCER_HEARTBEAT_TIMEOUT
  CONSUMER_HEARTBEAT_TIMEOUT
  INIT
  DEINIT
  RECVD_MEASUREMENT_MSG
  MEASUREMENT_SAMPLE_CHECK
  SENT_HEARTBEAT_MSG
  INIT
  DEINIT
  MEASUREMENT_SAMPLED
  SENT_MEASUREMENT_MSG
  SENT_HEARTBEAT_MSG
Mutators:
  heartbeat-delay-mutator
  sample-data-mutator

$ modality sut create .

$ modality sut delete test-sut

Also nested beneath modality sut are subcommands that list or inspect the specific entities that constitute the given SUT. The entity subcommands share a common option, --using, which can be used to override the in-use SUT when retrieving a SUT's definition data.

# modality sut component

# Usage

modality sut component
  list [-v] [--using <sut name>] [--color <auto | always | never>]
  inspect <name | id> [--using <sut name>]

# Description

List and inspect component definitions for the given SUT.

# Subcommands

  • list [-v] [--using <sut name>] [--color <auto | always | never>]: List components in the SUT.
    • -v: Provide (more) verbose output (-v, -vv).
    • --using <sut name>: Use the given SUT rather than the default one.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name | id> [--using <sut name>]: Print the definition of the given component.
    • <name | id>: Identifier for the component. Component ID can be found with modality sut component list -v.
    • --using <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality sut component list
NAME                TAGS  PROBES  EVENTS
consumer-component  []    1       5
monitor-component   []    1       6
producer-component  []    1       5

$ modality sut component inspect monitor-component
Name: monitor-component
ID: 35a90f7f-25c6-404e-a5df-ec3d51268723
Tags: []
SUT: example-sut
Probes:
  MONITOR_PROBE
Events:
  INIT
  DEINIT
  RECVD_HEARTBEAT_MSG
  HEARTBEAT_SENDER_ID_CHECK
  PRODUCER_HEARTBEAT_TIMEOUT
  CONSUMER_HEARTBEAT_TIMEOUT
Mutators:

# modality sut probe

# Usage

modality sut probe
  list [-v] [--using <sut name>] [--color <auto | always | never>]
  inspect <name | id> [--using <sut name>]

# Description

List and inspect probe definitions for the given SUT.

# Subcommands

  • list [-v] [--using <sut name>] [--color <auto | always | never>]: List probes in the SUT.
    • -v: Provide (more) verbose output (-v, -vv).
    • --using <sut name>: Use the given SUT rather than the default one.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name | id> [--using <sut name>]: Print the definition of the given probe.
    • <name | id>: Identifier for the probe. Probe ID can be found with modality sut probe list -v.
    • --using <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality sut probe list
NAME            TAGS
CONSUMER_PROBE  [c-example, consumer]
MONITOR_PROBE   [c-example, monitor]
PRODUCER_PROBE  [c-example, producer]

$ modality sut probe inspect CONSUMER_PROBE
Name: CONSUMER_PROBE
Description: Consumer probe
ID: 438896705
Component: consumer-component
Source: c-example/src/consumer/main.c#L65
Tags: [c-example, consumer]

# modality sut event

# Usage

modality sut event
  list [-v] [--using <sut name>] [--color <auto | always | never>]
  inspect <name | id> [--using <sut name>]

# Description

List and inspect event definitions for the given SUT.

# Subcommands

  • list [-v] [--using <sut name>] [--color <auto | always | never>]: List events in the SUT.
    • -v: Provide (more) verbose output (-v, -vv).
    • --using <sut name>: Use the given SUT rather than the default one.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name | id> [--using <sut name>]: Print the definition of the given event.
    • <name | id>: Identifier for the event. Event ID can be found with modality sut event list -v.
    • --using <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality sut event list
NAME                        TAGS
CONSUMER_HEARTBEAT_TIMEOUT  [heartbeat, monitor, timeout]
DEINIT                      [monitor]
DEINIT                      [consumer]
DEINIT                      [producer]
HEARTBEAT_SENDER_ID_CHECK   [heartbeat, monitor]
INIT                        [monitor]
INIT                        [consumer]
INIT                        [producer]
MEASUREMENT_SAMPLED         [measurement, producer]
MEASUREMENT_SAMPLE_CHECK    [consumer, measurement]
PRODUCER_HEARTBEAT_TIMEOUT  [heartbeat, monitor, timeout]
RECVD_HEARTBEAT_MSG         [heartbeat, monitor]
RECVD_MEASUREMENT_MSG       [consumer, measurement]
SENT_HEARTBEAT_MSG          [consumer, heartbeat]
SENT_HEARTBEAT_MSG          [heartbeat, producer]
SENT_MEASUREMENT_MSG        [measurement, producer]

$ modality sut event inspect MEASUREMENT_SAMPLED
Name: MEASUREMENT_SAMPLED
Description: Producer sampled raw measurement
ID: 3
Type Hint: I8
Kind: event
Component: producer-component
Source: c-example/src/producer/main.c#L128
Tags: [measurement, producer]

# modality sut mutator

# Usage

modality sut mutator
  list [-v] [--using <sut name>] [--color <auto | always | never>]
  inspect [--using <sut name>]

# Description

List and inspect mutator definitions for the given SUT.

# Subcommands

  • list [-v] [--using <sut name>] [--color <auto | always | never>]: List mutators in the SUT.
    • -v: Provide (more) verbose output (-v, -vv).
    • --using <sut name>: Use the given SUT rather than the default one.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name | id> [--using <sut name>]: Print the definition of the given mutator.
    • <name | id>: Identifier for the mutator. Mutator ID can be found with modality sut mutator list -v.
    • --using <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality sut mutator list
NAME                     TAGS
heartbeat-delay-mutator  [consumer, heartbeat]
sample-data-mutator      [measurement, producer]

$ modality sut mutator inspect sample-data-mutator
Name: sample-data-mutator
ID Hash: 11820918689057293432
Source: src/producer/main.c#L138
Tags: [producer, measurement]
Probes:
  PRODUCER_PROBE
Parameters:
  Name: sample
    Type: i8
    Minimum Effect Value: 0
    Nominal: [-50..50]
    Safety: [-100..100]
    Hard: [-128..127]

# modality session

# Usage

modality session
  list [-v] [--color <auto | always | never>]
  inspect <name> [--with-sut <sut name>]
  use <name> [--force]
  open <session name> <sut name> [--force]
  close <name> [--keep-alive]
  delete <session name> [--force]
  link <session name> <sut name> [--force]

# Description

Provide the ability to retrieve information regarding sessions and to set a session as the default session.

# Subcommands

  • list [-v] [--color <auto | always | never>]: List the sessions.
    • -v: Provide (more) verbose output (-v, -vv).
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name> [--with-sut <sut name>]: Print detailed information about the given session.
    • <name>: Name of the session.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.
  • use <name> [--force]: Set the default session.
    • --force: Use the given name as the default session even if it doesn't currently exist.
  • open <name> <sut name> [--force]: Create and open a new session.
    • <session name>: Name to give to the newly opened session.
    • <sut name>: Name of the SUT to associate with this session.
    • --force: Close any other open sessions that were using the connections needed for this session.
  • close <name> [--keep-alive]: Close the given session and tear down the connections. If --keep-alive is used, the connections remain intact, but data is no longer recorded until that session is reopened. If close is used without --keep-alive on a closed session whose connections are still open, it finalizes that session, i.e., it closes the connections. Otherwise, close should be idempotent.
    • <name>: Name of the session to close.
    • --keep-alive: Close the session but keep its connections intact.
  • delete <name> [--force]: Delete the given session.
    • <name>: Name of the session to delete.
    • --force: Bypass the user input confirmation check.
  • link <session name> <sut name> [--force]: (Re)assigns the default SUT definition associated with a session (only works for closed sessions).
    • <session name>: Name of the session.
    • <sut name>: Name of the SUT to associate with this session.
    • --force: Bypass the user input confirmation check.

# Examples


$ modality session list
NAME          SUT          STATUS
test-session  example-sut  CLOSED

$ modality session inspect test-session
Name: test-session
Opened at: 2021-05-11 09:57:04 UTC
Opened by: example-user
Status: closed
Linked SUT: example-sut
Observed Probes:
  CONSUMER_PROBE
  MONITOR_PROBE
  PRODUCER_PROBE
Observed Events:
  INIT
  DEINIT
  RECVD_HEARTBEAT_MSG
  HEARTBEAT_SENDER_ID_CHECK
  PRODUCER_HEARTBEAT_TIMEOUT
  CONSUMER_HEARTBEAT_TIMEOUT
  INIT
  DEINIT
  RECVD_MEASUREMENT_MSG
  MEASUREMENT_SAMPLE_CHECK
  SENT_HEARTBEAT_MSG
  INIT
  DEINIT
  MEASUREMENT_SAMPLED
  SENT_MEASUREMENT_MSG
  SENT_HEARTBEAT_MSG

# modality session scope

# Usage

modality session scope
  list [-v] [--using <session name>] [--color <auto | always | never>]
  inspect <name | instance-id> [--using <session name>]
  open <name> [--using <session name>]
  close <name> [--using <session name>]

# Description

List, inspect, open, or close scopes for a particular session.

# Subcommands

  • list [-v] [--using <session name>] [--color <auto | always | never>]: List the scopes for a session.
    • -v: Provide (more) verbose output (-v, -vv).
    • --using <session name>: Use the given session rather than the default one.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name | instance-id> [--using <session name>]: Print detailed information about the given scope.
    • <name | instance-id>: Identifier for the scope to inspect.
    • --using <session name>: Use the given session rather than the default one.
  • open <name> [--using <session name>]: Open the scope with the given name.
    • <name>: Name for the newly opened scope.
    • --using <session name>: Use the given session rather than the default one.
  • close <name> [--using <session name>]: Close the scope with the given name. Close is idempotent, if the given scope is already closed then close is a no-op.
    • <name>: Name of the scope to close.
    • --using <session name>: Use the given session rather than the default one.

# Examples


$ modality session scope list
NAME                                   STATUS  INSTANCES
Case Study: Failure Detector Failsafe  Closed  1
DroneOperational                       Closed  1
DroneSetup                             Closed  1

$ modality session scope inspect DroneSetup
Name: DroneSetup
Instances:
  Instance: 1
    Opened at: 2021-05-11 09:57:04 UTC
    Closed at: 2021-05-11 09:57:10 UTC
    Opened by: example-user

# modality session component

# Usage

modality session component
  list [-v] [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]
  inspect <name | id> [--using <session name>] [--with-sut <sut name>]

# Description

List and inspect component definitions for a particular session.

# Subcommands

  • list [-v] [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]: List the components observed in a particular session.
    • -v: Provide (more) verbose output (-v, -vv).
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name | id> [--using <session name>] [--with-sut <sut name>]: Print detailed information about a component that was observed while recording a session.
    • <name | id>: Identifier for the component. Component ID can be found with modality session component list -v.
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.

# Examples


$ modality session component list
NAME                TAGS  PROBES  EVENTS
consumer-component  []    1       5
monitor-component   []    1       5
producer-component  []    1       5

$ modality session component inspect consumer-component
Name: consumer-component
ID: 76d0538f-1ff2-4865-bc41-8219aebac409
Tags: []
SUT: example-sut
Probes:
  CONSUMER_PROBE
Events:
  INIT
  DEINIT
  RECVD_MEASUREMENT_MSG
  MEASUREMENT_SAMPLE_CHECK
  SENT_HEARTBEAT_MSG
Mutators:
  heartbeat-delay-mutator

# modality session probe

# Usage

modality session probe
  list [-v] [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]
  inspect <name | id> [--using <session name>] [--with-sut <sut name>]

# Description

List and inspect the observed probes for a particular session.

# Subcommands

  • list [-v] [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]: List the probes observed in a particular session.
    • -v: Provide (more) verbose output (-v, -vv).
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name | id> [--using <session name>] [--with-sut <sut name>]: Print detailed information about a probe that was observed while recording a session.
    • <name | id>: Identifier for the probe. Probe ID can be found with modality session probe list -v.
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.

# Examples


$ modality session probe list
NAME            TAGS                   EVENTS
CONSUMER_PROBE  [c-example, consumer]  853
MONITOR_PROBE   [c-example, monitor]   392
PRODUCER_PROBE  [c-example, producer]  1835

$ modality session probe inspect PRODUCER_PROBE
Name: PRODUCER_PROBE
Description: Producer probe
ID: 785847259
Component: producer-component
Source: c-example/src/producer/main.c#L68
Tags: [c-example, producer]

# modality session event

# Usage

modality session event
  list [--include-internal-events] [-v] [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]
  instances <name | id> [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]
  inspect <name | id | coordinate> [--using <session name>] [--with-sut <sut name>]

# Description

List and inspect the observed events for a particular session.

# Subcommands

  • list [--include-internal-events] [-v] [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]: List the events observed in a particular session.
    • --include-internal-events: Include modality's internal events in the output.
    • -v: Provide (more) verbose output (-v, -vv).
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • instances <name | id> [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]: List the occurrences of the given event in a particular session.
    • <name | id>: Identifier for the event. Event ID can be found with modality session event list -v.
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name | id> [--using <session name>] [--with-sut <sut name>]: Print detailed information about an event that was observed while recording a session.
    • <name | id>: Identifier for the event. Event ID can be found with modality session event list -v.
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.

# Examples


$ modality session event list
NAME                        TAGS                           COUNT
CONSUMER_HEARTBEAT_TIMEOUT  [heartbeat, monitor, timeout]  40
DEINIT                      [monitor]                      9
DEINIT                      [consumer]                     9
DEINIT                      [producer]                     9
HEARTBEAT_SENDER_ID_CHECK   [heartbeat, monitor]           167
INIT                        [monitor]                      9
INIT                        [consumer]                     9
INIT                        [producer]                     9
MEASUREMENT_SAMPLED         [measurement, producer]        1277
MEASUREMENT_SAMPLE_CHECK    [consumer, measurement]        378
RECVD_HEARTBEAT_MSG         [heartbeat, monitor]           167
RECVD_MEASUREMENT_MSG       [consumer, measurement]        378
SENT_HEARTBEAT_MSG          [consumer, heartbeat]          79
SENT_HEARTBEAT_MSG          [heartbeat, producer]          90
SENT_MEASUREMENT_MSG        [measurement, producer]        450

$ modality session event inspect CONSUMER_HEARTBEAT_TIMEOUT
Name: CONSUMER_HEARTBEAT_TIMEOUT
Description: Monitor detected consumer heartbeat timeout
ID: 6
Type Hint: 
Kind: failure
Component: monitor-component
Source: c-example/src/monitor/main.c#L172
Tags: [heartbeat, monitor, timeout]

$ modality session event instances CONSUMER_HEARTBEAT_TIMEOUT
(528479634:131080:7:8, CONSUMER_HEARTBEAT_TIMEOUT @ MONITOR_PROBE, outcome=FAIL)
(528479634:131080:8:4, CONSUMER_HEARTBEAT_TIMEOUT @ MONITOR_PROBE, outcome=FAIL)
(528479634:131080:9:4, CONSUMER_HEARTBEAT_TIMEOUT @ MONITOR_PROBE, outcome=FAIL)
(528479634:131083:13:4, CONSUMER_HEARTBEAT_TIMEOUT @ MONITOR_PROBE, outcome=FAIL)
(528479634:131083:14:4, CONSUMER_HEARTBEAT_TIMEOUT @ MONITOR_PROBE, outcome=FAIL)
(528479634:131086:18:4, CONSUMER_HEARTBEAT_TIMEOUT @ MONITOR_PROBE, outcome=FAIL)
...

# modality session mutator

# Usage

modality session mutator
  list [-v] [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]
  inspect <name | id> [--using <session name>] [--with-sut <sut name>]

# Description

List and inspect the observed mutators for a particular session.

# Subcommands

  • list [-v] [--using <session name>] [--with-sut <sut name>]: List the mutators observed in a particular session.
    • -v: Provide (more) verbose output (-v, -vv).
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name | id> [--using <session name>] [--with-sut <sut name>]: Print detailed information about a mutator that was observed while recording a session.
    • <name | id>: Identifier for the mutator. Mutator ID can be found with modality session mutator list -v.
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.

# Examples


$ modality session mutator list
NAME          TAGS
data-mutator  []

$ modality session mutator list -v
NAME                     TAGS
heartbeat-delay-mutator  [consumer, heartbeat]
sample-data-mutator      [measurement, producer]

$ modality session mutator inspect heartbeat-delay-mutator
Name: heartbeat-delay-mutator
ID Hash: 8813786539029173927
Source: src/consumer/main.c#L166
Tags: [consumer, heartbeat]
Probes:
  CONSUMER_PROBE
Parameters:
  Name: heartbeat-delay
    Type: u32
    Minimum Effect Value: 0
    Nominal: [0..1000]
    Safety: [0..2000]
    Hard: [0..3000]

# modality session mutation

# Usage

modality session mutation
  list [-v] [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]
  inspect <mutator name> <mutation instance id> [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]

# Description

List and inspect the mutations and their state for a particular session.

# Subcommands

  • list [-v] [--using <session name>] [--with-sut <sut name>] [--color <auto | always | never>]: List the mutations observed in a particular session.
    • -v: Provide (more) verbose output (-v, -vv).
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <mutator name> <mutation instance id> [--using <session name>] [--with-sut <sut name>]: Print detailed information about a mutation that was observed while recording a session.
    • <name | id>: Identifier for the mutation. Mutation ID can be found with modality session mutation list.
    • --using <session name>: Use the given session rather than the default one.
    • --with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.

# Examples


$ modality session mutation list
MUTATOR                  INSTANCE  PARAMETERS              STATE
sample-data-mutator      1         [sample=-82]            communicated
sample-data-mutator      2         [sample=-1]             injected
heartbeat-delay-mutator  3         [heartbeat-delay=611]   injected
sample-data-mutator      4         [sample=-57]            injected
sample-data-mutator      5         [sample=70]             injected
heartbeat-delay-mutator  6         [heartbeat-delay=615]   communicated
heartbeat-delay-mutator  7         [heartbeat-delay=1052]  injected
heartbeat-delay-mutator  8         [heartbeat-delay=673]   injected
heartbeat-delay-mutator  9         [heartbeat-delay=1392]  injected

$ modality session mutation inspect sample-data-mutator 4
Mutator: sample-data-mutator
Probe: PRODUCER_PROBE
Instance: 4
State: injected
Parameters:
  Name: sample
    Value: -57
Preconditions:
  MATCH
    MEASUREMENT_SAMPLED@PRODUCER_PROBE


# modality objective

# Usage

modality objective
  list [-v] [--color <auto | always | never>]
  inspect <name | instance-id> [-v] [--format <text | json>] [--using <session name>] [--color <auto | always | never>]
  create <config file path>
  delete <name> [--force]
  instances <name> [-v] [--color <auto | always | never>]
  sample <name> [region expression] [--using <session name>]
  apply <name> [region expression] [--disable-status-output] [--using <session name>]

# Description

Create and delete objectives, list them and their associations, and inspect their detailed contents and relationships.

# Subcommands

  • list [-v] [--color <auto | always | never>]: List the objectives.
    • -v: Provide (more) verbose output (-v, -vv).
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • inspect <name | instance-id> [-v] [--format <text | json>] [--using <session name>] [--color <auto | always | never>]: Print detailed information about the given objective, including its definition and a list of associated sessions and SUTs.
    • <name | instance-id>: Identifier for the objective. Instance ID can be found with modality objective instances <name>.
    • -v: Provide (more) verbose output (-v, -vv).
    • --format <text | json>: The printed output format. Defaults to text.
    • --using <session name>: Use the given session rather than the default one.
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • create <config file path>: Create a new objective from the given configuration file.
    • <config file path>: Path to a TOML file with the objective definition information.
  • delete <name> [--force]: Delete the objective with the given name. Delete is idempotent; if no objective with the given name exists, then no action is taken.
    • <name>: Name of the objective to delete.
    • --force: Bypass the user input confirmation check.
  • instances <name> [-v] [--color <auto | always | never>]: List the instances of use for the given objective.
    • <name>: Name of the objective to list instances of.
    • -v: Provide (more) verbose output (-v, -vv).
    • --color <auto | always | never>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.
  • sample <name> [region expression] [--using <session name>]: Run the measurement checks for the currently open instance of the given objective.
    • <name>: Name of the objective to sample.
    • region expression: Trace region to run measurements against. If not provided, defaults to the default session. For details see Region Expressions.
    • --using <session name>: Use the given session rather than the default one.
  • apply <name> [region expression] [--disable-status-output] [--using <session name>]: Create a new instance of an objective and apply it to pre-recorded data using the supplied region expression. If no region expression is provided it will use the default session.
    • <name>: Name of the objective to apply.
    • region expression: Trace region to apply objective to. If not provided, defaults to the default session. For details see Region Expressions.
    • --disable-status-output: Don't show the objective status summary information.
    • --using <session name>: Use the given session rather than the default one.

# Examples


$ modality objective list
NAME               SESSIONS  SUTS
example-objective  1         1

$ modality objective instances example-objective
NAME               INSTANCE  SESSION               SUT          STATUS
example-objective  1         2021-04-16T06-51-16Z  example-sut  STOPPED

$ modality objective inspect example-objective
Name: example-objective
Created at: 2021-04-16 13:51:16 UTC
Created by: example-user
SUTS: [example-sut]
Instances By Session
  2021-04-16T06-51-16Z
    [1]
Stopping Conditions
  Attempts: 20
  Time: 2min
  Passing Measurements: 15
  Failing Measurements: 5
Mutation Constraints
  Max Concurrent Mutations: 1
  Mutation Precondition: 
  Mutator Tags: []
Mutator Constraints: 2
  heartbeat-delay-mutator
    Probe: CONSUMER_PROBE
    Mutation Precondition: 
    Parameter Constraints: 1
      Parameter: heartbeat-delay
      Range: [500, 2000]
  sample-data-mutator
    Probe: PRODUCER_PROBE
    Mutation Precondition: MATCH MEASUREMENT_SAMPLED @ PRODUCER_PROBE
    Parameter Constraints: 1
      Parameter: sample
      Range: [-100, 100]
Measurements: 7
  Consumer lifecycle
    Should Pass: true
    Stopping Conditions
      Passing Measurements: 
      Failing Measurements: 
    Check:
      MATCH
          (INIT @ CONSUMER_PROBE) AS Init,
          (RECVD_MEASUREMENT_MSG @ CONSUMER_PROBE) AS RecvdMsg,
          (MEASUREMENT_SAMPLE_CHECK @ CONSUMER_PROBE AND outcome = PASS) AS SampleChecked,
          (DEINIT @ CONSUMER_PROBE) AS Fini
      WHEN
          Init -> RecvdMsg -> SampleChecked -> Fini

  Consumer to monitor heartbeat
    Should Pass: true
    Stopping Conditions
      Passing Measurements: 
      Failing Measurements: 
    Check:
      MATCH
          # 2 == HEARTBEAT_ID_CONSUMER
          (SENT_HEARTBEAT_MSG @ CONSUMER_PROBE AND payload = 2) AS SentMsg,
          (RECVD_HEARTBEAT_MSG @ MONITOR_PROBE AND payload = 2) AS RecvdMsg,
          (HEARTBEAT_SENDER_ID_CHECK @ MONITOR_PROBE AND outcome = PASS) AS SrcIdChecked
      WHEN
          SentMsg -> RecvdMsg -> SrcIdChecked

  Monitor lifecycle
    Should Pass: true
    Stopping Conditions
      Passing Measurements: 
      Failing Measurements: 
    Check:
      MATCH
          (INIT @ MONITOR_PROBE) AS Init,
          # 1 == HEARTBEAT_ID_PRODUCER
          (RECVD_HEARTBEAT_MSG @ MONITOR_PROBE AND payload = 1) AS RecvdMsgA,
          # 2 == HEARTBEAT_ID_CONSUMER
          (RECVD_HEARTBEAT_MSG @ MONITOR_PROBE AND payload = 2) AS RecvdMsgB,
          (HEARTBEAT_SENDER_ID_CHECK @ MONITOR_PROBE AND outcome = PASS) AS SrcIdChecked,
          (DEINIT @ MONITOR_PROBE) AS Fini
      WHEN
          (Init -> RecvdMsgA -> SrcIdChecked -> Fini)
          AND
          (Init -> RecvdMsgB -> SrcIdChecked -> Fini)

  Passing expectation never preceded by out-of-bounds value
    Should Pass: true
    Stopping Conditions
      Passing Measurements: 
      Failing Measurements: 
    Check:
      MATCH
          SENT_MEASUREMENT_MSG @ PRODUCER_PROBE
              AND (payload < -50 OR payload > 50) AS Sent,
          (RECVD_MEASUREMENT_MSG @ CONSUMER_PROBE) AS Recvd,
          (MEASUREMENT_SAMPLE_CHECK @ CONSUMER_PROBE AND outcome = PASS) AS Checked
      WHEN
          Sent -> Recvd WITHIN 3 STEPS
          AND
          Recvd -> Checked WITHIN 3 STEPS
      AGGREGATE
          count() = 0

  Producer lifecycle
    Should Pass: true
    Stopping Conditions
      Passing Measurements: 
      Failing Measurements: 
    Check:
      MATCH
          (INIT @ PRODUCER_PROBE) AS Init,
          (MEASUREMENT_SAMPLED @ PRODUCER_PROBE) AS Sample,
          (SENT_MEASUREMENT_MSG @ PRODUCER_PROBE) AS SentMsg,
          (DEINIT @ PRODUCER_PROBE) AS Fini
      WHEN
          Init -> Sample -> SentMsg -> Fini

  Producer to consumer communications
    Should Pass: true
    Stopping Conditions
      Passing Measurements: 
      Failing Measurements: 
    Check:
      MATCH
          (MEASUREMENT_SAMPLED @ PRODUCER_PROBE) AS Sample,
          (SENT_MEASUREMENT_MSG @ PRODUCER_PROBE) AS SentMsg,
          (RECVD_MEASUREMENT_MSG @ CONSUMER_PROBE) AS RecvdMsg,
          (MEASUREMENT_SAMPLE_CHECK @ CONSUMER_PROBE AND outcome = PASS) AS SampleChecked
      WHEN
          Sample -> SentMsg -> RecvdMsg -> SampleChecked

  Producer to monitor heartbeat
    Should Pass: true
    Stopping Conditions
      Passing Measurements: 
      Failing Measurements: 
    Check:
      MATCH
          # 1 == HEARTBEAT_ID_PRODUCER
          (SENT_HEARTBEAT_MSG @ PRODUCER_PROBE AND payload = 1) AS SentMsg,
          (RECVD_HEARTBEAT_MSG @ MONITOR_PROBE AND payload = 1) AS RecvdMsg,
          (HEARTBEAT_SENDER_ID_CHECK @ MONITOR_PROBE AND outcome = PASS) AS SrcIdChecked
      WHEN
          SentMsg -> RecvdMsg -> SrcIdChecked


# modality mutate

# Usage

modality mutate
  [--allow-outside-safety-range]
  [--retain-mutation-epoch]
  [--objective-name <objective-name>]
  [--using <session name>]
  [
    --precondition <trace-expression> |
    --precondition-file <trace-expression-path>
  ]
  [
    <mutator-expression> [mutation-argument...] |
    [--tags=<tags-expression>] [--probes=<probes-expression>]
  ]

# Description

mutate perturbs the system under test in the given session—either the default one or the one given via --using—into a new state. This command has two chief variants —- a manual variant in which a specific mutation to the system is dictated by the user, and an automatic variant in which the modality system is asked to perturb the system under test in some internally-automatically-determined fashion.

# Manual Mutation

Send a specific mutation to the system under test. A manual mutation takes a mutator expression, which is an expression that uniquely identifies a mutator. It takes the form

<mutator-definition-identifier>@<probe-identifier>

A mutator-definition-identifier is one of:

  • A mutator’s name (if it is global unique for the entire SUT)
  • The mutator’s component and its name, joined by a colon: comp:mut
  • The mutator’s component id and its name, joined by a colon.

Note that the unsigned integer which represents the hash of the mutator's definition can also be substituted for the mutator's name in any case.

A probe-identifier is one of:

  • A probe's name (if the probe's name is unique across all components)
  • probe's component name and the probe's name, joined by a colon: comp:probe
  • A probe's component id and its name, joined by a colon.

Note that the probe’s numeric id can be used in the place of its name in any case.

The mutation-argument values (space separated) are used to parameterize the mutation injection when it occurs. The acceptable argument values are specific to the mutator and are position-sensitive. The form for writing mutation arguments is as bare values, following the positional order of the mutator's definition for arguments. E.g., for a mutator named "my mutator" with three arguments (an integer argument "a", a boolean argument "b", and a float argument "c"), a mutation could be manually injected with positional argument values by:

$ modality mutate "my-mutator@my-probe" 1 false 4.5

Alternately, the names of the mutator arguments could be used to supply the arguments on the command line in any desired order. (Modality system will reorder them correctly prior to sending them to the SUT). E.g., for the same mutator with three arguments, a mutation could be manually injected like so:

$ modality mutate "my-mutator@my-probe" b=false a=1 c=4.5

Note, all arguments in the mutator's definition must always be specified in the CLI.

Optionally, the --precondition trace expression constrains when the mutation should occur. The mutation is only applied after the precondition trace expression has a result. The --precondition-file parameter is a (mutually exclusive) alternative to --precondition which behaves the same, save for reading the trace expression out of a file.

The --allow-outside-safety-range flag is necessary if the user would like to inject a mutation with arguments that fall outside of their defined safety ranges.

# Automatic Mutation

Send a modality-generated mutation into the system under test. This variant is identified by its lack of a specific mutator-expression parameter.

Whenever the automatic variant of modality mutate is called, modality will attempt to clear out all prior mutations.

# Options

--allow-outside-safety-range: Allow mutation to occur with values that are outside the defined safety ranges of mutator arguments.

--retain-mutation-epoch: Retain the current mutation epoch with this mutation. This allows adding additional mutations to the same mutation epoch as determined from a previous mutation. If this flag is not provided, a new mutation epoch is constructed for this mutation.

--objective-name <objective-name>: Objective name to be associated with this mutation. For automatic mutations, this objective may constrain what mutations get generated.

--precondition <precondition>: A trace expression precondition to any mutation. Mutually exclusive with --precondition-file.

--precondition-file <precondition-file>: A file containing a trace expression precondition. Mutually exclusive with --precondition.

--probes=<probes>: A semicolon delimited list of probe identifiers. Automatic mutation will only be done using mutators associated with one of the specified probes.

--tags=<tags>: A semicolon delimited list of mutator tags. Automatic mutation will only be done using mutators which have at least one of the specified tags.

--using <session name>: Use the given session rather than the default one.

# Examples

$ modality mutate "my-mutator@my-probe" 1 false 4.5

$ modality mutate --tags=battery

# modality unmutate

# Usage

modality unmutate [--using <session name>]

# Description

Clear all mutations across the SUT as soon as possible.

# Options

--using <session name>: Use the given session rather than the default one.

# Examples

$ modality unmutate

# modality execute

# Usage

modality execute
  <script>
  <--objective-file <objective-file> | --objective-name <objective-name>>
  [--stopping-condition <condition>...]
  [--disable-script-io-inherit]
  [--disable-status-output]
  [--using <session name>]

# Description

Give the outer-test-loop control to Modality, allowing it to repeatedly run the given script until a stopping condition from the given objective is met. The objective's measurements will be taken after each script execution.

If the --stopping-condition option is supplied, the stopping conditions in the objective’s definition are ignored.

# Options

--objective-file <objective-file>: Path to an objective definition file. The objective will be created and applied to the current session. Mutually exclusive with --objective-name.

--objective-name <objective-name>: Name of an existing objective to use for measurements and stopping conditions. Mutually exclusive with --objective-file.

--stopping-condition <stopping-condition>: Use the provided stopping condition(s) in place of the general stopping conditions defined in the objective. This option can be provided multiple times.

--disable-script-io-inherit: Don't inherit the user stimulus script stdout and stderr.

--disable-status-output: Don't show the objective status summary information while iterating.

--using <session name>: Use the given session rather than the default one.

# Examples

$ modality execute --objective-name example-objective --disable-script-io-inherit ./stimulus.sh
┌─ Objective Status ───────────────────────────────────────────────────────────┐
│                                                                              │
│  Objective:   example-objective                                              │
│  Instance:    1                                                              │
│  Created At:  2021-04-21 21:55:24 UTC                                        │
│  Created By:  example-user                                                   │
│  Stopped At:                                                                 │
│  Cycle Count: 0                                                              │
│                                                                              │
├─ Stopping Conditions Progress ───────────────────────────────────────────────┤
│                                                                              │
│  Time:                 00:00:00 / 00:02:00   [                        ] 0%   │
│  Passing Measurements: 0 / 15                [                        ] 0%   │
│  Failing Measurements: 0 / 5                 [                        ] 0%   │
│                                                                              │
└──────────────────────────────────────────────────────────────────────────────┘

# modality query

# Usage

modality query <<trace-expression> | --file <trace-expression-path>>
  [--expand]
  [--explain]
  [--format syslog | json | "{custom}"]
  [-v]
  [--limit <n>]
  [--count]
  [--using <session name>]
  [--with-sut <sut name>]

# Description

Execute a query and print the results. The query expression can be given directly, or via a path to a file containing the query. For further details on writing trace queries, see the query language reference. Query results, formatting, and verbosity can be controlled with flags. The --explain flag provides alternative functionality to understand why a query does or does not have results.

# Options

--file <trace-expression-path>: Path to a file containing a trace query. Mutually exclusive with a trace expression provided directly as a command line argument.

--expand: By default, query only shows explicit matches and probe interactions along a query path. To see all events, use this flag.

--explain: Provide a breakdown of the query and illustrate which parts of it had matches and which parts did not.

--format <text | syslog | json | "{custom}">: The printed output format, defaults to text. Same as log format, see here for details.

-v: Provide (more) verbose output (-v, -vv).

--limit <n>: Limit the number of returned results. If used together with --count, this acts as an upper limit on the resulting count.

--count: Only print the number of results, not their content.

--using <session name>: Use the given session rather than the default one.

--with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.

# Examples

example_query.tq:

MATCH
    (MEASUREMENT_SAMPLED@PRODUCER_PROBE) AS Sampled,
    (RECVD_MEASUREMENT_MSG@CONSUMER_PROBE) AS Received,
    (MEASUREMENT_SAMPLE_CHECK@CONSUMER_PROBE) AS Checked
WHEN
    Sampled -> Received -> Checked

$ modality query --file example_query.tq
Result 1:
═════════
│  *  Sampled(785847259:0:0:6, MEASUREMENT_SAMPLED @ PRODUCER_PROBE, payload=-1)
│  :  (1 elided events)
│    
ɮ  
  │  
*  │  Received(438896705:2:1:4, RECVD_MEASUREMENT_MSG @ CONSUMER_PROBE, payload=-1)
  │  
*  │  Checked(438896705:2:1:5, MEASUREMENT_SAMPLE_CHECK @ CONSUMER_PROBE, outcome=PASS)

Result 2:
═════════
│  *  Sampled(785847259:2:1:4, MEASUREMENT_SAMPLED @ PRODUCER_PROBE, payload=0)
│  :  (1 elided events)
│    
ɮ  
  │  
*  │  Received(438896705:3:2:5, RECVD_MEASUREMENT_MSG @ CONSUMER_PROBE, payload=0)
  │  
*  │  Checked(438896705:3:2:6, MEASUREMENT_SAMPLE_CHECK @ CONSUMER_PROBE, outcome=PASS)

Result 3:
═════════
│  *  Sampled(785847259:3:2:4, MEASUREMENT_SAMPLED @ PRODUCER_PROBE, payload=-1)
│  :  (1 elided events)
│    
ɮ  
  │  
*  │  Received(438896705:4:3:5, RECVD_MEASUREMENT_MSG @ CONSUMER_PROBE, payload=-1)
  │  
*  │  Checked(438896705:4:3:6, MEASUREMENT_SAMPLE_CHECK @ CONSUMER_PROBE, outcome=PASS)
...

# modality check

# Usage

modality check <<trace-expression> | --file <trace-expression-path>>
  [--format <text | jsonl>]
  [--timeout <duration>]
  [--using <session name>]
  [--with-sut <sut name>]

# Description

In general, check behaves as query does with the primary difference being that check only cares that there are results at all, not what those results are. Check can be thought of as determining whether the given expression is satisfiable in the targeted session’s trace data. Check uses exit codes to tell you whether the given expression was satisfied, where a 0 exit code is used to denote the affirmative, while 1 is used to denote the negative. You can also use --should-fail to invert these outcomes if you expect the expression you’ve given should have no results. For further details on writing trace queries, see the query language reference.

# Options

--file <trace-expression-path>: Path to a file containing a trace query. Mutually exclusive with a trace expression provided directly as a command line argument.

--format <text | jsonl>: The printed output format. Defaults to text.

--timeout <timeout>: A duration in seconds for which to continue reattempting the given check.

--using <session name>: Use the given session rather than the default one.

--with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.

# Examples

$ modality check "MATCH IMU_CONSISTENCY_PREFLIGHT_CHECK @ MPID_COMMANDER AND outcome = pass"
true

# modality log

# Usage

modality log
  [-v | -vv]
  [--from <from> --radius <radius>]
  [--format <syslog | "{custom}">]
  [--probe <name | id>]
  [--component <name | id>]
  [--using <session name>]
  [--color <always | never | auto>]

# Description

Print a linear, git-style log view of a trace.

# Options

-v: Provide (more) verbose output (-v, -vv).

--from <from>: Event coordinate to center output around. Must be used in combiation with --radius.

--radius <radius>: Number of events in each direction around the central --from coordinate to print. Must be used in combination with --from.

--format <syslog | "{custom}">: Specify the format for each event row. See below for details.

--probe <name | id>: Restrict output to the given probe.

--component <name | id>: Restrict output to the given component.

--using <session name>: Use the given session rather than the default one.

--color <always | never | auto>: Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handing color escape codes.

# Formatting

log provides a list of formatters which map to the data points that are available for any event:

Specifier Data
%ei Event id
%en Event name
%ef Event file
%el Event line
%ed Event description
%et Event type hint
%ep Event payload
%er Raw event payload
%ec Event coordinate
%pi Probe id
%pn Probe name
%pf Probe file
%pl Probe line
%pd Probe description
%ci Component id
%cn Component name
%rt Receive time

In the case that a probe's or event's metadata is not available, then the name (%en, %pn) specifiers fall back to their respective ids, while the others will just return an empty string.

From these, log provides a few standard output formats:

Default:

%en @ %pn (%ec)

If the -v (verbose 1) flag is used, the following is added to the output.

    description: "%ed"
    payload: %ep
    tags: %et
    source: "%ef#L%el"

If the -vv (verbose 2) flag is used, the following is added to the verbose 1 output.

    probe tags: %pt
    probe source: "%pf#L%pl"
    component: %cn

Syslog:

Syslog output follows the RFC 3164 format:

PRIORITY=<events = 6, expectations = 4> TIMESTAMP=%rt HOSTNAME=%pn MSG=%ed

syslog can be passed to the --format flag to get this pre-defined format.

# Examples


$ modality log --from 438896705:4:3:5 --radius 3
*    (438896705:3:2:6, MEASUREMENT_SAMPLE_CHECK @ CONSUMER_PROBE, outcome=PASS)
  
«  CONSUMER_PROBE merged a snapshot from PRODUCER_PROBE
  
  *  (785847259:4:3:2, MEASUREMENT_SAMPLED @ PRODUCER_PROBE, payload=-2)
  
*    (438896705:4:3:5, RECVD_MEASUREMENT_MSG @ CONSUMER_PROBE, payload=-1)
  
*    (438896705:4:3:6, MEASUREMENT_SAMPLE_CHECK @ CONSUMER_PROBE, outcome=PASS)
  
«  CONSUMER_PROBE merged a snapshot from PRODUCER_PROBE
  
  *  (785847259:5:4:2, MEASUREMENT_SAMPLED @ PRODUCER_PROBE, payload=-2)
  

# modality metrics coverage

# Usage

modality metrics coverage <region-expression>
  [-v]
  [--format <text | json>]
  [--with-sut <sut-name>]

# Description

Calculate coverage (percentage of known entities that were actually observed) for events, failures, expectations, and probes for the given region.

# Options

-v: Provide (more) verbose output (-v, -vv).

--format <text | json>: The printed output format. Defaults to text.

--with-sut <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality metrics coverage 'session="test"'
Coverage Summary:
  Event: 0.119
  Failure: 0.000
  Expectation state: 0.000
  Probes reporting: 0.778

# modality metrics complexity

# Usage

modality metrics complexity <component | probe | event> <region-expression>
  [-v]
  [--format <text | json>]
  [--with-sut <sut-name>]

# Description

Calculate a metric reflecting complexity at a component-, probe-, or event-level for the given region.

# Options

-v: Provide (more) verbose output (-v, -vv).

--format <text | json>: The printed output format. Defaults to text.

--with-sut <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality metrics complexity event 'session="test"'
Cyclomatic complexity: 89.000

$ modality metrics complexity probe 'session="2021-03-12T06-11-02Z"'
Cyclomatic complexity: 3.000

# modality metrics coupling

# Usage

modality metrics coupling <component | probe | event> <region-expression>
  [-v]
  [--format <text | json>]
  [--with-sut <sut-name>]

# Description

Calculate a metric reflecting coupling at a component-, probe-, or event-level for the given region.

# Options

-v: Provide (more) verbose output (-v, -vv).

--format <text | json>: The printed output format. Defaults to text.

--with-sut <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality metrics coupling probe 'session="test"'
Coupling factor: 1.000
Mean practical module coupling factor: 0.667
Per-node practical module coupling:
  CONSUMER_PROBE: 0.667
  MONITOR_PROBE: 0.667
  PRODUCER_PROBE: 0.667

# modality metrics pattern-frequency

# Usage

modality metrics pattern-frequency <doubles | triples> <region-expression>
  [-v]
  [--format <text | json>]
  [--with-sut <sut-name>]

# Description

Calculate the common two- or three-hop patterns and the count of their occurrences for the given region.

# Options

-v: Provide (more) verbose output (-v, -vv).

--format <text | json>: The printed output format. Defaults to text.

--with-sut <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality metrics pattern-frequency triples 'session="test"'
Pattern 1:
  Frequency: 449
    Event: MEASUREMENT_SAMPLED @ PRODUCER_PROBE
    Event: MEASUREMENT_SAMPLED @ PRODUCER_PROBE
    Event: SENT_MEASUREMENT_MSG @ PRODUCER_PROBE

Pattern 2:
  Frequency: 378
    Event: SENT_MEASUREMENT_MSG @ PRODUCER_PROBE
    Interaction: PRODUCER_PROBE -> CONSUMER_PROBE
    Event: RECVD_MEASUREMENT_MSG @ CONSUMER_PROBE

Pattern 3:
  Frequency: 378
    Event: MEASUREMENT_SAMPLED @ PRODUCER_PROBE
    Event: SENT_MEASUREMENT_MSG @ PRODUCER_PROBE
    Interaction: PRODUCER_PROBE -> CONSUMER_PROBE
...

# modality metrics fault-localization

# Usage

modality metrics fault-localization <spectrum-based | statistical> <region-expression>
  [--stimulus-only]
  [--format <text | json>]
  [--with-sut <sut-name>]

# Description

Given a region, locate some suspicious patterns in the system under test using either the spectrum based fault localization (SBFL) approach or statistical debugging (SD) approach.

# Options

--stimulus-only: Only stimulus-side expectation outcomes matter, any failure of any stimulus-side expectation means the related test region is a failure. Otherwise, any failure of any stimulus-side or SUT-side expectation means the related test region is a failure.

--format <text | json>: The printed output format. Defaults to text.

--with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.

# Examples


$ modality metrics fault-localization statistical 'session="test-session"'
Scope Failure Summary:
  Scope: DroneOperational
    Failing scope instances: [2021-05-10T10-37-39Z 3]
    Passing scope instances: []
    Failures:
      Expectation: FAILURE_DETECTOR_STATUS_ROLL_ACTIVE @ PX4_COMMANDER
      Expectation: NOMINAL_VOLTAGE_CHECK @ PX4_BATTERY

  Scope: DroneSetup
    Failing scope instances: []
    Passing scope instances: [2021-05-10T10-37-39Z 2]
    Failures:

  Scope: Case Study: Failure Detector Failsafe
    Failing scope instances: [2021-05-10T10-37-39Z 1]
    Passing scope instances: []
    Failures:
      Expectation: NOMINAL_VOLTAGE_CHECK @ PX4_BATTERY
      Expectation: FAILURE_DETECTOR_STATUS_ROLL_ACTIVE @ PX4_COMMANDER

Suspicious Predicates:
  Score: 0.551
  Kind: ExpectationFailureInSUT
  NOMINAL_VOLTAGE_CHECK @ PX4_BATTERY

  Score: 0.551
  Kind: ExpectationFailureInSUT
  FAILURE_DETECTOR_STATUS_ROLL_ACTIVE @ PX4_COMMANDER

# modality metrics falsifiability

# Usage

modality metrics falsifiability <region-expression>
  [--include-internals]
  [--stimulus-only]
  [--format <text | json>]
  [--with-sut <sut-name>]

# Description

Given a region, return a summary of failures within scopes.

# Options

--include-internals: Consider Modality's internal events/probes/components in the calculation.

--stimulus-only: Only stimulus-side expectation outcomes matter, any failure of any stimulus-side expectation means the related test region is a failure. Otherwise, any failure of any stimulus-side or SUT-side expectation means the related test region is a failure.

--format <text | json>: The printed output format. Defaults to text.

--with-sut <sut name>: Use the given SUT rather than the one currently associated with this session.

# Examples


$ modality metrics falsifiability 'session="test-session"'
Scope Failure Summary:
  Scope: DroneSetup
    Failing scope instances: []
    Passing scope instances: [2021-05-10T10-37-39Z 2]
    Failures:

  Scope: DroneOperational
    Failing scope instances: [2021-05-10T10-37-39Z 3]
    Passing scope instances: []
    Failures:
      Expectation: NOMINAL_VOLTAGE_CHECK @ PX4_BATTERY
      Expectation: FAILURE_DETECTOR_STATUS_ROLL_ACTIVE @ PX4_COMMANDER

  Scope: Case Study: Failure Detector Failsafe
    Failing scope instances: [2021-05-10T10-37-39Z 1]
    Passing scope instances: []
    Failures:
      Expectation: NOMINAL_VOLTAGE_CHECK @ PX4_BATTERY
      Expectation: FAILURE_DETECTOR_STATUS_ROLL_ACTIVE @ PX4_COMMANDER

# modality metrics similarity

# Usage

modality metrics similarity <region-expression-a> <region-expression-b>
  [--format <text | json>]
  [--with-sut <sut-name>]

# Description

Given two regions, return a metric reflecting their similarity.

# Options

--format <text | json>: The printed output format. Defaults to text.

--with-sut <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality metrics similarity 'session="test-session" and scope="DroneSetup"' 'session="test-session" and scope="DroneOperational"'
Doubles: 0.973
Triples: 0.984

# modality metrics loops

# Usage

modality metrics loops <component | probe | event> <region-expression>
  [-v]
  [--format <text | json>]
  [--with-sut <sut-name>]

# Description

Calculate the loops found within the given region, and the nodes which comprise the loop at either a component-, probe-, or event-level.

# Options

-v: Provide (more) verbose output (-v, -vv).

--format <text | json>: The printed output format. Defaults to text.

--with-sut <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality metrics loops event 'session="test-session"'
Loop: 1
  Origin: SENSORS_REQ_IN_PREFLIGHT_CHECKS @ PX4_COMMANDER
  Distinct events: 13
  Path:
    *  (SENSORS_REQ_IN_PREFLIGHT_CHECKS @ PX4_COMMANDER)
      
    *  (AIRFRAME_PREFLIGHT_CHECK @ PX4_COMMANDER)
      
    *  (MAGNETOMETER_PREFLIGHT_CHECK @ PX4_COMMANDER)
      
    *  (ACCEL_PREFLIGHT_CHECK @ PX4_COMMANDER)
      
    *  (GRYO_PREFLIGHT_CHECK @ PX4_COMMANDER)
      
    *  (BARO_PREFLIGHT_CHECK @ PX4_COMMANDER)
      
    *  (IMU_CONSISTENCY_PREFLIGHT_CHECK @ PX4_COMMANDER)
      
    *  (ARM_TRANSITION_PREFLIGHT_CHECK @ PX4_COMMANDER)
      
    *  (ARMDISARM_TRANSITION_ACCEPTED @ PX4_COMMANDER)
      
    *  (ARMDISARM_TRANSITION_RESULT_CHECK @ PX4_COMMANDER)
      
    *  (FAILURE_DETECTOR_UPDATE_STABLE @ PX4_COMMANDER)
      
    *  (FAILURE_DETECTOR_STATUS_ROLL_ACTIVE @ PX4_COMMANDER)
      
    *  (FAILURE_DETECTOR_STATUS_PITCH_ACTIVE @ PX4_COMMANDER)
      
    *  (SENSORS_REQ_IN_PREFLIGHT_CHECKS @ PX4_COMMANDER)
    
Loop: 2
  Origin: SENSORS_REQ_IN_PREFLIGHT_CHECKS @ PX4_COMMANDER
  Distinct events: 13
  Path:
    *  │  (SENSORS_REQ_IN_PREFLIGHT_CHECKS @ PX4_COMMANDER)
      │  
    *  │  (AIRFRAME_PREFLIGHT_CHECK @ PX4_COMMANDER)
      │  
    *  │  (MAGNETOMETER_PREFLIGHT_CHECK @ PX4_COMMANDER)
      │  
    *  │  (ACCEL_PREFLIGHT_CHECK @ PX4_COMMANDER)
      │  
    *  │  (GRYO_PREFLIGHT_CHECK @ PX4_COMMANDER)
      │  
    *  │  (BARO_PREFLIGHT_CHECK @ PX4_COMMANDER)
      │  
    *  │  (IMU_CONSISTENCY_PREFLIGHT_CHECK @ PX4_COMMANDER)
      │  
    *  │  (DATA_LINK_RECOVERY_PREFLIGHT_CHECK @ PX4_COMMANDER)
      │  
    *  │  (FAILURE_DETECTOR_UPDATE_STABLE @ PX4_COMMANDER)
    »╗  
    │  *  (FREEFALL @ PX4_LAND_DETECTOR)
    │    
    │  *  (GROUND_CONTACT @ PX4_LAND_DETECTOR)
    │    
    │  *  (MAYBE_LANDED @ PX4_LAND_DETECTOR)
    │    
    │  *  (LANDED @ PX4_LAND_DETECTOR)
    │    
    ɮ  
      │  
    *  │  (SENSORS_REQ_IN_PREFLIGHT_CHECKS @ PX4_COMMANDER)
...

# modality metrics topology

# Usage

modality metrics topology <component | probe | event> <region-expression>
  [-v]
  [--format <text | json>]
  [--with-sut <sut-name>]

# Description

Generate a state machine-like topology representing the unique events, probes, or components and their direct relationships for the given region.

# Options

-v: Provide (more) verbose output (-v, -vv).

--format <text | json>: The printed output format. Defaults to text.

--with-sut <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality metrics topology probe 'session="test-session"'
SOURCE             DESTINATION
PX4_BATTERY        PX4_COMMANDER
PX4_COMMANDER      PX4_LAND_DETECTOR
PX4_LAND_DETECTOR  GAZEBO_SIMULATOR
PX4_LAND_DETECTOR  PX4_COMMANDER
PX4_LAND_DETECTOR  PX4_NAVIGATOR
PX4_NAVIGATOR      GAZEBO_SIMULATOR
PX4_SIMULATOR      PX4_NAVIGATOR

# modality metrics summary

# Usage

modality metrics summary <region-expression>
  [-v]
  [--include-internals]
  [--format <text | json>]
  [--with-sut <sut-name>]

# Description

Given a region, return some general summary statistics.

# Options

-v: Provide (more) verbose output (-v, -vv).

--include-internals: Consider Modality's internal events/probes/components in the calculation.

--format <text | json>: The printed output format. Defaults to text.

--with-sut <sut name>: Use the given SUT rather than the default one.

# Examples


$ modality metrics summary 'session="simple-session"' -v
Interactions: 556
Passing Expectation Instances: 524
Failure Instances: 21
Scopes Begun: 8

Sessions With Mutations: 1
Injected Mutations: 5
Active Mutations: 5
Mutation Epochs: 17
Overlapping Mutations By Session:
  Session: 2021-05-12T11-21-30Z
    Injected Mutations:
      Instance: 6
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-57]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 2
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-81]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 7
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=1007]
        Associated Failures: 10
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 3
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-7]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 5
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-10]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
    Active Mutations:
      Instance: 3
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-7]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 2
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-81]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 6
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-57]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 5
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-10]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 7
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=1007]
        Associated Failures: 10
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
    Mutation epochs:
      [14, 11, 12, 13, 10, 15, 5, 17, 4, 3, 16, 18, 7, 9, 2, 6, 8]

Probes With Passing Expectations: 2
Probes With Failing Expectations: 1
Distinct Events: 15
Distinct Passing Expectations: 2
Distinct Failing Expectations: 1
Distinct Events By Component:
  consumer-component: 5
  monitor-component: 5
  producer-component: 5

Distinct Events With Payloads: 8
Distinct Payloads: 24
Event Payloads By Component:
  consumer-component
    MEASUREMENT_SAMPLE_CHECK: 3
    RECVD_MEASUREMENT_MSG: 3
    SENT_HEARTBEAT_MSG: 3
  monitor-component
    HEARTBEAT_SENDER_ID_CHECK: 3
    RECVD_HEARTBEAT_MSG: 3
  producer-component
    MEASUREMENT_SAMPLED: 3
    SENT_HEARTBEAT_MSG: 3
    SENT_MEASUREMENT_MSG: 3

# Region Expressions

A region expression is a predicate used to match some region or regions of causal histories. Region expressions are used in a variety of commands to designate what part of recorded execution the command should be evaluated against. Every region expression must include at least one session. Different conditions in a region expression can be combined with AND, OR, and parentheses.

# Allowed Fields

  • session: session name
    • Comparisons: =, !=
    • At least one session is required in every region expression.
    • More than one session may be used in a single region expression. If more than one session is used, clauses specifying session must be combined with OR, since sessions are non-overlapping by definition.
  • scope: scope name
    • Comparisons: =, !=
  • scope_instance: scope instance ID
    • Comparisons: =, !=
    • Scope instance IDs are allocated on a per-session basis, so field comparisons involving scope instance IDs should be appropriately combined with a session specification.
  • coordinates:
    • Comparisons: =, !=, <, >, <=, >=
    • coordinates < X means "the region causally preceding coordinates point X, excluding X"
    • coordinates > X means "the region causally following coordinates point X, excluding X"
    • coordinates <= X means "the region causally preceding coordinates point X, including X"
    • coordinates >= X means "the region causally following coordinates point X, including X"
  • probe: probe name or ID
    • Comparisons: =, !=
  • component: component name
    • Comparisons: =, !=
  • component_id: component UUID
    • Comparisons: =, !=

# Examples

'session = "session-a" OR session = "session-b"'

The union of all of session-a with all of session-b.

'session = "session-a" AND (scope = "scope-a" OR scope = "scope-b")'

The region comprised of the parts of session-a where either scope-a or scope-b was active.

'scope = "foo" AND (session = "session-a" OR session = "session-b")'

The union of the regions in session session-a and session session-b where a scope named foo was active.

'session = "session-a" AND coordinates > 7:9:4:2 AND scope = "bar"'

The region comprised of the part of session-a causally following point 7:9:4:2 where a scope bar was active.

'session = "session-a" AND scope = "baz" AND probe = "probe-a"'

The region comprised only of data from probe probe-a in session session-a where a scope baz was active.

# modality-probe

modality-probe is the command for modality's instrumentation-based manifest and header generation tools. Since Modality is designed to be used in resource-constrained situations these tools are provided to extract rich metadata from your instrumentation and save it for use from the CLI while keeping it out of your binary.

modality-probe commands
modality-probe manifest-gen
modality-probe header-gen
modality-probe mutator-gen

# modality-probe manifest-gen

# Usage

modality-probe manifest-gen <source-path>
  [--regen-component-id]
  [--component-name <component-name>]
  [--event-id-offset <event-id-offset>]
  [--exclude <exclude-patterns>...]
  [--file-extension <file-extensions>...]
  [-l, --lang <lang>]
  [-o, --output-path <output-path>]
  [--probe-id-range <probe-id-range>]

# Description

Generate Modality component definition files from instrumentation macro invocations.

# Options

--regen-component-id: Regenerate the component IDs instead of using existing IDs (if present).

--component-name <component-name>: Component name used when generating a component manifest. Defaults to component.

--event-id-offset <event-id-offset>: Integer to start numbering event IDs from. Defaults to 1.

--exclude <exclude-patterns>...: Exclude files that match the given pattern.

--file-extension <file-extensions>...: Limit the source code searching to files with matching extensions.

-l, --lang <lang>: Language, either c or rust. If not specified then guess based on file extensions.

-o, --output-path <output-path>: Output path for generated component files. Defaults to component.

--probe-id-range <probe-id-range>: Constrain the generated probe ID to a specific range. This can be either <inclusive_start>..<exclusive_end> or <inclusive_start>..=<inclusive_end>. The range values are unsigned 32-bit integers and must be non-zero.

# Examples

$ modality-probe manifest-gen --lang c --file-extension c --component-name test-component --output-path src/test-component src
Adding probe PRODUCER_PROBE, ID 118856317 to src/test-component/probes.csv
Adding probe CONSUMER_PROBE, ID 924402791 to src/test-component/probes.csv
Adding event PRODUCER_STARTED, ID 1 to src/test-component/events.csv
Adding event PRODUCER_SHUTDOWN, ID 2 to src/test-component/events.csv
Adding event PRODUCER_MEASUREMENT_SAMPLED, ID 3 to src/test-component/events.csv
Adding event PRODUCER_SAMPLE_DELTA_OK, ID 4 to src/test-component/events.csv
Adding event PRODUCER_MEASUREMENT_SENT, ID 5 to src/test-component/events.csv
Adding event CONSUMER_STARTED, ID 6 to src/test-component/events.csv
Adding event CONSUMER_SHUTDOWN, ID 7 to src/test-component/events.csv
Adding event CONSUMER_MEASUREMENT_RECVD, ID 8 to src/test-component/events.csv
Updated component test-component in src/test-component/Component.toml

# modality-probe header-gen

# Usage

modality-probe header-gen <component-path>
  [--rust-u32-types]
  [--include-guard-prefix <include-guard-prefix>]
  [-l, --lang <lang>]
  [-o, --output-path <output-path>]

# Description

Generate C or Rust header files with event/probe ID constants from component definition files.

# Options

--rust-u32-types: When generating Rust definitions, use bare u32 types instead of ProbeId/EventId types.

--include-guard-prefix <include-guard-prefix>: C header include guard prefix. Defaults to MODALITY_PROBE.

-l, --lang <lang>: Language to output the source in, either c or rust. Defaults to c.

-o, --output-path <output-path>: Write output to file instead of stdout.

# Examples

$ modality-probe header-gen --lang c --output-path src/include/component_definitions.h src/test-component
Generated definitions for component test-component in src/include/component-definitions.h

# modality-probe mutator-gen

# Usage

modality-probe mutator-gen <source-path>
  [--file-extension <file-extensions>...]
  [--definitions-output-path <definitions-output-path>]
  [--header-output-path <header-output-path>]
  [-o, --output-path <output-path>]

# Description

Generate mutator implementations and definition files from instrumentation macro invocations.

# Options

--file-extension <file-extensions>...: Limit the source code searching to files with matching extensions.

--definitions-output-path <definitions-output-path>: Output path where generated mutator definition files are placed. Defaults to mutators.

--header-output-path <header-output-path>: Output header files to this directory instead of alongside the source files in output-path.

-o, --output-path <output-path>: Output path where generated source code files are placed. Defaults to ..

# Examples

$ modality-probe mutator-gen --file-extension c --definitions-output-path src/test-component/mutators --header-output-path src/include --output-path src/mutators src