modality objective

Command: modality objective

Create and delete objectives, list them and their associations, perform analysis, and inspect their detailed contents and relationships. This command's functionality is exposed via subcommands:

# modality objective apply

Command: modality objective apply <name> [region expression]

Create a new instance of an objective and take its measurements against the pre-recorded data in the supplied region expression. When using this command the mutator guidance and stopping conditions in the objective are ignored since the data has already been recorded. If no region expression is provided it will use the default session.

# Example

$ modality objective apply example-objective
┌─ Objective Status ───────────────────────────────────────────────────┐
│                                                                      │
│  Objective:   example-objective                                      │
│  Instance:    5                                                      │
│  Created At:  2021-08-24 21:14:09 UTC                                │
│  Created By:  example@auxon.io                                       │
│  Stopped At:                                                         │
│  Cycle Count: 1                                                      │
│                                                                      │
├─ Stopping Conditions Progress ───────────────────────────────────────┤
│                                                                      │
│  Time:                 00:00:00 / 00:02:00   [                ] 0%   │
│  Passing Measurements: 0 / 10                [                ] 0%   │
│  Failing Measurements: 1 / 10                [=>              ] 10%  │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

# Options

<name> - Name of the objective to apply.

region expression - Trace region to apply the objective to. If not provided the objective will be applied to the default session. For details see Region Expressions.

--disable-status-output - Don't show the objective status summary information.

--using <session name>- Use the given session rather than the default one.

# modality objective create

Command: modality objective create <definition file path>

Create a new objective from the given objective definition file.

# Example

$ modality objective create ./sample-objective.toml 

$ modality objective list
NAME               TAGS                  SESSIONS  SUTS
example-objective  [example]             1         1
sample objective   [create-sample, foo]  1         1

# Options

<definition file path> - Path to a TOML file with the objective definition information.

# modality objective delete

Command: modality objective delete <name>

Delete the objective with the given name. Delete is idempotent; if no objective with the given name exists, then no action is taken.

# Example

$ modality objective delete sample-objective
Are you sure you want to delete objective 'sample-objective'? (yes/no)
y

$ modality objective list
NAME               TAGS         SESSIONS  SUTS
example-objective  [example]    1         1

# Options

<name> - Name of the objective to delete.

--force - Bypass the user input confirmation check.

# modality objective inspect

Command: modality objective inspect <objective name> [instance id]

This command has two variants:

  • Print detailed information about the given objective's definition and relationships. This will show the entire objective definition, list associated SUTs, and show instances of the objective by session.
  • Print information about a single instance of an objective. This includes a summary of measurement results and, at higher verbosity levels, lists of every mutation that was injected and of the measurement results for every execution run.

# Example

$ modality objective inspect example-objective
Name: example-objective
Created at: 2021-07-06 09:36:11 UTC
Created by: example-user
SUTS: [example-sut]
Instances By Session:
  Session: example-session
    Instances: [1]
Tags: [example]
Stopping Conditions:
  Attempts: 
  Time: 2min
  Passing Measurements: 10
  Failing Measurements: 10
Mutation Constraints:
  Max Concurrent Mutations: 1
  Mutation Precondition: 
  Mutator Tags: []
Mutator Constraints: 2
  Mutator: heartbeat-delay-mutator
    Probe: CONSUMER
    Mutation Precondition: 
    Parameter Constraints: 1
      Parameter: heartbeat-delay
      Range: [500, 2000]
  ...
Measurements: 8
  Measurement Name: Consumer heartbeat timing
    Stopping Conditions:
      Passing Measurements: 
      Failing Measurements: 
    Check:
      NEVER
          # 2 == HEARTBEAT_ID_CONSUMER
          RECVD_MSG @ MONITOR (_.payload = 2) 
          -> AFTER 600ms
          RECVD_MSG @ MONITOR (_.payload = 2)

  ...

# Options

<objective name> - Identifier for the objective to be inspected.

[instance id] - If an instance id is provided, the output will focus on that particular instance (i.e. run) of the given objective. Instance IDs can be found with modality objective instances <name>. The -v and -vv flags are particularly useful for this variant of the command.

--color <auto | always | never> - Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handling color escape codes.

-f, --format <text | json> - The printed output format. Defaults to text.

--include-internal - Include Modality's internal entities in the output.

--using <session name> - Use the given session rather than the default one.

-v, --verbose - Provide verbose output.

-vv - If an instance id has been given, provide even more verbose output.

# modality objective instances

Command: modality objective instances <name>

List the instances of the given objective. An objective instance represents a run of that objective, and each objective instance is associated with a single session. Objective instances can be created with the execute, objective apply, and objective sample commands.

# Example

$ modality objective instances example-objective
NAME               INSTANCE  SESSION               SUT          STATUS
example-objective  1         example-session       example-sut  STOPPED
example-objective  2         2021-04-16T06-51-16Z  example-sut  STOPPED

# Options

<name> - Name of the objective to list instances of.

--color <auto | always | never> - Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handling color escape codes.

-v, --verbose - Provide verbose output.

# modality objective list

Command: modality objective list

List all known objectives.

# Example

$ modality objective list
NAME               TAGS                  SESSIONS  SUTS
example-objective  [example]             1         1
sample objective   [create-sample, foo]  1         1

# Options

--color <auto | always | never> - Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handling color escape codes.

-v, --verbose - Provide verbose output.

# modality objective report

Command: modality objective report [name]

Produce a report about objective measurement outcomes for a session, including analysis of detected failures. See Objective Report Example for a full, compiled example of a produced report.

# Example

This command outputs markdown. A full, compiled example of this objective report can be seen here.

$ modality objective report
Run **example-session** had **39%** event coverage, **33%** failure coverage, **50%** of the expectation space covered and saw **100%** of probes reporting.
During this run Modality recorded **23789** events, injected **1** mutations, and saw **401** expectations.

This run included the use of the `example-objective` objective, which ran once. See below for more details on these runs.
# Run 1

## Failed Measurements

### Monitor heartbeat timeouts

The measurement **Monitor heartbeat timeouts** failed 5 times during this run.

**Failure 1**

This failure occurred in the presence of the following mutations:


| Mutator | Preconditions | Parameters |
|-|-|-|
| sample-data-mutator @ PRODUCER | SAMPLED@PRODUCER | `{"sample":68}` |

Causing the following error in the measurement:
```
Violation: Constraint Has Match
  │ 
1 │ NEVER *@*(((_.tag = 'timeout') AND (_.outcome = FAIL)))
  │             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  │             |
  │             This constraint caused this failure.
```
**Failure 2**

...

# Options

[name] - Name of the objective to sample. If no name is provided, the most interesting objective for the relevant session will be selected.

-f, --format <markdown | json | html> - The printed output format. Defaults to markdown.

--using <session name> - Use the given session rather than the default one.

# modality objective sample

Command: modality objective sample <name> [region expression]

Run the measurement checks for the currently open instance of the given objective.

# Example

$ modality objective sample example-objective

# Options

<name> - Name of the objective to sample.

region expression - Trace region to run measurements against. If not provided, defaults to the default session. For details see Region Expressions.

--using <session name> - Use the given session rather than the default one.