deviant experiment

Command: deviant experiment

Create and manage experiments.

Experiments are entities for grouping and guiding mutations. They are used to constrain the set of mutations that are proposed to apply against a system. Mutations that are created as part of an experiment are linked to the experiment for future data analysis.

This command's functionality is exposed via subcommands:

# deviant experiment create

Command: deviant experiment create <experiment name>

Create a new experiment. Creating experiments allows you to track related mutations.

# Example

$ deviant experiment create example \
    --mutator-filter '_.group = "foo" AND _.mycorp.safe_for_prod = true'

Created experiment 'example'
mutator-filter = '_.group = "foo" AND _.mycorp.safe_for_prod = true'

# Options

<experiment name> - Name of the newly created experiment

-f, --format <text | json> - The printed output format, defaults to text.

--mutator-filter <mutator predicate> - An optional logical expression that specifies which mutators ought to be used for this experiment.

--expected-mutator <expected-mutator> - Deviant will expect a mutator instance matching this filter expression to exist and to be available for use before this experiment solidifies its mutation suggestion strategy. Can be provided more than once to indicate multiple expected mutator instances. If --mutator-filter is also present it will be applied prior to counting up expected mutator matches. Take care to ensure all expected mutators also pass the --mutator-filter expression.

--spec <spec> - A specification of interest to the experiment. The outcome of the spec's evaluation will be used as part of the experiment's feedback loop for learning about the system under test. This may be a spec's name or a specific spec's version id. If a name is used, that spec's current version will be associated with the experiment. Can be provided more than once to indicate multiple specs of interest.

--approach <random | structured-breadth | learned-perturbation> - What goal is this experiment striving towards, combined with what broad-strokes strategy is it taking. Options are random, structured-breadth, or learned-perturbation.

# deviant experiment list

Command: deviant experiment list

List the names of all known experiments.

# Example

$ deviant experiment list

Experiments: 3
alhazen
avicenna
rhazes

# Options

-f, --format <text | json> - The printed output format, defaults to text.

# deviant experiment inspect

Command: deviant experiment inspect <experiment name>

Inspect a specific experiment

# Example

$ deviant experiment inspect testing

Experiment 'testing'
Definition:
approach = 'learned-perturbation'
specs = [
    'power',
    'vision',
    'imu2',
    'imu',
    'comms',
    'compute',
]

# Options

<experiment name> - Name of the experiment to inspect

-f, --format <text | json> - The printed output format, defaults to text.

# deviant experiment train

Command: deviant experiment train <experiment name>

Provide data for a learning-based experiment

When performing an experiment which learns from the system as it goes on (such as learned-pertubation), this command may be used to present new data for incorporation into the model.

This command has no effect with used with a non-learning-based experiment (such as random or structured-breadth).

# Example

$ deviant experiment train testing --segment "Run testing-42"

Training experiment 'testing' over 1 segment
Training complete for 1 segment.

--whole-workspace - Limit data to the workspace as a monolithic whole unit. May not be used with '--latest-segment' or '--segment'

--latest-segment - Limit data to the content of the latest segment. This will automatically update when a new segment is detected. If not made explicit here, the CLI will attempt to extract it from the configuration context. May not be used with '--segment' or '--whole-workspace'.

--segment <SEGMENT> - Limit all event-related commands to the segments which match this pattern. Patterns may include '*' or '?' in the style of glob. May not be used with '--latest-segment' or '--whole-workspace'.

--segmentation-rule <segmentation-rule> - Limit data to the segments produced by a segmentation rule with a name matching this pattern. Patterns may include '*' or '?' in the style of glob. May not be used with '--latest-segment' or '--whole-workspace'.

--workspace-name <workspace-name> - Name of the workspace of interest. If not made explicit here, the CLI will attempt to extract it from the configuration context.

# deviant experiment results

Command: deviant experiment results <experiment name>

View the results of an experiment.

# Example

$ deviant experiment results testing

Experiment 'testing'
Mutations: 225 explicitly linked, 0 found in scope
Mutators: 225

# Options

<experiment name> - Name of the experiment to inspect

-f, --format <text | json> - The printed output format, defaults to text.

--global - Instead of limiting to a single or set of segments, or a workspace, look for related mutations in Modality's entire database. If this flag is not present results are limited to those pertaining to the current segment(s) of the current workspace. May not be used with --latest-segment, --segment, --segmentation-rule, or --whole-workspace.

--workspace-name <workspace-name> - Name of the workspace in which to look for related mutations. If not specified the current default workspace (set with modality workspace use) will be used.

--whole-workspace - Look for related mutations in the workspace as a monolithic whole unit. May not be used with --global, --latest-segment, --segment, or --segmentation-rule.

--latest-segment - Look for related mutations in the single most recent segment (i.e. the segment which has most recently received an event or timeline attribute update) rather than the current default segment(s). The latest segment is automatically updated as new data are received. May not be used with --global, --segment, --segmentation-rule, or --whole-workspace.

--segment <segment pattern>... - Look for related mutations in the given segment(s) rather than the current default segment(s). Accepts one or more segment name patterns, which can include * or ? as wildcards in the style of glob. May not be used with --global, --latest-segment, or --whole-workspace.

--segmentation-rule <segmentation rule pattern>... - Look for related mutations in segments produced by the given segmentation rule(s) rather than the current default segment(s). Accepts one or more segmentation rule name patterns, which can include * or ? as wildcards in the style of glob. May not be used with --global, --latest-segment, or --whole-workspace.

--timeline-filter <timeline filter> - A timeline filter expression (opens new window) to restrict mutation-inspection to timelines that match the filter.

# deviant experiment impact

Command: deviant experiment impact <experiment name>

Find the impact of mutations injected over the course of an experiment.

All the segments associated with the given experiment will be inspected. Those that do not contain mutations will be pooled together as a baseline. Then, each segment which contains a mutation will be compared with the baseline to identify the impact of the mutation.

# Example

$ deviant experiment impact testing

Experiment Impact Report for Experiment "testing"

Specification Impact Scenario 1
  Associated Mutations:
      - Watchdog execution out-of-sync error =>
        - Mutation Id 3e9ae92b-bd18-4a10-a982-08422fe47c29 in Segment 'Run testing-124', no params
        - Mutation Id ae2857cb-60c5-48ed-90bf-503ac21c3ae6 in Segment 'Run testing-164', no params
        - Mutation Id f02262ab-dbcc-40dd-886b-b2e4249d3672 in Segment 'Run testing-181', no params
        - Mutation Id 0a11bfaa-ab29-4737-9478-b23fc6094df3 in Segment 'Run testing-208', no params
        - Mutation Id 63569205-ed20-46f4-8eb5-04eb869fab6d in Segment 'Run testing-68', no params
        - Mutation Id ee050978-1f1e-4b3e-a2a0-28f7ca4eed8b in Segment 'Run testing-9', no params

  Specification Impact:
    Specification "imu":
      Behavior: IMU subsystem samples are send every 100ms
        - passed in the baseline, but failed in the mutated segments
      
  System Behavior Impact:
    Timeline "correlation":
      Event "correlated_ir_events":
        Attribute "event_count":
          - was higher in the mutated cases (5.02 → 5.18)
          ...

# Options

<experiment name> - Name of the experiment to assess

-f, --format <text | json | html> - The printed output format, defaults to text. The html option produces the document which is shown in the VS Code extension's Experiment Impact outline.

--workspace-name <workspace-name> - Name of the workspace of interest. If not made explicit here, the CLI will attempt to extract it from the configuration context.

--alpha <alpha> - The p-value threshold to use when assessing statistical significance. Defaults to 0.05.

--relative-count-threshold <relative-count-threshold> - Threshold ratio for filtering out insignificant changes in event count; between 0.0 and 1.0. Defaults to 0.05.

--relative-cadence-threshold <relative-cadence-threshold> - Threshold ratio for filtering out insignificant changes in event cadence; between 0.0 and 1.0. Defaults to 0.05.

--ignore-event <ignore-event> - Ignore any events whose name matches this glob pattern.

--ignore-timeline <ignore-timeline> - Ignore any timelines whose name matches this glob pattern.

--ignore-attr <ignore-attr> - Ignore any attr keys which match this glob pattern.

-o, --output-file <OUTPUT_FILE> - Write output to the given file, instead of stdout.

← mutation →