conform spec

Command: conform spec

Create, manage, and evaluate specifications. Specifications describe the expected behavior of your system, and can specify nominal, recovery, and prohibited behavior. They are stored in Modality's backend so that you can update them, track versions, evaluate them against specified parts of recorded history, and store and recall evaluation results. For more information on how to write specifications see the SpeQTr language reference. (opens new window)

This command's functionality is exposed via subcommands:

spec create
spec delete
spec eval
spec inspect
spec list
spec update

# conform spec create

Command: conform spec create <specification name>

Create a new specification. Creating specifications allows you to track the results of their evaluations, and created specifications can also be updated to manage versions of the same specification over time. You must provide either the --file or the --from-stdin option to specify the SpeQTr expression for this specification.

# Example

$ conform spec create example --file example-spec.speqtr
Created spec 'example' at version '06980945-ac5e-4137-a75c-b50608837f05'

# Options

<specification name> - Name of the newly created specification.

--color <always | never | auto> - Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handling color escape codes.

--file <specification file path> - Path to a file containing the SpeQTr expression for this specification. Mutually exclusive with --from-stdin.

-f, --format <text | json> - The printed output format, defaults to text.

--from-stdin - Provide the SpeQTr expression for this specification through standard input. When providing an expression through standard input, you may pipe from another command like echo or cat, or type into the terminal and use Ctrl+D on a blank line to indicate the end of the input. Mutually exclusive with --file.

--quiet - If creation succeeds, only print out the ID of the created spec version.

# conform spec delete

Command: conform spec delete [specification name]

Delete one or all versions of a specification. If a specification name is provided, all versions of that specification will be deleted. Otherwise you can pass the --version option to specify a single version of a specification to delete. Either a specification name or the --version option must be provided.

# Example

$ conform spec delete example
Are you sure you want to delete specification 'example' at version 06980945-ac5e-4137-a75c-b50608837f05? (yes/no)
y
Deleted specification 'example'

# Options

--color <always | never | auto> - Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handling color escape codes.

--force - Bypass the user confirmation check.

-f, --format <text | json> - The printed output format. Defaults to text.

--version <specification version> - Only delete the given specification version.

# conform spec eval

Command: conform spec eval

Evaluate a specification. You can evaluate a stored specification either by name or by version ID. You can also evaluate a specification which has not already been created—this will create and evaluate the specification. There are a number of options to select which data to evaluate the specification against. If you do not provide any of these options it will be evaluated against the current default segment of the current default workspace, set with modality segment use (opens new window) and modality workspace use (opens new window).

# Example

$ conform spec eval --name example
No explicit workspace provided. Using the default workspace.
evaluation of spec "example"
[ ✔ ] segment "Run 80ba761d-4ce2-426b-9cae-a88c50bdacf5"
[ ✔ ]   behavior "Pipeline messaging"
All Evaluations Passed
Behaviors: [1: Passed] [0: Inconclusive] [0: Failed] (stored as 52715903-ade7-4c69-9d6e-cbbc6fa2aee3)

$ conform spec eval --name failing-example
No explicit workspace provided. Using the default workspace.
evaluation of spec "failing-example"
[ ✘ ] segment "Run 80ba761d-4ce2-426b-9cae-a88c50bdacf5"
[ ✘ ]   behavior "Pipeline messaging"
Failures Detected
Behaviors: [0: Passed] [0: Inconclusive] [1: Failed] (stored as c66a8db3-70b2-4c1d-b724-aa06abe17dd9)

   |
 1 | behavior 'Pipeline messaging'
   |
 6 |     nominal case 'it is received by the consumer'
 7 |         tx
 8 |            -> within 850us
   |               ^^^^^^^^^^^^ Did not match
 9 |         "Received measurement message"@consumer (_.interaction.remote_nonce = tx.nonce)
10 |     end
   |
   = error: When evaluating the behavior 'Pipeline messaging', no expected cases
            matched. The case 'it is received by the consumer' partially matched,
            but an event matching the subexpression 'within 850us' was not found.

# Options

--color <always | never | auto> - Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handling color escape codes.

--dry-run - Evaluate the specification without storing its results.

--file <specification file path> - Path to a file containing the SpeQTr expression to evaluate. A specification will be autogenerated for this expression. Mutually exclusive with --from-stdin, --version, and --name.

-f, --format <text | json> - The printed output format, defaults to text.

--from-stdin - Provide the SpeQTr expression to evaluate through standard input. A specification will be autogenerated for this expression. When providing an expression through standard input, you may pipe from another command like echo or cat, or type into the terminal and use Ctrl+D on a blank line to indicate the end of the input. Mutually exclusive with --file, --version, and --name.

--global - Instead of limiting to a single or set of segments, or a workspace, evaluate the specification against Modality's entire database. If this flag is not present results are limited to those pertaining to the current segment(s) of the current workspace. May not be used with --latest-segment, --segment, --segmentation-rule, or --whole-workspace.

--latest-segment - Evaluate the specification against the single most recent segment (i.e. the segment which has most recently received an event or timeline attribute update) rather than the current default segment(s). The latest segment is automatically updated as new data are received. May not be used with --global, --segment, --segmentation-rule, or --whole-workspace.

--name <specification name> - Name of the specification to evaluate. Mutually exclusive with --file, --from-stdin, and --version.

--quiet - Rather than printing the results of evaluating the specification, just print the result ID.

--segment <segment pattern>... - Evaluate the specification against the given segment(s) rather than the current default segment(s). Accepts one or more segment name patterns, which can include * or ? as wildcards in the style of glob. May not be used with --global, --latest-segment, or --whole-workspace.

--segmentation-rule <segmentation rule pattern>... - Evaluate the specification against segments produced by the given segmentation rule(s) rather than the current default segment(s). Accepts one or more segmentation rule name patterns, which can include * or ? as wildcards in the style of glob. May not be used with --global, --latest-segment, or --whole-workspace.

--timeline-filter <timeline filter> - A timeline filter expression (opens new window) to restrict evaluation to timelines that match the filter.

--validation-only - Just check that the specification is runnable, do not evaluate it.

-v, --verbose - Print verbose debugging information.

--version <specification version id> - ID of an exact specification version to evaluate. Mutually exclusive with --file, --from-stdin, and --name.

--whole-workspace - Evaluate the specification against the workspace as a monolithic whole unit. May not be used with --global, --latest-segment, --segment, or --segmentation-rule.

--with-all-failures - By default, when a specification fails, Conform's logical analysis functionality prints the part of the specification that most likely led to the failure. Pass --with-all-failures to make Conform not limit itself to a single guess per failure, instead printing every possible failure point of the specification.

--with-case-matches - Rather than simply printing a summary of passing and failing specifications, print information about how many times each case in each behavior of the specification was matched.

--with-lax-causality - When evaluating event relationships (opens new window) like FOLLOWED BY, trust all wall clocks across timelines. By default, to establish a sequential relationship between events, Conform requires a causal link, either because the events are on the same timeline or because the different timelines the events are on are joined by an explicit interaction. If you pass --with-lax-causality Conform evaluates your specification under the assumption that all wall clocks are perfectly synchronized across timelines, and that timelines can affect each other regardless of whether an explicit interaction is present. This means that relationships exist between all events, and timestamps are the only thing which determines event order.

--without-logical-analysis - Suppress printing analysis of what may have caused a specification to fail.

--workspace-name <workspace-name> - Name of the workspace to evaluate the specification against. If not specified the current default workspace (set with modality workspace use) will be used.

--with-top-level-fault-localization - When printing the outcome for failed behaviors, analyze the differences between top-level data scopes (e.g. workspace segments, whole workspaces), in cases where the behavior succeeded vs where it failed.

--with-behavior-level-fault-localization - When printing the outcome for failed behaviors, analyze the differences between behavior-level data regions (e.g. the microregions made as part of triggered behaviors) in cases where the behavior succeeded vs where it failed.

--fault-localization-grouping - An ordered list of attribute keys that is used to group the events analyzed as part of fault localization. If not specified, the grouping will be by event name and timeline name.

# conform spec inspect

Command: conform spec inspect <specification name or version>

Print detailed information about the given specification.

# Example

$ conform spec inspect test
Spec: test

Updated By:		phil
Updated At:		2022-08-07 02:07:44 UTC
Current Version:	0c7790d4-320b-4b2e-9630-7f0be29c9857

Other Versions:
	* 0c7790d4-320b-4b2e-9630-7f0be29c9857	Created at 2022-08-07 02:07:44 UTC by phil
	■ e8e08702-634d-477a-a649-f641b6383d3a	Created at 2022-08-05 22:09:18 UTC by phil
Spec Evaluations: 1
	Result Set 398c519b-6fae-4801-a74e-17763ad84d75 from 'test' at version 0c7790d4-320b-4b2e-9630-7f0be29c9857
		Behaviors: ['Monitoring the heartbeats
			generically','pipeline messaging']
		✔ Passing Regions: 1
		✘ Failing Regions: 1
		⯃ Unknown Regions: 0

# Options

<specification name or version> - Specification to inspect. You may provide either a specification name or a specification version. This command prints information about results associated with the given specification. If you provide a name, it also shows a list of other versions of the specification, if any.

--color <always | never | auto> - Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handling color escape codes.

-f, --format <text | json> - The printed output format, defaults to text.

--with-speqtr - Print the full SpeQTr expression for this specification.

# conform spec list

Command: conform spec list

List all known specifications along with their current version ID.

# Example

$ conform spec list
test at version 0c7790d4-320b-4b2e-9630-7f0be29c9857
other-test at version 148dee3e-c1b2-43c1-a614-b55331091bf3

# Options

--color <always | never | auto> - Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handling color escape codes.

-f, --format <text | json> - The printed output format, defaults to text.

# conform spec update

Command: conform spec update <specification name>

Update an existing specification, creating a new version of it.

# Example

$ conform spec update test --file new-test-spec.speqtr
Updated spec 'test' to version 'cac6191a-ac21-48ab-94f6-fc324d943c78'

# Options

--color <always | never | auto> - Whether to color the output. Default is auto, which will autodetect the target output location and whether or not it is capable of handling color escape codes.

--file <specification file path> - Path to a file containing the SpeQTr expression for this specification. Mutually exclusive with --from-stdin.

-f, --format <text | json> - The printed output format, defaults to text.

--from-stdin - Provide the SpeQTr expression for this specification through standard input. When providing an expression through standard input, you may pipe from another command like echo or cat, or type into the terminal and use Ctrl+D on a blank line to indicate the end of the input. Mutually exclusive with --file.

--quiet - If creation succeeds, only print out the ID for the updated spec version.

← result SpeQTr (opens new window) →