# Analyzing A Session

This document provides an overview of Modality's functionality for analyzing a session. It assumes you have already collected a session—a guide for collecting data can be found here. For a higher level overview of concepts see the Introduction. For terminology see the Glossary.

Modality provides many ways to learn about the behavior of your System Under Test (SUT). At the more superficial level are commands to list the entities observed in a session. For deeper insights, Modality can calculate a variety of metrics over any region of execution, and the query language allows you to ask complex questions of a collected trace. You can also use the log command to explore your trace from any point and discover things you might not think to query for.

# Getting an Overview

# Listing Observed Entities

The simplest information you can get from a session is a list of the entities that were observed. Note that these subcommands exist on both the modality sut and modality session commands—the difference is that the modality sut command lists all entities defined for a given SUT, while the modality session command lists all entities observed in that particular session.

All modality session commands will act on the current default session, which can be set with modality session use <session name> and viewed with modality status. Alternatively, all modality session commands accept a --using <session name> flag to specify the session that the command should act on.


$ modality session probe list
NAME               TAGS                                               EVENTS
GAZEBO_SIMULATOR   [control-plane, gazebo, gazebo-plugin, simulator]  22341
PX4_BATTERY        [battery, control-plane, library, power, px4]      69794
PX4_COMMANDER      [commander, control-plane, module, px4]            60955
PX4_LAND_DETECTOR  [control-plane, land-detector, module, px4]        397
PX4_NAVIGATOR      [module, navigator, px4]                           13
PX4_SIMULATOR      [control-plane, module, px4, simulator]            46998


$ modality session event list
NAME                                      TAGS                                         COUNT
ACCEL_PREFLIGHT_CHECK                     [preflight-checks]                           23
AIRFRAME_PREFLIGHT_CHECK                  [preflight-checks]                           23
ARMDISARM_TRANSITION_ACCEPTED             [arm-disarm, commander, px4]                 3
ARMDISARM_TRANSITION_RESULT_CHECK         [arm-disarm, commander, px4]                 3
ARM_TRANSITION_PREFLIGHT_CHECK            [commander, preflight-checks, px4]           1
AUTO_DISARM_INITIATED                     [arming, commander, px4]                     1
BARO_PREFLIGHT_CHECK                      [preflight-checks]                           23
BATTERY_STATUS_CHECKED                    [battery, commander, px4]                    7634
BATTERY_WARN_INCREASED_WHILE_ARMED        [battery, commander, px4]                    7634
BATTERY_WARN_LEVEL                        [battery, commander, px4]                    7634
COMPONENT_ARM_DISARM_CHECK                [commander, px4, vehicle-command]            1
...

In the simple example above we list the probes and events observed in this session. There are similar commands to see the components, expectations, mutators, mutations, scopes, and objectives observed, as well as modality session inspect <session name> to get some summary information about a session. These commands also take a verbose flag to show increasingly more information (-v, -vv, etc...):


$ modality session probe list -v
NAME               TAGS                                               EVENTS  COMPONENT      SOURCE
GAZEBO_SIMULATOR   [control-plane, gazebo, gazebo-plugin, simulator]  22341   gz-mutator     gazebo-mutator-plugin/gz_mutator.cc#L101
PX4_BATTERY        [battery, control-plane, library, power, px4]      69794   battery        battery/battery.cpp#L142
PX4_COMMANDER      [commander, control-plane, module, px4]            60955   commander      commander/Commander.cpp#L493
PX4_LAND_DETECTOR  [control-plane, land-detector, module, px4]        397     land-detector  land_detector/LandDetector.cpp#L91
PX4_NAVIGATOR      [module, navigator, px4]                           13      navigator      navigator/navigator_main.cpp#L115
PX4_SIMULATOR      [control-plane, module, px4, simulator]            46998   simulator      simulator/simulator.h#L148

$ modality session probe list -vv
NAME               TAGS                                               EVENTS  COMPONENT      SOURCE                                    ID         COMPONENT ID
GAZEBO_SIMULATOR   [control-plane, gazebo, gazebo-plugin, simulator]  22341   gz-mutator     gazebo-mutator-plugin/gz_mutator.cc#L101  886975341  d60874c7-2b5f-4fd0-b989-ddd5ab37e4ce
PX4_BATTERY        [battery, control-plane, library, power, px4]      69794   battery        battery/battery.cpp#L142                  611298183  34da0f69-f6d8-41a9-834b-f081370f4edb
PX4_COMMANDER      [commander, control-plane, module, px4]            60955   commander      commander/Commander.cpp#L493              43287440   6236bdff-18f6-4b8c-bc80-b79c5f7816ee
PX4_LAND_DETECTOR  [control-plane, land-detector, module, px4]        397     land-detector  land_detector/LandDetector.cpp#L91        115156549  9df71df1-9b99-4fb5-aa9b-f7f7a4fc58ff
PX4_NAVIGATOR      [module, navigator, px4]                           13      navigator      navigator/navigator_main.cpp#L115         504104136  9144c5c6-3bee-4066-9979-4820f71a6b62
PX4_SIMULATOR      [control-plane, module, px4, simulator]            46998   simulator      simulator/simulator.h#L148                177602422  7aca632a-0655-40f9-b45b-89d96a694306

For details on session commands see the CLI reference.

# Metrics

Another way Modality helps you gain an understanding of what's happening in your system is through a variety of metrics. These metrics are designed to help you think about characteristics of your SUT, which could be warning signs like increasing complexity or coupling, or simply ways to better understand the behavior of your SUT like seeing its topology or which patterns are most frequent. Many metrics can be calculated at either a component, probe, or event level to give you insights at various levels of detail.


$ modality metrics complexity probe 'session="first_session"'
Cyclomatic complexity: 3.0

$ modality metrics complexity event 'session="first_session"'
Cyclomatic complexity: 43.0

$ modality metrics complexity event 'session="first_session" and scope="figure_8"'
Cyclomatic complexity: 28.0

TIP

You can calculate these metrics over any region you specify, from a part of a session to the union of multiple sessions. Some general guidelines for region expressions are:

  • A region expression must specify at least one session.
  • You can specify session, scope, component, probe, or coordinates and can join these with AND, OR, and parentheses.

For details on region expression capabilities, see the reference.


$ modality metrics coverage 'session="first_session"'
Coverage Summary
  Event: 0.3183856502242152,
  Failure: 0.15,
  Expectation state: 0.2875,
  Probes reporting: 1.0

$ modality metrics coverage 'session="first_session" OR session="second_session"'
Coverage Summary
  Event: 0.34977578475336324,
  Failure: 0.15,
  Expectation state: 0.3,
  Probes reporting: 1.0

$ modality metrics coverage 'session="first_session" AND scope="figure_8"'
  Event: 0.008968609865470852,
  Failure: 0.0,
  Expectation state: 0.0,
  Probes reporting: 0.14285714285714285

A good metric for gaining a general understanding of a region is the summary metric, which prints overview information about the given region of a trace at various levels of detail. Passing the -v flag makes it include information about instances of events and mutations that occurred:


$ modality metrics summary 'session="simple-session"' -v
Interactions: 670
Passing Expectation Instances: 626
Failure Instances: 13
Scopes Begun: 11

Sessions With Mutations: 1
Injected Mutations: 9
Active Mutations: 9
Mutation Epochs: 23
Overlapping Mutations By Session:
  Session: 2021-05-04T09-36-09Z
    Injected Mutations:
      Instance: 4
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=1886]
        Associated Failures: 3
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 8
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-45]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 6
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=70]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 7
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=1275]
        Associated Failures: 2
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 9
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=593]
        Associated Failures: 2
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 5
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=1853]
        Associated Failures: 1
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 2
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-37]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 10
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=552]
        Associated Failures: 2
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 11
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=983]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
    Active Mutations:
      Instance: 7
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=1275]
        Associated Failures: 2
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 5
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=1853]
        Associated Failures: 1
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 6
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=70]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 4
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=1886]
        Associated Failures: 3
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 8
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-45]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 9
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=593]
        Associated Failures: 2
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 11
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=983]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
      Instance: 10
        Mutator: heartbeat-delay-mutator
        Probe: CONSUMER_PROBE
        Parameters: [heartbeat-delay=552]
        Associated Failures: 2
        Distinct Failure Tags: 5
        Probe Breadth: 1
        Probe Reach: 0
        Causal Reach: 0
      Instance: 2
        Mutator: sample-data-mutator
        Probe: PRODUCER_PROBE
        Parameters: [sample=-37]
        Associated Failures: 0
        Distinct Failure Tags: 0
        Probe Breadth: 0
        Probe Reach: 0
        Causal Reach: 0
    Mutation epochs:
      [10, 5, 14, 11, 23, 22, 3, 2, 20, 18, 24, 9, 21, 8, 19, 15, 12, 6, 17, 4, 7, 16, 13]

Probes With Passing Expectations: 2
Probes With Failing Expectations: 1
Distinct Events: 15
Distinct Passing Expectations: 2
Distinct Failing Expectations: 1
Distinct Events By Component:
  consumer-component: 5
  monitor-component: 5
  producer-component: 5

Distinct Events With Payloads: 8
Distinct Payloads: 24
Event Payloads By Component:
  consumer-component
    MEASUREMENT_SAMPLE_CHECK: 3
    RECVD_MEASUREMENT_MSG: 3
    SENT_HEARTBEAT_MSG: 3
  monitor-component
    HEARTBEAT_SENDER_ID_CHECK: 3
    RECVD_HEARTBEAT_MSG: 3
  producer-component
    MEASUREMENT_SAMPLED: 3
    SENT_HEARTBEAT_MSG: 3
    SENT_MEASUREMENT_MSG: 3

Many metrics, especially those which can provide warning signs of potential future problems, are meaningful mostly in comparison. For instance, the baseline complexity of a SUT can vary enormously depending on what it's doing and a high initial complexity is not necessarily problematic. But if a change creates a significant increase in complexity, that may be an indicator that that change is more likely to introduce bugs.


$ modality metrics complexity event 'session="baseline"'
Cyclomatic complexity: 94.0

$ modality metrics complexity event 'session="release_candidate"'
Cyclomatic complexity: 214.0

The above are just a few examples of Modality's metrics. To see the full list of metrics with more information about each see the reference.

# Drilling Down

After getting an overview of what your system is doing the next step to unlock the power of Modality is through the query language. The query language allows you to interrogate your system with a fine toothed comb, finding places where specific things occurred. Results show the events that match your query along with their coordinates, which serve to pinpoint that specific occurrence of that event and can, for example, be used in the modality log command to explore the region of the trace around that instant.

The simplest query simply looks for occurrences of an event:


$ modality query 'MATCH name="TAKE_OFF_DETECTED"'
Result 1:
=========
(name = 'TAKE_OFF_DETECTED')(115156549:10:10:2, TAKE_OFF_DETECTED @ PX4_LAND_DETECTOR)

Query conditions can be joined with logical connectives like AND and OR. In the specific case of querying for an event at a specific probe you can take advantage of the @ shorthand in the second example:


$ modality query 'MATCH name="TAKE_OFF_DETECTED" AND probe="PX4_LAND_DETECTOR"'
Result 1:
=========
((name = 'TAKE_OFF_DETECTED') AND (probe = 'PX4_LAND_DETECTOR'))(115156549:10:10:2, TAKE_OFF_DETECTED @ PX4_LAND_DETECTOR)

$ modality query 'MATCH TAKE_OFF_DETECTED@PX4_LAND_DETECTOR'
Result 1:
=========
TAKE_OFF_DETECTED@PX4_LAND_DETECTOR(115156549:10:10:2, TAKE_OFF_DETECTED @ PX4_LAND_DETECTOR)

Tags are an important piece of functionality that we saw in the guide to instrumentation. They can be attached to events and allow you to add custom ways to query for events and reason about your system:


$ modality query 'MATCH tag="vehicle-command"'
Result 1:
=========
(tag = 'vehicle-command')(43287440:532:124:11, RECVD_VEHICLE_CMD @ PX4_COMMANDER, payload=400)

Result 2:
=========
(tag = 'vehicle-command')(43287440:532:124:12, HANDLING_VEHICLE_CMD @ PX4_COMMANDER, payload=400)

Result 3:
=========
(tag = 'vehicle-command')(43287440:532:124:13, HANDLE_COMMAND_COMPONENT_ARM_DISARM @ PX4_COMMANDER, payload=true)

Result 4:
=========
(tag = 'vehicle-command')(43287440:532:124:24, COMPONENT_ARM_DISARM_CHECK @ PX4_COMMANDER, outcome=PASS)

Result 5:
=========
(tag = 'vehicle-command')(43287440:533:124:35, RECVD_VEHICLE_CMD @ PX4_COMMANDER, payload=176)

Result 6:
=========
(tag = 'vehicle-command')(43287440:533:124:36, HANDLING_VEHICLE_CMD @ PX4_COMMANDER, payload=176)
...

Querying for events with conditions on the payload can help you track down the problematic instances of a common event:


$ modality query 'MATCH GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND payload > 292'
Result 1:
=========
(GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND (payload > 292))(886975341:88:648:15, GROUND_TRUTH_VEL_UP @ GAZEBO_SIMULATOR, payload=293)

Result 2:
=========
(GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND (payload > 292))(886975341:88:649:6, GROUND_TRUTH_VEL_UP @ GAZEBO_SIMULATOR, payload=294)

Result 3:
=========
(GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND (payload > 292))(886975341:88:649:15, GROUND_TRUTH_VEL_UP @ GAZEBO_SIMULATOR, payload=295)

Result 4:
=========
(GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND (payload > 292))(886975341:88:651:8, GROUND_TRUTH_VEL_UP @ GAZEBO_SIMULATOR, payload=294)

Result 5:
=========
(GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND (payload > 292))(886975341:88:651:17, GROUND_TRUTH_VEL_UP @ GAZEBO_SIMULATOR, payload=293)

The modality query command can be invoked with various levels of verbosity, starting with -v:


$ modality query 'MATCH GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND payload > 292' -v
Result 1:
=========
(GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND (payload > 292))(886975341:88:648:15, GROUND_TRUTH_VEL_UP @ GAZEBO_SIMULATOR, payload=293, tags=[gazebo, ground-truth], src="gazebo-mutator-plugin/gz_mutator.cc#L466")

Result 2:
=========
(GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND (payload > 292))(886975341:88:649:6, GROUND_TRUTH_VEL_UP @ GAZEBO_SIMULATOR, payload=294, tags=[gazebo, ground-truth], src="gazebo-mutator-plugin/gz_mutator.cc#L466")

Result 3:
=========
(GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND (payload > 292))(886975341:88:649:15, GROUND_TRUTH_VEL_UP @ GAZEBO_SIMULATOR, payload=295, tags=[gazebo, ground-truth], src="gazebo-mutator-plugin/gz_mutator.cc#L466")

Result 4:
=========
(GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND (payload > 292))(886975341:88:651:8, GROUND_TRUTH_VEL_UP @ GAZEBO_SIMULATOR, payload=294, tags=[gazebo, ground-truth], src="gazebo-mutator-plugin/gz_mutator.cc#L466")

Result 5:
=========
(GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND (payload > 292))(886975341:88:651:17, GROUND_TRUTH_VEL_UP @ GAZEBO_SIMULATOR, payload=293, tags=[gazebo, ground-truth], src="gazebo-mutator-plugin/gz_mutator.cc#L466")

Another useful query is to look for expectations. All expectations have an outcome—in this case we look for all failing expectations:


$ modality query 'MATCH outcome=FAIL'
Result 1:
=========
(outcome = FAIL)(43287440:0:0:6, ACCEL_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=FAIL)

Result 2:
=========
(outcome = FAIL)(43287440:0:0:7, GRYO_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=FAIL)

Result 3:
=========
(outcome = FAIL)(43287440:0:0:9, IMU_CONSISTENCY_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=FAIL)

Since the query language allows for complicated selection and filtering, queries can quickly become unwieldy to type at the command line. The modality query command therefore accepts a --file flag, allowing you to write your queries in a file. Writing queries in files has the added benefit of saving your queries, letting you build up a trail of breadcrumbs from a manual investigation or a library of properties to check against every system run. The query file should be a text file, its extension doesn't matter.

In this example, we look for instances where an event with a specific payload is followed by another event with a specific payload:


LandingDetected.tq
MATCH
    (MAYBE_LANDED@PX4_LAND_DETECTOR AND payload = true) AS MaybeLanded,
    (LANDED@PX4_LAND_DETECTOR AND payload = true) AS Landed
WHEN
    MaybeLanded FOLLOWED BY Landed


$ modality query --file LandingDetected.tq
Result 1:
═════════
*  MaybeLanded(115156549:0:0:5, MAYBE_LANDED @ PX4_LAND_DETECTOR, payload=true)
  
*  Landed(115156549:0:0:6, LANDED @ PX4_LAND_DETECTOR, payload=true)

Result 2:
═════════
*  MaybeLanded(115156549:1:1:4, MAYBE_LANDED @ PX4_LAND_DETECTOR, payload=true)
  
*  Landed(115156549:1:1:5, LANDED @ PX4_LAND_DETECTOR, payload=true)

Result 3:
═════════
*  MaybeLanded(115156549:2:2:4, MAYBE_LANDED @ PX4_LAND_DETECTOR, payload=true)
  
*  Landed(115156549:2:2:5, LANDED @ PX4_LAND_DETECTOR, payload=true)

This example is similar to the above but looks for sequences of 3 events:


FullLandingDetected.tq
MATCH
    (GROUND_CONTACT@MPID_LAND_DETECTOR AND payload = true) AS GroundContact,
    (MAYBE_LANDED@MPID_LAND_DETECTOR AND payload = true) AS MaybeLanded,
    (LANDED@MPID_LAND_DETECTOR AND payload = true) AS Landed
WHEN
    GroundContact FOLLOWED BY MaybeLanded
    AND MaybeLanded FOLLOWED BY Landed


Result 1:
═════════
*  GroundContact(115156549:0:0:4, GROUND_CONTACT @ PX4_LAND_DETECTOR, payload=true)
  
*  MaybeLanded(115156549:0:0:5, MAYBE_LANDED @ PX4_LAND_DETECTOR, payload=true)
  
*  Landed(115156549:0:0:6, LANDED @ PX4_LAND_DETECTOR, payload=true)

Result 2:
═════════
*  GroundContact(115156549:1:1:3, GROUND_CONTACT @ PX4_LAND_DETECTOR, payload=true)
  
*  MaybeLanded(115156549:1:1:4, MAYBE_LANDED @ PX4_LAND_DETECTOR, payload=true)
  
*  Landed(115156549:1:1:5, LANDED @ PX4_LAND_DETECTOR, payload=true)

Result 3:
═════════
*  GroundContact(115156549:2:2:3, GROUND_CONTACT @ PX4_LAND_DETECTOR, payload=true)
  
*  MaybeLanded(115156549:2:2:4, MAYBE_LANDED @ PX4_LAND_DETECTOR, payload=true)
  
*  Landed(115156549:2:2:5, LANDED @ PX4_LAND_DETECTOR, payload=true)

If you include timing information with your instrumentation you can use it to further restrict your results based on time. Note that time comparisons are only valid where all events include time and where all events come from probes with the same wall clock ID:


ArmingSuccessful.tq
MATCH
    (ARMDISARM_TRANSITION_ACCEPTED@PX4_COMMANDER AND payload = true) AS TransitionedToArmed,
    (ARMDISARM_TRANSITION_RESULT_CHECK@PX4_COMMANDER AND outcome = PASS) AS TransitionAccepted
WHEN
    TransitionedToArmed FOLLOWED BY TransitionAccepted WITHIN 100ms


$ modality query --file ArmingSuccessful.tq 
Result 1:
═════════
*  ArmingCmd(43287440:66157:135:49, HANDLE_COMMAND_COMPONENT_ARM_DISARM @ PX4_COMMANDER, payload=true)

:  (10 elided events)

*  CmdSuccess(43287440:66157:135:60, COMPONENT_ARM_DISARM_CHECK @ PX4_COMMANDER, outcome=pass)

So far we have only looked at the most common building blocks of a modality query, the MATCH and WHEN clauses. There is also an AGGREGATE clause, to calculate aggregated statistics or conditions, and a FILTER clause to filter results after they have been selected with the MATCH and WHEN conditions.

Here is a simple example that calculates some statistics about the drone's velocity during this session:

AscentRateStats.tq
MATCH
    (GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR) AS AscentRate
AGGREGATE
    min(AscentRate.payload),
    max(AscentRate.payload),
    mean(AscentRate.payload),
    stddev(AscentRate.payload)
$ modality query --file AscentRateStats.tq 
max(AscentRate.payload): 295
mean(AscentRate.payload): 65.69897
min(AscentRate.payload): -376
stddev(AscentRate.payload): 127.541725

AGGREGATE clauses can also have boolean expressions if you are just interested in seeing whether a condition held true over the session:

AscentRateSatisfactory.tq
MATCH
    (GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR) AS AscentRate
AGGREGATE
    min(AscentRate.payload) > -300,
    max(AscentRate.payload) < 300,
    mean(AscentRate.payload) > 50,
    mean(AscentRate.payload) < 100,
    stddev(AscentRate.payload) < 150
$ modality query --file AscentRateSatisfactory.tq 
(max(AscentRate.payload) < 300): true
(mean(AscentRate.payload) < 100): true
(mean(AscentRate.payload) > 50): true
(min(AscentRate.payload) > -300): false
(stddev(AscentRate.payload) < 150): true

The FILTER and AGGREGATE clauses can be combined to calculate exactly the statistics you are interested in. In this case, we calculate the same aggregate statistics as above but filter out events where the payload is 0, i.e. when the drone is just sitting on the ground:

AscentRateStats.tq
MATCH
    (GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR) AS AscentRate
FILTER
    AscentRate.payload != 0.0
AGGREGATE
    min(AscentRate.payload),
    max(AscentRate.payload),
    mean(AscentRate.payload),
    stddev(AscentRate.payload)
$ modality query --file AscentRateStats.tq 
max(AscentRate.payload): 295
mean(AscentRate.payload): 66.139244
min(AscentRate.payload): -376
stddev(AscentRate.payload): 127.854546

# Checking Conditions

Where the query command finds instances of events that match your criteria, the check command allows you to see whether conditions hold across your entire session using the query language. modality check takes a query just like modality query, but instead of returning a list of results, it returns true if there are any results that match the query and false if the query cannot be matched in your session. Thus, by writing properties of your system as queries, you can use modality check to confirm that they hold.

In this example, we check the simple condition that the drone never exceeded a velocity of 300. Note that, since we want to confirm this never happened, false is the expected result:

$ modality check 'MATCH GROUND_TRUTH_VEL_UP@GAZEBO_SIMULATOR AND payload >= 300'
false

Just like in modality query, we can also refer to expectations in modality check. Here, as a first pass at evaluating the success of our test run, we check whether the session contains any failing expectations and discover that it does:

$ modality check 'MATCH outcome=FAIL'
true

modality check also supports the --file flag for longer queries. Here, we check whether the drone successfully armed during the session:

ArmingSuccessful.tq
MATCH
    (ARMDISARM_TRANSITION_ACCEPTED@MPID_COMMANDER AND payload = true) AS TransitionedToArmed,
    (ARMDISARM_TRANSITION_RESULT_CHECK@MPID_COMMANDER AND outcome = PASS) AS TransitionAccepted
WHEN
    TransitionedToArmed FOLLOWED BY TransitionAccepted WITHIN 100ms
$ modality check --file ArmingSuccessful.tq
true

# Exploring Your Session

The query and check commands are great for when you know what you are looking for, but when you need to look through a session to find the cause of some behavior or understand what your system is doing, you will want to combine them with the log command. This command allows you to look through the events in a portion of the session that you specify.

The most typical usage is to provide a starting coordinate and a radius around that coordinate so as to limit the amount of information printed at any one time. From there you can invoke the command again with a new central coordinate to move around the session:


$ modality log --from 43287440:0:0:7 --radius 5
*    (43287440:0:0:3, SENSORS_REQ_IN_PREFLIGHT_CHECKS @ PX4_COMMANDER, payload=true)
  
*    (43287440:0:0:4, AIRFRAME_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=PASS)
  
*    (43287440:0:0:5, MAGNETOMETER_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=PASS)
  
*    (43287440:0:0:6, ACCEL_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=FAIL)
  
*    (43287440:0:0:7, GRYO_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=FAIL)  <=== central coordinate
  
*    (43287440:0:0:8, BARO_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=PASS)
  
*    (43287440:0:0:9, IMU_CONSISTENCY_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=FAIL)
  
«  PX4_COMMANDER merged a snapshot from PX4_BATTERY
  
  *  (611298183:1:0:12, RAW_VOLTAGE @ PX4_BATTERY, payload=12.150001)
  
*    (43287440:1:0:12, BATTERY_WARN_LEVEL @ PX4_COMMANDER, payload=0)  <=== new coordinate of interest
  
  *  (611298183:1:0:13, STATE_RESET @ PX4_BATTERY)
  

$ modality log --from 43287440:1:0:12 --radius 5
*    (43287440:0:0:7, GRYO_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=FAIL)
  
*    (43287440:0:0:8, BARO_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=PASS)
  
*    (43287440:0:0:9, IMU_CONSISTENCY_PREFLIGHT_CHECK @ PX4_COMMANDER, outcome=FAIL)
  
«  PX4_COMMANDER merged a snapshot from PX4_BATTERY
  
  *  (611298183:1:0:12, RAW_VOLTAGE @ PX4_BATTERY, payload=12.150001)
  
*    (43287440:1:0:12, BATTERY_WARN_LEVEL @ PX4_COMMANDER, payload=0)
  
  *  (611298183:1:0:13, STATE_RESET @ PX4_BATTERY)
  
*    (43287440:1:0:13, BATTERY_WARN_INCREASED_WHILE_ARMED @ PX4_COMMANDER, payload=false)
  
  *  (611298183:1:0:14, FILTERED_THROTTLE @ PX4_BATTERY, payload=0)
  
*    (43287440:1:0:14, MAIN_STATE_OUTCOME_FROM_BATTERY_FAILSAFE @ PX4_COMMANDER, payload=0)
  
*    (43287440:1:0:15, BATTERY_STATUS_CHECKED @ PX4_COMMANDER)
  
*    (43287440:1:0:16, ARMDISARM_TRANSITION_ACCEPTED @ PX4_COMMANDER, payload=false)
  
*    (43287440:1:0:17, ARMDISARM_TRANSITION_RESULT_CHECK @ PX4_COMMANDER, outcome=PASS)
  

The log command prints its output as an ASCII graph, making it easier to visualize what is happening in the session. Each line in the graph is a probe, and communication between probes is shown with arrows. All output pertaining to probes is color coded to make it easier to distinguish between them.

A typical workflow, then, is to use modality query to find instances of events of interest, and then plug those coordinates into modality log to explore surrounding execution and build an understanding of what happened and why.