Modality Tutorial

In this tutorial we will take a tour of Modality's basic concepts and functionality. We will use Modality to collect data from an example system, and then explore the various ways Modality makes it possible for us to learn about our system's behavior.

# Setup

First, install Modality.

Once installed, create a user. For this tutorial we'll create a user named "tutorial". If you already have a Modality user, and a token with permissions for both query and ingest, you can skip this step

$ modality user create tutorial --use

# Example System Overview

The example system we use in this tutorial is a much simplified version of a set of interacting components you might see in an embedded system. There are three components in this system:

The producer takes measurements from the outside world via some sensor. It sends these measurement values to the consumer. In addition, the producer regularly sends heartbeat messages to the monitor.
The consumer receives the sensor measurements from the producer, and performs some work based on those measurements (in this example that work is simply a sleep whose duration is dictated by the measurement value). Just like the producer, the consumer periodically sends a heartbeat message to the monitor.
The monitor exists to make sure the other components are running correctly. It receives heartbeat messages from each of the producer and consumer, and reports an error if either component is late sending its heartbeat.

Here is a simple diagram of the example system:

# Run the System

The example system is included with the Modality client, so you should already have it installed. We'll run it for 4 seconds, to produce some trace data that is stored in Modality.

$ modality_monitored_pipeline_example 4

Running a pipeline of collaborating processes for 4 seconds. Sending traces to modality.
2022-07-20T00:36:32.927870Z  INFO monitored_pipeline::consumer: Starting up
2022-07-20T00:36:32.952271Z  INFO monitored_pipeline::producer: Starting up
2022-07-20T00:36:32.952480Z  INFO monitored_pipeline::producer: Producer sampled raw measurement sample=0
2022-07-20T00:36:32.952578Z  INFO monitored_pipeline::producer: Producer sending measurement message sample=0 destination="consumer"
2022-07-20T00:36:32.952696Z  INFO monitored_pipeline: Sending heartbeat measurement message destination="monitor"
2022-07-20T00:36:32.952968Z  INFO monitored_pipeline::consumer: Received measurement message sample=0 interaction.remote_timeline_id=c87758c7-8c0d-4ef1-a622-7c5e3a7ffabd interaction.remote_timestamp=1658277392952499123
2022-07-20T00:36:32.953580Z  INFO monitored_pipeline::monitor: Starting up
2022-07-20T00:36:32.953781Z  INFO monitored_pipeline::monitor: Received heartbeat message source="producer" interaction.remote_timeline_id=c87758c7-8c0d-4ef1-a622-7c5e3a7ffabd interaction.remote_timestamp=1658277392952604550
2022-07-20T00:36:32.954024Z  INFO monitored_pipeline::monitor: Monitor has observed a component for the first time for that component source="producer"
2022-07-20T00:36:33.116859Z  INFO monitored_pipeline::producer: Producer sampled raw measurement sample=-1
2022-07-20T00:36:33.117004Z  INFO monitored_pipeline::producer: Producer sending measurement message sample=-1 destination="consumer"
2022-07-20T00:36:33.193490Z  INFO monitored_pipeline::consumer: Received measurement message sample=-1 interaction.remote_timeline_id=c87758c7-8c0d-4ef1-a622-7c5e3a7ffabd interaction.remote_timestamp=1658277393116906800
2022-07-20T00:36:33.219735Z  INFO monitored_pipeline::producer: Producer sampled raw measurement sample=0
2022-07-20T00:36:33.219874Z  INFO monitored_pipeline::producer: Producer sending measurement message sample=0 destination="consumer"
...

Just that easily the data from this run of our system is in Modality, so it's worth recapping how it got there. This system is written in Rust. It uses tracing.rs, a popular Rust tracing framework, to collect diagnostic information. It also includes Modality's tracing.rs library, which automatically sends the trace data to the Modality daemon, modalityd.

Note

There are many different options for getting data from your system into Modality. The example system shows an easy approach for systems written in Rust. To learn about all the ways Modality lets you build a data collection setup that works for you, see Data Collection.

# Explore the Data

Modality uses workspaces to let you select the data you are interested in working with. It includes a default workspace which selects the data from the most recent run of your system. We'll use the default workspace for the rest of the tutorial, so let's select it:

$ modality workspace use default
Using 'default' as the default workspace

Now we can use the timeline and event commands to get a first look at some of our data. First we see that our system has 4 timelines, each with some basic attributes.

$ modality timeline list
Found 4 Timelines (as grouped by [timeline.name])
*@main
  # 1 timeline instances containing a total of 3 events
  # 4 attributes:
    # timeline.id, timeline.name, timeline.receive_time, timeline.run_id


*@producer
  # 1 timeline instances containing a total of 66 events
  # 4 attributes:
    # timeline.id, timeline.name, timeline.receive_time, timeline.run_id


*@monitor
  # 1 timeline instances containing a total of 17 events
  # 4 attributes:
    # timeline.id, timeline.name, timeline.receive_time, timeline.run_id


*@consumer
  # 1 timeline instances containing a total of 33 events
  # 4 attributes:
    # timeline.id, timeline.name, timeline.receive_time, timeline.run_id

Next, we'll look at the list of events from our system. This tells us various information about all the events in the trace, including their names, their attributes, and the number of times (instances) they were observed:

$ modality event list
Found 10 Events (as grouped by [event.name])
Joined thread@*
  # 3 event instances
  # 11 attributes:
    # event.component, event.name, event.severity, event.source.file,
    # event.source.line, event.source.module, event.timestamp, timeline.id,
    # timeline.name, timeline.receive_time, timeline.run_id


Starting up@*
  # 3 event instances
  # 10 attributes:
    # event.name, event.severity, event.source.file, event.source.line,
    # event.source.module, event.timestamp, timeline.id, timeline.name,
    # timeline.receive_time, timeline.run_id


Producer sampled raw measurement@*
  # 29 event instances
  # 11 attributes:
    # event.name, event.sample, event.severity, event.source.file,
    # event.source.line, event.source.module, event.timestamp, timeline.id,
    # timeline.name, timeline.receive_time, timeline.run_id


Producer sending measurement message@*
  # 29 event instances
  # 13 attributes:
    # event.destination, event.name, event.nonce, event.sample,
    # event.severity, event.source.file, event.source.line,
    # event.source.module, event.timestamp, timeline.id, timeline.name,
    # timeline.receive_time, timeline.run_id
...

Now, to get a graphical view of our system trace, we'll use modality log. This view shows an ASCII art graph of system execution on the left with event details on the right. Timelines are represented by the vertical lines, events are points on those lines, and interactions are arrows between timelines.

■             "Starting up" @ monitor   [35d3faab9351478f822a1d7b4cc88a83:4bc8]
║               name = Starting up
║               severity = info
║               source.file = tracing-modality/examples/monitored_pipeline.rs
║               source.line = 373
║               source.module = monitored_pipeline::monitor
║               timestamp = 2022-10-19 08:23:42.778083012 +00:00
║
║           ■             "Starting up" @ consumer   [fc5c3d8d977f4d768b7d5227d9031239:054b2f]
║           ║               name = Starting up
║           ║               severity = info
║           ║               source.file = tracing-modality/examples/monitored_pipeline.rs
║           ║               source.line = 304
║           ║               source.module = monitored_pipeline::consumer
║           ║               timestamp = 2022-10-19 08:23:42.778384295 +00:00
║           ║
║           ║           ■             "Starting up" @ producer   [8288fb81fb334257988fccc1fc31c090:0a445b]
║           ║           ║               name = Starting up
║           ║           ║               severity = info
║           ║           ║               source.file = tracing-modality/examples/monitored_pipeline.rs
║           ║           ║               source.line = 212
║           ║           ║               source.module = monitored_pipeline::producer
║           ║           ║               timestamp = 2022-10-19 08:23:42.778768147 +00:00
║           ║           ║
║           ║           ■             "Producer sampled raw measurement" @ producer   [8288fb81fb334257988fccc1fc31c090:0ba95e]
║           ║           ║               name = Producer sampled raw measurement
║           ║           ║               sample = -1
║           ║           ║               severity = info
║           ║           ║               source.file = tracing-modality/examples/monitored_pipeline.rs
║           ║           ║               source.line = 237
║           ║           ║               source.module = monitored_pipeline::producer
║           ║           ║               timestamp = 2022-10-19 08:23:42.778954854 +00:00
║           ║           ║
║           ║           ○             "Producer sending measurement message" @ producer   [8288fb81fb334257988fccc1fc31c090:0c4c59]
║  ╭────────────────────║               [Interaction i0000]
║  │        ║           ║               destination = consumer
║  │        ║           ║               name = Producer sending measurement message
║  │        ║           ║               nonce = -2186930594787535621
║  │        ║           ║               sample = -1
║  │        ║           ║               severity = info
║  │        ║           ║               source.file = tracing-modality/examples/monitored_pipeline.rs
║  │        ║           ║               source.line = 261
║  │        ║           ║               source.module = monitored_pipeline::producer
║  │        ║           ║               timestamp = 2022-10-19 08:23:42.779103063 +00:00
║  │        ║           ║
║  │        ║           ○             "Sending heartbeat message" @ producer   [8288fb81fb334257988fccc1fc31c090:0ceb26]
║  │  ╭─────────────────║               [Interaction i0001]
║  │  │     ║           ║               destination = monitor
║  │  │     ║           ║               name = Sending heartbeat message
║  │  │     ║           ║               nonce = -4634878031135775668
║  │  │     ║           ║               severity = info
║  │  │     ║           ║               source.file = tracing-modality/examples/monitored_pipeline.rs
║  │  │     ║           ║               source.line = 441
║  │  │     ║           ║               source.module = monitored_pipeline
║  │  │     ║           ║               timestamp = 2022-10-19 08:23:42.779286310 +00:00
║  │  │     ║           ║
○  │  │     ║           ║             "Received heartbeat message" @ monitor   [35d3faab9351478f822a1d7b4cc88a83:0da3c6]
║◀────╯     ║           ║               [Interaction i0001]
║  │        ║           ║               interaction.remote_nonce = -4634878031135775668
║  │        ║           ║               interaction.remote_timeline_id = 8288fb81-fb33-4257-988f-ccc1fc31c090
║  │        ║           ║               name = Received heartbeat message
║  │        ║           ║               severity = info
║  │        ║           ║               source = producer
║  │        ║           ║               source.file = tracing-modality/examples/monitored_pipeline.rs
║  │        ║           ║               source.line = 383
║  │        ║           ║               source.module = monitored_pipeline::monitor
║  │        ║           ║               timestamp = 2022-10-19 08:23:42.779478930 +00:00
║  │        ║           ║
■  │        ║           ║             "Monitor has observed a component for the first time for that component" @ monitor   [35d3faab9351478f822a1d7b4cc88a83:0eab60]
║  │        ║           ║               name = Monitor has observed a component for the first time for that component
║  │        ║           ║               severity = info
║  │        ║           ║               source = producer
║  │        ║           ║               source.file = tracing-modality/examples/monitored_pipeline.rs
║  │        ║           ║               source.line = 390
║  │        ║           ║               source.module = monitored_pipeline::monitor
║  │        ║           ║               timestamp = 2022-10-19 08:23:42.779755538 +00:00
║  │        ║           ║
║  │        ○           ║             "Received measurement message" @ consumer   [fc5c3d8d977f4d768b7d5227d9031239:0fa7fa]
║  ╰───────▶║           ║               [Interaction i0000]
║           ║           ║               interaction.remote_nonce = -2186930594787535621
║           ║           ║               interaction.remote_timeline_id = 8288fb81-fb33-4257-988f-ccc1fc31c090
║           ║           ║               name = Received measurement message
║           ║           ║               sample = -1
║           ║           ║               severity = info
║           ║           ║               source.file = tracing-modality/examples/monitored_pipeline.rs
║           ║           ║               source.line = 319
║           ║           ║               source.module = monitored_pipeline::consumer
║           ║           ║               timestamp = 2022-10-19 08:23:42.779906462 +00:00
║           ║           ║
...

# Drill Down With Queries

Now that we've taken a general look at our data we'll use queries to ask more specific questions about what our system was doing. Since query syntax is not the main focus of this document, explanations will be brief. To better understand queries and learn more about everything you can do with the SpeQTr language, see the SpeQTr docs. (opens new window)

Results May Vary

The example system we're using in this tutorial is not perfectly deterministic—it will produce slightly different data every run, depending on how your OS schedules the execution of the different threads. As such, when you run the queries below you are likely to see different results than are shown here. You can use this document's results as a guide for interpreting your own.

# Find a Specific Event

To start, we'll ask a universal question: were there any errors during this run? This query uses the * wildcard to match any event on any timeline, and further specifies that events should only match if they have an attribute named severity whose value is 'error'.

Note

This system uses an event attribute named severity to indicate whether an event represents an error. Note that this is not a universal approach. Modality lets you structure your data however is best for your specific use case. You could use severity to indicate errors, but you could also use the event name to indicate error, or use a different attribute. This means that, when writing queries for your own system, you need to remember to match the structure of the data you have put into Modality.

$ modality query "*@*(_.severity = 'error')"
Result 1:
═════════
■  "Detected heartbeat timeout" @ monitor  [35d3faab9351478f822a1d7b4cc88a83:649bceb0]
║    component=producer
║    severity=error
║    source.file=tracing-modality/examples/monitored_pipeline.rs
║    source.line=414
║    source.module=monitored_pipeline::monitor
║    timestamp=1666167824464463167ns
║    query.label=*@*(_.severity = 'error')
║

Result 2:
═════════
■  "Detected heartbeat timeout" @ monitor  [35d3faab9351478f822a1d7b4cc88a83:655d28a4]
║    component=producer
║    severity=error
║    source.file=tracing-modality/examples/monitored_pipeline.rs
║    source.line=414
║    source.module=monitored_pipeline::monitor
║    timestamp=1666167824477196872ns
║    query.label=*@*(_.severity = 'error')
║

Let's walk through what information is included in the query results:

The system reported 2 errors during this run.
Both errors are the same: the monitor detected a heartbeat timeout from the producer component.
On the top line, each result shows the event and timeline name, followed by event coordinates, which uniquely identify the event instance.
Next, we see event attributes. For these events, they include the relevant component and event severity, as well as information about where we can find this error event in the source code, the timestamp, and the label assigned when running the query.

So, this query informs us that there were 2 instances where the producer heartbeat triggered a timeout error. Let's continue our investigation.

# Match a Sequence of Events

The next question we'll ask is, "Show me all the pairs of consecutive heartbeats that the monitor received from the producer." This example demonstrates how easy it is to query for sequences of events. Sequences of events let you look for events in context, which is crucial to understanding your system. Rather than just "Show me every instance of X", you often need to see e.g. "every instance of X that follows an instance of Y with a payload less than 50" to zero in on problem areas.

Note

This tutorial takes a slightly roundabout approach. In a real investigation you would probably start from the results of the first query, working back from where the error occurred to see what the system was doing at that time. This tutorial is giving examples of the types of questions you can ask with Modality, so it doesn't always take the shortest route of investigation.

$ modality query "'Received heartbeat message'@monitor(_.source = 'producer') FOLLOWED BY 'Received heartbeat message'@monitor(_.source = 'producer')"

Result 1:
═════════
╔«═╝  8288fb81-fb33-4257-988f-ccc1fc31c090 interacted with monitor at 35d3faab9351478f822a1d7b4cc88a83:0da3c6
║  ║
■  ║  "Received heartbeat message" @ monitor  [35d3faab9351478f822a1d7b4cc88a83:0da3c6]
║  ║    interaction.remote_nonce=-4634878031135775668
║  ║    interaction.remote_timeline_id=8288fb81-fb33-4257-988f-ccc1fc31c090
║  ║    severity=info
║  ║    source=producer
║  ║    source.file=tracing-modality/examples/monitored_pipeline.rs
║  ║    source.line=383
║  ║    source.module=monitored_pipeline::monitor
║  ║    timestamp=1666167822779478930ns
║  ║    query.label='Received heartbeat message'@monitor(_.source = 'producer')
║  ║
╔«═╝  8288fb81-fb33-4257-988f-ccc1fc31c090 interacted with monitor at 35d3faab9351478f822a1d7b4cc88a83:20747d7d
║  ║
■  ║  "Received heartbeat message" @ monitor  [35d3faab9351478f822a1d7b4cc88a83:20747d7d]
║  ║    interaction.remote_nonce=-6025894133084304786
║  ║    interaction.remote_timeline_id=8288fb81-fb33-4257-988f-ccc1fc31c090
║  ║    severity=info
║  ║    source=producer
║  ║    source.file=tracing-modality/examples/monitored_pipeline.rs
║  ║    source.line=383
║  ║    source.module=monitored_pipeline::monitor
║  ║    timestamp=1666167823321250844ns
║  ║    query.label='Received heartbeat message'@monitor(_.source = 'producer') [2]
║  ║

Result 2:
═════════
╔«═╝  8288fb81-fb33-4257-988f-ccc1fc31c090 interacted with monitor at 35d3faab9351478f822a1d7b4cc88a83:20747d7d
║  ║
■  ║  "Received heartbeat message" @ monitor  [35d3faab9351478f822a1d7b4cc88a83:20747d7d]
║  ║    interaction.remote_nonce=-6025894133084304786
║  ║    interaction.remote_timeline_id=8288fb81-fb33-4257-988f-ccc1fc31c090
║  ║    severity=info
║  ║    source=producer
║  ║    source.file=tracing-modality/examples/monitored_pipeline.rs
║  ║    source.line=383
║  ║    source.module=monitored_pipeline::monitor
║  ║    timestamp=1666167823321250844ns
║  ║    query.label='Received heartbeat message'@monitor(_.source = 'producer')
║  ║
╔«═╝  8288fb81-fb33-4257-988f-ccc1fc31c090 interacted with monitor at 35d3faab9351478f822a1d7b4cc88a83:403df013
║  ║
■  ║  "Received heartbeat message" @ monitor  [35d3faab9351478f822a1d7b4cc88a83:403df013]
║  ║    interaction.remote_nonce=5473947973528223796
║  ║    interaction.remote_timeline_id=8288fb81-fb33-4257-988f-ccc1fc31c090
║  ║    severity=info
║  ║    source=producer
║  ║    source.file=tracing-modality/examples/monitored_pipeline.rs
║  ║    source.line=383
║  ║    source.module=monitored_pipeline::monitor
║  ║    timestamp=1666167823854899763ns
║  ║    query.label='Received heartbeat message'@monitor(_.source = 'producer') [2]
║  ║
...

Our results show us every consecutive pair of heartbeats that the monitor received from the producer. This is interesting, but it does not nail down our problem. The missing piece is a time constraint—heartbeats don't just need to be received, they need to be received within a time limit. It would be possible to look through these heartbeat pairs and compare their timestamps to find the problem areas, but that would be tedious and time consuming. Let's have Modality do it for us instead.

# Query With Time Constraints

We are now ready to write a query to find the places where an error occurred. The question we are asking here is, "Show me all the places where at least 600 milliseconds elapsed before the monitor received a heartbeat from the producer." We can write this query with a simple modification to the relationship specifier in the previous query, adding AFTER 600 ms.

Let's run the query and see where our errors occurred:

$ modality query "'Received heartbeat message'@monitor(_.source = 'producer') FOLLOWED BY AFTER 600 ms 'Received heartbeat message'@monitor(_.source = 'producer')"

Result 1:
═════════
╔«═╝  8288fb81-fb33-4257-988f-ccc1fc31c090 interacted with monitor at 35d3faab9351478f822a1d7b4cc88a83:403df013
║  ║
■  ║  "Received heartbeat message" @ monitor  [35d3faab9351478f822a1d7b4cc88a83:403df013]
║  ║    interaction.remote_nonce=5473947973528223796
║  ║    interaction.remote_timeline_id=8288fb81-fb33-4257-988f-ccc1fc31c090
║  ║    severity=info
║  ║    source=producer
║  ║    source.file=tracing-modality/examples/monitored_pipeline.rs
║  ║    source.line=383
║  ║    source.module=monitored_pipeline::monitor
║  ║    timestamp=1666167823854899763ns
║  ║    query.label='Received heartbeat message'@monitor(_.source = 'producer')
║  ║
╔«═╝  8288fb81-fb33-4257-988f-ccc1fc31c090 interacted with monitor at 35d3faab9351478f822a1d7b4cc88a83:6585ba4c
║  ║
■  ║  "Received heartbeat message" @ monitor  [35d3faab9351478f822a1d7b4cc88a83:6585ba4c]
║  ║    interaction.remote_nonce=-1546364400581164495
║  ║    interaction.remote_timeline_id=8288fb81-fb33-4257-988f-ccc1fc31c090
║  ║    severity=info
║  ║    source=producer
║  ║    source.file=tracing-modality/examples/monitored_pipeline.rs
║  ║    source.line=383
║  ║    source.module=monitored_pipeline::monitor
║  ║    timestamp=1666167824479940281ns
║  ║    query.label='Received heartbeat message'@monitor(_.source = 'producer') [2]
║  ║

Modality has pointed us to a single place where the producer heartbeat was late. You may remember, however, that the original query found two errors. Our next example will confirm the cause of the other error.

# Querying Across Timelines

So far we have only looked at sequences of events on a single timeline. Many times, however, the context you need to answer a question spans multiple different components of your system. This example shows how easy it is to write a single query for a sequence of events across timelines. The question we are asking is "Show me all the places where the producer sent a measurement message and then the consumer received it".

$ modality query "'Producer sending measurement message'@producer FOLLOWED BY 'Received measurement message'@consumer"

Result 1:
═════════
■  ║  "Producer sending measurement message" @ producer  [8288fb81fb334257988fccc1fc31c090:0c4c59]
║  ║    destination=consumer
║  ║    nonce=-2186930594787535621
║  ║    sample=-1
║  ║    severity=info
║  ║    source.file=tracing-modality/examples/monitored_pipeline.rs
║  ║    source.line=261
║  ║    source.module=monitored_pipeline::producer
║  ║    timestamp=1666167822779103063ns
║  ║    query.label='Producer sending measurement message'@producer
║  ║
╚═»╗  producer interacted with consumer at fc5c3d8d977f4d768b7d5227d9031239:0fa7fa
║  ║
║  ■  "Received measurement message" @ consumer  [fc5c3d8d977f4d768b7d5227d9031239:0fa7fa]
║  ║    interaction.remote_nonce=-2186930594787535621
║  ║    interaction.remote_timeline_id=8288fb81-fb33-4257-988f-ccc1fc31c090
║  ║    sample=-1
║  ║    severity=info
║  ║    source.file=tracing-modality/examples/monitored_pipeline.rs
║  ║    source.line=319
║  ║    source.module=monitored_pipeline::consumer
║  ║    timestamp=1666167822779906462ns
║  ║    query.label='Received measurement message'@consumer
║  ║

Result 2:
═════════
■  ║  "Producer sending measurement message" @ producer  [8288fb81fb334257988fccc1fc31c090:06b07d0e]
║  ║    destination=consumer
║  ║    nonce=-1048534873025976780
║  ║    sample=-2
║  ║    severity=info
║  ║    source.file=tracing-modality/examples/monitored_pipeline.rs
║  ║    source.line=261
║  ║    source.module=monitored_pipeline::producer
║  ║    timestamp=1666167822889337122ns
║  ║    query.label='Producer sending measurement message'@producer
║  ║
╚═»╗  producer interacted with consumer at fc5c3d8d977f4d768b7d5227d9031239:09c786ed
║  ║
║  ■  "Received measurement message" @ consumer  [fc5c3d8d977f4d768b7d5227d9031239:09c786ed]
║  ║    interaction.remote_nonce=-1048534873025976780
║  ║    interaction.remote_timeline_id=8288fb81-fb33-4257-988f-ccc1fc31c090
║  ║    sample=-2
║  ║    severity=info
║  ║    source.file=tracing-modality/examples/monitored_pipeline.rs
║  ║    source.line=319
║  ║    source.module=monitored_pipeline::consumer
║  ║    timestamp=1666167822940424169ns
║  ║    query.label='Received measurement message'@consumer
║  ║
...

# Query Aggregation

So far we have looked at using queries to find specific places in the trace that match a given pattern. Queries also have aggregation capabilities, allowing you to perform various calculations. The question we are asking is "I want to get a statistical picture of some runtime parameter reported by my system", in this case the sensor measurement taken by the producer.

This example also demonstrates running a query which is saved in a file. Since complex queries can get quite long, it is often useful to work with them in files and then use the --file option.

# aggregate-example.speqtr
'Producer sampled raw measurement'@producer AS measurement
AGGREGATE
  std_dev(measurement.sample) AS meas_stddev,
  mean(measurement.sample) AS meas_mean

$ modality query --file aggregate-example.speqtr
Aggregates:
  meas_stddev = 1.3163022337173969
  meas_mean = -2.675675675675676

# JSON Output

Since Modality can answer complex questions about your system, it's a natural fit in CI pipelines and other testing scenarios. In these cases processing and reporting results is key. As such, virtually all Modality commands have the option of formatting their output as JSON.

$ modality query --file /vagrant/aggregate-example.speqtr --format json
[{"label":"meas_stddev","value":{"Scalar":1.3163022337173969}},{"label":"meas_mean","value":{"Scalar":-2.675675675675676}}]

# Select Data of Interest

Thus far in this tutorial we have taken a tour of analysis functions after a single run of our system. We know all of our results have been related to this run, since that's the only data in Modality. Once you start collecting more data you need some way of telling Modality what you're interested in investigating. Workspaces and segmentation are the tools that let you do this.

Modality compiles all the data you ever collect into a single database, building an empirical model of your system that allows you to ask complex questions, across disparate parts and over time. Workspaces let you create a view on your Modality database that only includes the relevant data for what you're trying to accomplish.

In addition, many use cases involve some related set of data that is split into discrete chunks. Segmentation rules let you tell Modality how to split up your data. Modality will then automatically create segments in your workspace that you can analyze, individually or together.

The default workspace we're using in this tutorial automatically segments by run_id, an attribute which many Modality integrations add to your data. This automatically segments data from separate runs of your system.

# Collect More Data

To see segmentation in action we will run the example system again. This time, to keep the segment small, let's just run for one second.

$ modality_monitored_pipeline_example 1

Running a pipeline of collaborating processes for 1 seconds. Sending traces to modality.
2022-10-18T00:38:28.533174Z  INFO monitored_pipeline::monitor: Starting up
2022-10-18T00:38:28.533422Z  INFO monitored_pipeline::producer: Starting up
2022-10-18T00:38:28.533783Z  INFO monitored_pipeline::producer: Producer sampled raw measurement sample=-1
2022-10-18T00:38:28.533945Z  INFO monitored_pipeline::producer: Producer sending measurement message sample=-1 nonce=-2186930594787535621 destination="consumer"
2022-10-18T00:38:28.534131Z  INFO monitored_pipeline: Sending heartbeat message destination="monitor" nonce=-4634878031135775668
2022-10-18T00:38:28.534246Z  INFO monitored_pipeline::monitor: Received heartbeat message source="producer" interaction.remote_timeline_id=729a8cfe-315e-4472-8ae7-3c33ddf6b926 interaction.remote_nonce=-4634878031135775668
2022-10-18T00:38:28.534357Z  INFO monitored_pipeline::monitor: Monitor has observed a component for the first time for that component source="producer"
2022-10-18T00:38:28.535274Z  INFO monitored_pipeline::consumer: Starting up
2022-10-18T00:38:28.535427Z  INFO monitored_pipeline::consumer: Received measurement message sample=-1 interaction.remote_timeline_id=729a8cfe-315e-4472-8ae7-3c33ddf6b926 interaction.remote_nonce=-2186930594787535621
2022-10-18T00:38:28.635071Z  INFO monitored_pipeline::producer: Producer sampled raw measurement sample=-2
2022-10-18T00:38:28.636118Z  INFO monitored_pipeline::producer: Producer sending measurement message sample=-2 nonce=-1048534873025976780 destination="consumer"
...

# Work With New Data

First, let's confirm that we now have 2 segments as expected.

$ modality segment list
'by-run-id':
 Name: 'Run 36deea7f-71d9-45b8-87a2-2ddc26d9994d'
 Name: 'Run 5bf70ad4-8263-4509-bd45-42649b2fc460'

Since these segments are auto-generated, a UUID is used for their names. When writing your own segmentation rules you can provide a template for naming the segments.

Let's now make sure we are working with our latest run. The modality segment use command has a --latest option—this will select the most recent segment, and it will update if we collect more data.

$ modality segment use --latest
Using the latest segment as the default segment

Now, let's run our first query again and see if our results have changed.

$ modality query "*@*(_.severity = 'error')"
$

No results. It looks like, with the system only running for 1 second, we didn't see any heartbeat timeout errors. This confirms that we are only working with the data from our latest run.

Workspaces

There are many different ways that you can use workspaces. One example would be a workspace for all the data produced by some system, and other workspaces for other systems. Another could be a workspace for CI, where data is segmented by CI runs, and separate workspaces for bench testing or integration testing with different segmentation rules. Workspaces and segmentation are designed to be flexible so you can adapt them to your needs.

← Server Collect Data →