How to Integrate Modality Into C or Bare Metal Systems
# Summary
Integrating Modality into your system enables Modality's observability and mutation features in your system. In this guide we will walk through steps to do the following:
Integration Steps
- Add Modality to your build, linking its libraries and using its code-scanning functions.
- Add a probe instrumentation macro, the hook for Modality's instrumentation.
- Call functions to record communication between components in your system.
- Add event instrumentation macros to record what your system is doing.
- Optionally, add mutators to create fault injection points. Mutators require the following steps:
- Minor additional build steps.
- Add mutator instrumentation macros.
- Inform Modality of mutators in your system.
- Declare connections for Modality to communicate with your system.
- Send data to Modality however works for your system.
- Handle control messages from Modality.
- Create a SUT in the CLI.
At the end of this guide you will be ready to collect data from your running system and then analyze it.
# Supported Platforms
This guide provides general integration steps for any system written in C, using the C Instrumentation SDK.
The full list of integration guides includes platform-specific Modality integrations that provide some automatic instrumentation, such as for FreeRTOS and RTIC.
# Integrate with Your Build
To use Modality's Instrumentation SDK you must adjust your build. Modality's CMake Integrations assist with CMake projects.
The
ModalityProbe
config package contains helpers to run Modality's manifest and code generation tools. Load it in your CMakeLists.txt to take advantage of those functions.Link with the modality probe libraries.
LibModalityProbe
provides the bulk of Modality's instrumentation functionality, whileLibModalityProbeSupport
adds utilities to make processing control messages easier.Call CMake functions to run Modality's manifest and header generation tools as part of your build.
- Modality lets you split your system into components — logical groupings of system functionality that you can define however you'd like.
- Call
modality_probe_generate_manifest
for each component in your system, specifying the directory containing source files for that component and where to output the generated manifests. - Call
modality_probe_generate_header
for each generated component directory to create the required header file for that component. You will need to#include
this header everywhere you have instrumentation for that component. - Note that these tools will save the optional tags and descriptions from your instrumentation as system info in Modality but will remove them from your binary.
You're now ready to use Modality's instrumentation macros. Simply #include
<modality/probe.h>
and the appropriate component header in files where you want to collect events. When you build, the tools will scan your source for new Modality instrumentation and update generated files as appropriate.
Example: CMakeLists.txt with Modality
cmake_minimum_required(VERSION 3.5)
project(cmake-example)
# 1. Load ModalityProbe config package
find_package(ModalityProbe REQUIRED)
add_executable(cmake-example main.c)
# 2. Link Modality's libraries
target_link_libraries(
cmake-example
PRIVATE
# Required
ModalityProbe::LibModalityProbe
# Optional: needed if using control message convenience utilities from the Modality probe support library
ModalityProbe::LibModalityProbeSupport)
# 3. Call manifest and header generation functions
modality_probe_generate_manifest(
TARGET cmake-example
DEPENDS main.c
COMPONENT_NAME "my-component"
OUTPUT_PATH "my-component"
EXCLUDES "build"
FILE_EXTENSIONS "c"
SOURCE_PATH ".")
modality_probe_generate_header(
TARGET cmake-example
OUTPUT_FILE "generated_component_definitions.h"
COMPONENT_PATH "my-component")
# How to Add a Probe
Probes are the hooks that all of Modality's instrumentation hangs from. You must initialize a probe before any other Modality functionality can work, from a simple event to an advanced generative test suite.
# 1. Insert a probe macro
Select a thread of system execution that you plan to instrument with events and/or mutators.
Make sure that you only use each Modality probe on a single thread. Modality probes are not thread-safe.
Initialize a Modality probe with the
MODALITY_PROBE_INIT
macro.
dest uint8_t* |
---|
The storage buffer you must allocate for the probe. Probes use this storage as a ring buffer, overwriting the oldest data if it gets full. |
dest_size size_t |
---|
The size of the probe's buffer. You should balance your resource constraints with the frequency and expected types of your event reports. |
probe_id uint32_t |
---|
Unique ID for this probe. We strongly recommend providing an identifier for this field, which the manifest generation tools will recognize and generate a uint32_t ID for. This then provides a token that identifies the probe's unique name, which is listed in command line descriptions of your system and used in many user-written queries. |
time_res uint32_t |
---|
Clock resolution lets Modality warn you if time-based queries are unsatisfiable. |
Use the macro MODALITY_PROBE_TIME_RESOLUTION_UNSPECIFIED if a wall clock's resolution is unknown. |
wc_id uint16_t |
---|
Wall clock ID is used to allow time-based queries across different probes. |
Each probe should use a single clock for all time-related measurements. |
If this probe does not use wall clock time or does not share a wall clock with any other probe, use MODALITY_PROBE_WALL_CLOCK_ID_LOCAL_ONLY . |
next_sid_fn modality_probe_next_sequence_id_fn |
---|
Can be used to handle probe restarts during a collection session. If you don't need to handle system restarts during a single session, set this field to NULL . See the reference for details. |
next_sid_state void* |
---|
Can be used to handle probe restarts during a collection session. If you don't need to handle system restarts during a single session, set this field to NULL . See the reference for details. |
probe modality_probe** |
---|
The pointer to the newly-initialized probe instance. This output parameter must be available everywhere you will record events or attach mutators. |
# Optional Parameters
MODALITY_TAGS(“<tag>”[,”<tag>”]) Optional macro |
---|
No-op macro. Use MODALITY_TAGS to add any number of strings you want to associate with this probe. Used in investigative queries and system requirements. Modality's manifest generation tool scans for this value—it will never be included in your binary. |
description Optional string |
---|
This string helps to identify a probe inside the Modality CLI. Modality's manifest generation tool scans for this value—it will never be included in your binary. |
Example: Probe Macro
#include <modality/probe.h>
/* The header file you generated for this component with 'modality_probe_generate_header' */
#include "generated_component_definitions.h"
modality_probe *g_producer_probe = MODALITY_PROBE_NULL_INITIALIZER;
/* Example Modality probe initialization */
MODALITY_PROBE_INIT(
&g_producer_probe_buffer[0], // probe storage buffer
sizeof(g_producer_probe_buffer), // size of probe buffer
PRODUCER_PROBE, // probe name
MODALITY_PROBE_TIME_RESOLUTION_UNSPECIFIED, // clock resolution
MODALITY_PROBE_WALL_CLOCK_ID_LOCAL_ONLY, // wall clock ID
NULL, // next_sid_fn
NULL, // next_sid_state
&g_producer_probe, // output parameter,
// points to initialized probe
MODALITY_TAGS("c-example", "measurement", "producer"), // optional tags
// associated with this probe
"Measurement producer probe"); // optional probe description
# 2. Send snapshots between threads
Snapshots establish links between threads of execution for Modality's internal processing. Call functions to produce, send, and merge snapshots when threads communicate:
For every time a thread is going to communicate with any other thread, produce a snapshot by using
modality_probe_produce_snapshot
.The pointer to probe (sender) parameter should point to the probe on the thread that is going to send a message.
The produced snapshot should be included in the message sent to the other thread.
For every time a message is received from another thread, use
modality_probe_merge_snapshot
to merge the snapshot included in the message.Snapshots must be merged before any other message handling occurs.
The pointer to probe (receiver) parameter should point to the probe on the thread that is receiving the message.
If you are sending binary blobs you can use
modality_probe_produce_snapshot_bytes
to produce a snapshot as a byte buffer, andmodality_probe_merge_snapshot_bytes
to merge such a snapshot.
Example: Snapshot Producer Function
#include <modality/probe.h>
modality_probe_causal_snapshot snapshot;
err = modality_probe_produce_snapshot(
g_producer_probe, // pointer to probe (sender)
&snapshot); // pointer to produced snapshot
assert(err == MODALITY_PROBE_ERROR_OK);
Example: Snapshot Merge Function
#include <modality/probe.h>
err = modality_probe_merge_snapshot(
g_consumer_probe, // pointer to probe (receiver)
&message.snapshot); // pointer to received snapshot
assert(err == MODALITY_PROBE_ERROR_OK);
# Your probe is complete
Events added to this thread should point to this probe.
Mutators and mutator announcements added to this thread should point to this probe.
# How to Add an Event
Events record the fact that something happened in logical time, along with a payload and/or wall clock time. From this foundation, you can observe your system's behavior at any level of detail.
# Insert an event macro
Pick something that would be interesting to know about what your system is doing. Based on the type of system behavior that you want to observe, select one of the event macros below and insert it into your system code.
probe modality_probe* |
---|
Pointer to the Modality probe for the event's thread. |
event_id uint32_t |
---|
Unique ID for this event. We strongly recommend providing an identifier for this field, which the manifest generation tools will recognize and generate a uint32_t ID for. This then provides a token that identifies the event's name. Event names are listed in command line descriptions of your system, and are used in many user-written queries. |
... varies |
---|
Event macros may have other mandatory parameters specified in their descriptions, like payload, timestamp, and failure condition. |
MODALITY_TAGS(“<tag>”[,”<tag>”]) Optional macro |
---|
No-op macro. Use MODALITY_TAGS to add any number of strings you want to associate with an event. Used in investigative queries and system requirements. Modality's manifest generation tool scans for this value—it will never be included in your binary. |
description Optional string |
---|
This string helps to identify an event inside the Modality CLI. Modality's manifest generation tool scans for this value—it will never be included in your binary. |
Example: Event macro
#include <modality/probe.h>
/* The header file you generated for this component with 'modality_probe_generate_header' */
#include "generated_component_definitions.h"
/* Example basic event */
MODALITY_PROBE_RECORD(g_producer_probe, PRODUCER_STARTED);
# Event Macro: Basic Event
- If you simply want to record that something has happened in logical time, use the most basic type of event macro.
See the Instrumentation SDK for a complete list of event macros.
Example: Basic Event
#include <modality/probe.h>
/* The header file you generated for this component with 'modality_probe_generate_header' */
#include "generated_component_definitions.h"
MODALITY_PROBE_RECORD(
g_producer_probe, // pointer to probe
PRODUCER_STARTED, // event name
MODALITY_TAGS("producer"), // optional event tags
"Measurement producer started"); // optional event description
# Event Macro: Event with Payload
If you want to record some piece of system state, use an event macro with an associated payload.
The payload of an event is always 4 bytes, but Modality provides convenience macros for many different types of payloads.
Example: Event with Payload
#include <modality/probe.h>
/* The header file you generated for this component with 'modality_probe_generate_header' */
#include "generated_component_definitions.h"
const int8_t sample = g_producer_measurement.m + (int8_t) (-1 + (rand() % 4));
MODALITY_PROBE_RECORD_W_I8(
g_producer_probe, // pointer to probe
PRODUCER_MEASUREMENT_SAMPLED, // event name
sample, // payload
MODALITY_TAGS("producer", "measurement-sample"), // optional event tags
"Measurement producer sampled a value for transmission"); // optional event description
# Event Macro: Event with Time
If you want to be able to query an event in terms of wall clock time, use a macro to record time.
If you want to query multiple events in terms of wall clock time, all of the queried events need to record time.
For any given time-based query, all the queried events need to point to probes with the same wall clock ID.
If the wall clock ID is
MODALITY_PROBE_WALL_CLOCK_ID_LOCAL_ONLY
for an event's probe, then time-based queries on that event can only include events from that same probe.The timestamp (
time_ns
in this example) is limited to 61 bits of nanoseconds.
Example: Event with Time
#include <modality/probe.h>
/* The header file you generated for this component with 'modality_probe_generate_header' */
#include "generated_component_definitions.h"
uint64_t time_ns = 1;
MODALITY_PROBE_RECORD_W_TIME(
g_producer_probe, // pointer to probe
PRODUCER_STARTED, // event name
time_ns, // timestamp
MODALITY_TAGS("producer"), // optional event tags
"Measurement producer started"); // optional event description
# Event Macro: Event with Payload and Time
If you want to record an event with both a piece of state and wall clock time, use one of the macros which combines a payload and time.
As an event with payload, the features of the event with payload macro apply here as well.
As an event with time, the features of the event with time macro apply here as well.
Example: Event with Payload and Time
#include <modality/probe.h>
/* The header file you generated for this component with 'modality_probe_generate_header' */
#include "generated_component_definitions.h"
uint64_t time_ns = 1;
const int8_t sample = g_producer_measurement.m + (int8_t) (-1 + (rand() % 4));
MODALITY_PROBE_RECORD_W_I8_W_TIME(
g_producer_probe, // pointer to probe
PRODUCER_MEASUREMENT_SAMPLED, // event name
sample, // payload
time_ns, // timestamp
MODALITY_TAGS("producer", "measurement sample"), // optional event tags
"Measurement producer sampled a value for transmission"); // optional event description
# Event Macro: Expectation Event
If you want to capture a conditional failure, use a macro for an expectation.
Expectations make it easier to highlight critical events. They are taken into consideration by certain Modality metrics and by Modality's generative testing mutation planner.
Expectations record a boolean
outcome
indicating success (PASS
) or failure (FAIL
).For the failure condition, which is
(sample - g_producer_measurement.m) <= 2
in this example, true represents success and false represents failure, so you can think of this condition as an invariant of your system: something that should always be true.If you want to associate a severity level with an expectation or failure event, insert the optional
MODALITY_SEVERITY()
macro with an integer value between 1 and 10. Severity lets you prioritize potential failures of varying importance.
Example: Expectation Event
#include <modality/probe.h>
/* The header file you generated for this component with 'modality_probe_generate_header' */
#include "generated_component_definitions.h"
MODALITY_PROBE_EXPECT(
g_producer_probe, // pointer to probe
PRODUCER_SAMPLE_DELTA_OK, // expectation name
(sample - g_producer_measurement.m) <= 2, // failure condition
MODALITY_SEVERITY(6), // optional expectation severity
MODALITY_TAGS("producer"), // optional expectation tags
"Measurement delta within ok range"); // optional expectation description
# Event Macro: Expectation with Time
If you want to capture a conditional failure along with its wall clock time, use a macro for an expectation with time.
As an expectation, the features of the expectation macro apply here as well.
As an event with time, the features of the event with time macro apply here as well.
Example: Expectation Event with Time
#include <modality/probe.h>
/* The header file you generated for this component with 'modality_probe_generate_header' */
#include "generated_component_definitions.h"
MODALITY_PROBE_EXPECT_W_TIME(
g_producer_probe, // pointer to probe
PRODUCER_SAMPLE_DELTA_OK, // expectation name
(sample - g_producer_measurement.m) <= 2, // failure condition
time_ns, // timestamp
MODALITY_SEVERITY(6), // optional expectation severity
MODALITY_TAGS("producer"), // optional expectation tags
"Measurement delta within ok range"); // optional expectation description
# Event Macro: Failure Event
If you want to catch a specific point in execution that always indicates a failure, record an event using the failure macro.
A recorded failure event has the same implications on Modality metrics as an expectation with an
outcome
ofFAIL
.Failure events make it easier to instrument unambiguous problems in your system.
Example: Failure Event
#include <modality/probe.h>
/* The header file you generated for this component with 'modality_probe_generate_header' */
#include "generated_component_definitions.h"
MODALITY_PROBE_FAILURE(
g_producer_probe, // pointer to probe
BAD_THING_HAPPENED, // failure event name
MODALITY_SEVERITY(10), // optional failure event severity
MODALITY_TAGS("problem"), // optional failure event tags
"A bad thing happened"); // optional failure event description
# Event Macro: Failure Event with Time
If you want to catch a specific point that always indicates a failure along with the time that event happens, record an event using the failure with time macro.
A recorded failure event has the same implications on Modality metrics as an expectation with an
outcome
ofFAIL
.As an event with time, the features of the event with time macro apply here as well.
Example: Failure Event with Time
#include <modality/probe.h>
/* The header file you generated for this component with 'modality_probe_generate_header' */
#include "generated_component_definitions.h"
MODALITY_PROBE_FAILURE_W_TIME(
g_producer_probe, // pointer to probe
BAD_THING_HAPPENED, // failure event name
time_ns, // timestamp
MODALITY_SEVERITY(10), // optional failure event severity
MODALITY_TAGS("problem"), // optional failure event tags
"A bad thing happened"); // optional failure event description
# How to Add a Mutator Pro
Mutators allow you to define pieces of state that can be manipulated. Mutators are essential to many Modality use cases, such as finding undiscovered issues or reproducing bugs.
Pro Mutation is a feature of the full version of Modality.
# 1. Adjust your build for mutators
Modality's mutator functionality requires a few additional changes to your build process.
Add the generated mutator
.c
files to your build.Add generated mutator header files to your include path.
Call
modality_probe_generate_mutators
to scan for mutator macros and generate headers and source files accordingly. This CMake function comes in theModalityProbe
config package.
Example: CMakeLists.txt with Modality Mutators
cmake_minimum_required(VERSION 3.5)
project(cmake-example)
find_package(ModalityProbe REQUIRED)
add_executable(
cmake-example
main.c
# 1. Add generated mutator .c files to your build
generated_mutators/sample_data_mutator.c)
target_link_libraries(...)
target_include_directories(
cmake-example
PRIVATE
# 2. Add generated mutator header files to your include path
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/generated_mutators>)
modality_probe_generate_manifest(...)
modality_probe_generate_header(...)
# 3. Call modality_probe_generate_mutators to scan for mutator macros and
# generate files accordingly
modality_probe_generate_mutators(
TARGET cmake-example
DEPENDS main.c
FILE_EXTENSIONS "c"
COMPONENT_PATH "my-component"
OUTPUT_PATH "generated_mutators"
EXCLUDES "build"
SOURCE_PATH ".")
# 2. Insert a mutator macro
Identify a piece of system state that you want Modality to be able to change. Mutators can manipulate any parameter in your system.
Use the MODALITY_MUTATOR
macro to declare a mutator that can change a system variable. You must #include <modality/mutator.h>
for access to the macro, and also #include <your-generated-mutator.h>
in the file where you invoke MODALITY_MUTATOR
for mutator functionality to work.
probe modality_probe* |
---|
Pointer to the Modality probe for that mutator's thread. |
mutator_name identifier |
---|
Identifier for the mutator's name, used in objectives and the CLI. |
param_name identifier |
---|
Identifier for the system parameter that this mutator changes for use in Modality's CLI. |
param_def macro |
---|
Use the MODALITY_PARAM_DEF macro to define the system parameter that this mutator will be able to change. |
MODALITY_PARAM_DEF specifies parameter type, minimum effect value, and ranges of allowable values for that parameter. |
data_ptr void* |
---|
Pointer to the variable in the original system code that may be mutated. |
data_size size_t |
---|
The mutable system variable's size, in bytes. |
MODALITY_TAGS(“<tag>”[,”<tag>”]) Optional macro |
---|
No-op macro. Use MODALITY_TAGS to add any number of strings you want to associate with a mutator. Used in queries, mutation constraints, and system requirements. Modality's mutator generation tool scans for this value—it will never be included in your binary. |
Example: Mutator Macro
#include <modality/probe.h>
#include <modality/mutator.h>
/* The header file you generated with 'modality-probe mutator-gen' */
#include "battery_mutator.h"
/* Example mutator:
Generates a mutator over the filtered voltage parameter */
MODALITY_MUTATOR(
_probe, // pointer to probe
BATTERY_MUTATOR, // mutator name
voltage, // mutation parameter name in Modality
MODALITY_PARAM_DEF( // parameter definition
F32, // parameter type
12.15f, // minimum effect value
NOMINAL(11.5f, 12.3f), // nominal range
SAFETY(6.0f, 12.3f), // safety range
HARD(2.1f, 12.3f)), // hard range
&_voltage_filtered_v, // system variable to mutate
sizeof(_voltage_filtered_v), // size of system variable
MODALITY_TAGS("px4", "battery", "power")); // optional mutator tags
# 3. Send mutator announcements
Add a function to produce announcement data and send that announcement to the Modality daemon.
Each mutator must be announced to the Modality daemon at least once per data-recording session.
- Mutators cannot be used until they are announced, so it is generally best to announce all mutators during system setup.
Use the
modality_probe_announce_mutators
function to produce announcement data for all the mutators on a given probe.Deliver the announcement data to one of the
collector_connections
specified in your Modality configuration file.- Modality gracefully handles receiving duplicate mutator announcements, so you can send announcement data repeatedly if necessary.
Note: Handling Control Messages
The probe for a mutator's thread also needs to handle control messages from the Modality daemon, which is covered below in How to Communicate with Modality.
Example: Mutator Announcement Function
#include <modality/probe.h>
/* Announcement sent to socket Modality daemon is listening on */
size_t announcement_size;
err = modality_probe_announce_mutators(
_probe, // pointer to probe
&_announcement_buffer[0], // announcement destination buffer
sizeof(_announcement_buffer), // destination buffer size
&announcement_size); // output announcement size
assert(err == MODALITY_PROBE_ERROR_OK);
if(announcement_size != 0)
{
const ssize_t sendto_status = sendto(
_report_socket,
&_announcement_buffer[0],
announcement_size,
0,
(const struct sockaddr*) &_collector_addr,
sizeof(_collector_addr));
assert(sendto_status != -1);
}
# How to Communicate with Modality
To report events and trigger mutations, your system needs to handle communications to and from the Modality daemon.
# 1. Create a Modality configuration file
Your Modality configuration file specifies connections between your system under test and the Modality daemon. It also handles user and license information.
Create a TOML file named
Modality.toml
.collector_connections
lists the connections the Modality daemon will listen on for your system to send probe reports and mutator announcements.control_connections
lists the destinations to which the Modality daemon sends control messages for your system to handle.
See the Modality configuration file reference for more details on configuring Modality.
Example: Modality Configuration File
# Modality.toml
collector_connections = [
"udp://192.168.122.76:34333",
"http://192.168.122.76:34444"
]
control_connections = [
"udp://192.168.122.76:34555",
"http://regular-http-address.com/path"
]
# 2. Send reports to Modality
Add functions to regularly create reports from each probe and send those reports to Modality.
Use the
modality_probe_report
function to generate a report from all the events in a probe's buffer, and then clear that buffer.You must call
modality_probe_report
regularly enough to clear the probe buffer before it fills up. If a probe buffer fills up, it will begin overwriting the oldest data.- After a session has been recorded, you can call
modality session probe reporting-details
to see information about reports and whether events were overwritten.
- After a session has been recorded, you can call
Once created, reports need to be delivered to the Modality daemon for future use. Depending on your system resources and the volume of events generated, you may:
With network access, simply send reports to a socket that the Modality daemon is listening on, as specified in
collector_connections
of your Modality configuration file.Optionally, save reports to durable storage for later use with Modality.
Optionally, batch reports from multiple probes and send them periodically.
Example: Report Creation and Delivery Over a Socket
#include <modality/probe.h>
static void send_report(modality_probe * const probe)
{
size_t report_size;
const size_t err = modality_probe_report(
probe, // probe to generate report from
&g_report_buffer[0], // report buffer
sizeof(g_report_buffer), // size of report buffer
&report_size); // size of created report
assert(err == MODALITY_PROBE_ERROR_OK);
if(report_size != 0)
{
const ssize_t status = sendto(
g_report_socket,
&g_report_buffer[0],
report_size,
0,
(const struct sockaddr*) &g_collector_addr,
sizeof(g_collector_addr));
assert(status != -1);
}
}
# 3. Handle control messages from Modality
Add handling for control messages that arrive from the Modality daemon into your system. Control messages enable mutations and scopes.
Listen on the
control_connections
you specify in your Modality configuration file to process control messages.Control messages can be targeted to a specific probe or broadcast to all probes. How you handle incoming control messages depends on your system architecture and constraints.
If you want to have a central point listen and then route control messages, use the
modality_probe_get_control_message_destination
function to get a message's destination.If you want to have each probe listen on its own connection, use the
modality_probe_process_control_message
function for each probe. It will ignore a message if it is not meant for this probe and indicate that the message should be forwarded.Probes are single threaded, so make sure to call
modality_probe_process_control_message
on the same thread that the probe is logging events on.
Example: Control Message Handling
#include <modality/probe.h>
/* Status function called periodically */
void Battery::updateBatteryStatus(
hrt_abstime timestamp,
float voltage_v,
float current_a,
bool connected,
modality_probe *probe)
{
uint8_t *bytes;
/* Check for new control messages */
size_t length = get_control_message(&bytes);
if(length != 0)
{
size_t should_forward;
size_t err = modality_probe_process_control_message(
probe,
bytes,
length,
&should_forward);
assert(err == MODALITY_PROBE_ERROR_OK);
}
...
# How to Create a Modality SUT
To complete instrumentation, write a System Under Test (SUT) definition file and then create your SUT in Modality.
# 1. Check generated files
If you added Modality's CMake functions for observability and mutation to your build you should have manifest files already generated for all of your instrumentation. If you are not using CMake or prefer not to run the code and manifest generation commands as part of your build, you can use the CLI to manually generate the required files.
# 2. Create a System Under Test definition file
- Write a
SUT.toml
definition file to collect all of the components that compose your overall System Under Test (SUT). - Your SUT requires a
name
. - Optionally, use an array of
tags
to help identify your system. - Use an array of
component_paths
to all of the directories containing the component manifest files. These are the directories created bymodality_probe_generate_manifest()
.
Example: System Under Test (SUT) Definition File
# SUT.toml
name = "drone"
tags = ["drone", "simulator"]
component_paths = [
"./src/lib/battery/battery-component",
"./src/modules/commander/commander-component",
"./src/modules/land_detector/land-detector-component",
"./src/modules/simulator/simulator-component",
"./src/modules/navigator/navigator-component",
"./gazebo-mutator-plugin/gz-mutator-component"
]
# 3. Create and confirm your System Under Test (SUT) in Modality
Call
modality sut create <directory containing SUT.toml>
for Modality to create a system under test from your SUT definition file.Call
modality sut list
to confirm that your system under test was created, thenmodality use <system-under-test-name>
to set it as the default SUT.Call
modality sut component list
to check your SUT's components, then callmodality sut probe list
to check your SUT's probes.
$ modality sut list
NAME COMPONENTS SESSIONS
drone 6 1
$ modality sut use drone
$ modality sut component list
NAME TAGS PROBES EVENTS
battery [battery, px4] 1 10
commander [commander, px4] 1 76
gz-mutator [simulator, px4] 1 6
land-detector [land-detector, px4] 1 5
navigator [navigator, px4] 1 5
simulator [simulator, px4] 1 1
$ modality sut probe list
NAME TAGS
GAZEBO_SIMULATOR [control-plane, gazebo, gazebo-plugin, simulator]
PX4_BATTERY [battery, control-plane, library, power, px4]
PX4_COMMANDER [commander, control-plane, module, px4]
PX4_LAND_DETECTOR [control-plane, land-detector, module, px4]
PX4_NAVIGATOR [module, navigator, px4]
PX4_SIMULATOR [control-plane, module, px4, simulator]
# Next Steps
Your system is now instrumented for Modality—congratulations!
Your testing and development teams can now collect session data from your system in action, and then analyze sessions separately.
If you've robustly instrumented your system with both events and mutators, you can now take advantage of Modality's showcase use cases, such as finding undiscovered issues, reproducing a bug with generative testing, and automatically validating system requirements.
Note - This Guide is Linear, Modality is Flexible
This guide shows just one possible approach to instrumenting Modality. In practice, you have control over every element of instrumentation.
While this guide uses C macros, Modality is architecture-agnostic.
While this guide assumes zero advance work, Modality instrumentation iterates quickly. Once you've set up probes and communication with Modality, adding additional events and mutators is straightforward.
While this guide advocates for fully instrumenting your system, teams can start collecting session data after setting up one probe with one event.
If you see a constraint relative to your own system, there's an excellent chance Modality can flex to fit your needs.
See the Instrumentation SDK for extra macros, and Using Modality for inspiration.
Better yet, contact us with any questions (opens new window).