Conform Concepts
TIP
Conform is built on top of Auxon's Modality database. This document builds on the concepts described in the Modality documentation. (opens new window)
# Executable Specifications
Most specification documents are dead the day after they are written. They are read once (if you're lucky), and never updated. Determining whether the system you built actually matches the spec is often not even considered.
Conform allows you to execute your specifications, which are written in the SpeQTr language. This has huge advantages:
The specification is a substantial part of the system test suite.
There are clear reasons to keep the spec updated, so it actually happens.
You can automatically determine if your system meets the spec, as frequently as you want to run your tests. You can use this to track progress towards implementation of the spec.
SpeQTr is specifically designed to be familiar to anyone with experience writing specifications. Its behaviors follow a familiar "happy path / recovery cases" format which is very amenable to incremental development of the specifications. For example, a systems engineer might start by specifying the nominal behavior of the system, while a QA team or functional safety specialist can later add detail to the expected system behavior under various failure scenarios.
SpeQTr also supports machine readable annotations (like Doxygen or Javadoc). This is an extensible place to keep any kind of workflow-specific metadata you might need: information about a responsible engineer, IDs from your requirements tracking system, or anything else.
# Trace-based Testing
Writing and maintaining automated system integration tests is well known to be a slog. You need to orchestrate the system well enough to get it into the right initial state, then inject the right bit of stimulus at the right place (and time), wait for the system to do its thing, and then somehow figure out if the system did what it was supposed to. All of these usually require access to non-standard APIs or interfaces. If you manage to get such a test suite up and going, it is very fragile and expensive to maintain. Many teams don't even try.
Conform implements trace-based testing, which solves these problems. It decouples exercising your system from verifying its behavior. Instead of constructing a test harness to exercise the system and check its behavior, you instead only need to exercise the system. The system is instrumented to report what it's doing as the scenario executes as trace data recorded in Modality (typically using a platform-provided tracing or logging library you're probably already using). Then, you use Conform to check that, whatever the system did, it followed what's written in the spec.
Pulling integration tests apart into these two pieces is way better:
You can iterate on your specs (assertions) independently of your execution script. This alone can save weeks of development time.
You can run new tests against previously recorded executions. Found a new bug? Write a spec to check for it, then run it over your historical data to see when it was introduced.
The tests themselves are far less flaky. The error-prone back-and-forth between your test harness and system under test is gone. Instead, you can use the conditional matching features of SpeQTr to match specifications to their preconditions, whenever they happen.
Instead of writing assertions that work only in the context of a test suite, you're writing an executable SpeQTr specification. It's not harder to write (it's easier in most cases), but you can use it in a lot more cases. For example, you can collect data from a field test, or from live systems if you can run them tracing enabled, and run your specs against the collected traces.
When normal integration tests fail, figuring out what went wrong is a chore. Conform can do a lot of this automatically, based on the rich traces stored in Modality combined with intelligent error reporting. When you need to do manual analysis, you have a huge amount of context available in Modality to understand what was happening around the failure.
# Behaviors
When you're writing a SpeQTr specification for Conform, you're writing down a bunch of system behaviors. These represent something that your system should (or shouldn't) do. Conform evaluates each behavior against your recorded system executions, validating that the system behaved as specified, or telling you where it did not.
# Triggered Behaviors
Most behaviors are "triggered": they look for an occurrence of an event in the recorded trace as a trigger, and then assert that it must be followed by a given event pattern. For example, you might have a trigger of "begin update process", and look for the pattern of events that signify the successful update process.
When you specify the trigger condition for your behavior (in a when
block), you can choose to specify both a start and an end trigger
condition, or just a start. If you have both a start and end, that
defines a window of events which Conform will look at for verifying
the body of the behavior. If you have just the start, the window
between one occurrence of the trigger and the next one is used.
# Global Behaviors
Other behaviors simply specify something that should always occur (perhaps every trace should contain an event recording the software version), or should never occur (lp0 should never catch on fire).
# Nominal and Recovery Cases
Within any behavior, you can specify a number of cases. Each of these is a pattern of behavior that you expect the system to be able to take, in that situation. These are separated into nominal and recovery cases.
The idea of this separation is to be able to specify the normal behavior of the system in a nominal case, then have several recovery cases describing what the system should do when some part of the execution deviates from normal (nominal) operation. Recovery cases can refer directly to nominal cases, so it's easy to write a behavior as a single nominal case with a number of recovery cases that work as off-ramps from the nominal one.
# Unspecified Behavior
When Conform is evaluating a behavior and none of the cases match, this is unspecified behavior. This is a spec failure, and Conform will tell you which part of the trace it was looking at when the failure occurred. It will also look at the trace to see if it was close to one of the specified cases, to help you more quickly diagnose what went wrong.
# SpeQTr
For more details on writing specifications with Conform, see the SpeQTr reference documentation. (opens new window)