What is Deviant?

Deviant is an intelligent, automated tool for stress-testing and identifying edge cases and latent risk in cyber-physical systems.

Deviant allows you to a perturb a system and characterize its behavior under variable, interesting conditions. It injects stresses or changes called "mutations" (e.g. timing issues, signal corruption, etc.) into a system and observe the effects using Auxon Modality and Conform.

You can use Deviant by hand, as a flexible, software-defined fault-injection framework. However, it's most powerful when used as automated, intelligent agent for proposing and performing experiments over many runs of your system, under multitudes of Deviant-proposed scenarios. When used this way, Deviant leverages reinforcement learning to automatically choose new mutations based on the system's behavior. This helps to to rapidly uncover latent issues in system design or implementation, and also to enrich the model of your systems' behavior that's being constructed in Modality.

Deviant can be applied to individual components, to subsystems, to complete integrated systems, or even large fleets and "systems of systems" all at once. Deviant and Modality's flexible infrastructure for interacting with your systems allows it to work with software components (SiL), hardware components (HiL), simulators, and even your process workflow tooling (e.g. CI, test harnesses, requirements management tools, etc.). This flexibility also enables experimentation and analysis that bridges the cause-and-effect gap between the digital and physical domains.

Deviant as an Automated Adversarial Agent

When you provide Deviant with an experiment configuration, it works in conjunction with a system test driver to automatically select and inject new mutations. Experiments offer configurable goals (e.g. breaking vs. optimizing) and define the bounds of mutations to be used. For example, you may be interested in mutations targeting one component of your system or you may want to limit the range of some parameters based on known safe limits for a hardware test rig, so you don't permanently damage expensive hardware.

Once you turn it loose, Deviant observes the system's behavior (through the execution traces collected in Modality) to see how it reacts to the active mutations. This helps Deviant decide what mutations to propose and inject in the next cycle to most effeciently progress to the experiment's goals.

The results of experments have two distinct contexts. We call them in-context and out-of-context:

• "In-context" is when experiments have been run with knowledge of your systems' desired or expected behavior as expressed through Conform specs. Deviant's results will tell you which mutator scenarios had a logical impact on your specs passing or failing, and which behavior or case branches couldn't be excercised at all.

• "Out-of-context" is when experiments have been run without knowledge of your systems' desired or expected behavior. In this case, Deviant's results are similar to a sensitivity analysis or statistical impact characterization of how your systems' observed behavior changed during the experiments.

Understanding Stress and Impact

Deviant was inspired by the tools like Finite Element Analysis, which are used by Structural and Mechanical Engineers to understand how a structure reacts to different kinds of stress. In the world of integrated software and cyber-physical systems, we need to think about 'stress' and 'reaction' in different ways.

The Stress applied by Deviant is the collection of different mutations that were injected into the system over the course of an experiment. When you're running an experiment, Deviant generates system stress in many different dimensions, based on the experiment configuration.

Impact is observed when the system's behavior changes. We see this as changes in the trace events emitted by the system under test. Deviant looks for many kinds of changes when analyzing impact, including event presence / disappearence, changes to event attributes and statistical distrubution, and statistical changes to event cadence (rate of ocurrence).

Localized Experiment Impact for a Satellite System