Deviant Concepts

TIP

Deviant is built on top of Auxon's Modality database and uses many of the same concepts. These are described in detail in the Modality documentation (opens new window).

# Experiments

An experiment is a plan that tells Deviant how to automatically stress test your system. Mutations generated for an experiment are all grouped together, so their results can be analyzed together as well. Every experiment has a unique name, and may optionally have extra metadata that controls the sorts of mutations suggested within that experiment.

Most experiments will specify a "mutator filter," a logical expression that specifies which mutators shall be used for this experiment. This filter is described using the SpeQTr event predicate (opens new window) syntax.

You can also choose an experiment "approach", which selects the algorithm Deviant uses when choosing mutations. There are currently three supported approaches:

  • random: Choose mutators and mutator parameters completely at random
  • structured-breadth: Use a space-filling-curve algorithm to choose mutators and mutator parameters. This starts with a rough breakdown of mutators and their parameter spaces, then iterates back over them in increasing levels of detail as you keep creating mutations.
  • learned-pertubation: Use a reinforcement learning algorithm to suggest mutators and mutator parameters based on how the system has responded to previous mutations. The goal of this method is to generate behaviors that are different from anything previously seen, so it will tend to stay away from mutators and mutator parameter choices which generate system behaviors that are similar to those already observed.
$ deviant experiment create testing \
     --mutator-filter '_.group = "foo" AND _.mycorp.safe_for_prod = true'
     --approach learned-pertubation

Having created an experiment, you can ask Deviant to create mutations for it.

$ deviant mutation create --experiment testing

Since we've used the learned-pertubation approach here, we should also tell Deviant to update its model after collection some data from the system, under mutation:

$ deviant experiment train testing --latest-segment

Continue your experiment by repeating these steps many times (hundreds or thousands of times). After you've collected enough data, you can generate an experiment impact report, from the CLI, or from the Auxon VS Code Extension:

Experiment Impact Report Button

Experiment Impact Report Outline

Experiment Impact Report Visualization

# Mutators

A mutator is a well-described mutation actuator. It is responsible for enacting mutations and making them real. Each mutator declares the bounds of mutations that it is capable of injecting.

Every mutator is defined by its metadata attributes, much like modality events (opens new window). These attributes are key-value pairs responsible for describing what kind of change to the system the mutator can cause, as well as where the mutator resides relative to the rest of the system under test.

Attribute keys are strings. Keys have a hierarchical namespace where each segment is separated by the . character. Attribute values can be of a number of types: strings, numbers, etc.

Some example mutator attribute keys and values might be:

  • mutator.name = "message bus delayer"
  • mutator.description = "Introduce delivery delay to some messages passing through the bus"
  • mutator.layer = "operational"
  • mutator.statefulness = "intermittent"
  • mutator.mycorp.build_number = 199
  • mutator.mycorp.deployment.pod.id = 45926
  • mutator.mycorp.safe_for_prod = true

If a mutator has any parameters available for its mutations, the mutator's attributes also describe the expected parameters and their data types.

For example:

  • mutator.params.delay.name = "delay"
  • mutator.params.delay.description = "the amount of time to delay messages"
  • mutator.params.delay.value_type = "Nanoseconds"
  • mutator.params.delay.value_min = 0
  • mutator.params.delay.value_max = 2500
  • mutator.params.delay.default_value = 1000
  • mutator.params.delay.least_effect_value = 0
  • mutator.params.delay.value_distribution.kind = "continuous"
  • mutator.params.delay.value_distribution.scaling = "linear"
  • mutator.params.route.name = "route"
  • mutator.params.route.description = "which bus route to target"
  • mutator.params.route.value_type = "String"
  • mutator.params.route.value_distribution.kind = "discrete"
  • mutator.params.route.value_distribution.option_set.alpha = "a"
  • mutator.params.route.value_distribution.option_set.beta = "b"
  • mutator.params.route.value_distribution.option_set.gamma = "c"

See the tutorials for examples of how to use a premade mutators or host your own custom mutator.

# Mutations

A mutation is a concrete attempt to change something about the system under test. Mutations come with zero or more parameters that define the exact changes to the system that should be made.

For example, if you wanted to see how your system performs under certain environmental conditions, you could have a mutation which changes the value reported by a temperature sensor. That mutation's parameters could be set to reflect the exact Fahrenheit/Celsius degree value you are interested in.

Mutations are created by users via Deviant, which then pushes the mutation to the appropriate mutator via its mutation-plane protocol.

The main way to create mutations is via Deviant's CLI:

$ deviant mutation create --params 'temp=42' --params 'bar="extra"' \
                          --mutator-id='ed109c30-a5e8-488e-868d-ae2800b4d628'

While mutations can be any interesting change to the system under test, it may be useful to think of them as equivalent to "faults" from a traditional fault-injection testing perspective.

Each mutation is defined in terms of a single, specific, mutator.

# Mutator Server

Deviant applies mutations by way of requests sent to a mutation server; a long-running process which provides an implementation of the Mutator HTTP API for describing and operating one or more mutators.

Mutator servers can either be directly integrated into your system, for cases where internal system access is required to perform the mutation; or exist standalone, for cases where internal system access is not needed, such as when mutations are performed via external facing API calls.

This API is then used by Auxon's modality-reflector (opens new window) relay process to connect deviant, by way of modalityd, to a given mutator server.

Mutator servers that are integrated within a system can be used by configuring a modality-reflector instance with the mutator server's HTTP API URL, which will query the server for mutators and make them available within Deviant.

Alternatively, mutator servers that are able to operate from outside a system may be structured to operate as a modality-reflector plugin for convenience. Plugins are hosted by the reflector, running as a subprocess and provided with structured configuration.

# Injecting Mutations

Deviant's most primitive operation is injecting mutations into your system.

A mutator implementation can be as simple as an off-the-shelf plugin that interacts with a standard API exposed by your system or as deeply integrated as custom mutator built especially for performing a given mutation. The full range of mutations that are available to use on your system is determined by the mutators that it exposes. Each mutator defines the range of mutation that it has the capability to inject a for a particular kind of mutation, which the system is expected to be able to cope with.

Deviant can suggest possible mutations for you to inject, or you can inject mutations of your own design.

Deviant comes with pre-made mutators for various environments, and users can develop custom mutators with Auxon's Deviant SDK. For more information about developing custom mutators, refer to the following resources:

  • Overall Auxon SDK: https://github.com/auxoncorp/auxon-sdk/
  • Example Mutator implementations in Rust: https://github.com/auxoncorp/auxon-sdk/tree/main/client-libraries/rust/examples
  • Documentation for Rust libraries used in mutator development:
    • Mutator Protocol: https://docs.rs/auxon-sdk/latest/auxon_sdk/mutator_protocol/
    • Mutation Plane Client: https://docs.rs/auxon-sdk/latest/auxon_sdk/mutation_plane_client/
    • Mutator Server: https://docs.rs/auxon-sdk/latest/auxon_sdk/mutator_server/

← Client →