Chaos Monkey for Spring Boot Reference Guide

1. Getting started

There are two ways to activate Chaos Monkey for Spring Boot (CM4SB) in your existing Spring Boot app. Either by adding it to your regular app dependencies (eg. pom.xml/build.gradle(.kts)) or instead, you include it as an external dependency when starting your Spring Boot app.

1.1. Adding Chaos Monkey for Spring Boot

1.1.1. As dependency

1.) Add Chaos Monkey for Spring Boot as dependency for your project

Maven

<dependency>
    <groupId>de.codecentric</groupId>
    <artifactId>chaos-monkey-spring-boot</artifactId>
    <version>3.1.0</version>
</dependency>

Gradle

implementation 'de.codecentric:chaos-monkey-spring-boot:3.1.0'

Gradle Kotlin

implementation("de.codecentric:chaos-monkey-spring-boot:3.1.0")

If you want to use the latest snapshot version, add the sonatype staging repository oss.sonatype.org/content/repositories/snapshots/

2.) Start your Spring Boot App with the chaos-monkey spring profile enabled. We also pass some properties to assault services with latency.

Command line example:

Spring profile

java -jar your-app.jar --spring.profiles.active=chaos-monkey --chaos.monkey.enabled=true --chaos.monkey.watcher.service=true --chaos.monkey.assaults.latencyActive=true

System property (alternative)

java -jar your-app.jar -DLOAD_CHAOS_MONKEY=true --chaos.monkey.enabled=true --chaos.monkey.watcher.service=true --chaos.monkey.assaults.latencyActive=true

Passing all arguments through command line is a bit cumbersome, but you can also specify the properties in your application.properties file, head over to Minimal CM4SB configuration via property file to find out how.

1.1.2. Using external dependency jar file

If you dont want to add Chaos Monkey for Spring Boot to your dependencies (see As dependency), you can use the external jar solution as an alternative.

You can extend your existing application with the CM4SB and add it as an external dependency at startup.For this it is necessary to use the PropertiesLauncher of Spring Boot.

You can specify the necessary properties to activate the CM4SB as an external property file using the parameter --spring.config.location.

Spring Boot external config documentation

This special JAR file is also available through Maven Central:

Maven

<dependency>
    <groupId>de.codecentric</groupId>
    <artifactId>chaos-monkey-spring-boot</artifactId>
    <classifier>jar-with-dependencies</classifier>
    <version>3.1.0</version>
</dependency>

Gradle

implementation 'de.codecentric:chaos-monkey-spring-boot:3.1.0:jar-with-dependencies'

Gradle Kotlin

implementation("de.codecentric:chaos-monkey-spring-boot:3.1.0:jar-with-dependencies")

Start your Spring Boot application, add Chaos Monkey for Spring Boot JAR and properties.

command-line

java -cp your-app.jar
    -Dloader.path=chaos-monkey-spring-boot-3.1.0-jar-with-dependencies.jar
        org.springframework.boot.loader.PropertiesLauncher
    --spring.profiles.active=chaos-monkey
    --spring.config.location=file:./chaos-monkey.properties

Each Chaos Monkey version is built for a specific Spring Boot version, which can be found on the release page. Chaos Monkey might break when it is added as external jar and you’re using a different Spring Boot version in your application.

It is your decision how you want to integrate and use the CM4SB. There is no difference between the two possibilities. However, it’s easier to add it as a dependency (see As dependency), so you don’t need to adjust the starting of your application.

1.2. Minimal CM4SB configuration via property file

In this example, we want to make latency attacks on our @Service annotated classes.

application.properties:

spring.profiles.active=chaos-monkey
chaos.monkey.enabled=true

chaos.monkey.watcher.service=true
chaos.monkey.assaults.latencyActive=true

application.yml:

spring:
  profiles:
    active: chaos-monkey
chaos:
  monkey:
    enabled: true
    watcher:
      service: true
    assaults:
      latencyActive: true

Configuration via property file offers all available options, but requires a restart to apply changes. To avoid this, you can configure it via Actuator.

2. Spring Boot Actuator Endpoints

Chaos Monkey for Spring Boot offers you some built in endpoints exposed via JMX or HTTP. This allows you to change configuration at runtime.

Spring Boot Actuator needs to be included in your project dependencies if you want to use this feature.

2.1. Enable chaosmonkey endpoint

application.properties:

management.endpoint.chaosmonkey.enabled=true
management.endpoint.chaosmonkeyjmx.enabled=true

# include all endpoints
management.endpoints.web.exposure.include=*

# include specific endpoints
management.endpoints.web.exposure.include=health,info,chaosmonkey

application.yml:

management:
  endpoint:
    chaosmonkey:
      enabled: true
    chaosmonkeyjmx:
      enabled: true

  endpoints:
    web:
      exposure:
        # include all endpoints
        include: "*"
        # include specific endpoints
        include:
          - health
          - info
          - chaosmonkey

2.2. HTTP Endpoint

ID	Description	Methods
/chaosmonkey	Running Chaos Monkey configuration	GET
/chaosmonkey/status	Is Chaos Monkey enabled or disabled?	GET
/chaosmonkey/enable	Enable Chaos Monkey	POST
/chaosmonkey/disable	Disable Chaos Monkey	POST
/chaosmonkey/watchers	Running Watchers configuration.	GET
/chaosmonkey/watchers	Change Watchers Configuration	POST
/chaosmonkey/assaults	Running Assaults configuration	GET
/chaosmonkey/assaults	Change Assaults configuration	POST
/chaosmonkey/assaults/runtime/attack	Execute configured runtime Assault	POST

Description

Methods

/chaosmonkey

Running Chaos Monkey configuration

GET

/chaosmonkey/status

Is Chaos Monkey enabled or disabled?

GET

/chaosmonkey/enable

Enable Chaos Monkey

POST

/chaosmonkey/disable

Disable Chaos Monkey

POST

/chaosmonkey/watchers

Running Watchers configuration.

GET

/chaosmonkey/watchers

Change Watchers Configuration

POST

/chaosmonkey/assaults

Running Assaults configuration

GET

/chaosmonkey/assaults

Change Assaults configuration

POST

/chaosmonkey/assaults/runtime/attack

Execute configured runtime Assault

POST

2.2.1. Examples

GET Chaos Monkey Configuration

/chaosmonkey - Response 200 OK

{
  "chaosMonkeyProperties": {
    "enabled": true,
    "togglePrefix": "chaos.monkey"
  },
  "assaultProperties": {
    "level": 3,
    "latencyRangeStart": 1000,
    "latencyRangeEnd": 3000,
    "latencyActive": true,
    "exceptionsActive": false,
    "exception": {
      "type": "java.lang.RuntimeException",
      "method": "<init>",
      "arguments": [
        {
          "className": "java.lang.String",
          "value": "Chaos Monkey - RuntimeException"
        }
      ]
    },
    "killApplicationActive": false,
    "memoryActive": false,
    "memoryMillisecondsHoldFilledMemory": 90000,
    "memoryMillisecondsWaitNextIncrease": 1000,
    "memoryFillIncrementFraction": 0.15,
    "memoryFillTargetFraction": 0.25,
    "cpuActive": true,
    "cpuMillisecondsHoldLoad": 90000,
    "cpuLoadTargetFraction": 0.9,
    "runtimeAssaultCronExpression": "OFF"
  },
  "watcherProperties": {
    "controller": false,
    "restController": true,
    "service": false,
    "repository": false,
    "component": false,
    "restTemplate": false,
    "webClient": false,
    "actuatorHealth": false
  }
}

GET Chaos Monkey Status

Case: Chaos Monkey is running.

/chaosmonkey/status - Response 200 OK

{
  "enabledAt": "2022-03-18T10:01:54.465+01:00",
  "enabledFor": {
    "raw": "PT45.221S",
    "formatted": "45 seconds"
  },
  "enabled": true
}

Case: Chaos Monkey is activated, but will not launch any attacks.

/chaosmonkey/status - Response 200 OK

{
  "enabled": false
}

POST Chaos Monkey enable

/chaosmonkey/enable - Response 200 OK

{
  "enabledAt": "2022-03-18T10:06:22.094+01:00",
  "enabled": true
}

POST Chaos Monkey disable

/chaosmonkey/disable - Response 200 OK

{
  "disabledAt": "2022-03-18T10:06:57.185+01:00",
  "enabledFor": {
    "raw": "PT35.091S",
    "formatted": "35 seconds"
  },
  "enabled": false
}

GET Watchers

/chaosmonkey/watchers - Response 200 OK

{
  "controller": false,
  "restController": true,
  "service": false,
  "repository": false,
  "component": false,
  "restTemplate": false,
  "webClient": false,
  "actuatorHealth": false
}

POST Watchers

Request to enable/disable Watchers

/chaosmonkey/watchers - Request

{
  "controller": true,
  "restController": true,
  "service": true,
  "repository": true,
  "component": false,
  "restTemplate": false,
  "webClient": false,
  "actuatorHealth": false
}

/chaosmonkey/watchers - Response 200 OK

Watcher config has changed

GET Assaults

/chaosmonkey/assaults - Response 200 OK

{
  "level": 3,
  "latencyRangeStart": 1000,
  "latencyRangeEnd": 3000,
  "latencyActive": true,
  "exceptionsActive": false,
  "exception": {
      "type": "java.lang.RuntimeException",
      "method": "<init>",
      "arguments": [
        {
          "className": "java.lang.String",
          "value": "Chaos Monkey - RuntimeException"
        }
      ]
    },
  "killApplicationActive": false,
  "killApplicationCronExpression": "OFF",
  "memoryActive": false,
  "memoryMillisecondsHoldFilledMemory": 90000,
  "memoryMillisecondsWaitNextIncrease": 1000,
  "memoryFillIncrementFraction": 0.15,
  "memoryFillTargetFraction": 0.25,
  "memoryCronExpression": "OFF",
  "cpuActive": true,
  "cpuMillisecondsHoldLoad": 90000,
  "cpuLoadTargetFraction": 0.9,
  "cpuCronExpression": "OFF",
  "runtimeAssaultCronExpression": "OFF"
}

POST Assaults

Request to enable Latency & Exception Assault

/chaosmonkey/assaults - Request

{
  "level": 5,
  "latencyRangeStart": 2000,
  "latencyRangeEnd": 5000,
  "latencyActive": true,
  "exceptionsActive": true,
  "killApplicationActive": false
}

/chaosmonkey/assaults - Response 200 OK

Assault config has changed

Define specific method attacks

/chaosmonkey/assaults - Request

{
  "level": 5,
  "latencyRangeStart": 2000,
  "latencyRangeEnd": 5000,
  "latencyActive": true,
  "exceptionsActive": true,
  "killApplicationActive": false,
  "watchedCustomServices": [
    "com.example.chaos.monkey.chaosdemo.controller.HelloController.sayHello",
    "com.example.chaos.monkey.chaosdemo.controller.HelloController.sayGoodbye"
  ]
}

/chaosmonkey/assaults - Response 200 OK

Assault config has changed

Define custom Exceptions

/chaosmonkey/assaults - Request

{
  "level": 5,
  "latencyRangeStart": 2000,
  "latencyRangeEnd": 5000,
  "latencyActive": true,
  "exceptionsActive": true,
  "killApplicationActive": false,
  "exception": {
    "type": "java.lang.IllegalArgumentException",
    "method": "<init>",
    "arguments": [
      {
        "className": "java.lang.String",
        "value": "custom illegal argument exception"
      }
    ]
  }
}

/chaosmonkey/assaults - Response 200 OK

Assault config has changed

2.2.2. POST Assault Attack

/chaosmonkey/assaults/runtime/attack - Response 200 OK

Started runtime assaults

2.3. JMX Endpoint

Some settings can also be made via JMX, but detailed changes to the assaults are not possible.

Operation	Description	Response example
enableChaosMonkey	Enable Chaos Monkey	enabledAt=2022-03-18T10:19:51.108+01:00 enabled=true
disableChaosMonkey	Disable Chaos Monkey	enabled=false
getAssaultProperties	Running Assault configuration	level=3 latencyRangeStart=1000 latencyRangeEnd=3000 latencyActive=true exceptionsActive=false killApplicationActive=false restartApplicationActive=false
getWatcherProperties	Running Watcher configuration	controller=true restController=false service=true repository=false component=false
toggleLatencyAssault	Toggle Latency Assault status	New value (true/false)
toggleExceptionAssault	Toggle Exception Assault status	New value (true/false)
toggleKillApplicationAssault	Toggle KillApplication Assault status	New value (true/false)
toggleMemoryAssault	Toggle Memory Assault status	New value (true/false)
toggleCpuAssault	Toggle CPU Assault status	New value (true/false)
isChaosMonkeyActive	Is Chaos Monkey active or not	true or false

Operation

Description

Response example

enableChaosMonkey

Enable Chaos Monkey

enabledAt=2022-03-18T10:19:51.108+01:00 enabled=true

disableChaosMonkey

Disable Chaos Monkey

enabled=false

getAssaultProperties

Running Assault configuration

level=3 latencyRangeStart=1000 latencyRangeEnd=3000 latencyActive=true exceptionsActive=false killApplicationActive=false restartApplicationActive=false

getWatcherProperties

Running Watcher configuration

controller=true restController=false service=true repository=false component=false

toggleLatencyAssault

Toggle Latency Assault status

New value (true/false)

toggleExceptionAssault

Toggle Exception Assault status

New value (true/false)

toggleKillApplicationAssault

Toggle KillApplication Assault status

New value (true/false)

toggleMemoryAssault

Toggle Memory Assault status

New value (true/false)

toggleCpuAssault

Toggle CPU Assault status

New value (true/false)

isChaosMonkeyActive

Is Chaos Monkey active or not

true or false

2.4. Enable springdoc-openapi integration

Enable wanted endpoints like described in Enable chaosmonkey endpoint. Then include springdoc-openapi in your dependencies and configure the following:

application.properties:

springdoc.show-actuator=true
chaos.monkey.apidoc.enabled=true

application.yml:

springdoc:
  show-actuator: true
chaos.monkey:
    apidoc:
      enabled: true

Chaosmonkey will now show up as a group in your Swagger UI.

3. Chaos Toggles and Name Mapping

Chaos Monkey for Spring Boot supports toggles, also known as feature flags. Feature flags are evaluated on every request, allowing the ability to conditionally enable an assault.

Toggles provide:

An additional mechanism to enable or disable assaults
Allow the ability to limit chaos assaults to a specific user or role
Provides an extension mechanism for users to implement their own logic

3.1. Implementation

3.1.1. Defaults

By default, all toggles are enabled and respect the Chaos Monkey properties used for configuration.

Toggles only control if an already configured assault will be used. The assault must be configured via the typical properties or actuator before a toggle will be evaluated.

3.1.2. Using Unleash to control toggles

Unleash is an open source service and SDK that can be used to configure toggles.

Chaos Monkey includes a minimal configuration for Unleash. To leverage it:

Include the Unleash Java SDK on your classpath
Configure the Unleash SDK as needed
Enable the property chaos.monkey.toggle.unleash.enabled=true

When using Unleash all toggles are DISABLED by default. So if the Chaos Monkey properties are enabled, but they don’t seem to be functioning, check your Unleash toggles.

The default ChaosToggleNameMapper is in place and will check for toggles named the following (based on the target of an assault):

chaos.monkey.controller
chaos.monkey.component
chaos.monkey.repository
chaos.monkey.restController
chaos.monkey.service
chaos.monkey.unknown

If the assault has been enabled via properties then enabling that toggle in Unleash with the matching name will allow the corresponding assault to happen.

3.2. Creating your own

3.2.1. ChaosToggleNameMapper

To control the names of the toggles, you can provide a bean that implements the ChaosToggleNameMapper interface.

ChaosToggleNameMapper interface:

public interface ChaosToggleNameMapper {
  String mapName(ChaosTarget type, String name);
}

The supplied name to the interface is the class name and method being assaulted.

Java

public class MyVeryDetailedToggleNameMapper extends DefaultChaosToggleNameMapper {
  public MyAppToggleMapper(String prefix) {
    super(prefix);
  }

  @Override
  public String mapName(ChaosTarget type, String name) {
    return togglePrefix + "." + type.getName() + "." + name;
  }
}

Kotlin

class MyVeryDetailedToggleNameMapper(prefix: String) : DefaultChaosToggleNameMapper(prefix) {
    override fun mapName(type: ChaosTarget, name: String): String {
        return togglePrefix + "." + type.getName() + "." + name
    }
}

This would create an extremely detailed toggle name so that individual methods and classes could be toggled individually:

chaos.monkey.controller.com.example.chaos.monkey.toggledemo.controller.HelloController.sayHello

3.2.2. ChaosToggles Interface

The ChaosToggles interface is where the actual check to enable or disable an assault is.

ChaosToggles interface:

public interface ChaosToggles {
  boolean isEnabled(String toggleName);
}

To add you own logic to control which toggles are enabled, you can provide a bean that implements the ChaosToggles interface.

Java

public class MyParamChaosToggles implements ChaosToggles {
  @Override
  public boolean isEnabled(String toggleName) {
    HttpServletRequest curRequest =
                    ((ServletRequestAttributes) RequestContextHolder.currentRequestAttributes()).getRequest();

    String chaosParam =  curRequest.getParameter("chaos");
    return chaosParam != null && toggleName.startsWith(chaosParam);
  }
}

Kotlin

class MyParamChaosToggles : ChaosToggles {
    override fun isEnabled(toggleName: String): Boolean {
        val curRequest = (RequestContextHolder.currentRequestAttributes() as ServletRequestAttributes).request
        val chaosParam = curRequest.getParameter("chaos")
        return chaosParam != null && toggleName.startsWith(chaosParam)
    }
}

This implementation would enable chaos attacks for requests that include a chaos request parameter that starts with the toggle name.

curl http://localhost:8080/hello?chaos=chaos.monkey.repository

3.3. Toggle Recommendations

By leveraging toggles it is possible to leave Chaos Monkey enabled in all environments, and conditionally trigger assaults based on feature flags or custom logic.

Make sure that any logic in custom ChaosToggles implementations are quick, since they can potentially be called multiple times per request.

4. Configuration

Chaos Monkey for Spring Boot can be customized to your planned experiment. You can decide which attacks you want to run and which parts of your application should be attacked.

4.1. Properties

Table 1. Main
Property	Description	Values	Default
chaos.monkey.enabled	Determine whether should execute or not	TRUE or FALSE	FALSE

Table 2. Assaults
Property	Description	Values	Default
chaos.monkey.assaults.level	How many requests are to be attacked. 1 each request, 5 each 5th request is attacked	1-10000	1
chaos.monkey.assaults.deterministic	Whether the level should be used as a deterministic value (attack every x requests) or a chance (on average, 1 in x requests will be attacked)	TRUE or FALSE	FALSE
chaos.monkey.assaults.latencyRangeStart	Minimum latency in ms added to the request	Integer.MIN_VALUE, Integer.MAX_VALUE	1000
chaos.monkey.assaults.latencyRangeEnd	Maximum latency in ms added to the request	Integer.MIN_VALUE, Integer.MAX_VALUE	3000
chaos.monkey.assaults.latencyActive	Latency assault active	TRUE or FALSE	FALSE
chaos.monkey.assaults.exceptionsActive	Exception assault active	TRUE or FALSE	FALSE
chaos.monkey.assaults.exception.type	Exception to be thrown	Class name	java.lang.RuntimeException
chaos.monkey.assaults.exception.method	Exception constructor or static creator method	<init> for constructor or method name	<init>
chaos.monkey.assaults.exception.arguments[0].type	Exception argument types (repeatable)	Class name	java.lang.String
chaos.monkey.assaults.exception.arguments[0].value	Exception argument types (repeatable)	primitive or json value matching the respective type	"Chaos Monkey - RuntimeException"
chaos.monkey.assaults.killApplicationActive	AppKiller assault active	TRUE or FALSE	FALSE
chaos.monkey.assaults.killApplication.cron.expression	Cron expression like `/1 * * * ?` can be set to enable chaos monkey AppKiller assault on a schedule	Any valid cron expression (or OFF)	OFF
chaos.monkey.assaults.memoryActive	Memory assault active	TRUE or FALSE	FALSE
chaos.monkey.assaults.memoryMillisecondsHoldFilledMemory	Duration to assault memory when requested fill amount is reached in ms.	min=1500, max=Integer.MAX_VALUE	90000
chaos.monkey.assaults.memoryMillisecondsWaitNextIncrease	Time in ms between increases of memory usage.	min=100, max=30000	1000
chaos.monkey.assaults.memoryFillIncrementFraction	Fraction of one individual memory increase iteration. `1.0` equals 100 %.	min=0.01, max=1.0	0.15
chaos.monkey.assaults.memoryFillTargetFraction	Final fraction of used memory by assault. `0.95` equals 95 %.	min=0.01, max=0.95	0.25
chaos.monkey.assaults.memory.cron.expression	Cron expression like `/1 * * * ?` can be set to enable chaos monkey memory assault on a schedule	Any valid cron expression (or OFF)	OFF
chaos.monkey.assaults.cpuActive	CPU assault active	TRUE or FALSE	FALSE
chaos.monkey.assaults.cpuMillisecondsHoldLoad	Duration to assault cpu when requested load is reached in ms.	min=1500, max=Integer.MAX_VALUE	90000
chaos.monkey.assaults.cpuLoadTargetFraction	Final fraction of used cpu by assault. `0.95` equals 95 %.	min=0.1, max=1.0	0.9
chaos.monkey.assaults.cpu.cron.expression	Cron expression like `/1 * * * ?` can be set to enable chaos monkey cpu assault on a schedule	Any valid cron expression (or OFF)	OFF
chaos.monkey.assaults.watchedCustomServices	Limits watched packages/classes/methods	List of fully qualified packages, class and/or method names	Empty list

Table 3. Watcher
Property	Description	Values	Default
chaos.monkey.watcher.controller	Controller watcher active	TRUE or FALSE	FALSE
chaos.monkey.watcher.restController	RestController watcher active	TRUE or FALSE	FALSE
chaos.monkey.watcher.service	Service watcher active	TRUE or FALSE	FALSE
chaos.monkey.watcher.repository	Repository watcher active	TRUE or FALSE	FALSE
chaos.monkey.watcher.component	Component watcher active	TRUE or FALSE	FALSE
chaos.monkey.watcher.actuatorHealth	Spring actuator health watcher active	TRUE or FALSE	FALSE
chaos.monkey.watcher.beans	Bean watcher for given beans	List of bean names	Empty list
chaos.monkey.watcher.beanClasses	Bean watcher for given beans classes	List of bean classes	Empty list
chaos.monkey.watcher.exclude	Exclude classes from proxying. Use if proxying a class breaks it.	List of class names	Empty list

5. Watcher

A watcher is a Chaos Monkey for Spring Boot component, that will scan your app for beans based on one of the conditions described in Watcher Types.

With the help of Spring AOP, Chaos Monkey recognizes the execution of a public method in a watched bean and will either not execute any action or start one of its assaults. You can customize the behaviour by configuration.

5.1. Watcher Types

5.1.1. Annotation Watchers

These watchers find your beans based on the following annotations:

@Controller
@RestController
@Service
@Repository
@Component

5.1.2. Actuator Watchers

Chaos Monkey can also watch the

Spring Boot HealthIndicators

and assault the health checks with exceptions and latency. Exceptions will let the HealthIndicators go into status DOWN and the latency assault postpone the response.

5.1.3. Outgoing Request Watchers

Chaos Monkey also provides watchers for:

RestTemplate
WebClient

The watchers will inject assaults for all RestTemplate and WebClient beans in the Spring Context based on the given assault configuration.

RestTemplates and WebClients which are created not as a bean via new RestTemplate() and WebClient.create() are not getting instrumented.

These watchers don’t use AOP. Instead, they inject respective customizers.

5.1.4. Alternative Bean Watcher

Chaos Monkey can watch any Bean in your application by name or class.

The chaos monkey profile must be active and bean names/classes must be configured when the beans are created. Adding bean names/classes to the configuration after the beans have been created will not enable them to be watched.

5.2. Customize Watcher

You can customize the behavior of all watchers using the property watchedCustomServices and decide which classes and public methods should be attacked. You can set the property to reference public methods, classes or packages.

If watchedCustomServices is not set, all classes and public methods recognized by the enabled watchers are attacked.

You can either maintain the list in your application properties or adjust it at runtime using the Spring Boot Actuator Endpoint.

See following examples:

Chaos Monkey Spring Boot Actuator Endpoint (/actuator/chaosmonkey/assaults)

{
  "level": 3,
  "latencyActive": true,
  "latencyRangeStart": 1000,
  "latencyRangeEnd": 3000,
  "watchedCustomServices": [
    "com.example.chaos.monkey.chaosdemo.controller.HelloController.sayHello",
    "com.example.chaos.monkey.chaosdemo.service.HelloService"
  ]
}

application.properties:

chaos.monkey.enabled=true
chaos.monkey.watcher.controller=true
chaos.monkey.assaults.level=1
chaos.monkey.assaults.latency-active=true
chaos.monkey.assaults.watchedCustomServices=com.example.chaos.monkey.chaosdemo.controller.HelloController.sayHello,com.example.chaos.monkey.chaosdemo.service.HelloService

application.yml:

chaos:
  monkey:
    enabled: true
    watcher:
      controller: true
    assaults:
      level: 1
      latency-active: true
      watched-custom-services:
        - com.example.chaos.monkey.chaosdemo.controller.HelloController.sayHello
        - com.example.chaos.monkey.chaosdemo.service.HelloService

This list only restricts the attacked classes found by watchers. If a class or method wasn’t attacked when this list was unset, it won’t be attacked if it is added to the list.

6. Assaults

Assaults are the heart of Monkey’s Chaos and he makes use of them based on your configuration. The following assaults are provided:

6.1. Request Assaults

These assaults attack some point of your application and are triggered by a /chaosmonkey/watchers - Response 200 OK, for example when a method of your component is called.

6.1.1. Latency Assault

If Latency Assault is enabled, latency is added to a request. You control the number of requests where this should occur via the level.

6.1.2. Exception Assault

You can determine at runtime whether an exception should occur when the method is used.

Thanks to the great community, the Chaos Monkey for Spring Boot can throw any kind of RuntimeException. You can configure the required exception at runtime via the Actuator Endpoint.

6.2. Runtime Assaults

These assaults attack your whole application.

You need to trigger these Assaults yourself either using Assault Scheduling or the Chaos Monkey Endpoint.

6.2.1. AppKiller Assault

When the configured methods are called in the application, the Chaos Monkey will shut down the application.

6.2.2. Memory Assault

Memory Assaults attack the memory of the Java Virtual Machine.

The Memory assault is heavily depending on the Java version you are using. We are testing with the default garbage collectors of each Java version. On Java 8 the fill rate is limited to 256 MB per slice!

6.2.3. CPU Assault

CPU Assaults attack the CPU of the Java Virtual Machine.

6.3. Chaos Monkey Assault Scheduler

You can schedule Chaos Monkey Runtime Assaults (Memory, CPU, AppKiller) using cron expressions. See also Configuration

7. Metrics

Since version 2.0.1 of Chaos Monkey for Spring Boot, there are some metrics available. Thanks to micrometer, we are now able to deliver some insights about all calls and attacks recognized by the CM4SB.

Spring Boot’s application metrics facade supports numerous monitoring systems, including:

Atlas
Datadog
Ganglia
Graphite
Influx
JMX
New Relic
Prometheus
SignalFx
Simple (in-memory)
StatsD
Wavefront

7.1. CM4SB Metrics

If you have micrometer.io on your classpath and enabled, you are able to view and export these kind of insights. Take a look at Spring Docs to see how it works.

Metrics are available via Spring Boot Actuator endpoint, if you are exposing them in simple mode.

endpoint example:

/actuator/metrics

Beside all other metrics, deliverd by default, you will find some CM4SB metrics. They will always start with "chaos.monkey.*".

7.1.1. Chaos Monkey Application Metrics

Request count

total = all requests recognized
assaulted = all attacked requests

chaos monkey metric request count:

chaos_monkey_application_request_count_total
chaos_monkey_application_request_count_assaulted

7.1.2. Watcher Metrics

For all types of CM4SB Watchers, you will find a metric.

chaos monkey metric request count:

chaos_monkey_assault_component_watcher_total
chaos_monkey_assault_controller_watcher_total
chaos_monkey_assault_repository_watcher_total
chaos_monkey_assault_restController_watcher_total
chaos_monkey_assault_service_watcher_total

Assault Latency

*_latency_count_gauge = indicates the current average (gauge) level of all latency attacks
*_latency_count_total = number of latency attacks

chaos monkey metric latency count in ms:

chaos_monkey_assault_latency_count_gauge
chaos_monkey_assault_latency_count_total

Assault Exception

Number of thrown exception attacks

chaos monkey metric exception:

chaos_monkey_assault_exception_count

Assault KillApp

Number of Application kills, there is a delay before the app is killed (5s) and the produce of this metric. So, be sure this metric is collected by your favorite system. Otherwise it is lost.

chaos monkey metric:

chaos_monkey_assault_killapp_count

8. FAQ

8.1. I’m using kotlin and my classes are not attacked

If you want chaos monkey to attack your classes they need to be open (=non-final). We recommend using the Kotlin Spring plugin, which automatically opens Spring annotated classes for you:

Maven

<plugin>
    <artifactId>kotlin-maven-plugin</artifactId>
    <groupId>org.jetbrains.kotlin</groupId>
    <version>${kotlin.version}</version>

    <configuration>
        <compilerPlugins>
            <plugin>spring</plugin>
        </compilerPlugins>
    </configuration>

    <dependencies>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-maven-allopen</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
    </dependencies>
</plugin>

Gradle

plugins {
  id "org.jetbrains.kotlin.plugin.spring" version "<kotlin version>"
}

Gradle Kotlin

plugins {
  kotlin("plugin.spring") version "<kotlin version>"
}

8.2. When I add chaos monkey I get a `NullpointerException` on startup

Some classes cannot be proxied correctly by spring. The most common case for this are filters extending GenericFilterBean. To prevent these classes from being proxied, you can either:

mark them as final
don’t mark them as spring @Component (or @Service, @Controller, …). Instead, register them as @Bean inside a configuration.

9. Changes in 3.1.0

Built with Spring Boot 3.2.1

9.1. Bug Fixes

#402 Always register outgoing watchers so they can be enabled later

9.2. Improvements

#411 Allow custom exceptions to fall through in outgoing watchers (rest template, webclient). This could also slightly change the behaviour/output of outgoing watchers when not using a custom exception.

9.3. New Features

#408 Watch beans by type

9.4. Contributors

This release was only possible because of these great humans ❤️:

Thank you for your support!