1. What is the goal of Chaos Monkey?

Inspired by PRINCIPLES OF CHAOS ENGINEERING and by my work in distributed system, with a focus on Spring Boot, I wanted to test the resulting applications better and especially during operation.

After writing many unit and integration tests, a code coverage from 70% to 80%, this unpleasant feeling remains, how our baby behaves in production?

Many questions remain unanswered:

  • Will our fallbacks work?

  • How does the application behave with network latency?

  • What if one of our services breaks down?

  • Service Discovery works, but is our Client-Side-Load-Balancing also working?

As you can see, there are many more questions and open topics you have to deal with.

That was my start to take a deep dive into Chaos Engineering and I started this little project to share my thoughts and experience.

A detailed explanation of when and why you should use the Chaos Monkey for Spring Boot can be found here.

2. Getting started

How to activate Chaos Monkey for Spring Boot in your existing Spring Boot App?

There are two ways to integrate the Chaos Monkey for Spring Boot into your existing Spring Boot app. Either it is an integral part of your project and is listed in the POM or you can include it as an external dependency when starting your Spring Boot App.

2.1. Dependency in your POM file

1.) Let´s activate Chaos Monkey for Spring Boot, only 2 steps are required

pom.xml
<dependency>
    <groupId>de.codecentric</groupId>
    <artifactId>chaos-monkey-spring-boot</artifactId>
    <version>2.0.1</version>
</dependency>

2.) Start your Spring Boot App Command-Line example:

command-line
java -jar your-app.jar --spring.profiles.active=chaos-monkey

2.2. External dependency JAR file

You can extend your existing application with the CM4SB and add it as an external dependency at startup. For this it is necessary to use the PropertiesLauncher of Spring Boot.

You can specify the necessary properties to activate the CM4SB as an external property file using the parameter --spring.config.location.

This special JAR file is also available via Maven:

pom.xml
<dependency>
    <groupId>de.codecentric</groupId>
    <artifactId>chaos-monkey-spring-boot</artifactId>
    <classifier>jar-with-dependencies</classifier>
    <version>2.0.1</version>
</dependency>

Start your Spring Boot application, add Chaos Monkey for Spring Boot JAR and properties.

command-line
java -cp your-app.jar
    -Dloader.path=chaos-monkey-spring-boot-2.0.1-jar-with-dependencies.jar
        org.springframework.boot.loader.PropertiesLauncher
    --spring.profiles.active=chaos-monkey
    --spring.config.location=file:./chaos-monkey.properties

It is your decision how you want to integrate and use the CM4SB. There is no difference between the two possibilities.

2.3. Minimal property file to run CM4SB

Activate the watcher you need for your experiments. In this example we want to make attacks on our @Service and @RestController annotated classes.

chaos.monkey.enabled=true is optional, you can enable/disable the Chaos Monkey at runtime! (default=false)
application.properties:
spring.profiles.active=chaos-monkey
chaos.monkey.enabled=true

chaos.monkey.watcher.controller=false
chaos.monkey.watcher.restController=true
chaos.monkey.watcher.service=true
chaos.monkey.watcher.repository=false
application.yml:
spring:
  profiles:
    active: chaos-monkey
chaos:
  monkey:
    enabled: true
    watcher:
      component: false
      controller: false
      repository: false
      rest-controller: true
      service: true

3. Spring Boot Actuator Endpoints

Chaos Monkey for Spring Boot offers you some build in endpoints exposed via JMX or HTTP.


3.1. Enable chaosmonkey endpoint

application.properties:
management.endpoint.chaosmonkey.enabled=true
management.endpoint.chaosmonkeyjmx.enabled=true

# inlcude all endpoints
management.endpoints.web.exposure.include=*

# include specific endpoints
management.endpoints.web.exposure.include=health,info,chaosmonkey

3.2. HTTP Endpoint

ID Description Methods

/chaosmonkey

Running Chaos Monkey configuration

GET

/chaosmonkey/status

Is Chaos Monkey enabled or disabled?

GET

/chaosmonkey/enable

Enable Chaos Monkey

POST

/chaosmonkey/disable

Disable Chaos Monkey

POST

/chaosmonkey/watcher

Running Watcher configuration.

NOTE: Watcher cannot be changed at runtime, they are Spring AOP components that have to be created when the application starts.

GET

/chaosmonkey/assaults

Running Assaults configuration

GET

/chaosmonkey/assaults

Change Assaults configuration

POST

3.2.1. Examples

GET Chaos Monkey Configuration
/chaosmonkey - Response 200 OK
{
"chaosMonkeyProperties":{
"enabled": true
},
"assaultProperties":{
"level": 3,
"latencyRangeStart": 1000,
"latencyRangeEnd": 3000,
"latencyActive": true,
"exceptionsActive": false,
"killApplicationActive": false,
"watchedCustomServices": []
},
"watcherProperties":{
"controller": true,
"restController": false,
"service": true,
"repository": false,
"component": false
}
}
GET Chaos Monkey Status

Case: Chaos Monkey is running.

/chaosmonkey/status - Response 200 OK
Ready to be evil!

Case: Chaos Monkey is activated, but will not launch any attacks.

/chaosmonkey/status - Response 503 Service Unavailable
You switched me off!
POST Chaos Monkey enable
/chaosmonkey/enable - Response 200 OK
Chaos Monkey is enabled
POST Chaos Monkey disable
/chaosmonkey/disable - Response 200 OK
Chaos Monkey is disabled
GET Watcher
Watcher cannot be changed at runtime, they are Spring AOP components that have to be created when the application starts.
/chaosmonkey/watcher - Response 200 OK
{
"controller": true,
"restController": false,
"service": true,
"repository": false,
"component": false
}
GET Assaults
/chaosmonkey/assaults - Response 200 OK
{
"level": 3,
"latencyRangeStart": 1000,
"latencyRangeEnd": 3000,
"latencyActive": true,
"exceptionsActive": false,
"killApplicationActive": false,
"watchedCustomServices": []
}
POST Assaults
/chaosmonkey/assaults - Request
{
"level": 5,
"latencyRangeStart": 2000,
"latencyRangeEnd": 5000,
"latencyActive": true,
"exceptionsActive": true,
"killApplicationActive": false,
"watchedCustomServices": [ "com.example.chaos.monkey.chaosdemo.controller.HelloController.sayHello","com.example.chaos.monkey.chaosdemo.controller.HelloController.sayGoodbye"]
}
/chaosmonkey/assaults - Response 200 OK
Assault config has changed

3.3. JMX Endpoint

Some settings can also be made via JMX, but detailed changes to the assaults are not possible.

JMX Console
Operation Description Response example

enableChaosMonkey

Enable Chaos Monkey

Chaos Monkey is enabled

disableChaosMonkey

Disable Chaos Monkey

Chaos Monkey is disabled

getAssaultProperties

Running Assault configuration

level=3 latencyRangeStart=1000 latencyRangeEnd=3000 latencyActive=true exceptionsActive=false killApplicationActive=false restartApplicationActive=false

getWatcherProperties

Running Watcher configuration

controller=true restController=false service=true repository=false component=false

toggleLatencyAssault

Toggle Latency Assault status

New value (true/false)

toggleExceptionAssault

Toggle Exception Assault status

New value (true/false)

toggleKillApplicationAssault

Toggle KillApplication Assault status

New value (true/false)

isChaosMonkeyActive

Is Chaos Monkey active or not

true or false

4. Configuration

Chaos Monkey for Spring Boot can be customized to your planned experiment. You can decide which attacks you want to run and which parts of your application should be attacked.

Except for the Watcher, you can also influence the behavior of the Chaos Monkey during runtime.

4.1. Properties

Property Description Values Default

chaos.monkey.enabled

Determine whether should execute or not

TRUE or FALSE

FALSE

chaos.monkey.assaults.level

How many requests are to be attacked. 1 each request, 5 each 5th request is attacked

1-10000

5

chaos.monkey.assaults.latencyRangeStart

Minimum latency in ms added to the request

Integer.MIN_VALUE, Integer.MAX_VALUE

3000

chaos.monkey.assaults.latencyRangeEnd

Maximum latency in ms added to the request

Integer.MIN_VALUE, Integer.MAX_VALUE

15000

chaos.monkey.assaults.latencyActive

Latency assault active

TRUE or FALSE

TRUE

chaos.monkey.assaults.exceptionsActive

Exception assault active

TRUE or FALSE

FALSE

chaos.monkey.assaults.killApplicationActive

AppKiller assault active

TRUE or FALSE

FALSE

chaos.monkey.watcher.controller

Controller watcher active

TRUE or FALSE

FALSE

chaos.monkey.watcher.restController

RestController watcher active

TRUE or FALSE

FALSE

chaos.monkey.watcher.service

Service watcher active

TRUE or FALSE

TRUE

chaos.monkey.watcher.repository

Repository watcher active

TRUE or FALSE

FALSE

chaos.monkey.watcher.component

Component watcher active

TRUE or FALSE

FALSE

5. Watcher

A watcher is a Chaos Monkey for Spring Boot component, that will scan your app for a specific type of annotation.

Following Spring annotation are supported:

  • @Controller

  • @RestController

  • @Service

  • @Repository

  • @Component

With the help of Spring AOP, Chaos Monkey recognizes the execution of a public method and will either not execute any action or start one of its assaults. You can customize the behave by configuration.

5.1. Customize Watcher

You can customize the behavior of all watchers using the property watchedCustomServices and decide which classes and public methods should be attacked. If no signatures are stored, all classes and public methods, recognized by the watchers are attacked by default.

You can either maintain the list in your application properties or adjust it at runtime using the Spring Boot Actuator Endpoint.

Example Chaos Monkey Spring Boot Actuator Endpoint
{
"level": 3,
"latencyRangeStart": 1000,
"latencyRangeEnd": 3000,
"latencyActive": true,
"exceptionsActive": false,
"killApplicationActive": false,
 "watchedCustomServices": [ "com.example.chaos.monkey.chaosdemo.controller.HelloController.sayHello","com.example.chaos.monkey.chaosdemo.controller.HelloController.sayGoodbye"]
}
Example application.properties:
...
chaos.monkey.watcher.controller=true
chaos.monkey.assaults.level=1

chaos.monkey.assaults.watchedCustomServices=com.example.chaosdemo.HelloController.sayHello,com.example.chaosdemo.HelloController.sayGoodbye

...
The listed classes and their methods must have been found by one of the watchers. Only the behaviour of the watchers that are activated can be influenced.

6. Assaults

Assaults are the heart of Monkey’s Chaos and he makes use of them based on your configuration. Following assaults are actual provided:

  1. Latency Assault

    1. adds random latency to the request

  2. Exception Assault

    1. throws Chaos Monkey Runtime Exception, if he wants to (random)

  3. AppKiller Assault

    1. Yeah, what can I say? It destroys the entire application!

You can customize the behave of each Assault by configuration.

7. Metrics

Since version 2.0.1 of Chaos Monkey for Spring Boot, there are some metrics available. Thanks to micrometer, we are now able to deliver some insights about all calls and attacks recognized by the CM4SB.

Spring Boot’s application metrics facade supports numerous monitoring systems, including:

  • Atlas

  • Datadog

  • Ganglia

  • Graphite

  • Influx

  • JMX

  • New Relic

  • Prometheus

  • SignalFx

  • Simple (in-memory)

  • StatsD

  • Wavefront

7.1. CM4SB Metrics

If you have micrometer.io on your classpath and enabled, you are able to view and export these kind of insights. Take a look at Spring Docs to see how it works.

Metrics are available via Spring Boot Actuator endpoint, if you are exposing them in simple mode.

endpoint example:
/actuator/metrics

Beside all other metrics, deliverd by default, you will find some CM4SB metrics. They will always start with "chaos.monkey.*".

7.1.1. Chaos Monkey Application Metrics

Request count
  • total = all requests recognized

  • assaulted = all attacked requests

chaos monkey metric request count:
chaos_monkey_application_request_count_total
chaos_monkey_application_request_count_assaulted

7.1.2. Watcher Metrics

For all types of CM4SB Watchers, you will find a metric.

chaos monkey metric request count:
chaos_monkey_assault_component_watcher_total
chaos_monkey_assault_controller_watcher_total
chaos_monkey_assault_repository_watcher_total
chaos_monkey_assault_restController_watcher_total
chaos_monkey_assault_service_watcher_total
Assault Latency
  • *_latency_count_gauge = indicates the current average (gauge) level of all latency attacks

  • *_latency_count_total = number of latency attacks

chaos monkey metric latency count in ms:
chaos_monkey_assault_latency_count_gauge
chaos_monkey_assault_latency_count_total
Assault Exception

Number of thrown exception attacks

chaos monkey metric exception:
chaos_monkey_assault_exception_count
Assault KillApp

Number of Application kills, there is a delay before the app is killed (5s) and the produce of this metric. So, be sure this metric is collected by your favorite system. Otherwise it is lost.

chaos monkey metric:
chaos_monkey_assault_exception_count

8. Changes in 2.0.1

8.1. Spring Boot Version

  • supported and tested Spring Boot versions, up to 2.0.6

  • supported and tested Spring Framework versions, up to 5.0.10

8.2. Watcher

  • Filtering to classes and method names. By default all public methods of a monitored class are attacked by the Chaos Monkey, now single classes and methods can be defined and changed at runtime.

8.3. Metrics

  • CM4SB now provides metrics about all recognized classes and methods. In addition, it provides metrics for all assaults and their use.

8.4. AOP

  • Switch from compile-time weaving to load-time weaving to bring up the possibility to run the CM4SB as external loaded JAR.

    .application.properties:
java -cp ./target/demo.jar
    -Dloader.path=chaos-monkey-spring-boot-2.0.1.jar
        org.springframework.boot.loader.PropertiesLauncher
    --spring.profiles.active=chaos-monkey

8.5. Bugs fixed

  • Repository Watcher was not initilized

  • Repository Watcher was not triggered by calling default Spring Data methods -