1. What is the goal of Chaos Monkey?

Inspired by PRINCIPLES OF CHAOS ENGINEERING and by my work in distributed system, with a focus on Spring Boot, I wanted to test the resulting applications better and especially during operation.

After writing many unit and integration tests, a code coverage from 70% to 80%, this unpleasant feeling remains, how our baby behaves in production?

Many questions remain unanswered:

  • Will our fallbacks work?

  • How does the application behave with network latency?

  • What if one of our services breaks down?

  • Service Discovery works, but is our Client-Side-Load-Balancing also working?

As you can see, there are many more questions and open topics you have to deal with.

That was my start to take a deep dive into Chaos Engineering and I started this little project to share my thoughts and experience.

A detailed explanation of when and why you should use the Chaos Monkey for Spring Boot can be found here.

2. Getting started

How to activate Chaos Monkey for Spring Boot in your existing Spring Boot App?

1.) Let´s activate Chaos Monkey for Spring Boot, only 2 steps are required

pom.xml
<dependency>
    <groupId>de.codecentric</groupId>
    <artifactId>chaos-monkey-spring-boot</artifactId>
    <version>2.0.0</version>
</dependency>

2.) Start your Spring Boot App Command-Line example:

command-line
java -jar your-app.jar --spring.profiles.active=chaos-monkey
chaos.monkey.enabled=true is optional, you can enable/disable the Chaos Monkey at runtime! (default=false)
application.properties:
spring.profiles.active=chaos-monkey
chaos.monkey.enabled=true
application.yml:
spring:
  profiles:
    active: chaos-monkey
chaos:
  monkey:
    enabled: true

3. Spring Boot Actuator Endpoints

Chaos Monkey for Spring Boot offers you some build in endpoints exposed via JMX or HTTP.


3.1. Enable chaosmonkey endpoint

application.properties:
management.endpoint.chaosmonkey.enabled=true
management.endpoint.chaosmonkeyjmx.enabled=true

# inlcude all endpoints
management.endpoints.web.exposure.include=*

# include specific endpoints
management.endpoints.web.exposure.include=health,info,chaosmonkey

3.2. HTTP Endpoint

ID Description Methods

/chaosmonkey

Running Chaos Monkey configuration

GET

/chaosmonkey/status

Is Chaos Monkey enabled or disabled?

GET

/chaosmonkey/enable

Enable Chaos Monkey

POST

/chaosmonkey/disable

Disable Chaos Monkey

POST

/chaosmonkey/watcher

Running Watcher configuration.

NOTE: Watcher cannot be changed at runtime, they are Spring AOP components that have to be created when the application starts.

GET

/chaosmonkey/assaults

Running Assaults configuration

GET

/chaosmonkey/assaults

Change Assaults configuration

POST

3.2.1. Examples

GET Chaos Monkey Configuration
/chaosmonkey - Response 200 OK
{
"chaosMonkeyProperties":{
"enabled": true
},
"assaultProperties":{
"level": 3,
"latencyRangeStart": 1000,
"latencyRangeEnd": 3000,
"latencyActive": true,
"exceptionsActive": false,
"killApplicationActive": false,
"restartApplicationActive": false
},
"watcherProperties":{
"controller": true,
"restController": false,
"service": true,
"repository": false,
"component": false
}
}
GET Chaos Monkey Status

Case: Chaos Monkey is running.

/chaosmonkey/status - Response 200 OK
Ready to be evil!

Case: Chaos Monkey is activated, but will not launch any attacks.

/chaosmonkey/status - Response 503 Service Unavailable
You switched me off!
POST Chaos Monkey enable
/chaosmonkey/enable - Response 200 OK
Chaos Monkey is enabled
POST Chaos Monkey disable
/chaosmonkey/disable - Response 200 OK
Chaos Monkey is disabled
GET Watcher
Watcher cannot be changed at runtime, they are Spring AOP components that have to be created when the application starts.
/chaosmonkey/watcher - Response 200 OK
{
"controller": true,
"restController": false,
"service": true,
"repository": false,
"component": false
}
GET Assaults
/chaosmonkey/assaults - Response 200 OK
{
"level": 3,
"latencyRangeStart": 1000,
"latencyRangeEnd": 3000,
"latencyActive": true,
"exceptionsActive": false,
"killApplicationActive": false,
"restartApplicationActive": false
}
POST Assaults
/chaosmonkey/assaults - Request
{
"level": 5,
"latencyRangeStart": 2000,
"latencyRangeEnd": 5000,
"latencyActive": true,
"exceptionsActive": true,
"killApplicationActive": false,
"restartApplicationActive": false
}
/chaosmonkey/assaults - Response 200 OK
Assault config has changed

3.3. JMX Endpoint

Some settings can also be made via JMX, but detailed changes to the assaults are not possible.

JMX Console
Operation Description Response example

enableChaosMonkey

Enable Chaos Monkey

Chaos Monkey is enabled

disableChaosMonkey

Disable Chaos Monkey

Chaos Monkey is disabled

getAssaultProperties

Running Assault configuration

level=3 latencyRangeStart=1000 latencyRangeEnd=3000 latencyActive=true exceptionsActive=false killApplicationActive=false restartApplicationActive=false

getWatcherProperties

Running Watcher configuration

controller=true restController=false service=true repository=false component=false

toggleLatencyAssault

Toggle Latency Assault status

New value (true/false)

toggleExceptionAssault

Toggle Exception Assault status

New value (true/false)

toggleKillApplicationAssault

Toggle KillApplication Assault status

New value (true/false)

isChaosMonkeyActive

Is Chaos Monkey active or not

true or false

4. Configuration

Chaos Monkey for Spring Boot can be customized to your planned experiment. You can decide which attacks you want to run and which parts of your application should be attacked.

Except for the Watcher, you can also influence the behavior of the Chaos Monkey during runtime.

4.1. Properties

Property Description Values Default

chaos.monkey.enabled

Determine whether should execute or not

TRUE or FALSE

FALSE

chaos.monkey.assaults.level

How many requests are to be attacked. 1 each request, 5 each 5th request is attacked

1-10

5

chaos.monkey.assaults.latencyRangeStart

Minimum latency in ms added to the request

Integer.MIN_VALUE, Integer.MAX_VALUE

3000

chaos.monkey.assaults.latencyRangeEnd

Maximum latency in ms added to the request

Integer.MIN_VALUE, Integer.MAX_VALUE

15000

chaos.monkey.assaults.latencyActive

Latency assault active

TRUE or FALSE

TRUE

chaos.monkey.assaults.exceptionsActive

Exception assault active

TRUE or FALSE

FALSE

chaos.monkey.assaults.killApplicationActive

AppKiller assault active

TRUE or FALSE

FALSE

chaos.monkey.watcher.controller

Controller watcher active

TRUE or FALSE

FALSE

chaos.monkey.watcher.restController

RestController watcher active

TRUE or FALSE

FALSE

chaos.monkey.watcher.service

Service watcher active

TRUE or FALSE

TRUE

chaos.monkey.watcher.repository

Repository watcher active

TRUE or FALSE

FALSE

chaos.monkey.watcher.component

Component watcher active

TRUE or FALSE

FALSE

5. Watcher

A watcher is a Chaos Monkey for Spring Boot component, that will scan your app for a specific type of annotation.

Following Spring annotation are supported:

  • @Controller

  • @RestController

  • @Service

  • @Repository

  • @Component

With the help of Spring AOP, Chaos Monkey recognizes the execution of a public method and will either not execute any action or start one of its assaults. You can customize the behave by configuration.

6. Assaults

Assaults are the heart of Monkey’s Chaos and he makes use of them based on your configuration. Following assaults are actual provided:

  1. Latency Assault

    1. adds random latency to the request

  2. Exception Assault

    1. throws Chaos Monkey Runtime Exception, if he wants to (random)

  3. AppKiller Assault

    1. Yeah, what can I say? It destroys the entire application!

You can customize the behave of each Assault by configuration.

7. Changes in 2.0.0

7.1. Watcher

  • new Watcher for @Component

7.2. Endpoints

  • Spring Boot Actuator Endpoint /chaosmonkey/*

  • JMX Endpoint

7.3. Bugs fixed

7.4. Contributors

Thank you for your support!