Circuit breaker implementation in spring boot2

What is it

Circuit breaker is a design pattern used in modern software development. It is used to detect failures and encapsulates the logic of preventing a failure from constantly recurring, during maintenance, temporary external system failure or unexpected system difficulties. The Circuit breaker pattern helps to prevent such a catastrophic cascading failure across multiple systems. The circuit breaker pattern allows you to build a fault tolerant and resilient system that can survive gracefully when  key services are either unavailable or have high latency.

White box libraries

There are many libraries which claim that they have successfully implemented the Circuit Breaker Design Pattern. Few of them are:

1. Hystrix: Opensource Library designed by Netflix. This is the best library available as of today but it is no longer in active development, and is currently in maintenance mode. Because of this reason we have kept it out from our scope.
2. Sentinel: Opensource Library designed and maintained by Alibaba. Currently active and have good integration with spring boot as well as open feign. We can consider it.
3. Resilience4j: Opensource library designed and maintained by opensource community. Relatively young and provides integration with spring-boot2 and open feign. We will see the details below.
4. Failsafe: Opensource library designed and maintained by opensource community. No analysis done as it seems it may not have good integration with our existing architecture.
5. jRugged: Opensource library designed and maintained by Comcast community. No analysis done as it seems it may not have good integration with our existing architecture.
6. Spring Cloud CircuitBreaker: Opensource library designed and maintained by Spring Community. Currently in incubator stage and only SNAPSHOT builds are available on spring repo. 

Black box integrations

1. ISTIO: Istio monitors the system from the outside and does not know how the system works internally. It should be integrated with Kubernetes in order to achieve the desired functionality.

Circuit Breaker States

1. CLOSED: The circuit is normally behaving and routing all the calls to desired services.
2. OPEN: The circuit is OPEN and does not allows calls to the service.
3. HALF_OPEN: This is intermediate state where circuit breaker tries to check if the service, which was reported erroneous, is back to normal function. In this state the circuit breaker allows few calls to the service being called. If the calls gets successful then the circuit breaker state changes to CLOSED or OPEN again.

The following diagram depicts these 3 states

Behaviour in CLOSED State

Behaviour in OPEN State

Behaviour in HALF-OPEN State

Resilience4j

Resilience4j is a lightweight, easy-to-use fault tolerance library inspired by Netflix Hystrix, but designed for Java 8 and functional programming. There are 4 core functions of this library:

1. CircuitBreaker
2. BulkHead
3. RateLimiter
4. Retry

Our scope is limited to use the CircuitBreaker function.
The following are the properties which are of our interest:

• registerHealthIndicator: Registers the circuit breakers with the actuator /health endpoint
• ringBufferSizeInClosedState: The number of calls after which circuit breaker can take any action. It does not matter if all the calls are Success or all of them are failure. The circuit breaker will not take action until the ring buffer is full. Default 100
• ringBufferSizeInHalfOpenState: The size of the ring buffer when the CircuitBreaker is half-open. This ring buffer is used when the breaker transitions from open to half-open to decide whether the circuit is healthy or not. This is helpful when we does not want to CLOSE the circuit and wait till the ringBuffer is full again in cases when the underlying system is still down. Default 10
• automaticTransitionFromOpenToHalfOpenEnabled: This property will allow automatic state transitions from open to half open circuit and allows limited traffic to flow to check if circuit can be closed. Default false
• waitDurationInOpenState: The time in seconds to wait in OPEN state. Default 60
• failureRateThreshold: The failure rate in % after which the circuit transition will trigger. Default 50
• recordExceptions: List of exceptions which should count as failure. Default empty i.e. all the exceptions will be recorded as a failure.
• ignoreExceptions: List of exceptions which should be ignored. Default empty i.e. no exceptions will be ignored an all of them will be recorded as failure.

To integrate a circuit breaker with spring boot microservice, we need the following dependencies on the classpath:

<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>io.github.resilience4j</groupId> <artifactId>resilience4j-spring-boot2</artifactId> <version>0.17.0</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-aspects</artifactId> </dependency> <dependency> <groupId>io.github.resilience4j</groupId> <artifactId>resilience4j-feign</artifactId> <version>0.17.0</version> </dependency>

We want to limit the calls to the downstream if it is down or is behaving unexpectedly. Here with resilience4j we can use Circuit Breaker function at the following 2 places:

1. At any class like controller or service or dao etc.
2. At the Feign interface.

1.   Service:

• For any service class annotate the class with @CircuitBreaker annotation on either the class level or on each method. For eg @CircuitBreaker(name = "methodA”, fallbackMethod = “errorMethodA”
• The fallback method is the method which will be executed if there are any exceptions from the service which is annotated with @CircuitBreaker. The fallback method should be kept in the same class with the same signature except 1 additional parameter i.e. Throwable throwable.
• If there are multiple fallbackMethod methods, the method that has the most closest match will be invoked, for example if you try to recover from NumberFormatException, the method with signature String fallback(String parameter, IllegalArgumentException exception)} will be invoked.
• When the fallback is added then it gets executed on all the kind exceptions no matter if the circuit is OPEN or CLOSED or HALF_OPEN. If the circuit is OPEN then all the calls to the method will end to the fallback method without calling the underlying layer.
• You can define one global fallback method with an exception parameter only if multiple methods has the same return type and you want to define the same fallback method for them once and for all.
• You can define a list of exceptions which should be ignored by circuit breaker OR which should ONLY be recorded by circuit breaker.

Example service:

@Service public class Resilience4jService { @Autowired private RestTemplate restTemplate; @CircuitBreaker(name = "helloWorld", fallbackMethod = "errorInternal") public String helloWorld(String type) { if (type.equals("error")) { System.out.println("1. Call received means circuit is closed"); throw new RuntimeException("This is runtime exception"); } else { return "Hello world from Resilience4jService"; } } @CircuitBreaker(name = "callRemoteService", fallbackMethod = "errorExternal") public String callRemoteService(String success) { System.out.println("2. Call received means circuit is closed"); if (Objects.nonNull(success)) { return restTemplate.getForObject(URI.create("http://localhost:8090/sample?success="+success), String.class); } return restTemplate.getForObject(URI.create("http://localhost:8090/sample"), String.class); } public String alwaysError() { throw new RuntimeException("I will always throw error"); } /** * Fallback method for 'helloWorld' method * @param methodParam * @param throwable * @return */ public String errorInternal(String methodParam, Throwable throwable) { return "This is error from fallback triggered by internal error"; } /** * Fallback method for 'callRemoteService' method * @param throwable * @return */ public String errorExternal(Throwable throwable) { return "This is error from fallback triggered by external error"; } }

Feign:

• As of now there is no annotation support of resilience4j with openfeign library for ex. like the annotation @FeignClient provided by spring cloud.
• I have submitted a PR here to the resilience4j community to allow using @FeignClient with @CircuitBreaker. If this PR gets approved and merged then we need not to do the following DSL configuration.
• For now we need to use raw openfeign i.e. to remove the @FeignClient annotation on the client.
• For now we need to configure the Feign client in one of the @Configuration class. Sample code which registers MyFeignClient is as following:

@Configuration public class ResilienceFeignConfig { @Bean public MyFeignClient myFeignClient(CircuitBreakerRegistry registry) { CircuitBreaker circuitBreaker = registry.circuitBreaker("myFeignClient"); // "myFeignClient" is picked from application.yml by property resilience4j.circuitbreaker.instances.myFeignClient MyFeignClient requestFailedFallback = () -> "Fallback on FeignException"; MyFeignClient circuitBreakerFallback = () -> "CircuitBreaker is open!"; FeignDecorators decorators = FeignDecorators.builder() .withCircuitBreaker(circuitBreaker) //.withFallbackFactory(MyFallback::new) .withFallback(requestFailedFallback, FeignException.class) .withFallback(circuitBreakerFallback, CallNotPermittedException.class) .build(); return Resilience4jFeign.builder(decorators) // This is needed to use Spring mvc annotations on feign client .contract(new SpringMvcContract()) .target(MyFeignClient.class, "http://localhost:8090/"); } }

The FeignClient should be changed to the following:

public interface MyFeignClient { @GetMapping(path = "/sample") String feignMessage(); }

• The working sample POC could be found at: https://github.com/harishmbm/resilience4j-springboot
• The support of @FeignClient with @CircuitBreaker is not going to be coming in near future from the community.

application.yml configuration

resilience4j.circuitbreaker: configs: default: registerHealthIndicator: true ringBufferSizeInClosedState: 4 ringBufferSizeInHalfOpenState: 2 automaticTransitionFromOpenToHalfOpenEnabled: true waitDurationInOpenState: 20s failureRateThreshold: 50 eventConsumerBufferSize: 10 shared: registerHealthIndicator: true ringBufferSizeInClosedState: 4 ringBufferSizeInHalfOpenState: 2 waitDurationInOpenState: 20s failureRateThreshold: 50 eventConsumerBufferSize: 10 ignoreExceptions: - com.resilience4j.exception.BusinessException instances: helloWorld: baseConfig: default ringBufferSizeInClosedState: 4 registerHealthIndicator: true recordExceptions: - java.lang.RuntimeException callOtherService: baseConfig: default ringBufferSizeInClosedState: 4 registerHealthIndicator: true ignoreExceptions: # We need to ignore 4xx errors returned by the server as they are valid business case. - org.springframework.web.client.HttpClientErrorException recordExceptions: # We need to record all http errors - org.springframework.web.client.RestClientException myFeignClient: baseConfig: default ringBufferSizeInClosedState: 4 registerHealthIndicator: true ignoreExceptions: - feign.FeignException.BadRequest - feign.FeignException.Unauthorized - feign.FeignException.Forbidden - feign.FeignException.NotFound - feign.FeignException.MethodNotAllowed - feign.FeignException.NotAcceptable - feign.FeignException.Conflict - feign.FeignException.Gone - feign.FeignException.UnsupportedMediaType - feign.FeignException.TooManyRequests - feign.FeignException.UnprocessableEntity recordExceptions: - java.net.SocketTimeoutException - java.net.ConnectException - feign.FeignException.InternalServerError - feign.FeignException.NotImplemented - feign.FeignException.BadGateway - feign.FeignException.GatewayTimeout - feign.FeignException.ServiceUnavailable - feign.RetryableException management: health: status: http-mapping: DOWN: 200 management.endpoints.web.exposure.include: '*' management.endpoint.health.show-details: always management.metrics.tags.application: harish-demo management.metrics.distribution.percentiles-histogram.http.server.requests: true management.metrics.distribution.percentiles-histogram.resilience4j.circuitbreaker.calls: true

Following are the cases tested to verify the fallback.

S.No	Case	Type	Application Configuration in yml	Result
1	No fallback defined	Circuit breaker at FeignClient		Fallback did not executed
2	Fallback defined	Circuit breaker at FeignClient	Ignored FeignException	Fallback Executed
3	Fallback defined	Circuit breaker at FeignClient	Recorded FeignException	Fallback Executed
4	Http errors from endpoint 2 (call to 3rd party)	Circuit breaker at Service method for endpoint 2	Recorded RestClientException	Fallback executed and Circuit opened after 4 calls. Endpoint 1 still working fine
5	Internal errors from endpoint 1 (no calls to 3rd party)	Circuit breaker at Service method for endpoint 1	Recorded RuntimeException	Fallback executed and Circuit opened after 4 calls. Endpoint 2 still working fine
6	Bad Request from 3rd party	Circuit breaker at Service method for endpoint 2	Ignored HttpClientErrorException	Fallback executed, ignored actual server error and circuit remained closed.	https://github.com/resilience4j/resilience4j/issues/563

Bugs in library:

1. Fallback does not works properly https://github.com/resilience4j/resilience4j/issues/560
2. No proper support with @FeignClient annotation https://github.com/resilience4j/resilience4j/issues/559. PR is raised by me here https://github.com/resilience4j/resilience4j/pull/579
3. Gulps the original error message sent by the underlying layer and wraps the response in its own json format. https://github.com/resilience4j/resilience4j/issues/563. This may be from Feign.
   

Note: Since we have exception handlers defined per service hence we can add a handler method which can listen on the exceptions thrown by the circuit breaker. This was we can build our own custom message if circuit is OPEN. See example as below:

@ControllerAdvice(assignableTypes = Resilience4jController.class) public class BenefitControllerAdvice extends ResponseEntityExceptionHandler { @ExceptionHandler(io.github.resilience4j.circuitbreaker.CallNotPermittedException.class) public ResponseEntity<Object> handleAll(Exception ex) { return buildResponseEntity(INTERNAL_SERVER_ERROR); } public ResponseEntity<Object> buildResponseEntity(HttpStatus httpStatus) { return status(httpStatus).body("{Circuit breaker is OPEN or HALF_OPEN}"); } }