The Circuit Breaker

Retry handles failures that pass quickly. Now suppose product-service is down, not for a moment but for minutes. Every call from order-service will fail, retried or not. Calling anyway hurts both sides:

• product-service gets extra load while it is trying to recover, so it may take longer to come back.
• order-service wastes its own resources: each call holds a thread and adds latency until it times out, only to fail.

order-service should stop calling product-service while it is failing and start again once it has recovered. But how does it know when to stop and when to restart? That knowledge is what a circuit breaker provides: it prevents an application from making repeated calls to a downstream service that is likely to fail.

The idea:

1. Count the calls that fail. Once failures pass a threshold, stop calling: fail every call immediately, without touching the network or blocking a thread.
2. After a wait, let a few trial calls through.
3. If they succeed, resume normal calls; if not, stop again.

The three states

• CLOSED: the normal state. All calls go through to the downstream service, while the breaker tracks their results.
• OPEN: no calls go through; every call fails at once.
• HALF_OPEN: only a limited number of trial calls go through, and their results decide the next state.

Think of an electrical circuit: when it is closed, current flows; when it opens, the flow stops. "Closed" is the healthy state, which is easy to get backwards.

Step 1: annotate the call

Dependencies as before (resilience4j-spring-boot3, spring-boot-starter-aop).

@Component
public class OrderService {

    @Autowired
    ProductClient productClient;

    @CircuitBreaker(name = "productService", fallbackMethod = "fallback")
    public String invokeProductAPI(String id) {
        return productClient.getProductById(id);
    }

    public String fallback(String id, Throwable ex) {
        return "not able to invoke product service";
    }
}

The fallback runs for every failed call: a call that reached product-service and failed, and a call the open circuit refused.

Step 2: configure it

resilience4j.circuitbreaker.instances.productService.sliding-window-type=COUNT_BASED
resilience4j.circuitbreaker.instances.productService.sliding-window-size=10
resilience4j.circuitbreaker.instances.productService.minimum-number-of-calls=5
resilience4j.circuitbreaker.instances.productService.failure-rate-threshold=50
resilience4j.circuitbreaker.instances.productService.wait-duration-in-open-state=10s
resilience4j.circuitbreaker.instances.productService.permitted-number-of-calls-in-half-open-state=3
resilience4j.circuitbreaker.instances.productService.automatic-transition-from-open-to-half-open-enabled=true

The window: which calls count. In CLOSED state, the breaker computes the failure rate over a sliding window of recent calls:

• sliding-window-type=COUNT_BASED with sliding-window-size=10: the last 10 calls.
• sliding-window-type=TIME_BASED with sliding-window-size=10: the calls of the last 10 seconds, however many.

(This window only measures failures. It has nothing to do with the rate limiter's windows in chapter 6.)

minimum-number-of-calls: the breaker does not compute a failure rate until the window holds at least this many calls. Without it, one failure out of two calls would be a 50% failure rate and could open the circuit: a false alarm based on too little data.

failure-rate-threshold: the percentage of failed calls at which the circuit opens.

wait-duration-in-open-state: how long the circuit stays OPEN before trying again.

permitted-number-of-calls-in-half-open-state: how many trial calls HALF_OPEN lets through.

automatic-transition-from-open-to-half-open-enabled: with true, the breaker moves to HALF_OPEN on its own when the wait ends. With false (the default), it stays OPEN until the first call after the wait, which then triggers the move.

Step 3: watch it work

Run with product-service failing, and the wait shortened to 3 s for the test. After each call, print the state:

call 1   downstream called -> fallback (IllegalStateException)   CLOSED   buffered 1
call 2   downstream called -> fallback (IllegalStateException)   CLOSED   buffered 2
call 3   downstream called -> fallback (IllegalStateException)   CLOSED   buffered 3
call 4   downstream called -> fallback (IllegalStateException)   CLOSED   buffered 4
call 5   downstream called -> fallback (IllegalStateException)   OPEN     failure rate 100%
call 6   (not called)      -> fallback (CallNotPermittedException)   OPEN
call 7   (not called)      -> fallback (CallNotPermittedException)   OPEN
         ... 3 s later: HALF_OPEN
trial 1  downstream called -> ok                                  HALF_OPEN
trial 2  downstream called -> fallback (IllegalStateException)   HALF_OPEN
trial 3  downstream called -> ok                                  CLOSED

• Calls 1 to 4: the window has fewer than 5 calls (minimum-number-of-calls), so no failure rate is computed (-1) and the circuit stays closed.
• Call 5: now there are 5 calls, all failed: 100% ≥ 50%. The circuit opens.
• Calls 6 and 7: product-service is not called at all; the breaker throws CallNotPermittedException at once.
• After the wait, the circuit moves to HALF_OPEN by itself.
• Trials: 1 of the 3 trial calls failed, a failure rate of 33%, below the 50% threshold, so the circuit closes. Had 2 or 3 of them failed, it would have gone back to OPEN for another wait.

So HALF_OPEN does not need 100% success: the trial calls are judged against the same failure-rate-threshold.

Which exceptions count as failures

By default every exception thrown by the method is recorded as a failure. To choose, list them (comma-separated, subclasses included):

resilience4j.circuitbreaker.instances.productService.record-exceptions=java.io.IOException,feign.FeignException$FeignServerException
resilience4j.circuitbreaker.instances.productService.ignore-exceptions=java.lang.IllegalArgumentException

With record-exceptions, only those count as failures. With ignore-exceptions, those count neither as failures nor as successes. As with retry, a 4xx caused by bad input says nothing about product-service's health, so it is usually ignored.

How it works inside

The aspect hands each call to Resilience4j's CircuitBreakerStateMachine, which holds the current state and the transition rules. After each recorded result it checks whether the state must change. For example, in CLOSED:

private void checkIfThresholdsExceeded(Result result) {
    if (Result.hasExceededThresholds(result) && isClosed.compareAndSet(true, false)) {
        publishCircuitThresholdsExceededEvent(result, circuitBreakerMetrics);
        transitionToOpenState();
    }
}

Moves out of CLOSED and HALF_OPEN are driven by call results. But OPEN → HALF_OPEN must happen after a time, with no call involved. When the circuit opens with automatic transition enabled, the OpenState constructor schedules the move:

if (circuitBreakerConfig.isAutomaticTransitionFromOpenToHalfOpenEnabled()) {
    ScheduledExecutorService scheduledExecutorService = schedulerFactory.getScheduler();
    transitionToHalfOpenFuture = scheduledExecutorService
            .schedule(this::toHalfOpenState, waitDurationInMillis, TimeUnit.MILLISECONDS);
}

schedule(task, delay, unit) hands the task to a ScheduledThreadPoolExecutor to run once the delay has passed.

How a scheduled executor waits

A ScheduledThreadPoolExecutor keeps its tasks in a delayed queue: a priority queue ordered by when each task is due, so the task due soonest is at the head.

1. An idle worker thread looks at the head of the queue.
2. If that task is not due yet, the thread blocks for exactly the remaining delay; it does not spin.
3. When the delay ends, the thread wakes, takes the task and runs it.
4. Other idle threads look at the next task in the queue and wait for it the same way.

The same structure can implement many "do this later" features, such as messages that delete themselves after 24 hours.

Next: The resilience patterns protect calls between services. Chapter 10 turns to the front door: an API gateway that gives clients one entry point and routes each request to the right service.

Check yourself

1. Why is it harmful to keep calling a service that is down?
2. What happens to calls in each of the three states?
3. Why is CLOSED the healthy state?
4. What is the difference between a count-based and a time-based sliding window?
5. Why does minimum-number-of-calls exist?
6. With window 10, minimum 5 and threshold 50%, on which call does the circuit open if every call fails?
7. What exception does an OPEN circuit throw, and is the downstream called?
8. What does automatic-transition-from-open-to-half-open-enabled=false change?
9. With 3 trial calls and a 50% threshold, one trial fails and two succeed. What happens?
10. Which exceptions count as failures by default, and how do you change that?
11. How does the breaker move from OPEN to HALF_OPEN without any call?
12. How does a ScheduledThreadPoolExecutor wait for a task that is not yet due?

Answers

1. It adds load to a service that is trying to recover, and wastes the caller's threads and time on calls that will fail anyway.
2. CLOSED: all calls go through and are tracked. OPEN: none go through; they fail immediately. HALF_OPEN: a limited number of trial calls go through to test recovery.
3. Like an electrical circuit: closed means the flow (of calls) passes; open means it is cut.
4. Count-based uses the last N calls; time-based uses all calls from the last N seconds.
5. To avoid judging on too few calls: one failure out of two would be 50% and could open the circuit on a false alarm.
6. The fifth: before it there are fewer than 5 calls; at 5, the failure rate is 100%.
7. CallNotPermittedException; the downstream is not called.
8. The breaker no longer moves to HALF_OPEN on a timer; it stays OPEN until the first call after the wait duration triggers the move.
9. The trial failure rate is 33%, below 50%, so the circuit closes.
10. All exceptions. Use record-exceptions to count only listed ones, and ignore-exceptions to exclude some.
11. When it opens, it schedules the transition task on a ScheduledExecutorService with the wait duration as the delay.
12. Its tasks sit in a delayed queue ordered by due time; an idle worker blocks for the remaining delay of the head task, then wakes and runs it.