Circuit Breaker: Fault Tolerance in Distributed Microservices

The Electrical Fuse Box Analogy

In your home electrical system, every room draws power from a central breaker panel. If an appliance in your kitchen malfunctions and draws a dangerous power surge, the breaker switch for the kitchen trips instantly, physically severing the circuit. The lights in the kitchen turn off, but the rest of the house remains powered. The refrigerator continues running, the living room lights stay on, and your house is protected from an electrical fire. Once you unplug the broken toaster, you reset the breaker switch to restore normal current.

In a microservices architecture, a Circuit Breaker acts as that electrical fuse box. In distributed systems, services constantly depend on one another. If the Payment Service experiences database locks or network degradation and begins taking thirty seconds to respond, calling services like the Order Service will wait, keep connections open, exhaust thread pools, and eventually crash. Soon, the entire platform collapses in a cascading failure.

A circuit breaker wraps remote calls. It monitors failure rates. When failures exceed a defined threshold, the circuit breaker trips open, rejecting subsequent calls immediately without waiting, and directs traffic to a graceful fallback method.

This lecture covers cascading failures, Resilience4j CircuitBreaker architecture, sliding window metrics, state transitions (CLOSED, OPEN, HALF_OPEN), and fallback method design.

---

The Danger of Cascading Failures

Consider a chain of microservice invocations:

[User Request] -> [ Order Service ] -> [ Payment Service ] -> [ External Bank API ]

What happens when the External Bank API slows down or stops responding?

1. Payment Service threads wait for the External Bank API, holding open HTTP connections.
2. Payment Service's thread pool becomes completely exhausted.
3. Order Service calls Payment Service, which now fails to respond.
4. Order Service's thread pool fills up with blocked requests.
5. Soon, Order Service can no longer handle even simple read requests for existing orders.
6. The failure cascades from the bank API all the way to the frontend user interface.

Bank API Outage -> Payment Service Thread Exhaustion -> Order Service Crash -> Platform Down

A circuit breaker breaks this chain by failing fast, shedding load, and providing an alternate response before thread pools deplete.

---

Resilience4j Circuit Breaker States

A circuit breaker operates as a finite state machine with three primary states:

          Failures exceed threshold
     +---------------------------------> [ OPEN ]
     |                                      |
     |                                      | Wait duration expires
     |                                      v
 [ CLOSED ] <------------------------ [ HALF OPEN ]
(Normal Flow)    Trial calls succeed         |
     ^                                       | Trial calls fail
     +---------------------------------------+

1. CLOSED (Normal Operation)

• The circuit breaker is fully connected.
• All requests are permitted to pass through to the downstream service.
• The circuit breaker monitors responses, recording successes, failures, and slow calls in a rolling sliding window.
• As long as the failure percentage stays below the configured threshold, the circuit remains CLOSED.

2. OPEN (Fail Fast)

• When the failure rate or slow call rate exceeds the configured threshold, the circuit trips to OPEN.
• Zero requests are permitted through to the downstream service.
• All incoming requests fail immediately or execute the configured fallback method with zero network latency.
• This gives the struggling downstream service room to recover and prevents calling services from exhausting thread pools.

3. HALF_OPEN (Trial Probe)

• After a configurable wait duration in the OPEN state (e.g. thirty seconds), the circuit transitions to HALF_OPEN.
• A small, limited number of trial requests (e.g. five requests) are permitted to pass through to the downstream service.
• If the trial requests succeed, the downstream service has recovered. The circuit transitions back to CLOSED.
• If even one trial call fails, the circuit trips directly back to OPEN for another wait period.

---

Sliding Window Metrics: Count Based vs Time Based

To evaluate whether the failure rate has exceeded the threshold, Resilience4j records results inside a sliding window:

1. Count Based Sliding Window (COUNT_BASED)

Maintains an array recording the last N calls (e.g. the last 100 calls). When call number 101 arrives, the oldest call record is evicted:

resilience4j.circuitbreaker.instances.paymentService.slidingWindowType=COUNT_BASED
resilience4j.circuitbreaker.instances.paymentService.slidingWindowSize=100

2. Time Based Sliding Window (TIME_BASED)

Maintains an aggregation of calls over the last N seconds (e.g. the last sixty seconds):

resilience4j.circuitbreaker.instances.paymentService.slidingWindowType=TIME_BASED
resilience4j.circuitbreaker.instances.paymentService.slidingWindowSize=60

---

Implementing Resilience4j Circuit Breaker in Spring Boot

1. Add Dependencies

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-circuitbreaker-resilience4j</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

2. Configure Properties in application.properties

# Sliding window configuration
resilience4j.circuitbreaker.instances.paymentService.slidingWindowType=COUNT_BASED
resilience4j.circuitbreaker.instances.paymentService.slidingWindowSize=10
resilience4j.circuitbreaker.instances.paymentService.minimumNumberOfCalls=5

# Thresholds
resilience4j.circuitbreaker.instances.paymentService.failureRateThreshold=50
resilience4j.circuitbreaker.instances.paymentService.slowCallRateThreshold=50
resilience4j.circuitbreaker.instances.paymentService.slowCallDurationThreshold=2s

# State transition timing
resilience4j.circuitbreaker.instances.paymentService.waitDurationInOpenState=10s
resilience4j.circuitbreaker.instances.paymentService.permittedNumberOfCallsInHalfOpenState=3
resilience4j.circuitbreaker.instances.paymentService.automaticTransitionFromOpenToHalfOpenEnabled=true

Key configuration meanings:

• minimumNumberOfCalls=5: The circuit will not evaluate failure rates until at least five calls have been recorded.
• failureRateThreshold=50: If 50% or more calls fail, trip to OPEN.
• waitDurationInOpenState=10s: Stay in OPEN state for ten seconds before entering HALF_OPEN.

---

Using @CircuitBreaker and Fallback Methods

Annotate the method making the remote call with @CircuitBreaker:

package com.example.orderservice.service;

import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;

@Service
public class OrderService {

    @Autowired
    private RestTemplate restTemplate;

    @CircuitBreaker(name = "paymentService", fallbackMethod = "paymentFallback")
    public String processPayment(String orderId, double amount) {
        System.out.println("Calling Payment Service for order: " + orderId);
        String url = "http://PAYMENT-SERVICE/payment/charge?orderId=" + orderId + "&amount=" + amount;
        return restTemplate.postForObject(url, null, String.class);
    }

    // Fallback method must have identical parameter types plus Throwable as final parameter
    public String paymentFallback(String orderId, double amount, Throwable throwable) {
        System.err.println("Circuit Breaker activated! Reason: " + throwable.getMessage());
        return "PAYMENT_PENDING: Payment service is currently unavailable. Order " + orderId + " queued for later processing.";
    }
}

Strict Rules for Fallback Methods:

1. The fallback method must reside in the same class as the annotated method.
2. The fallback method signature must match the annotated method's parameter list plus a Throwable parameter at the end.
3. The return type of the fallback method must be identical to the return type of the original method.

---

Monitoring Circuit Breaker via Actuator

Resilience4j publishes metrics directly to Spring Boot Actuator endpoints:

1. View all circuit breakers:

   GET http://localhost:8081/actuator/circuitbreakers

2. View circuit breaker events (state transitions, rejected calls, fallbacks):

   GET http://localhost:8081/actuator/circuitbreakerevents

Sample actuator response showing CLOSED state:

{
  "circuitBreakers": {
    "paymentService": {
      "state": "CLOSED",
      "failureRate": "0.0%",
      "slowCallRate": "0.0%",
      "failureRateThreshold": "50.0%",
      "slowCallRateThreshold": "50.0%",
      "bufferedCalls": 4,
      "failedCalls": 0,
      "slowCalls": 0
    }
  }
}

---

Interview Questions & Pitfalls

Q1: What are the three primary states of a Circuit Breaker, and what causes the transitions between them?

The three states are CLOSED (normal operation), OPEN (downstream service is failing; calls are rejected immediately), and HALF_OPEN (trial state testing whether downstream has recovered). The transition from CLOSED to OPEN occurs when failure rate or slow call rate exceeds the threshold. Transition from OPEN to HALF_OPEN occurs after a configured wait duration. Transition from HALF_OPEN to CLOSED occurs if trial calls succeed, or back to OPEN if trial calls fail.

Q2: What is the purpose of the minimumNumberOfCalls configuration in Resilience4j?

minimumNumberOfCalls prevents the circuit breaker from tripping prematurely on low sample sizes. For instance, if you configure a 50% failure rate threshold and the very first call fails, the instantaneous failure rate is 100%. Without minimumNumberOfCalls, a single transient failure would trip the circuit. Setting minimumNumberOfCalls=10 ensures at least ten calls are recorded before calculating failure rates.

Q3: What are the strict signature requirements for a fallback method in Resilience4j?

The fallback method must belong to the same class, return the exact same type as the annotated method, accept all parameters of the annotated method in identical order, and accept a final parameter of type Throwable (or a specific exception subclass) to inspect the error that triggered the fallback.

Q4: What is the difference between a count based sliding window and a time based sliding window?

A count based sliding window evaluates the last N invocations regardless of when they occurred. A time based sliding window evaluates calls that occurred within the last N seconds, evicting calls older than the time window.

Q5: What happens when a request arrives while the circuit breaker is in the OPEN state?

The circuit breaker intercepts the call immediately without executing the target method. It throws a CallNotPermittedException with zero network delay. If a fallback method is defined, Resilience4j executes the fallback directly.