Fault Tolerance and the Rate Limiter

Services now find each other and spread their calls. This chapter starts the module on what happens when things go wrong.

Fault-tolerant microservices

A fault-tolerant microservice keeps working when a service it depends on fails. Instead of crashing or passing the failure on, it handles it gracefully.

Microservices talk over the network, so one unhealthy service can bring down others. Normally the flow is simple: a client calls order-service, order-service calls product-service, and the responses come back.

Now a bug makes product-service slow: every call takes 60 seconds.

1. Each call from order-service to product-service blocks a thread for 60 seconds.
2. A burst of requests arrives at order-service. Every one of them blocks a thread waiting on product-service.
3. order-service's thread pool runs out. It starts rejecting requests, including ones that never touch product-service.
4. order-service's own callers now wait or fail, and their threads pile up.
5. Meanwhile the extra load makes product-service even slower.

The problem started in product-service and spread to everything upstream. This is a cascading failure.

Resilience4j

Resilience4j is a library of fault-tolerance mechanisms. The next chapters cover five of them, in this order:

RateLimiter → Bulkhead → TimeLimiter → CircuitBreaker → Retry

The order is logical. For example, limiting traffic should come before retrying it: retrying requests that the rate limiter will reject anyway only wastes work.

The rate limiter

A rate limiter controls how many requests are allowed in a time window. It protects a service from sudden traffic spikes, such as a burst from a misbehaving client or a denial-of-service attack. Rejected requests typically get HTTP 429 Too Many Requests.

There are several algorithms.

Fixed window counter

Split time into fixed windows and count the requests in each. Once the count reaches the limit, reject the rest of the window.

Limit 5, window 10 s: in window 0–10 s, five requests are accepted. In window 10–20 s, five are accepted and the sixth is rejected.

Weakness: bursts at the edges. If five requests arrive at the very end of one window and five at the very start of the next, ten requests pass in about one second, double the intended rate.

Sliding log

Store the timestamp of every accepted request. For a new request, count the timestamps in the last X seconds (the window, which slides with the clock). If the count has reached the limit, reject.

Limit 5, window 10 s: requests at 2, 5, 7 and 9 s are accepted. At 11 s the window is 1–11 s and holds four requests, so the request is accepted. At 12 s the window 2–12 s already holds five, so a new request is rejected.

Weaknesses: memory for one timestamp per accepted request, and the overhead of removing timestamps that fall out of the window. It is, however, exact.

Sliding window counter with sub-windows

Divide the main window into equal sub-windows, and keep a counter per sub-window instead of timestamps. To decide, add up the counters of the sub-windows in the current main window. The main window moves forward one sub-window at a time.

Limit 5, window 10 s, sub-windows of 2 s: a request at 9 s sums sub-windows 0–2, 2–4, 4–6, 6–8, 8–10 = 1+1+1+1+0 = 4, which is below 5, so it is accepted. After 10 s the window jumps to 2–12 s; sub-window 0–2 expires and 10–12 is added.

Weaknesses: more complex than a fixed window, and memory for the sub-window counters (less than one timestamp per request, though).

Sliding window counter with weighted window

Drop the sub-windows: keep only the count of each fixed window, and estimate the sliding window from the current and previous counts, weighted by how much the sliding window overlaps each:

requests in sliding window ≈ count(current fixed window) + overlap% × count(previous fixed window)

Limit 5, fixed windows of 10 s, five requests in 0–10 s:

• At 11 s, the sliding window (1–11 s) overlaps the previous window by 90%: 0 + 0.9 × 5 = 4.5, rounded up to 5. The limit is reached, so the request is rejected.
• At 12 s, the overlap is 80%: 0 + 0.8 × 5 = 4. Below the limit, so the request is accepted.

Weakness: it assumes requests were spread evenly across the previous window. If all five came at its very end, the true count for 2–12 s is six, over the limit, yet the request is allowed. It is an approximation.

Token bucket

A bucket holds tokens up to a capacity. Each request takes one token; with no token left, the request is denied. A refiller adds tokens at a fixed rate; tokens that arrive when the bucket is full overflow and are lost.

Capacity 4, one token every 6 s: requests at 1, 2, 3 and 4 s take the four tokens; the request at 5 s is denied; at 6 s a token is added and the next request gets it.

Weakness: bursts. After an idle period the bucket is full, and that many requests pass at once. Choose the capacity carefully: a capacity of 1,000 allows 1,000 requests in one second.

Leaky bucket

Requests go into a queue and are processed at a fixed rate, like water leaking from a hole in a bucket. If the queue is full, new requests overflow (HTTP 429).

Weaknesses: added latency, since requests wait in the queue, and the queue size is a trade-off: too small rejects a lot, too large adds a lot of latency and memory.

Step 1: Resilience4j in Spring Boot

Add the Resilience4j Spring Boot 3 module and Spring Boot's AOP starter:

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

The annotations work through AOP. Without the AOP starter they are silently ignored: the application starts, the annotated method runs every time, and nothing is ever limited.

Step 2: annotate the call

Keep the call to product-service in a service class. The controller stays plain:

@RestController
@RequestMapping("/orders")
public class OrderController {

    @Autowired
    OrderService orderService;

    @GetMapping("/{id}")
    public String callProductAPI(@PathVariable String id) {
        return orderService.invokeProductAPI(id);
    }
}

The Feign client from earlier chapters:

@FeignClient(name = "product-service")
public interface ProductClient {

    @GetMapping("/products/{id}")
    String getProductById(@PathVariable("id") String id);
}

Put @RateLimiter on the method that makes the call:

@Component
public class OrderService {

    @Autowired
    ProductClient productClient;

    @RateLimiter(name = "productRateLimiter", fallbackMethod = "rateLimitedFallback")
    public String invokeProductAPI(String id) {
        return productClient.getProductById(id);
    }

    public String rateLimitedFallback(String id, Throwable t) {
        return "Rate limit exceeded. Try later";      // or throw a custom exception
    }
}

• name identifies the rate limiter instance; its settings come from properties under that name. Each limiter is separate: productRateLimiter, salesRateLimiter...
• fallbackMethod names the method to call instead when the call is not permitted.

Because it works through an AOP proxy, the annotated method must be public, on a Spring bean, and called from another bean. A call from inside the same class skips the proxy and the limiter (as with any Spring AOP annotation).

Step 3: configure the limiter

# 2 calls per 10-second period; a call waits up to 1 s for a permission before being rejected
resilience4j.ratelimiter.instances.productRateLimiter.limit-for-period=2
resilience4j.ratelimiter.instances.productRateLimiter.limit-refresh-period=10s
resilience4j.ratelimiter.instances.productRateLimiter.timeout-duration=1s

Property	Meaning
limit-for-period	permissions available in each period
limit-refresh-period	length of a period; at the start of each one, permissions are reset to limit-for-period
timeout-duration	how long a call waits for a permission before being rejected (0 = reject at once)

Resilience4j's default limiter splits time into periods of limit-refresh-period. At the start of each period it resets the available permissions to limit-for-period. That works like a token bucket of capacity limit-for-period that is refilled completely at the start of every period. Unused permissions do not carry over.

Four quick calls in a row:

product 0       0 ms
product 1       0 ms
FALLBACK(RequestNotPermitted)   1001 ms
FALLBACK(RequestNotPermitted)   1001 ms

The first two take the period's permissions. The third waits the 1-second timeout-duration, gets nothing, and is rejected with RequestNotPermitted, which sends it to the fallback. After the 10-second period ends, two more calls pass.

Fallback rules

The fallback method must match the original method: same return type, same parameters, plus one extra parameter for the exception.

• It runs for any exception, not only rejections. If getProductById throws (product-service is down, say), the fallback is called with that exception, here IllegalStateException. A message like "rate limit exceeded" would then be wrong. Check the exception, or narrow the parameter type.
• The exception type selects the fallback. With fallback(String id, RequestNotPermitted e), only rate-limit rejections go to the fallback; other exceptions propagate unchanged. You can declare several fallbacks with the same name and different exception types; the most specific match is used.
• A fallback with the wrong signature is never called. If the parameters do not match, Resilience4j finds no fallback and the original exception (RequestNotPermitted: RateLimiter 'productRateLimiter' does not permit further calls) is thrown to the caller.

In a real API, the usual pattern is to throw a custom exception from the fallback and map it to 429 Too Many Requests with an @ExceptionHandler.

How it works: AOP

@RateLimiter is handled by an @Around aspect:

1. Intercept the call to the annotated method.
2. Ask the rate limiter named in the annotation for a permission (waiting up to timeout-duration).
3. If granted, call proceed(), running the method. If not, throw RequestNotPermitted.
4. If anything threw, look for a matching fallback.

The same annotation works whatever the method does inside: Feign, RestTemplate or RestClient.

Your own limiter

Since it is only an aspect, you can write your own with a different algorithm, the same way as the custom interceptor in the Spring Boot book. Define an annotation:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface CustomRateLimiter {
    int limit();              // max requests
    int windowInSeconds();    // sliding window duration
}

Write the aspect:

@Aspect
@Component
public class RateLimiterAspect {

    @Around("@annotation(customRateLimiter)")
    public Object rateLimit(ProceedingJoinPoint pjp, CustomRateLimiter customRateLimiter)
            throws Throwable {

        // your algorithm (sliding log, sliding window...): reject or continue
        return pjp.proceed();
    }
}

Use it:

@CustomRateLimiter(limit = 5, windowInSeconds = 60)
public String getProducts() {
    return "List of products";
}

Next: A rate limiter caps how many calls come in. Chapter 7 caps how many run at the same time, with the bulkhead, so one slow dependency cannot take every thread.

Check yourself

1. What makes a microservice fault-tolerant?
2. Describe how a slow product-service causes a cascading failure.
3. In what order do the next chapters apply the five Resilience4j mechanisms, and why does the rate limiter come before retry?
4. What is the weakness of the fixed window counter?
5. What does the sliding log store, and what does that cost?
6. How does the sliding window counter with sub-windows decide, and how does its window move?
7. Why can the weighted sliding window allow more requests than the limit?
8. What is the main risk of a token bucket?
9. What are the trade-offs of a leaky bucket?
10. What happens to @RateLimiter if spring-boot-starter-aop is missing?
11. With limit-for-period=2, limit-refresh-period=10s, timeout-duration=1s, what happens to four quick calls?
12. When is a fallback with parameters (String id, Throwable t) called?
13. What happens if the fallback's signature does not match?

Answers

1. It keeps working when a service it depends on fails, handling the failure gracefully instead of crashing or passing it on.
2. Each call blocks an order-service thread for 60 s; a burst uses up order-service's thread pool, so it rejects all requests; its callers then block in turn, and the extra load slows product-service further.
3. RateLimiter, Bulkhead, TimeLimiter, CircuitBreaker, Retry. Retrying requests the rate limiter will reject only wastes work.
4. Bursts at window edges: up to twice the limit can pass in a short time spanning two windows.
5. A timestamp per accepted request. It costs memory and the work of removing expired timestamps.
6. It sums the counters of the sub-windows inside the main window and rejects at the limit; the window jumps forward one sub-window at a time.
7. It assumes the previous window's requests were spread evenly; if they were bunched at its end, the estimate is too low.
8. Bursts: a full bucket after idle time lets that many requests through at once.
9. Requests wait in the queue (more latency), and the queue size must balance rejections against latency and memory.
10. It is silently ignored: the method runs on every call with no limiting.
11. The first two pass; the third and fourth each wait 1 s, are rejected with RequestNotPermitted, and go to the fallback.
12. For any exception thrown by the call or the limiter (a RequestNotPermitted, or e.g. product-service failing).
13. No fallback is found, and the original exception is thrown to the caller.