The Bulkhead and the Time Limiter

Chapter 6 limited how many requests come in. This chapter limits how many calls go out at the same time.

Rate limiter or bulkhead?

The two are easy to confuse, so set them side by side, from the point of view of order-service:

	Rate limiter	Bulkhead
Protects order-service from	its clients (callers)	its downstream services
Limits	requests per time window	concurrent calls
Concurrency	not considered: 10 per minute may come all at once or one by one	the whole point

Two problems a bulkhead solves

Use case 1: a fragile downstream. product-service is small and can handle only three concurrent requests. It has no protection of its own. order-service must never have more than three calls to it in flight; a fourth must wait or be rejected. The semaphore bulkhead solves this.

Use case 2: a slow downstream starving a fast one. order-service has two endpoints:

• API 1 calls product-service, which answers in about 100 ms.
• API 2 calls payment-service, which takes about 5 s.

Traffic to API 2 spikes. Each call holds an order-service thread for 5 seconds. Even if the calls are not simultaneous, the slow ones pile up until every thread in order-service's pool is waiting on payment-service. Now API 1 cannot run either, although its downstream is fast and its traffic is normal. One slow dependency took the whole service down.

The name comes from ships: a hull is divided into watertight compartments, bulkheads, so a leak floods one compartment and not the whole ship. Give payment calls their own small pool of threads, say five. However many payment calls arrive, they cannot use more than those five, and API 1 keeps working. The thread-pool bulkhead solves this. (It is a narrow case of the noisy neighbor problem, where one tenant uses up resources shared with everyone else.)

The semaphore bulkhead

It counts the calls currently inside the protected method and allows at most N at once, using a semaphore. A semaphore holds N permits: a thread takes one to enter and returns it on leaving; with none left, the thread waits or gives up. The call runs on the caller's own thread.

Step 1: dependencies

The same as chapter 6: resilience4j-spring-boot3 plus spring-boot-starter-aop.

Step 2: annotate the call

@Component
public class OrderService {

    @Autowired
    ProductClient productClient;

    @Bulkhead(name = "productService",
              type = Bulkhead.Type.SEMAPHORE,
              fallbackMethod = "productFallback")
    public String invokeProductAPI(String id) {
        return productClient.getProductById(id);
    }

    public String productFallback(String id, Throwable t) {
        return "Too many concurrent requests, please try again later";
    }
}

SEMAPHORE is the default type. The fallback follows the same rules as in chapter 6.

Step 3: configure it

# at most 2 calls at once; a third is rejected immediately
resilience4j.bulkhead.instances.productService.max-concurrent-calls=2
resilience4j.bulkhead.instances.productService.max-wait-duration=0

• max-concurrent-calls: permits in the semaphore (default 25).
• max-wait-duration: how long a call waits for a permit before being rejected (default 0). It takes a number and a unit: 0, 300ms, 2s, 1m.

Four simultaneous requests, with product-service taking 2 seconds:

product 1                                @2015 ms
product 3                                @2015 ms
FALLBACK 2  BulkheadFullException        @14 ms
FALLBACK 4  BulkheadFullException        @14 ms

Two get permits and run; the other two find none and, with no wait allowed, are rejected at once with BulkheadFullException.

The thread-pool bulkhead

It gives the protected calls a dedicated thread pool. The caller's thread hands the call to that pool and is free; only the pool's threads ever wait on the downstream.

A quick reminder: thread pools

(The full story is in ThreadPoolExecutor in depth.) A pool has a core size, a maximum size and a queue:

1. While fewer than core threads are busy, a new task gets a thread.
2. When all core threads are busy, tasks wait in the queue.
3. When the queue is full, the pool creates extra threads, up to max.
4. When max threads are busy and the queue is full, the task is rejected.

Step 1: annotate the call

@Component
public class OrderService {

    @Autowired
    ProductClient productClient;

    @Bulkhead(name = "productService",
              type = Bulkhead.Type.THREADPOOL,
              fallbackMethod = "productFallback")
    public CompletableFuture<String> invokeProductAPI(String id) {
        System.out.println("Thread name is: " + Thread.currentThread().getName());
        return CompletableFuture.completedFuture(productClient.getProductById(id));
    }

    public CompletableFuture<String> productFallback(String id, Throwable t) {
        return CompletableFuture.completedFuture("Product Service is busy");
    }
}

The method must return CompletableFuture (or another CompletionStage). The aspect does roughly this:

CompletableFuture.supplyAsync(() -> yourMethodBody(), bulkheadThreadPoolExecutor);

It submits your whole method body as a task to the bulkhead's pool and immediately returns a future to the caller. So:

• Inside the method, use CompletableFuture.completedFuture(...), which only wraps the result you already computed. Do not call supplyAsync yourself: that would run the call on yet another pool (the common pool), not the bulkhead's.
• Put only the downstream call in the method. Everything inside runs on the bulkhead's threads.
• Write the fallback too. Without one, a rejected call completes the future with BulkheadFullException, and the client gets a 500 error.

Step 2: configure the pool

# 3 threads (core = max), plus room for 2 waiting calls; the 6th concurrent call is rejected
resilience4j.thread-pool-bulkhead.instances.productService.core-thread-pool-size=3
resilience4j.thread-pool-bulkhead.instances.productService.max-thread-pool-size=3
resilience4j.thread-pool-bulkhead.instances.productService.queue-capacity=2

Six simultaneous requests, product-service taking 2 seconds:

pool 3 on bulkhead-productService-1
pool 4 on bulkhead-productService-2
pool 5 on bulkhead-productService-3
                                              <- calls 1 and 2 wait in the queue
Product Service is busy 6 BulkheadFullException     (immediately)
product 3, 4, 5                               total 2005 ms
pool 1 on bulkhead-productService-2
pool 2 on bulkhead-productService-3
product 1, 2                                  total 4006 ms

• The calls run on bulkhead-productService-N threads, not Tomcat's http-nio-... threads: the dedicated pool is in use.
• Three run at once, two wait in the queue, and the sixth is rejected straight away.
• When the first three finish (2 s), the queued two take the freed threads and finish at 4 s.
• Each caller got its future back within a few milliseconds; only get() on it waited.

The time limiter

A thread-pool bulkhead frees the caller's thread, but the caller still waits on the future. If product-service hangs, it waits forever. The time limiter puts a deadline on an asynchronous call: if the future has not completed in time, it completes it with a TimeoutException.

For blocking calls you already have deadlines: the connect and read timeouts of Feign, RestTemplate and RestClient (chapters 1 to 3). The time limiter is for calls that return a future or a reactive type (Mono, Flux), so it pairs naturally with the thread-pool bulkhead:

@TimeLimiter(name = "productService", fallbackMethod = "timeoutFallback")
@Bulkhead(name = "productService", type = Bulkhead.Type.THREADPOOL)
public CompletableFuture<String> invokeProductAPI(String id) {
    return CompletableFuture.completedFuture(productClient.getProductById(id));
}

public CompletableFuture<String> timeoutFallback(String id, Throwable t) {
    return CompletableFuture.completedFuture("Product service timed out");
}

resilience4j.timelimiter.instances.productService.timeout-duration=1s

With a 3-second product-service, the caller gets Product service timed out (from a TimeoutException) after 1 second.

The time limiter frees the caller. The bulkhead thread still runs the slow call until it ends, so keep the HTTP client's own read timeout as well. Reactive programming with WebFlux, where the time limiter is used most, is outside this book.

Next: The time limiter stops waiting on a slow call, but the next call goes straight back to the same struggling service. Chapter 8 adds retry for failures that pass, and chapter 9 the circuit breaker for failures that do not.

Check yourself

1. Whom does a rate limiter protect order-service from, and whom does a bulkhead protect it from?
2. Which bulkhead fits a downstream that can handle only three concurrent requests?
3. How can a slow payment-service make a fast endpoint unavailable, and which bulkhead prevents it?
4. On which thread does a semaphore-bulkhead call run? A thread-pool-bulkhead call?
5. What do max-concurrent-calls and max-wait-duration control?
6. What exception reaches the fallback when a bulkhead is full?
7. Why must a thread-pool-bulkhead method return CompletableFuture, and why use completedFuture instead of supplyAsync inside it?
8. With core = max = 3 and queue = 2, what happens to six simultaneous calls?
9. What does the time limiter do, and when do you use it instead of a read timeout?
10. When the time limiter fires, does the slow call stop?

Answers

1. The rate limiter protects it from its clients (requests per time window); the bulkhead protects it from its downstream services (concurrent calls going out).
2. The semaphore bulkhead with max-concurrent-calls=3.
3. Its slow calls hold order-service threads until the whole pool is blocked, so even API 1 finds no thread. A thread-pool bulkhead gives payment calls a limited pool of their own.
4. Semaphore: the caller's own thread. Thread pool: a thread from the bulkhead's dedicated pool (bulkhead-<name>-N).
5. The number of calls allowed in at once, and how long a call waits for a free slot before rejection.
6. BulkheadFullException.
7. The aspect runs the method body on the bulkhead's pool and hands the caller a future at once. completedFuture only wraps the result; supplyAsync would move the call to another pool (the common pool), bypassing the bulkhead's.
8. Three run on the pool's threads, two wait in the queue and run when threads free up, and the sixth is rejected immediately.
9. It completes an asynchronous call's future with a TimeoutException if it takes too long, freeing the caller. Use it for futures and reactive types; for blocking HTTP calls, the client's read timeout does the job.
10. No; the bulkhead thread keeps running it. Only the caller stops waiting, so keep a read timeout on the HTTP client too.