Rate Limiter: Fault Tolerance in Distributed Microservices

The Real World Analogy

Imagine a popular amusement park with a single, very popular ride. Without any queuing system, hundreds of people could rush the ride simultaneously, overwhelming its safety mechanisms and breaking down the machinery. The park solution is a turnstile: only a fixed number of visitors may enter the ride queue per hour. Everyone else must wait outside. The ride runs smoothly, no one gets hurt, and the visitor experience remains positive for all.

A rate limiter is exactly this turnstile for your microservice. It controls how many requests are allowed into your service within a given time window. Without it, a sudden traffic spike — accidental or malicious — can overwhelm your service, starve it of threads, and cause cascading failure across your entire system.

What Is a Fault Tolerant Microservice?

A fault tolerant microservice is one that continues to work correctly even when downstream systems fail or when traffic spikes occur. Instead of crashing or propagating failure, it handles the situation gracefully — either by returning a cached response, a meaningful error, or by queuing the request for later.

Without fault tolerance, a single slow or failing downstream service can bring down your entire system through cascading failure. Consider this chain: a buggy deployment in the Product service causes it to take sixty seconds to respond. The Order service calls Product, waits sixty seconds, and all its threads are now blocked. As more traffic pours in, all Order service threads are consumed waiting for Product. The thread pool exhausts itself. The Order service starts rejecting requests. Now clients and other services that depend on Order also start failing. One bad deployment cascades into a full system outage.

Rate limiter is the first line of defense in the recommended fault tolerance order: Rate Limiter → Bulkhead → Time Limiter → Circuit Breaker → Retry.

What Does Rate Limiter Do?

Rate limiter controls the number of requests allowed to a service within a specific time window. Its primary purpose is to protect your service from sudden traffic spikes, denial of service attacks, and abusive clients.

Key distinction: Rate limiter protects your application from its clients. It governs incoming traffic from external callers. It does NOT address concurrency in downstream calls — that is the job of the bulkhead pattern covered in the next chapter.

Rate Limiting Algorithms

Before examining implementation, understanding the underlying algorithms is critical for interviews and for choosing the right approach for your system.

1. Fixed Window Counter

Counts requests in a fixed time window. If the count exceeds the limit, requests are rejected.

Example: Limit = 5 requests per 10 second window.

Window 1 (0–10s):  R1 R2 R3 R4 R5  → All 5 accepted
                   R6               → Rejected (limit reached)
Window 2 (10–20s): R1 R2 R3 R4 R5  → All 5 accepted in new window

Disadvantage: If all 5 requests arrive at the very end of Window 1 and another 5 arrive at the very start of Window 2, then 10 requests pass through within a single 1-second period — double the intended limit. This is the boundary burst problem.

2. Sliding Log

Stores the exact timestamp of every accepted request. To decide whether to accept a new request, counts how many accepted requests fall within the last N seconds (sliding window).

Example: Window = 10s, limit = 5.

Accept  R1 at 10:00:02  → store timestamp, count=1
Accept  R2 at 10:00:05  → store timestamp, count=2
Accept  R3 at 10:00:07  → store timestamp, count=3
Accept  R4 at 10:00:09  → store timestamp, count=4
Window slides to 10:00:01–10:00:11
Accept  R5 at 10:00:11  → count=4, limit=5 → accepted, count=5
Window slides to 10:00:02–10:00:12
R6 at 10:00:12  → count=5, limit=5 → rejected

When the window slides, timestamps that fall outside the window must be cleaned up (garbage collected).

Disadvantage: Requires storing a timestamp per request. High traffic means significant memory usage. Cleanup overhead adds complexity.

3. Sliding Window Counter with sub windows

Divides the main window into equal size sub windows. Each sub window maintains a request count. As the main window slides, expired sub windows are dropped and new ones are added.

Example: Main window = 10s, sub window = 2s, limit = 5.

Sub-windows: [0–2, 2–4, 4–6, 6–8, 8–10]
Each sub-window tracks its own count.
At any point, sum the counts of all active sub-windows.
Main window slides by one sub-window size (2s) at a time.

More memory efficient than sliding log but more complex than fixed window.

4. Sliding Window Counter with Weighted Window

Combines fixed window counters with a sliding calculation using proportional weight. The algorithm estimates how many requests fall within the current sliding window based on the percentage overlap with the previous fixed window.

totalRequests = currentWindowCount + (overlapPercentage × previousWindowCount)

Disadvantage: The estimate can be inaccurate when requests are not uniformly distributed. If all requests arrived at the very end of the previous window, the weighted estimate undercounts them, potentially allowing more requests through than intended.

5. Token Bucket

A bucket holds a fixed number of tokens. Each incoming request consumes one token. A refiller adds tokens at a fixed rate. If no token is available, the request is rejected.

Example: Bucket capacity = 4, refill rate = 1 token per 6 seconds.

t=0:  Bucket has 4 tokens
R1: consume 1 token → 3 left (accepted)
R2: consume 1 token → 2 left (accepted)
R3: consume 1 token → 1 left (accepted)
R4: consume 1 token → 0 left (accepted)
R5: no token → rejected
t=6:  Refiller adds 1 token → bucket has 1
R6: consume 1 token → accepted

Advantage: Allows short bursts up to the bucket capacity.

Disadvantage: If capacity is set too high without careful planning, a large burst is possible when accumulated tokens are all consumed at once.

6. Leaky Bucket

Requests enter a queue. The queue drains at a strictly constant rate. If the queue is full, incoming requests are rejected (overflow).

Requests → [Queue (fixed size)] → Processed at constant rate
                                       ↑
                               If queue is full: HTTP 429

Advantage: Smooth, constant output rate regardless of bursty input.

Disadvantage: Increases latency because requests wait in the queue. Queue size must be tuned carefully — too small and many requests are rejected; too large and latency spikes.

Resilience4j default: Token bucket algorithm, because it is simple to configure and handles moderate bursts gracefully.

Implementation with Resilience4j

Dependency

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

Feign Client Interface

package com.example.order.client;

import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;

// Using service discovery via the logical service name
@FeignClient(name = "product-service")
public interface ProductClient {

    @GetMapping("/products/{id}")
    String getProductById(@PathVariable("id") String id);
}

Service with @RateLimiter Annotation

package com.example.order.service;

import io.github.resilience4j.ratelimiter.annotation.RateLimiter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;

@Service
public class OrderService {

    private static final Logger log = LoggerFactory.getLogger(OrderService.class);

    private final ProductClient productClient;

    public OrderService(ProductClient productClient) {
        this.productClient = productClient;
    }

    // name: matches the instance name in application.properties
    // fallbackMethod: called when a request is rejected by the rate limiter
    @RateLimiter(name = "productRateLimiter", fallbackMethod = "rateLimitedFallback")
    public String invokeProductApi(String productId) {
        log.info("Calling product service for productId={}", productId);
        return productClient.getProductById(productId);
    }

    // Fallback signature rules:
    //   - Same return type as the original method
    //   - Same parameters as the original method
    //   - One additional Throwable parameter at the end
    // If signature does not match, framework uses its default fallback.
    public String rateLimitedFallback(String productId, Throwable ex) {
        log.warn("Rate limit exceeded for productId={}. Error: {}", productId, ex.getMessage());
        return "Rate limit exceeded. Please try again later.";
    }
}

Controller

package com.example.order.controller;

import com.example.order.service.OrderService;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class OrderController {

    private final OrderService orderService;

    public OrderController(OrderService orderService) {
        this.orderService = orderService;
    }

    @GetMapping("/order/product/{id}")
    public String getProduct(@PathVariable String id) {
        return orderService.invokeProductApi(id);
    }
}

Configuration in application.properties

Resilience4j rate limiter uses the token bucket algorithm. Three properties control its behavior:

# limitForPeriod: maximum tokens in the bucket (= bucket capacity)
resilience4j.ratelimiter.instances.productRateLimiter.limit-for-period=2

# limitRefreshPeriod: how often the bucket is refilled (and how much)
# After every 10 seconds, 2 tokens are added (matching limitForPeriod)
resilience4j.ratelimiter.instances.productRateLimiter.limit-refresh-period=10s

# timeoutDuration: how long a request should wait for a token before being rejected
# 0s means reject immediately if no token available
resilience4j.ratelimiter.instances.productRateLimiter.timeout-duration=1s

How these interact:

With limit-for-period=2 and limit-refresh-period=10s:

• The bucket holds a maximum of 2 tokens.
• Every 10 seconds, 2 tokens are added (up to the bucket capacity).
• With timeout-duration=1s: if no token is available, the request waits up to 1 second before being rejected.

Observed behavior:

Hit 1 → Token consumed → accepted
Hit 2 → Token consumed → accepted
Hit 3 → No token, waits 1s → rejected → "Rate limit exceeded"
... wait 10 seconds for refill ...
Hit 4 → Token consumed → accepted
Hit 5 → Token consumed → accepted
Hit 6 → No token → rejected

How AOP Wires the Rate Limiter Internally

When Spring sees the @RateLimiter annotation, AOP generates a proxy around your method. The proxy intercepts every call and:

1. Checks whether a token is available in the token bucket.
2. If available, consumes the token and calls proceed() — executing your actual method.
3. If unavailable, waits up to timeout-duration for a token.
4. If still unavailable after the timeout, invokes the fallback method.

You can replicate this behavior manually, which is useful for writing custom rate limiting algorithms:

package com.example.order.aspect;

import java.lang.annotation.*;

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface CustomRateLimiter {
}

package com.example.order.aspect;

import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.springframework.stereotype.Component;

import java.util.concurrent.Semaphore;

@Aspect
@Component
public class CustomRateLimiterAspect {

    // A simple semaphore-based rate limit: max 5 concurrent passes through
    private final Semaphore semaphore = new Semaphore(5);

    @Around("@annotation(CustomRateLimiter)")
    public Object applyRateLimit(ProceedingJoinPoint joinPoint) throws Throwable {
        boolean acquired = semaphore.tryAcquire();
        if (!acquired) {
            throw new RuntimeException("Custom rate limit exceeded");
        }
        try {
            return joinPoint.proceed();
        } finally {
            semaphore.release();
        }
    }
}

This example shows the core AOP pattern: intercept, enforce the policy, call proceed, release resources.

Rate Limiter vs. Bulkhead: A Critical Distinction

This is one of the most common points of confusion in interviews:

Concern	Rate Limiter	Bulkhead
Protects from	Clients (incoming traffic)	Downstream services (outgoing concurrency)
Measures	Requests per time window	Concurrent requests in flight
Talks about concurrency?	No	Yes
Example	Max 100 requests per minute from all clients	Max 3 concurrent calls to Payment service

Rate limiter says: "You may send me at most N requests per period." Bulkhead says: "I will send at most N concurrent requests to my downstream."

---

Interview Questions and Pitfalls

Q1: What is the purpose of a rate limiter and when would you use it?

A: A rate limiter controls how many requests are allowed to a service within a given time window. It protects the service from sudden traffic spikes, abusive clients, and denial of service attacks. Use it when you need to ensure that no single client or group of clients can overwhelm your service with more requests than it can safely handle.

Q2: What is the thundering herd problem and how does rate limiting help?

A: The thundering herd problem occurs when many clients simultaneously send requests to a service that has just recovered from a brief outage or when all clients retry at exactly the same time. This synchronized surge can overwhelm the recovering service and cause it to fail again. Rate limiting prevents this by capping the number of requests that pass through, regardless of how many clients are simultaneously trying.

Q3: Explain the difference between the token bucket and leaky bucket algorithms.

A: Token bucket allows requests up to the current token count (bucket capacity), enabling short bursts when tokens have accumulated. It rejects requests only when the bucket is empty. Leaky bucket queues requests and processes them at a strictly constant rate, providing a smooth output but introducing latency for queued requests. Token bucket is burst tolerant; leaky bucket enforces a uniform rate.

Q4: What are the three key properties of Resilience4j's rate limiter?

A: limit-for-period defines the bucket capacity — the maximum number of tokens (requests) allowed per refresh period. limit-refresh-period defines how often the bucket is refilled. timeout-duration defines how long a request should wait for a token before being rejected. Setting timeout to 0s causes immediate rejection when no token is available.

Q5: What happens if the rate limiter fallback method signature is incorrect?

A: If the fallback method does not match the return type and parameters of the original method (with an extra Throwable at the end), the framework cannot locate the correct fallback. It falls back to the Resilience4j default behavior, which typically throws a RequestNotPermitted exception directly to the caller without any custom handling. Always verify the fallback method signature matches exactly.

Q6: Why should rate limiter be applied before retry in the fault tolerance chain?

A: If retry is applied first, the service attempts to retry calls that the rate limiter would ultimately reject. This wastes computation retrying work that will never succeed because the rate limit quota is already exhausted. Applying rate limiter first ensures that only traffic within the allowed quota reaches the retry mechanism.

Q7: Pitfall — setting limit-for-period too high without considering burst capacity.

A: If you set the bucket capacity very high (e.g., 10,000 tokens) without careful analysis of what your downstream can actually handle, a sudden burst could allow 10,000 simultaneous requests through before the bucket depletes. Always size limit-for-period based on what your service and downstream can reliably handle, not based on expected peak demand.