The Retry Pattern

In a distributed system, a call to a downstream service can fail because of a transient problem: a brief network glitch, a timeout, an instance restarting. Sent again a moment later, the same request often succeeds. Retry does exactly that.

Done carelessly, retrying causes more harm than good, so this chapter spends as much time on when and how as on the code.

Why retry in the service, not in the client?

If a call fails, why not return the error and let the client call again? Because work is not free. Suppose order-service has done 90% of a request's processing (validation, database reads, computations) and the last step, calling product-service, hits a transient error. If the client retries, order-service redoes the 90% before reaching the same call. Retrying just the failed call saves that cost across thousands of requests, and the user simply sees a slightly slower success instead of an error.

When to retry

Do not retry permanent failures. A 4xx response means the request itself is wrong: invalid input, missing authentication, unknown resource. Sending the same request again fails the same way.

Do not retry non-idempotent operations. An operation is idempotent if performing it several times has the same effect as performing it once. Suppose service A calls service B's "create order" endpoint. B inserts the row, but its response is slow and A times out. A retries, and B inserts a second row. The timeout did not mean the work was not done. Retry such a call only if the endpoint is designed to be idempotent, for example by accepting an idempotency key and ignoring repeats.

Rule of thumb: retry 5xx errors. A 5xx means the request was valid but the system failed to process it. That can be transient, and it signals that the request did not succeed.

Retry network errors and timeouts only for idempotent endpoints, for the reason above.

Response	Meaning	Retry?
200 OK	success	no
400 Bad Request	invalid input	no
401 Unauthorized	authentication needed	no
403 Forbidden	access denied	no
404 Not Found	resource does not exist	no
409 Conflict	state conflict	no
429 Too Many Requests	rate limited	yes, after a delay
500 Internal Server Error	server failure	yes
502 Bad Gateway	upstream failure	yes
503 Service Unavailable	down or overloaded	yes
network error, timeout	connection failure	only if idempotent

How to retry: four strategies

1. Fixed interval

Wait the same delay before every retry. With 4 attempts (the original call plus 3 retries) and 2 s: calls at 0, 2, 4 and 6 s.

• Pros: simple, easy to configure and debug.
• Cons: retry storms, also called the thundering herd problem. When a service has an outage under heavy traffic, thousands of clients retry on the same schedule. Every 2 seconds a wave of retries hits the recovering service on top of the new traffic, and it may spend all its capacity on retries.

2. Exponential backoff

The delay grows after each failure:

delay = base × multiplier^{(failed retries so far)}

With base 100 ms and multiplier 2: 100 ms, 200 ms, 400 ms, 800 ms... Usually capped at a maximum delay.

• Pros: gives a failing service time to recover; spreads retries out somewhat.
• Cons: after a short outage, users may wait longer than needed (the outage ended at 500 ms, but the next retry is due at 700 ms); and every client following the same formula still retries in sync.

3. Exponential backoff with jitter

Add randomness to each exponential delay so clients drift apart. The retries no longer arrive in waves.

• Pros: the best fit for distributed systems; it spreads retries over time and makes retry storms unlikely.
• Cons: harder to reason about and debug.

4. Custom interval

Compute each delay yourself, for example following the Fibonacci sequence: 1, 1, 2, 3, 5, 8...

• Pros: full control.
• Cons: your code to write and maintain.

Step 1: a fixed-interval retry

Dependencies as before: resilience4j-spring-boot3 and spring-boot-starter-aop.

Put @Retry on the method that makes the downstream call:

@Component
public class OrderService {

    @Autowired
    ProductClient productClient;

    @Retry(name = "productService", fallbackMethod = "productFallback")
    public String invokeProductAPI(String id) {
        return productClient.getProductById(id);
    }

    public String productFallback(String id, Throwable t) {
        return "Product Service is busy";
    }
}

The whole method is retried, not just the HTTP call. If the method also writes to a database or charges a card before the call, that runs again on every attempt. Keep only the code you mean to repeat inside a @Retry method.

The fallback runs once, after the last attempt fails.

Configure the instance:

# fixed interval: 3 attempts in total, 2 s apart
resilience4j.retry.instances.productService.max-attempts=3
resilience4j.retry.instances.productService.wait-duration=2s

max-attempts counts the original call: 3 means one call and two retries. Fixed interval is the default strategy (defaults: 3 attempts, 500 ms).

With product-service stopped:

attempt   +0 ms
attempt   +2003 ms
attempt   +4003 ms
fallback  +4005 ms

Step 2: choose which exceptions to retry

By default Resilience4j retries every exception. Feign turns a 400 response into FeignException.BadRequest, and with the configuration above, that gets retried three times. That breaks the first rule. Tell the retry what to skip, or what to retry:

# never retry client errors (4xx)
resilience4j.retry.instances.productService.ignore-exceptions=feign.FeignException$FeignClientException

# or: retry only these (and their subclasses)
resilience4j.retry.instances.productService.retry-exceptions=feign.FeignException$FeignServerException,feign.RetryableException

FeignClientException is the parent of all 4xx Feign exceptions and FeignServerException of all 5xx; RetryableException covers I/O failures such as timeouts and refused connections. With the exception ignored, a 400 goes straight to the fallback after one attempt, while a 500 is still retried.

Step 3: exponential backoff

Only the configuration changes:

resilience4j.retry.instances.productService.max-attempts=4
resilience4j.retry.instances.productService.wait-duration=1s
resilience4j.retry.instances.productService.enable-exponential-backoff=true
resilience4j.retry.instances.productService.exponential-backoff-multiplier=2

wait-duration becomes the initial delay, and each later delay is multiplied by 2:

attempt   +0 ms
attempt   +1002 ms      (waited 1 s)
attempt   +3003 ms      (waited 2 s)
attempt   +7004 ms      (waited 4 s)
fallback  +7004 ms

Resilience4j computes it as initial × multiplier^(attempt − 1), so the first retry waits 1 s × 2⁰, the second 1 s × 2¹, the third 1 s × 2². Add exponential-max-wait-duration to cap the delay.

Step 4: exponential backoff with jitter

Add one property:

resilience4j.retry.instances.productService.enable-randomized-wait=true

Resilience4j then picks each delay at random around the exponential value, within ± randomized-wait-factor (default 0.5). So the first retry waits between 0.5 and 1.5 s, the second between 1 and 3 s, the third between 2 and 6 s. Two runs:

run 1:  +0   +680   +2656   +8259 ms
run 2:  +0   +733   +3141   +8380 ms

Some descriptions of jitter use "full jitter", a random value between 0 and the exponential delay. Resilience4j's randomized wait is the "± factor" variant above. Both break up synchronized retries.

How @Retry works

As with the other Resilience4j annotations, an AOP aspect intercepts the call. Building the retry takes three parts:

1. IntervalFunction: computes the delay before each retry. It has factory methods for every strategy above: IntervalFunction.of(2000) (fixed), ofExponentialBackoff(...), ofExponentialRandomBackoff(...).
2. RetryConfig: max attempts, the interval function, which exceptions to retry.
3. Retry: the object built from the config, which runs the call.

Roughly what the properties in Step 1 turn into:

RetryConfig config = RetryConfig.custom()
        .maxAttempts(3)
        .intervalFunction(IntervalFunction.of(2000))        // fixed 2 s
        .retryExceptions(IOException.class)                 // optional filter
        .build();

Retry retry = Retry.of("productService", config);

Then the aspect runs your method inside the retry: on an exception it asks the interval function for the delay, sleeps, and calls again until an attempt succeeds or the attempts run out.

Step 5: a custom interval

There is no property for "Fibonacci", so supply the interval function in code. Two ways.

With the annotation, using a RetryConfigCustomizer bean that adjusts the instance configured in properties:

@Configuration
public class RetryConfiguration {

    @Bean
    public RetryConfigCustomizer productServiceRetry() {
        IntervalFunction fibonacci = attempt -> {           // attempt = 1, 2, 3...
            long a = 1, b = 1;
            for (int i = 1; i < attempt; i++) { long next = a + b; a = b; b = next; }
            return a * 100;                                  // 100, 100, 200, 300, 500 ms
        };
        return RetryConfigCustomizer.of("productService",
                builder -> builder.intervalFunction(fibonacci));
    }
}

resilience4j.retry.instances.productService.max-attempts=6

Two pitfalls, both found by running it:

• The customizer only applies to an instance that exists in properties. Without at least one productService property, it is silently ignored and the defaults run.
• Do not also set wait-duration for that instance: the application then fails at startup with The intervalFunction was configured twice.

Six attempts, waiting 100, 100, 200, 300 and 500 ms between them.

Without the annotation, build the Retry yourself and wrap the call, exactly what the aspect does for you:

@Configuration
public class RetryConfiguration {

    @Bean
    public Retry customRetry() {
        IntervalFunction interval = attempt -> 2000L;        // your logic here
        RetryConfig config = RetryConfig.custom()
                .maxAttempts(6)
                .intervalFunction(interval)
                .retryExceptions(Exception.class)
                .build();
        return Retry.of("customRetry", config);
    }
}

@Component
public class OrderService {

    @Autowired ProductClient productClient;
    @Autowired Retry customRetry;

    public String invokeProductAPI(String id) {
        try {
            return customRetry.executeSupplier(() -> productClient.getProductById(id));
        } catch (Exception e) {
            return "all retries failed";                     // the fallback
        }
    }
}

Retry with other mechanisms

When several Resilience4j annotations sit on one method, they nest in a fixed order, Retry outermost:

Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( your method ) ) ) ) )

So every retry attempt passes through the rate limiter and uses up a permission; retries cannot exceed the rate limit. If the limiter rejects an attempt, the retry sees RequestNotPermitted, and, unless told otherwise, retries that too, wasting the waits. With a limit of 1 call per minute and 3 attempts, the method body runs once, and the fallback finally receives RequestNotPermitted. Add the rate limiter's and circuit breaker's rejections to ignore-exceptions when you combine them:

resilience4j.retry.instances.productService.ignore-exceptions=\
  io.github.resilience4j.ratelimiter.RequestNotPermitted,\
  io.github.resilience4j.circuitbreaker.CallNotPermittedException

Next: Retries help when a failure is brief. When product-service is down for minutes, retrying every call only adds load. Chapter 9's circuit breaker notices a failing service and stops calling it for a while.

Check yourself

1. What is a transient failure?
2. Why retry inside order-service instead of letting the client call again?
3. Why should 4xx responses not be retried?
4. Why is retrying a timed-out "create order" call dangerous, and when is it safe?
5. What problem does a fixed retry interval cause under heavy load?
6. What does exponential backoff improve, and what problem remains?
7. How does jitter help?
8. With max-attempts=3 and wait-duration=2s, how many calls are made and when?
9. Which exceptions does @Retry retry by default, and how do you stop it retrying 4xx Feign errors?
10. With exponential backoff, initial 1 s and multiplier 2, what are the delays for 4 attempts?
11. What range can the first delay fall in with randomized wait (factor 0.5) and initial 1 s?
12. What are the three building blocks behind @Retry?
13. How do you use a custom interval with the annotation, and what two mistakes make it fail?
14. In what order do Retry and RateLimiter nest, and what does that mean for retries?

Answers

1. A short, temporary problem, such as a network glitch, a timeout or a restarting instance, that a later attempt may not hit.
2. The work already done in the request (say 90% of it) would be repeated if the client called again; retrying only the failed call saves that cost and gives a better experience.
3. They mean the request itself is invalid; the same request will fail the same way.
4. The first call may have succeeded on the server even though the client timed out, so a retry can create a duplicate. It is safe when the endpoint is idempotent.
5. Retry storms (thundering herd): all clients retry in synchronized waves that can overwhelm the recovering service.
6. It gives the failing service growing breathing room and spreads retries; but clients using the same formula still retry in sync, and users may wait longer than the outage lasted.
7. Randomizing each delay makes clients drift apart, so retries are spread out instead of arriving in waves.
8. Three calls: at 0, 2 and 4 s; then the fallback.
9. All exceptions. Add ignore-exceptions=feign.FeignException$FeignClientException (or list only the retryable ones in retry-exceptions).
10. 1 s, 2 s, 4 s.
11. Between 0.5 s and 1.5 s.
12. IntervalFunction (the delay), RetryConfig (attempts, interval, exceptions), and the Retry object built from the config.
13. Register a RetryConfigCustomizer bean for the instance that sets intervalFunction. It is ignored if the instance has no properties at all, and the app fails to start if wait-duration is also set.
14. Retry is outermost: Retry(...RateLimiter(...)). Every attempt goes through the rate limiter and consumes a permission; a rejection (RequestNotPermitted) is retried unless ignored.