Appearance
Retry Pattern: Fault Tolerance in Distributed Microservices
The Real World Analogy
Imagine you are calling a government office to get a document processed. The phone rings but nobody picks up. You have two choices: hang up and ask the original requester to call again themselves, or wait a few seconds and retry the call yourself. If you already spent ten minutes navigating the phone menu and providing your details, asking the original caller to start over wastes all that prior effort. The smart move is to retry the last step yourself.
Distributed microservices face exactly this situation dozens of times per second. A service may complete ninety percent of its work before a transient network hiccup causes the final downstream call to fail. Forcing the client to retry the entire request from scratch is wasteful and expensive. Instead, the service itself should retry only the failing downstream step.
Why Transient Failures Happen and Why Retry Matters
In a distributed system, calls to downstream services can fail because of transient issues: brief network timeouts, momentary connection resets, or a downstream pod restarting after a deploy. These issues are temporary and a second attempt moments later often succeeds.
The cost of NOT retrying internally is significant. Suppose one hundred clients are simultaneously calling the Order service, which internally calls Product. Each client request takes significant CPU, memory, and thread resources. If ninety percent of the processing is done and the downstream call fails, asking all one hundred clients to restart their requests means repeating that ninety percent work unnecessarily. The computational waste scales directly with traffic volume.
However, retry is a double edged sword. Improperly implemented retry causes more harm than good. You must understand when to retry, when not to retry, and how to retry safely.
When to Retry and When Not to Retry
Do Not Retry on Permanent Failures (4xx errors)
4xx HTTP errors represent validation failures. The client sent an incorrect request. No matter how many times you retry the same bad request, it will always fail. Retrying wastes resources and accomplishes nothing.
400 Bad Request → Do NOT retry
401 Unauthorized → Do NOT retry
403 Forbidden → Do NOT retry
404 Not Found → Do NOT retryException: 429 Too Many Requests is a rate limiting signal. You can retry with a delay, which gives the rate limiter time to replenish its quota.
Do Not Retry on non idempotent Operations
Idempotent means that retrying a request multiple times produces the same result as calling it once. A safe retry must not cause duplicate processing.
Consider this scenario: Service A calls the POST /orders endpoint of Service B. Service B inserts a row into its database and then takes a long time to respond. Service A receives a timeout and retries. Now Service B inserts a second row. You end up with two orders when only one was intended.
POST /orders is non idempotent. Retrying it causes duplicates.
Contrast this with GET /products/123 which is idempotent. Calling it twice returns the same product data without side effects.
Rule: Retry only on idempotent or read only operations, or design your downstream API to handle duplicate requests gracefully using idempotency keys.
Retry on 5xx Errors, Network Errors, and Timeouts
5xx errors indicate server side problems. The client request is valid; the server is temporarily broken.
500 Internal Server Error → Retry
502 Bad Gateway → Retry
503 Service Unavailable → Retry
Network timeout → Retry (only if idempotent)
Connection reset → RetryTypes of Retry Strategies
1. Fixed Interval Retry
The simplest strategy: wait a constant time between each retry.
Configuration example:
- Maximum attempts: 4 (1 original + 3 retries)
- Wait duration: 2 seconds
Attempt 1 (original): t=0s
Attempt 2 (retry 1): t=2s
Attempt 3 (retry 2): t=4s
Attempt 4 (retry 3): t=6sAdvantage: Simple to configure and debug.
Disadvantage: High risk of the thundering herd problem. If one thousand clients all retry at the exact same fixed interval, a surge of retry traffic hits the downstream service simultaneously, potentially overwhelming it further and preventing recovery.
2. Exponential Backoff
Delay increases exponentially between retries using this formula:
delay = baseDelay × (factor ^ failedAttempts)With baseDelay = 1000ms and factor = 2:
Attempt 1 (original): t=0ms
Attempt 2 (retry 1): t=1000ms (1000 × 2^0)
Attempt 3 (retry 2): t=2000ms (1000 × 2^1)
Attempt 4 (retry 3): t=4000ms (1000 × 2^2)Advantage: Reduces load on a struggling downstream service. By increasing the wait time, the downstream gets breathing room to recover.
Disadvantage: Still deterministic. If all clients use the same baseDelay and factor, they still retry at the same times. Thundering herd is reduced but not eliminated. Also, if the outage resolves quickly, the exponentially growing delay makes clients wait longer than necessary.
3. Exponential Backoff with Jitter
Adds randomness to the exponential backoff delay to spread retries across time:
delay = random(0, min(maxDelay, baseDelay × factor^failedAttempts))Each client now picks a different random delay, making simultaneous retry surges extremely unlikely.
Advantage: Best defense against thundering herd problems.
Disadvantage: Slightly harder to predict and debug because timing is non deterministic.
4. Custom Interval Retry
You define your own delay sequence entirely. For example, Fibonacci delays: 1, 1, 2, 3, 5, 8 seconds.
Advantage: Full control over retry timing.
Disadvantage: You must write and maintain the logic yourself without framework assistance.
Implementation with Resilience4j
Resilience4j is the recommended library for building fault tolerant microservices in Spring Boot. It provides retry, rate limiter, circuit breaker, bulkhead, and time limiter in one dependency.
Dependency
xml
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-spring-boot3</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>Fixed Interval Retry
Service layer with @Retry annotation:
java
package com.example.order.service;
import io.github.resilience4j.retry.annotation.Retry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
@Service
public class OrderService {
private static final Logger log = LoggerFactory.getLogger(OrderService.class);
private final ProductClient productClient;
public OrderService(ProductClient productClient) {
this.productClient = productClient;
}
// Apply retry only on the method that calls downstream.
// If you put @Retry on a method with heavy business logic,
// ALL that logic reruns on every retry — be very intentional.
@Retry(name = "productService", fallbackMethod = "productServiceFallback")
public String invokeProductApi(String productId) {
log.info("Calling product service for productId={} at {}", productId, System.currentTimeMillis());
return productClient.getProductById(productId);
}
// Fallback is called after all retry attempts are exhausted.
// Signature: same return type, same parameters, plus Throwable at the end.
public String productServiceFallback(String productId, Throwable ex) {
log.warn("All retries failed for productId={}. Cause: {}", productId, ex.getMessage());
return "Product service is busy. Please try again later.";
}
}application.properties for fixed interval:
properties
# Instance name matches the name in @Retry(name = "productService")
resilience4j.retry.instances.productService.max-attempts=3
resilience4j.retry.instances.productService.wait-duration=2s
# Default strategy is fixed interval — no extra configuration neededOutput trace (service not started, all calls fail):
Calling product service at 20:56:10.100
Calling product service at 20:56:12.102 ← +2s delay
Calling product service at 20:56:14.106 ← +2s delay
Product service is busy. Please try again later.Exponential Backoff
properties
resilience4j.retry.instances.productService.max-attempts=4
resilience4j.retry.instances.productService.wait-duration=1s
resilience4j.retry.instances.productService.enable-exponential-backoff=true
resilience4j.retry.instances.productService.exponential-backoff-multiplier=2Output trace:
Original call at 21:22:40.100
Retry 1 at 21:22:41.100 ← +1s (1000 × 2^0)
Retry 2 at 21:22:43.100 ← +2s (1000 × 2^1)
Retry 3 at 21:22:47.100 ← +4s (1000 × 2^2)
Product service is busy.Exponential Backoff with Jitter
Only one additional property is needed:
properties
resilience4j.retry.instances.productService.max-attempts=4
resilience4j.retry.instances.productService.wait-duration=1s
resilience4j.retry.instances.productService.enable-exponential-backoff=true
resilience4j.retry.instances.productService.exponential-backoff-multiplier=2
resilience4j.retry.instances.productService.enable-randomized-wait=trueEach retry delay is now random within the computed exponential bound. Multiple clients will retry at different times, preventing thundering herd.
Custom Retry (Manual Construction)
When you need a completely custom delay strategy (e.g., Fibonacci intervals), you cannot rely on @Retry with application.properties. You must construct the Retry object manually.
java
package com.example.order.config;
import io.github.resilience4j.retry.Retry;
import io.github.resilience4j.retry.RetryConfig;
import io.github.resilience4j.core.IntervalFunction;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.time.Duration;
@Configuration
public class RetryConfiguration {
@Bean
public Retry customProductRetry() {
// Custom interval function: define delay for each attempt number.
// attempt = 1 for first retry, 2 for second, etc.
IntervalFunction fibonacciInterval = attempt -> {
long[] fib = {1000, 1000, 2000, 3000, 5000, 8000};
int index = (int) Math.min(attempt - 1, fib.length - 1);
return fib[index]; // return milliseconds
};
RetryConfig config = RetryConfig.custom()
.maxAttempts(6)
.intervalFunction(fibonacciInterval)
.retryExceptions(Exception.class)
.build();
return Retry.of("customProductRetry", config);
}
}Service using custom retry (manual wrapping — no @Retry annotation):
java
package com.example.order.service;
import io.github.resilience4j.retry.Retry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import java.util.function.Supplier;
@Service
public class OrderServiceManual {
private static final Logger log = LoggerFactory.getLogger(OrderServiceManual.class);
private final ProductClient productClient;
private final Retry customProductRetry;
public OrderServiceManual(ProductClient productClient, Retry customProductRetry) {
this.productClient = productClient;
this.customProductRetry = customProductRetry;
}
public String invokeProductApi(String productId) {
// Wrap the downstream call inside Retry.decorateSupplier.
// This is exactly what AOP does internally when @Retry is used.
Supplier<String> decorated =
Retry.decorateSupplier(customProductRetry,
() -> {
log.info("Calling product API at {}", System.currentTimeMillis());
return productClient.getProductById(productId);
});
try {
return decorated.get();
} catch (Exception e) {
log.error("All retries exhausted for productId={}", productId, e);
return "Fallback: all retries failed.";
}
}
}How Resilience4j Retry Works Internally (AOP Internals)
Resilience4j uses Spring AOP to generate proxy code at runtime. Understanding the internals helps you reason about behavior and write custom retry when needed.
The framework operates in three conceptual layers:
1. IntervalFunction This is the class containing the delay computation logic. It has static factory methods:
IntervalFunction.of(Duration)— fixed intervalIntervalFunction.ofExponentialBackoff(...)— exponential delayIntervalFunction.ofExponentialRandomBackoff(...)— exponential with jitter
Based on your application.properties configuration, AOP selects the appropriate method.
2. RetryConfig Holds all configuration: max attempts, the interval function, which exceptions to retry on, etc.
3. Retry object Created from RetryConfig. AOP creates this object, then wraps your annotated method inside retry.executeSupplier(...) or equivalent.
When your method throws an exception, the retry object checks:
- Is this exception in the retry list?
- Are there remaining attempts?
If yes, it calls the interval function to compute the next delay, sleeps, then calls your method again. After all attempts are exhausted, it calls your fallback method.
Critical Rules Summary
| HTTP Status | Retry? | Notes |
|---|---|---|
| 2xx | No | Already succeeded |
| 400, 401, 403, 404 | No | Client error, will always fail |
| 429 | Yes, with delay | Rate limited — wait for quota reset |
| 500, 502, 503 | Yes | Server errors, likely transient |
| Network timeout | Yes, if idempotent | Ensure no duplicate side effects |
The Recommended Order of Fault Tolerance Mechanisms
Apply fault tolerance mechanisms in this logical order: Rate Limiter → Bulkhead → Time Limiter → Circuit Breaker → Retry.
Rate limiter should be applied before retry. If you apply retry first, you waste computation retrying requests that the rate limiter will ultimately reject anyway. Only traffic that passes the rate limiter is worth retrying.
Interview Questions and Pitfalls
Q1: Why should we retry internally in a service rather than asking the client to retry?
A: When a request fails after completing a large portion of its processing, asking the client to retry means repeating all that expensive computation from scratch. Internal retry retries only the failing downstream step, preserving the work already done. This is especially critical at high traffic volumes where repeated full request restarts multiply resource waste.
Q2: What is the thundering herd problem in the context of retry?
A: When many clients all retry at the same fixed interval after a downstream outage, they simultaneously flood the recovering service with retry traffic in addition to new incoming traffic. The downstream, which is already struggling, gets overwhelmed again. Exponential backoff with jitter solves this by randomizing retry timing, spreading retries across time so no single burst occurs.
Q3: Why should we never retry on 4xx errors?
A: 4xx errors indicate that the client request itself is invalid. The input data fails validation. Retrying the same invalid request will always produce the same 4xx response no matter how many attempts are made. The only way to fix a 4xx is to fix the request. Retrying wastes CPU, memory, threads, and network bandwidth with zero chance of success.
Q4: What does idempotency mean and why does it matter for retry?
A: An idempotent operation produces the same result when executed once or multiple times without additional side effects. GET requests are naturally idempotent. POST requests that create resources are typically not idempotent — retrying a POST can create duplicate records. Before enabling retry on a downstream call, confirm that the downstream API is idempotent. If it is not, either make the API idempotent using idempotency keys or do not retry that specific call.
Q5: What is the difference between fixed interval, exponential backoff, and jitter?
A: Fixed interval waits the same duration between every retry. It is simple but risky at scale because all clients retry at the same time. Exponential backoff increases the wait time exponentially, giving the downstream service breathing room to recover and reducing retry surges. However, all clients using the same base and multiplier still retry at the same times. Adding jitter introduces randomness to the delay, spreading retries across different times and virtually eliminating synchronized retry storms.
Q6: What happens to the fallback method signature when using @Retry?
A: The fallback method must have the same return type and the same parameters as the original method. The only difference is an additional Throwable parameter at the end. If the signature does not match, the framework cannot identify the fallback and will use a default behavior. For example, if the original method is String invokeProduct(String productId), the fallback must be String fallback(String productId, Throwable ex).
Q7: Pitfall — placing @Retry on a method with heavy business logic.
A: If you annotate a method that contains database writes, external API calls, and business logic with @Retry, ALL of that logic reruns on every retry. This can cause partial duplicate work, performance problems, and unexpected side effects. Always place @Retry on the narrowest method that covers only the downstream call you want to retry.