Distributed Logging (Part 4) | End to End Distributed Logging with MDC, Correlation ID and Trace ID

The Amazon Delivery Tracking Number Analogy

Imagine ordering a laptop online. Your package does not travel on a single delivery truck. It moves from an automated fulfillment warehouse, to a regional sorting airport, onto a cargo plane, into a local delivery hub, and finally onto a local courier van. Each facility operates independently and keeps its own digital ledger.

How does customer support answer when you call asking, "Where is my laptop?" They do not search by your name across fifty different warehouses. They search by a single tracking number (e.g. TRK-882194). That single tracking number is stamped on the box at the warehouse, scanned at the airport, scanned at the distribution center, and recorded on the delivery driver's handheld scanner.

In a microservices architecture, a single user action — like clicking "Confirm Purchase" — can trigger a cascade of HTTP and messaging requests spanning six different microservices. If each service logs to its own file with generic timestamps, debugging an error is a nightmare. Distributed Logging with MDC (Mapped Diagnostic Context) and Correlation IDs is that package tracking number. A unique identifier is assigned at the API Gateway and propagated across every downstream HTTP call, ensuring every log entry produced by any service for that request shares the exact same ID.

This lecture covers the distributed tracing problem, SLF4J Mapped Diagnostic Context (MDC), Spring Web interceptors, propagating correlation IDs via HTTP headers, and thread pool caveats with asynchronous execution.

---

The Distributed Debugging Problem

Consider a standard ecommerce transaction:

User Click -> [ API Gateway ] -> [ Order Service ] -> [ Payment Service ] -> [ Bank Gateway ]
                                         |
                                         v
                                  [ Stock Service ]

Suppose the user encounters an error message on screen: 500 Internal Server Error.

When the on call engineer opens the logs in Kibana or Datadog:

• There are four thousand concurrent users active on the platform.
• The logs contain fifty thousand entries per minute.
• Which specific log line in the Payment Service corresponds to this user's failed click in the Order Service?

Without a shared identifier, it is impossible to reconstruct the sequence of events. You cannot correlate logs across process boundaries.

---

What Is MDC (Mapped Diagnostic Context)?

SLF4J provides MDC (org.slf4j.MDC), a thread local key value storage mechanism designed specifically for logging frameworks.

When you put a key into MDC:

MDC.put("correlationId", "req-12345");

Logback automatically injects that value into every subsequent log line emitted on that thread, formatted using %X{correlationId} in your log pattern:

<pattern>%d{ISO8601} [%thread] [%X{correlationId}] %-5level %logger - %msg%n</pattern>

Output:

2026-09-05 15:00:01.100 [http-nio-8080-exec-1] [req-12345] INFO  OrderService - Order creation initiated
2026-09-05 15:00:01.150 [http-nio-8080-exec-1] [req-12345] INFO  OrderService - Calling payment service

Notice that you never had to manually pass correlationId into your log.info() calls. MDC automatically enriches all log statements on that thread.

---

The End to End Architecture

Here is how correlation IDs propagate across an entire distributed system:

[ Incoming Request ]
        |
        v
[ API Gateway ]
  - Checks for incoming 'X-Correlation-Id' header
  - If missing, generates UUID: "c-9901"
  - Injects 'X-Correlation-Id: c-9901' into downstream request
        |
        v
[ Order Service ]
  - Filter extracts 'X-Correlation-Id' -> puts into MDC
  - Logs show: [c-9901] Creating order
  - RestTemplate/RestClient interceptor extracts MDC -> adds header to outgoing call
        |
        v
[ Payment Service ]
  - Filter extracts 'X-Correlation-Id' -> puts into MDC
  - Logs show: [c-9901] Charging credit card
  - Filter cleans up MDC in finally block

Now, searching for c-9901 in your centralized logging dashboard displays the complete chronological story of that transaction across all services.

---

Implementation Step 1: Inbound HTTP Interceptor

Create a servlet filter or Spring HandlerInterceptor to extract the correlation ID from incoming HTTP requests and populate MDC:

package com.example.orderservice.interceptor;

import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.slf4j.MDC;
import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;

import java.util.UUID;

@Component
public class CorrelationIdInterceptor implements HandlerInterceptor {

    public static final String CORRELATION_ID_HEADER = "X-Correlation-Id";
    public static final String MDC_KEY = "correlationId";

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
        String correlationId = request.getHeader(CORRELATION_ID_HEADER);

        // If upstream did not supply a correlation id, generate one
        if (correlationId == null || correlationId.isBlank()) {
            correlationId = UUID.randomUUID().toString();
        }

        // Put into MDC for current thread
        MDC.put(MDC_KEY, correlationId);

        // Echo back in response header so caller can track it
        response.setHeader(CORRELATION_ID_HEADER, correlationId);

        return true;
    }

    @Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) {
        // Critical: always clean up thread local to prevent memory leaks and thread pollution
        MDC.remove(MDC_KEY);
    }
}

Register the interceptor in Spring:

package com.example.orderservice.config;

import com.example.orderservice.interceptor.CorrelationIdInterceptor;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;

@Configuration
public class WebMvcConfig implements WebMvcConfigurer {

    @Autowired
    private CorrelationIdInterceptor correlationIdInterceptor;

    @Override
    public void addInterceptors(InterceptorRegistry registry) {
        registry.addInterceptor(correlationIdInterceptor).addPathPatterns("/**");
    }
}

---

Implementation Step 2: Outbound HTTP Interceptor

When OrderService calls PaymentService, it must propagate the active correlation ID in the outgoing HTTP request headers.

For RestTemplate:

package com.example.orderservice.config;

import org.slf4j.MDC;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.ClientHttpRequestInterceptor;
import org.springframework.web.client.RestTemplate;

import java.util.Collections;

@Configuration
public class RestTemplateConfig {

    @Bean
    public RestTemplate restTemplate() {
        RestTemplate restTemplate = new RestTemplate();

        ClientHttpRequestInterceptor interceptor = (request, body, execution) -> {
            String correlationId = MDC.get("correlationId");
            if (correlationId != null) {
                request.getHeaders().add("X-Correlation-Id", correlationId);
            }
            return execution.execute(request, body);
        };

        restTemplate.setInterceptors(Collections.singletonList(interceptor));
        return restTemplate;
    }
}

For OpenFeign (RequestInterceptor):

package com.example.orderservice.config;

import feign.RequestInterceptor;
import org.slf4j.MDC;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class FeignClientConfig {

    @Bean
    public RequestInterceptor feignCorrelationIdInterceptor() {
        return requestTemplate -> {
            String correlationId = MDC.get("correlationId");
            if (correlationId != null) {
                requestTemplate.header("X-Correlation-Id", correlationId);
            }
        };
    }
}

---

The Thread Pool Trap with MDC

Because MDC is backed by ThreadLocal, asynchronous operations break MDC propagation:

@Async
public void runInBackground() {
    // RUNS ON A DIFFERENT WORKER THREAD!
    // MDC is empty here! correlationId is lost!
    log.info("Processing asynchronous task");
}

When execution shifts to a worker thread from an @Async thread pool or CompletableFuture, the new thread does not inherit the parent thread's ThreadLocal context.

The Solution: TaskDecorator

Configure a TaskDecorator on your Spring ThreadPoolTaskExecutor:

package com.example.orderservice.config;

import org.slf4j.MDC;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.task.TaskDecorator;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

import java.util.Map;
import java.util.concurrent.Executor;

@Configuration
public class AsyncConfig {

    @Bean(name = "taskExecutor")
    public Executor taskExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(5);
        executor.setMaxPoolSize(10);
        executor.setQueueCapacity(100);
        executor.setThreadNamePrefix("async-worker-");

        // Copy MDC context from caller thread to worker thread
        executor.setTaskDecorator(new MdcTaskDecorator());
        executor.initialize();
        return executor;
    }

    public static class MdcTaskDecorator implements TaskDecorator {
        @Override
        public Runnable decorate(Runnable runnable) {
            // Captured on calling thread
            Map<String, String> contextMap = MDC.getCopyOfContextMap();

            return () -> {
                try {
                    // Restored on worker thread
                    if (contextMap != null) {
                        MDC.setContextMap(contextMap);
                    }
                    runnable.run();
                } finally {
                    // Cleaned up on worker thread
                    MDC.clear();
                }
            };
        }
    }
}

---

Interview Questions & Pitfalls

Q1: What is the underlying data structure supporting SLF4J MDC, and what memory leak risk does it introduce?

MDC is backed by ThreadLocal. In web servers that use thread pools (such as embedded Tomcat or Netty worker pools), threads are never destroyed; they are returned to the pool to handle subsequent requests. If you fail to clean up MDC in an afterCompletion or finally block, stale correlation IDs will pollute unrelated subsequent requests, and retained object references can cause severe memory leaks. Always call MDC.remove() or MDC.clear().

Q2: What is the difference between a Correlation ID and a Trace ID?

A Correlation ID is typically an application level identifier generated at the ingress gateway to tie related log lines together across services. A Trace ID is an OpenTelemetry and W3C standard identifier used in distributed tracing frameworks to map complete spans, parent child latency graphs, and timing trees across microservices. In modern architectures, systems often use the Trace ID as the correlation ID.

Q3: Why does MDC data vanish when executing code inside @Async methods or CompletableFuture?

MDC values live in the ThreadLocal storage of the initiating HTTP thread. When execution delegates to a different thread in a thread pool, the worker thread has its own empty ThreadLocal map. To propagate MDC across thread boundaries, you must use a TaskDecorator on your thread pool executor to capture the context map and copy it onto the worker thread.

Q4: How do you configure Logback to automatically print MDC values in every log statement?

In your logback-spring.xml pattern encoder, use the %X{keyName} syntax. For example, %X{correlationId} extracts the value associated with the correlationId key from MDC and prints it inside your log layout.

Q5: What is the purpose of returning X-Correlation-Id in the HTTP response headers?

Returning the correlation ID in the response headers allows the frontend client or calling API partner to capture the ID. If an API returns an error or times out, the client can submit that exact correlation ID in a support ticket, allowing engineering to find all relevant logs across all backend services in seconds.