Distributed Tracing in Depth | Micrometer and OpenTelemetry

The Relay Race Baton Analogy

Imagine an Olympic four by one hundred meter relay race. You want to understand why your team took forty seconds to complete the race instead of thirty eight seconds. If you only look at individual snapshots of each runner, you cannot see where time was lost. Did runner number one have a slow start off the blocks? Did the baton pass from runner two to runner three take an extra half second? Did runner four stumble on the final turn?

To diagnose performance, the team attaches an electronic chip to the baton. The chip records the total race time, the split time for each leg of the track, and the exact handover duration between athletes.

In distributed microservices, Distributed Tracing is that electronic relay baton. When a user submits an order, a request flows through the API Gateway, passes to the Order Service, invokes the Payment Service, updates the Inventory Service, and writes to a database. Distributed tracing tracks that request across every hop, measuring exact latencies, identifying slow database queries, and visualizing the entire execution tree in tools like Zipkin or Jaeger.

This lecture covers logs versus traces, OpenTelemetry and W3C trace context standards, Spring Boot 3 Micrometer Tracing architecture, spans and trace IDs, and integrating with Zipkin for visual latency analysis.

Logs vs Traces: Understanding the Difference

A classic system design interview question is: "If we already have distributed logging, why do we need distributed tracing?"

Characteristic	Distributed Logging (ELK, Logback, MDC)	Distributed Tracing (Micrometer, OpenTelemetry, Zipkin)
Primary Question	"What happened?"	"Where was time spent?"
Data Structure	Discrete text or JSON event records	Directed Acyclic Graph (tree of spans with timing metadata)
Performance Insight	Captures messages, stack traces, and variable values	Measures exact duration of each inter service call and database query
Visualization	Search tables and text queries in Kibana	Waterfall latency flame graphs and dependency maps
Overhead	Higher storage footprint per event	Lightweight binary headers; often sampled (e.g. 5% in production)

Logs tell you that an error occurred on line 52 of Payment Service. Tracing tells you that the entire request took 4.2 seconds, and 3.8 of those seconds were spent waiting on a slow PostgreSQL table lock in Inventory Service.

Core Concepts: Trace, Span, and Parent ID

Distributed tracing relies on standardized data models governed by the W3C Trace Context specification:

[ Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736 ]
   |
   +---> Span A: API Gateway (Total: 420ms)
            |
            +---> Span B: Order Service (Total: 380ms) [Parent: Span A]
                     |
                     +---> Span C: Inventory DB Call (20ms) [Parent: Span B]
                     |
                     +---> Span D: Payment Service HTTP (340ms) [Parent: Span B]
                              |
                              +---> Span E: External Bank API (310ms) [Parent: Span D]

1. Trace ID

A single globally unique 16 byte hexadecimal string representing the entire journey of a request from client entry to final response across all microservices. Every span in that call chain shares the exact same Trace ID.

2. Span

A single unit of work with a start time and duration. Examples include an HTTP request, a database query, or publishing an event to a Kafka topic. A span contains:

Span ID: Unique 8 byte identifier for this specific unit of work.
Parent Span ID: Identifies the calling span that initiated this work.
Timestamps: Start time and end time (yielding duration).
Tags: Key value pairs with metadata (e.g. http.status_code=200, db.system=postgresql).
Events: Point in time annotations (e.g. cache_miss).

3. W3C `traceparent` Header

Standard header used to propagate context across HTTP boundaries:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              |  |                                |                |
        version  Trace ID                          Parent Span ID   Sampled flag (01=yes)

Spring Boot 3 Tracing Architecture: Micrometer Tracing

In Spring Boot 2, Spring Cloud Sleuth was the standard tracing tool. In Spring Boot 3, Spring Cloud Sleuth was retired. Tracing was migrated into the Micrometer core ecosystem under Micrometer Tracing.

Micrometer Tracing acts as a facade (similar to SLF4J for logging). It allows you to write tracing code against a single API, while plugging in different underlying tracing bridges (Brave or OpenTelemetry) and reporting backends (Zipkin, Jaeger, Wavefront):

       [ Spring Boot 3 Application ]
                     |
        [ Micrometer Tracing API ]  (Facade)
                     |
      +--------------+--------------+
      |                             |
[ Brave Bridge ]        [ OpenTelemetry (OTel) Bridge ]
      |                             |
      +--------------+--------------+
                     |
          [ Tracer Reporters ]
          - Zipkin Reporter
          - OTLP (OpenTelemetry Protocol)
          - Jaeger

Implementing Micrometer Tracing with Zipkin

1. Add Dependencies in `pom.xml`

xml

<!-- Spring Boot Actuator exposes metrics and observation endpoints -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<!-- Micrometer Tracing Bridge using OpenTelemetry -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>

<!-- Zipkin reporter to ship traces to Zipkin server -->
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>

2. Configure `application.properties`

properties

server.port=8081
spring.application.name=order-service

# Management & Actuator
management.endpoints.web.exposure.include=health,info,metrics,prometheus

# Tracing Configuration
management.tracing.enabled=true

# Sampling probability: 1.0 = 100% of requests traced (ideal for development)
# For production, set to 0.05 (5%) or 0.10 (10%) to minimize network overhead
management.tracing.sampling.probability=1.0

# Zipkin server endpoint
management.zipkin.tracing.endpoint=http://localhost:9411/api/v2/spans

Running Zipkin Server via Docker

You can spin up a local Zipkin dashboard using a single Docker command:

bash

docker run -d -p 9411:9411 openzipkin/zipkin

Open your browser at http://localhost:9411. Once your microservices make calls, Zipkin displays interactive waterfall visualizations showing every span and its duration.

Creating Custom Spans in Java

While Spring Boot automatically instruments RestTemplate, RestClient, WebClient, and Spring MVC controllers, you can also create custom spans to measure private algorithms or internal business routines:

java

package com.example.orderservice.service;

import io.micrometer.tracing.Span;
import io.micrometer.tracing.Tracer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class OrderProcessingService {

    @Autowired
    private Tracer tracer;

    public void processOrder(String orderId) {
        // Create and start custom span
        Span customSpan = tracer.nextSpan().name("fraud-detection-check").start();

        try (Tracer.SpanInScope ws = tracer.withSpan(customSpan.tag("order.id", orderId))) {
            // Simulated heavy calculation
            executeFraudCheckAlgorithm(orderId);
            customSpan.event("fraud_check_passed");
        } catch (Exception e) {
            customSpan.error(e);
            throw e;
        } finally {
            // End the span to record its duration
            customSpan.end();
        }
    }

    private void executeFraudCheckAlgorithm(String orderId) {
        try {
            Thread.sleep(85);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
}

Sampling Strategies in Production

Tracing every single request in a production system handling fifty thousand requests per second would generate petabytes of trace data and saturate network links.

To balance visibility with performance, tracing frameworks use sampling:

Always Sample (1.0): Used strictly in local development or staging environments to debug individual requests.
Probabilistic Sampling (0.05 to 0.10): Samples five to ten percent of random requests. Statistically sufficient to capture slow database queries, intermittent network spikes, and general latency distributions.
Adaptive / Rate Limiting Sampling: Guarantees sampling a fixed ceiling of requests per second (e.g. 100 requests per second maximum), protecting tracing collector nodes from overload during unexpected traffic spikes.

Interview Questions & Pitfalls

Q1: What replaced Spring Cloud Sleuth in Spring Boot 3?

Spring Cloud Sleuth was retired in Spring Boot 3. It was replaced by Micrometer Tracing, which lives within the core Micrometer project. Micrometer Tracing acts as a vendor neutral facade supporting both OpenTelemetry and Brave tracer engines.

Q2: What is the difference between a Trace ID and a Span ID?

A Trace ID is a unique identifier shared by all units of work across all microservices involved in handling a single high level user request. A Span ID is a unique identifier representing one specific operation (an HTTP call, a database query) within that trace.

Q3: What is the W3C traceparent header?

The traceparent header is the industry standard format for propagating distributed tracing context over HTTP. It contains four fields separated by hyphens: version, trace ID (16 bytes), parent span ID (8 bytes), and trace flags (such as whether the request was sampled).

Q4: Why should production systems avoid setting management.tracing.sampling.probability=1.0?

Tracing 100% of production traffic generates massive network bandwidth and storage overhead. It can degrade application throughput and overwhelm trace collector nodes (Zipkin, Jaeger). Setting sampling to 5% or 10% provides statistically accurate performance data with minimal overhead.

Q5: How does Micrometer Tracing correlate traces with your existing SLF4J logs?

When Micrometer Tracing is enabled, it automatically injects the active traceId and spanId into SLF4J's Mapped Diagnostic Context (MDC). If your Logback pattern includes %X{traceId} and %X{spanId}, your standard application log lines automatically display the active trace IDs, linking logs and traces together.

Distributed Tracing in Depth | Micrometer and OpenTelemetry ​

The Relay Race Baton Analogy ​

Logs vs Traces: Understanding the Difference ​

Core Concepts: Trace, Span, and Parent ID ​

1. Trace ID ​

2. Span ​

3. W3C traceparent Header ​

Spring Boot 3 Tracing Architecture: Micrometer Tracing ​

Implementing Micrometer Tracing with Zipkin ​

1. Add Dependencies in pom.xml ​

2. Configure application.properties ​

Running Zipkin Server via Docker ​

Creating Custom Spans in Java ​

Sampling Strategies in Production ​

Interview Questions & Pitfalls ​