Client-Side Load Balancing

A load balancer spreads traffic across the instances of a service so no single instance is overloaded. Chapter 4 showed that Feign already does this: four calls alternated between two product-service instances. This chapter looks at how, and how to change it.

Server-side and client-side

Server-side load balancing: a separate, central component sits in front of the instances. The caller sends every request to it and it forwards each one. The caller has no balancing logic. Examples: Nginx, AWS Elastic Load Balancer.

Client-side load balancing: the caller does it, usually through a library. It gets the list of instances from service discovery, picks one, and calls it directly. Examples: Spring Cloud LoadBalancer, Netflix Ribbon (now retired), a service mesh sidecar such as Istio's.

This chapter covers the client side. A central entry point in front of services returns with the API gateway in chapter 10.

From manual choice to @LoadBalanced

Chapter 4 chose the instance by hand:

List<ServiceInstance> instances = discoveryClient.getInstances("product-service");
URI uri = instances.get(0).getUri();                 // always the first: no balancing
String response = restTemplate.getForObject(uri + "/products/" + id, String.class);

Hand it to Spring Cloud LoadBalancer instead.

Step 1: the dependency

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-loadbalancer</artifactId>
</dependency>

(The Eureka client starter already includes it.)

Step 2: mark the RestTemplate

@Configuration
public class Config {

    @Bean
    @LoadBalanced
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }
}

@LoadBalanced adds a LoadBalancerInterceptor to this RestTemplate. Every request it sends is intercepted; the interceptor looks up the instances, applies the load-balancing algorithm and rewrites the URL to the chosen host and port.

Step 3: use the service ID as the host

server.port=8081
spring.application.name=order-service
eureka.client.service-url.defaultZone=http://localhost:8761/eureka

product.service.baseurl=http://product-service

@RestController
@RequestMapping("/orders")
public class OrderController {

    @Autowired
    RestTemplate restTemplate;

    @Value("${product.service.baseurl}")
    String productBaseURL;

    @GetMapping("/{id}")
    public void callProductAPI(@PathVariable String id) {
        String response = restTemplate.getForObject(
                productBaseURL + "/products/" + id, String.class);
        System.out.println("Response from Product api call is: " + response);
    }
}

http://product-service/products/1 is not a real address. product-service is the service ID, and the interceptor replaces it with a real instance. No DiscoveryClient, no choosing code.

With three product-service instances, eight calls go round in order:

[8083, 8084, 8082, 8083, 8084, 8082, 8083, 8084]

A RestClient works the same way: annotate a RestClient.Builder bean with @LoadBalanced and build clients from it.

What the interceptor does

Recall from chapter 1 that RestTemplate runs its interceptors just before executing the request. LoadBalancerInterceptor does two things there:

1. loadBalancerClientFactory.getInstance(serviceId): get the load-balancing algorithm for this service ID.
2. loadBalancer.choose(request): ask service discovery for the instances, then let that algorithm pick one.

Every algorithm belongs to a service ID

The interceptor's first step hides the key idea: a load-balancing algorithm is always attached to one service ID. An order-service calling product-service, sales-service and payment-service has one load balancer per service, and each can use a different algorithm: random for product-service, round robin for sales-service, something custom for payment-service.

Spring Cloud keeps them apart by creating a small Spring context for each service ID, holding that service's load balancer and instance supplier. Each context is created lazily, on the first call to that service, not at startup.

The default algorithm is RoundRobinLoadBalancer, for every service. Spring Cloud LoadBalancer ships two algorithm classes:

Other strategies come from filtering the instance list rather than from new algorithm classes. Set spring.cloud.loadbalancer.configurations to weighted, zone-preference, health-check, same-instance-preference or request-based-sticky-session, and round robin then runs over the filtered list. For anything else (least connections, response time), write your own algorithm (below) or use a service mesh such as Istio.

Changing the algorithm for one service

Use random for product-service. Two pieces.

1. Name the service and its configuration on the application class:

@SpringBootApplication
@LoadBalancerClient(name = "product-service",
                    configuration = LoadBalancerProductClientConfig.class)
public class OrderServiceApplication { ... }

2. Define the algorithm in that configuration:

public class LoadBalancerProductClientConfig {      // no @Configuration

    @Bean
    public ReactorLoadBalancer<ServiceInstance> productClientLoadBalancer(
            LoadBalancerClientFactory factory) {

        return new RandomLoadBalancer(
                factory.getLazyProvider("product-service", ServiceInstanceListSupplier.class),
                "product-service");
    }
}

The RandomLoadBalancer constructor takes two arguments:

• a ServiceInstanceListSupplier (wrapped in a lazy ObjectProvider): the object that asks service discovery for the instances of product-service;
• the service ID this algorithm is attached to.

This configuration class is not part of the main application context: Spring Cloud loads it into product-service's own context, the first time product-service is called. Calls are now random:

[8084, 8083, 8084, 8084, 8084, 8082, 8082, 8084]

Leave @Configuration off

It is tempting to annotate LoadBalancerProductClientConfig with @Configuration. If you do, and it sits inside your component-scan package, it is also loaded into the main context, and every service's context inherits its bean from there. Then every service uses this random balancer, and since that balancer is tied to product-service, calls to sales-service end up on product-service instances:

sales-service -> [8083, 8084, 8083, 8082, ...]     <- product-service ports!

Classes passed to @LoadBalancerClient must either have no @Configuration annotation or live outside component scanning.

One algorithm for one service, another for the rest

To set a default for all services plus exceptions, use @LoadBalancerClients:

@SpringBootApplication
@LoadBalancerClients(
        defaultConfiguration = LoadBalancerGlobalConfig.class,
        value = {
                @LoadBalancerClient(name = "product-service",
                                    configuration = LoadBalancerProductClientConfig.class)
        })
public class OrderServiceApplication { ... }

The global configuration cannot hardcode a service ID: it serves every service. The service's context knows its own ID, under the property LoadBalancerClientFactory.PROPERTY_NAME:

public class LoadBalancerGlobalConfig {

    @Bean
    @ConditionalOnMissingBean
    public ReactorLoadBalancer<ServiceInstance> globalLoadBalancer(
            LoadBalancerClientFactory factory, Environment environment) {

        String serviceId = environment.getProperty(LoadBalancerClientFactory.PROPERTY_NAME);

        return new RoundRobinLoadBalancer(
                factory.getLazyProvider(serviceId, ServiceInstanceListSupplier.class),
                serviceId);
    }
}

This is why the per-service contexts matter: the method runs once in each service's context, with that service's ID.

@ConditionalOnMissingBean is required. product-service's context loads its own configuration first, then the default one. Without the condition, it ends up with two load balancers (random and round robin), cannot choose between them, and every call to product-service fails with an IllegalStateException. With the condition, the global bean is skipped wherever a service already has one.

The result, logged on first use:

>>> product config: creating RandomLoadBalancer
>>> global config: creating RoundRobinLoadBalancer for sales-service

product-service -> random
sales-service   -> [8092, 8093, 8092, 8093, ...]

Each bean is created once per service, on the first call, and reused for every call after.

A custom algorithm

Implement ReactorServiceInstanceLoadBalancer and its choose method, taking the same two constructor arguments as the built-in ones:

public class MyCustomLoadBalancer implements ReactorServiceInstanceLoadBalancer {

    private final ObjectProvider<ServiceInstanceListSupplier> serviceInstanceSuppliers;
    private final String serviceId;

    public MyCustomLoadBalancer(ObjectProvider<ServiceInstanceListSupplier> serviceInstanceSuppliers,
                                String serviceId) {
        this.serviceInstanceSuppliers = serviceInstanceSuppliers;
        this.serviceId = serviceId;
    }

    @Override
    public Mono<Response<ServiceInstance>> choose(Request request) {
        return serviceInstanceSuppliers
                .getIfAvailable()
                .get()                         // ask service discovery for the instances
                .next()
                .map(instances -> {
                    if (instances == null || instances.isEmpty()) {
                        return new EmptyResponse();
                    }
                    // your algorithm here (demo: always the first instance)
                    return new DefaultResponse(instances.get(0));
                });
    }
}

RoundRobinLoadBalancer and RandomLoadBalancer have exactly this shape; only the line that picks an instance differs. Plug it in like the random one:

public class LoadBalancerProductClientConfig {

    @Bean
    public ReactorLoadBalancer<ServiceInstance> productClientLoadBalancer(
            LoadBalancerClientFactory factory) {
        return new MyCustomLoadBalancer(
                factory.getLazyProvider("product-service", ServiceInstanceListSupplier.class),
                "product-service");
    }
}

Every call to product-service now goes to the first instance: [8082, 8082, 8082, ...].

With Feign

Nothing changes. Feign uses the same load balancer for @FeignClient(name = "product-service"), so the same @LoadBalancerClient / @LoadBalancerClients configuration applies:

@SpringBootApplication
@EnableFeignClients
@LoadBalancerClient(name = "product-service",
                    configuration = LoadBalancerProductClientConfig.class)
public class OrderServiceApplication { ... }

Next: Load balancing spreads healthy traffic. The next chapters protect a service when traffic or failures get out of hand, starting in chapter 6 with the rate limiter.

Check yourself

1. What is the difference between server-side and client-side load balancing? Give an example of each.
2. What does @LoadBalanced add to a RestTemplate?
3. In http://product-service/products/1, what is product-service?
4. What two steps does LoadBalancerInterceptor perform?
5. Why is every load-balancing algorithm tied to a service ID?
6. When is a service's load-balancer configuration loaded?
7. What is the default algorithm, and which two algorithm classes ship with Spring Cloud LoadBalancer?
8. What two arguments does RandomLoadBalancer take, and what does each do?
9. What goes wrong if the class passed to @LoadBalancerClient is a scanned @Configuration?
10. How does a global configuration find out which service it is configuring?
11. Why does the global load balancer bean need @ConditionalOnMissingBean?
12. What do you implement for a custom algorithm?

Answers

1. Server-side: a central component in front of the instances forwards each request (Nginx, ELB). Client-side: the caller gets the instance list and picks one itself (Spring Cloud LoadBalancer, Istio sidecar).
2. A LoadBalancerInterceptor that, for every request, resolves the service ID in the URL to a chosen instance.
3. The service ID (the target's spring.application.name), which the load balancer replaces with a real host and port.
4. It gets the algorithm for the service ID (loadBalancerClientFactory.getInstance(serviceId)) and asks it to choose an instance (choose), which fetches the instances from service discovery.
5. Each service can need a different algorithm, so Spring Cloud keeps a separate load balancer (in a separate small context) for each service ID.
6. Lazily, the first time that service is called; then it is reused.
7. Round robin; RoundRobinLoadBalancer and RandomLoadBalancer.
8. A lazy provider of ServiceInstanceListSupplier, which fetches the instances from service discovery, and the service ID the algorithm is attached to.
9. It is also loaded into the main context, so every service inherits its balancer, which is tied to product-service: calls to other services go to product-service instances.
10. From the environment property LoadBalancerClientFactory.PROPERTY_NAME in the service's own context.
11. A service with its own configuration would otherwise get two load balancer beans and fail; the condition skips the global one there.
12. ReactorServiceInstanceLoadBalancer, with a constructor taking the supplier provider and service ID, and a choose method that picks from the instances.