API Gateway: Routing and Load Balancing

So far each client has called services directly: order-service on 8081, product-service on 8082. A real system has many clients (web, mobile, partners) and many services (orders, products, invoices, sales...). Letting every client know every service's address does not scale.

An API gateway is a single entry point in front of all the microservices. Clients send every request to the gateway; the gateway forwards it to the right service. Clients never talk to services directly.

Because every request passes through one place, that place is where cross-cutting concerns belong:

1. Routing: forward each request to the right service.
2. Load balancing across a service's instances.
3. Authentication (for example, validating a JWT) once, instead of in every service.
4. Rate limiting at the front door, before traffic reaches the services.
5. Resilience: circuit breakers, retries.
6. Request and response transformation.
7. Monitoring and logging of all traffic.

This chapter covers the first two; chapter 11 covers authentication.

Why routing in one place matters

Services change. A large service gets split into two; two small ones get merged; one moves to new machines. If clients call services directly, every client must update its addresses and routing for each such change, and with a hundred clients that makes every change expensive.

With a gateway, clients only know the gateway. A split or merge means updating the gateway's routes, one application, and no client notices.

Step 1: two plain services

product-service (port 8082):

@RestController
@RequestMapping("/products")
public class ProductController {

    @GetMapping("/{id}")
    public ResponseEntity<String> getProduct(@PathVariable String id) {
        return ResponseEntity.ok("fetch the product details with id: " + id);
    }
}

order-service (port 8081):

@RestController
@RequestMapping("/orders")
public class OrderController {

    @GetMapping("/{id}")
    public ResponseEntity<String> getOrder(@PathVariable String id) {
        return ResponseEntity.ok("fetch the order details with id: " + id);
    }
}

Step 2: the gateway application

Create a third project on start.spring.io with the Gateway dependency (listed as "Reactive Gateway" in recent versions):

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>

with the Spring Cloud BOM from chapter 3.

Spring Cloud Gateway is built on Spring WebFlux (reactive, non-blocking), not Spring MVC. Do not add spring-boot-starter-web. If it is present, the gateway refuses to start:

Spring MVC found on classpath, which is incompatible with Spring Cloud Gateway.
Please set spring.main.web-application-type=reactive or remove spring-boot-starter-web dependency.

Step 3: define the routes

A route says: requests matching these conditions go to this destination. In the gateway's application.properties:

spring.application.name=apigateway
server.port=8083

# Route 0: /products/** -> product-service
spring.cloud.gateway.routes[0].id=product-service
spring.cloud.gateway.routes[0].uri=http://localhost:8082
spring.cloud.gateway.routes[0].predicates[0]=Path=/products/**

# Route 1: /orders/** -> order-service
spring.cloud.gateway.routes[1].id=order-service
spring.cloud.gateway.routes[1].uri=http://localhost:8081
spring.cloud.gateway.routes[1].predicates[0]=Path=/orders/**

routes is a list: index 0, 1, 2... one entry per route. Each has:

• id: a unique name for the route.
• uri: where matching requests go (host and port of the service).
• predicates: also a list; the conditions a request must meet. All of a route's predicates must match.

Path=/products/** matches any path starting with /products/. The path is forwarded unchanged, so GET localhost:8083/products/1 becomes GET localhost:8082/products/1.

Other predicates match on the method, headers, query parameters, host, time and more. For example, to send only GET and POST requests to product-service:

spring.cloud.gateway.routes[0].predicates[1]=Method=GET,POST

Path is by far the most used; Method is the next most common.

Start all three applications and call only the gateway:

GET localhost:8083/products/1  ->  fetch the product details with id: 1
GET localhost:8083/orders/1    ->  fetch the order details with id: 1

A request that matches no route gets 404. So does one that matches the path but fails another predicate: with Method=GET on the product route, a POST /products/1 returns 404.

Step 4: load balancing through service discovery

uri=http://localhost:8082 sends everything to one instance. With several product-service instances, the gateway should find them and spread the load, which is chapters 4 and 5 again.

Register the services with Eureka. Add the Eureka client to product-service and order-service:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>

# product-service
server.port=8082
spring.application.name=product-service
eureka.client.service-url.defaultZone=http://localhost:8761/eureka

(order-service the same, with its own name and port.) The Eureka server is the one from chapter 4, on port 8761.

Make the gateway a Eureka client too, so it can look services up. Add the same spring-cloud-starter-netflix-eureka-client dependency, then point the routes at service IDs with the lb:// scheme:

spring.application.name=apigateway
server.port=8083
eureka.client.service-url.defaultZone=http://localhost:8761/eureka

spring.cloud.gateway.routes[0].id=product-service
spring.cloud.gateway.routes[0].uri=lb://product-service
spring.cloud.gateway.routes[0].predicates[0]=Path=/products/**

spring.cloud.gateway.routes[1].id=order-service
spring.cloud.gateway.routes[1].uri=lb://order-service
spring.cloud.gateway.routes[1].predicates[0]=Path=/orders/**

lb://product-service tells the gateway to use Spring Cloud LoadBalancer: fetch the instances of product-service from Eureka, choose one with the load-balancing algorithm (round robin by default), and forward the request there. Everything from chapter 5, including changing the algorithm per service, applies unchanged.

With two product-service instances on 8082 and 8084:

GET /products/1 -> 8084
GET /products/2 -> 8082
GET /products/3 -> 8084
GET /products/4 -> 8082

This is client-side load balancing inside the gateway: the gateway is the client of product-service.

Next: Every request now enters through the gateway, which makes it the natural place to check who is calling. Chapter 11 adds authentication at the gateway.

Check yourself

1. What is an API gateway, and why does its position matter?
2. Name five concerns a gateway can handle.
3. How does a gateway make splitting or merging services cheaper?
4. Why must a Spring Cloud Gateway application not include spring-boot-starter-web?
5. What are a route's id, uri and predicates?
6. With predicates Path=/products/** and Method=GET, what happens to POST /products/1?
7. What does GET localhost:8083/products/1 become with uri=http://localhost:8082?
8. What does lb://product-service mean, and what does the gateway need for it to work?

Answers

1. A single entry point in front of all microservices: clients call only the gateway, which forwards each request. Since all traffic passes through it, cross-cutting concerns can be handled there once.
2. Any five of: routing, load balancing, authentication, rate limiting, resilience (circuit breaker, retry), request/response transformation, monitoring and logging.
3. Clients only know the gateway, so only the gateway's routes change; no client has to be updated.
4. The gateway runs on Spring WebFlux; with Spring MVC on the classpath it fails to start.
5. id: a unique name; uri: where matching requests go; predicates: the conditions (all must match) a request must meet to use the route.
6. It matches no route (all predicates must match), so the gateway returns 404.
7. GET localhost:8082/products/1: the path is forwarded unchanged.
8. Forward to an instance of the service named product-service, chosen by Spring Cloud LoadBalancer from the instances in service discovery. The gateway must be a discovery client (e.g. Eureka client) and the services must be registered.