Reverse Proxy vs Load Balancer vs API Gateway

The question

Put a layer 4 load balancer in front of six gRPC servers, and send traffic from four clients. Two servers sit idle and four run hot. Add six more servers and nothing changes. Swap in a layer 7 proxy and the load spreads evenly, but now that box must decrypt and parse every request. Then make one server three times slower: round robin, the default almost everywhere, keeps feeding it its full share and its queue grows by seconds.

Reverse proxies, load balancers and API gateways are usually compared by feature lists. The comparison that explains the behaviour above is different: what each box can read, and whether it decides once per connection or once per request.

What they have in common

All three are intermediaries that work for the server side: the client connects to them, they open their own connection to a server behind them (the backend, upstream or origin), and the client cannot tell how many servers are behind. HTTP’s own specification calls this kind of intermediary a gateway (RFC 9110, section 3.7); most engineers call it a reverse proxy. So “gateway” meant “reverse proxy” long before anyone sold an API gateway, which is half of the naming confusion.

The other half is that these are roles, not kinds of machine. nginx, HAProxy and Envoy can each play all three; Kong is an API gateway built on nginx; AWS’s Application Load Balancer is called a load balancer but routes by path and can check logins.

The layer numbers come from the OSI model, a numbered stack of networking jobs in which each layer wraps the one above it. Three layers matter here. Layer 3 is IP: addresses that get a packet to a machine. Layer 4 is TCP or UDP: port numbers that get bytes to one program on that machine and, for TCP, a connection that delivers them in order. Layer 7 is the application protocol, here HTTP: method, path, headers and body. Layers 5 and 6 are rarely named in practice, and TLS sits between 4 and 7, encrypting all of layer 7. So GET /orders/42 over HTTPS travels as IP packets from 203.0.113.7 to the server, inside a TCP connection from port 51234 to port 443, carrying TLS records whose encrypted contents are the HTTP request. A box that reads only the outer two wrappers knows who is talking to which port, and nothing about orders.

What separates the roles is which of those layers a box reads:

• A layer 4 (L4) box sees TCP or UDP: addresses, ports, bytes. It never decrypts TLS, so it cannot read a path, a header or an API key, and the unit it can balance is a connection.
• A layer 7 (L7) box terminates the client’s connection, decrypts it and parses HTTP. It can read and change everything, and the unit it can balance is a request.

Terminating a connection means being its far end. The client’s TCP and TLS handshakes complete with the proxy, not with your server; the proxy decrypts the bytes, reads the request, and sends it on over a second, separate connection to a backend, often one it opened earlier and keeps for reuse. A layer 4 box does not terminate TLS: the TLS session runs from the client to the backend and the box passes its encrypted bytes along. Some L4 balancers (Google’s Maglev, Linux IPVS) forward the client’s packets so that even the TCP connection runs end to end; others, such as HAProxy in TCP mode, accept the TCP connection and open their own, but still never see inside the TLS.

Enlarge figure

Passing through versus terminating. Behind a layer 4 balancer the client's TLS session runs all the way to the app, unread. A layer 7 proxy completes the client's handshake itself, reads the request, and forwards it on a second connection of its own.

So one request through a layer 7 proxy rides two connections, each with its own handshake and timeout. Every box adds such a hop, and another component that fails for every request at once.

Each one on its own

Reverse proxy

Its job: stand in front of an application and handle the parts of HTTP the application should not have to. It is layer 7: it sees the whole request and response.

• TLS termination. It holds the certificate and decrypts; the TLS page covers re-encryption and passthrough.
• Buffering slow clients. Say Gunicorn runs 4 sync workers (each serves one request at a time, from its first byte to its last), each request needs 20 ms of real work, and 100 users on poor mobile links each take 2 s to upload a request. Without a proxy, each worker waits 2 s for bytes before 20 ms of work: 2 requests a second. With nginx in front holding the slow uploads (it is built for thousands of idle connections) and handing each complete request over in one go, the same 4 workers serve about 200 a second. The same mechanism defends against slowloris, an attack that ties up workers by sending bytes very slowly.
• Caching, compression, static files, and header rewriting: X-Forwarded-For, redirects to HTTPS, stripping internal headers from responses.

Enlarge figure

A reverse proxy in front of one app. It reads the whole request, which is what lets it end TLS, buffer, cache and rewrite headers. With a single upstream, nothing is being balanced.

Typical tools: nginx, Caddy, HAProxy, Envoy, Traefik; CDNs are reverse proxies run as a service. Reach for it in front of any app server that faces the internet, from day one; it is the cheapest of the three to run.

Load balancer

Its job: spread work over a pool of identical copies, and stop sending work to copies that are unhealthy. Whether it can do that per request depends entirely on its layer.

Enlarge figure

What each box can read in one HTTPS request. A layer 4 balancer reads addresses and ports and forwards the rest as bytes it cannot decrypt. Anything that ends TLS can read and change every header and the body.

A layer 4 balancer picks a backend when a connection opens and sends every packet of that connection there. Whether that spreads requests depends on how many requests share a connection, and that depends on the HTTP version. HTTP/1.1 carries one request at a time on a connection: the next request waits until the previous response has fully arrived, so a browser that wants six files at once opens up to six connections to the same host. HTTP/2 multiplexes: it cuts every request and response into small frames, labels each frame with a stream number (one stream per request), and interleaves the frames of many streams on one connection. Servers commonly allow 100 or more streams in progress at once (RFC 9113, section 6.5.2 recommends no fewer than 100), so a client has no reason to open a second connection. gRPC runs on HTTP/2, and a gRPC client usually keeps one connection per server address for the life of its process.

Per-connection balancing works when connections are short and numerous, as with HTTP/1.1 browsers. It breaks when they are long-lived and multiplexed. A gRPC client that sends 60 requests a second for an hour sends 216,000 requests on one connection, and the L4 balancer made one decision for all of them. The whole client lands on one backend, and the balancer cannot move individual requests, because it cannot see them. This is why gRPC behind a plain Kubernetes Service is famously uneven: a Service is a stable virtual address whose rules, written by kube-proxy on every node, pick one pod per new connection, which is an L4 mechanism (Kubernetes blog, 2018).

A layer 7 balancer is a reverse proxy with a pool behind it. It terminates the client’s connections, keeps its own pool of connections to every backend, and picks a backend for each request. That costs TLS and parsing on every request, and buys the right unit of balancing.

Two terms carry the rest of this section. A request is in flight, or outstanding, from the moment the balancer sends it to a backend until the response comes back; meanwhile it is either being worked on or waiting in that backend’s queue. Utilisation is how busy a backend is: the rate requests arrive divided by the rate it can finish them. A backend that finishes 100 requests a second and receives 60 is at 60%, and requests rarely wait. At 120 a second it is past 100%: 20 more arrive each second than leave, so after 10 s there are 200 waiting and a new request waits 2 s behind them; after a minute, 12 s. The queue never settles at a new level. It grows for as long as arrivals exceed the service rate. Even below 100%, requests arrive in random clumps, so short queues form and waits rise steeply as utilisation nears 100%.

The algorithm matters once requests are the unit:

• Round robin: each backend in turn. nginx’s default. Fair only if every backend is equally fast.
• Least connections / least outstanding requests: the backend with the fewest in flight. A slow backend accumulates in-flight requests, so it automatically gets fewer new ones.
• Power of two choices: pick two backends at random, send to the less loaded one. Nearly as good as checking every backend, and far cheaper at large pool sizes; Envoy’s least-request policy uses it (Envoy docs).
• Hashing a client address or key, when the same client must reach the same backend.

Why two random picks are enough deserves a number. Finding the least loaded of all backends means reading every backend’s count on every request, and when many balancer copies see the same “least loaded” backend at the same moment, they all send to it together and overload it. Two random picks avoid both problems and still dodge bad backends almost every time. With 10 backends of which 3 are overloaded, one random pick lands on an overloaded one 30% of the time; two picks lose only when both are overloaded: 3/10 × 2/9, about 7%.

The sim below reports p95 latency. Sort the response times of every request in a recent window and read the value that 95% of them are at or under. If 94 requests out of 100 take 20 ms and 6 take 2 s, the median (p50) is 20 ms and the average about 140 ms, but p95 is 2 s: the slow end of the distribution, the tail, is where those 6 users live, and an average hides them. The tail matters more than it looks. A page that makes 20 backend calls waits for the slowest one, and if each call has a 5% chance of being slow, about 64% of page loads hit at least one (1 − 0.95²⁰).

Here is all of it at once, on the same traffic: four long-lived connections at 60 requests a second each, six backends that can each serve 100 a second.

SimulationSame traffic, same backends, three balancers

PlayStepReset

What to watch for. With 4 connections and 6 backends, the layer 4 lane leaves two backends idle while the others work harder; the layer 7 lanes use all six. Raise connections to 8: two layer 4 backends now get two connections each, more than they can serve, and their queues grow without limit while the layer 7 lanes stay fast. Then set connections back to 4 and make backend 1 three times slower: round robin keeps sending it its full share and its queue explodes; least outstanding requests notices and sends it less.

Three effects to take away. Per-connection balancing leaves backends idle and overloads others as soon as connections outnumber or unevenly share the backends. Round robin is blind to a slow backend and builds a queue there that dominates the tail latency: at a third of the speed, backend 1 finishes about 33 requests a second but still receives its 40, so its queue gains about 7 every second. Least outstanding requests reacts to slowness without any health check at all, because a slow server’s in-flight count is the signal.

One trap in the other direction: a freshly started backend has nothing in flight, so least outstanding requests (and least connections) send it a burst of traffic while its caches are cold, its connection pools are empty and, on the JVM, the JIT compiler has not yet turned its hot code into machine code. The fix is slow start: ramp a new backend’s share up over 30 to 60 seconds. Envoy has a slow start mode for its round robin and least-request balancers; check that whatever you run has one before you autoscale.

A health check is the balancer asking each backend “can you serve?” and removing the ones that say no or do not answer. Active checks call each backend on a timer (GET /healthz every few seconds) and mark it down only after several failures in a row, so one lost packet does not eject a healthy server. That makes them slow by design: checks every 5 s with 3 failures needed leave a crashed backend in the pool for up to 15 s, which at 100 req/s is 1,500 requests sent to it. Passive checks watch real traffic and react to the failures themselves: in open-source nginx, one failed attempt takes a backend out for 10 s by default (max_fails=1, fail_timeout=10s in the upstream module, as of October 2026). Draining stops new work to a backend during a deploy while in-flight requests finish.

Enlarge figure

A load balancer in front of three copies of the same app. Its decisions are which copy gets each connection or request, and which copies to skip. Here the middle one failed its health check and gets nothing.

Layer 4 still earns its place. It is far cheaper per byte: Google’s Maglev fills a 10 Gbps link with small packets on one ordinary Linux machine, and Cloudflare’s Unimog spreads connections across every server in each data centre with XDP (a Linux hook that runs the balancer’s code on each packet as the network driver receives it, before the rest of the kernel’s network stack). Reach for L4 for non-HTTP traffic (databases, UDP game protocols, MQTT), for fixed IP addresses partners can allow-list, for TLS passthrough, and in front of your L7 tier to scale it. Reach for L7 whenever connections are long-lived, routing depends on the request, or backends differ in speed.

Who balances the load balancer? One L4 machine is a single point of failure, so large sites run many and spread traffic over them at layer 3, in the routers. A router learns where to send packets from route announcements: a machine tells it “send packets for 203.0.113.10 to me”, in practice over BGP, the internet’s routing protocol. If four balancers all announce 203.0.113.10, the router has four equally good next hops, and ECMP (equal-cost multi-path routing) picks one for each packet by hashing the packet’s addresses, ports and protocol. Every packet of a connection carries the same values, so it hashes the same way: a connection from 198.51.100.7 port 51234 keeps reaching, say, balancer 3.

Adding or removing a balancer changes the set that hash picks from, and some connections start arriving at a balancer that never saw them. Maglev answers with two mechanisms. Every Maglev machine picks a backend from the same consistent hashing table, so whichever machine a packet reaches, it goes to the same backend. “Consistent” means that when a backend joins or leaves, most connections keep their backend; a plain hash mod N would move about three quarters of them when N goes from 4 to 3. And each machine keeps a table of connections it already forwards, so those stay put even while the backend set changes.

Across regions, the same announcement trick runs on the internet. With anycast, every data centre announces the same address, and each network on the way delivers packets to the announcement it sees as nearest: a user in Mumbai connecting to Cloudflare’s DNS resolver at 1.1.1.1 reaches a data centre near Mumbai, and a user in Frankfurt reaches one in Frankfurt at the same address. “Nearest” is in routing terms (fewest networks to cross, adjusted by each network’s own policies), not in kilometres. DNS-based balancing (the DNS page) instead hands out different addresses per region. It is load balancing all the way down; only the layer, and what each layer can see, changes.

Typical tools: L4: AWS Network Load Balancer, Google Cloud network load balancers, HAProxy in TCP mode, Linux IPVS. L7: AWS Application Load Balancer, nginx, HAProxy, Envoy.

API gateway

Its job: be the single front door for many services that outside clients call as an API, and decide about the caller before any service sees the call. Everything a reverse proxy sees, plus who is calling and on which plan.

• Authentication: reject missing, unknown or expired keys and tokens (401) at the edge, and pass the verified identity inward, ideally as a signed token rather than a plain header.
• Per-caller rate limits and quotas: 100 req/s on the free plan, 2,000 on the paid one, a million a month. The rate limiting page covers the counters.
• Routing by path, version or header to different services; transformation between protocols (JSON over HTTP in front of gRPC); aggregation of several service calls into one.
• The API product: developer portal, self-service keys, usage analytics and billing.

Why put this in one place? With twelve services, each validating keys and counting calls its own way, revoking a leaked key means twelve services must notice, and changing a plan limit means twelve teams make the change. With a gateway, both are one change. Aggregation has a number too: a phone on a 150 ms round trip that needs three services pays at least 450 ms calling them in sequence; a gateway next to the services makes the three calls in about a millisecond each and the phone pays one round trip. One small gateway per kind of client is the backends for frontends pattern (Sam Newman).

Enlarge figure

An API gateway in front of three different services. It turns callers away before any service sees them (401 for no valid key, 429 for too many calls) and routes the rest by path. The boxes behind it are different services, not copies of one.

Typical tools: Kong, Apache APISIX, Tyk, Envoy Gateway, Spring Cloud Gateway, and managed services: Amazon API Gateway, Azure API Management, Google Apigee. Reach for it when callers you do not control (partners, third-party developers, mobile apps you cannot update quickly) reach several services as one API. With one service and your own frontend as the only caller, a gateway mostly adds a hop.

How they overlap

Enlarge figure

An API gateway is a reverse proxy with API jobs added. Load balancing is a job both a reverse proxy and a plain layer 4 balancer do, so it straddles the line between them. The only box that is not a reverse proxy at all is the layer 4 balancer, which never reads HTTP.

Read it from the outside in. Anything that terminates HTTP is a reverse proxy. A reverse proxy with a pool is a layer 7 load balancer. A reverse proxy that also knows the caller is an API gateway. The layer 4 balancer is the only different kind of thing, and it shares exactly one job with the rest.

When you evaluate a product, ignore its name and ask three questions. Does it terminate HTTP, or only forward bytes? Does it know who the caller is, or only what the request is? Does it decide per connection or per request? A “load balancer” that routes by path and checks logins is doing gateway work; a Kubernetes “Gateway” (the Gateway API’s Gateway and HTTPRoute) mostly describes routing and balancing, which is the reverse-proxy role.

Side by side

	Reverse proxy	Load balancer	API gateway
Main job	HTTP chores for an app	Spread work over identical copies	One front door for many APIs and their callers
Layer	7	4 or 7	7
Unit it decides on	Request	Connection (L4) or request (L7)	Request, plus the caller
Knows	The whole request	Addresses and ports (L4); the whole request (L7)	The request and who sent it
Typical features	TLS, buffering, caching, rewrites	Algorithms, health checks, draining	Auth, per-caller limits and quotas, routing, transformation, portal
Behind it	One app or a few	Copies of one service	Different services
Cost per request	Low	Very low (L4); like a proxy (L7)	Highest: identity lookups, shared counters, often billed per call
Typical failure	Buffering breaks streaming; bad forwarded headers	Uneven load, slow detection of slow backends	Bottleneck, or a home for business logic

How to choose

Enlarge figure

Each yes adds one role, usually to a box you already have. Most systems stop after the first or second question, and many small ones never need a separate gateway at all.

1. One app on one server: a reverse proxy (nginx, Caddy) for TLS and slow clients, or nothing if your platform already runs one.
2. More than one copy: load balancing, usually as configuration on the proxy you have or your cloud’s managed L7 balancer. Use least outstanding requests if backends can differ in speed.
3. Many services or outside callers: an API gateway, behind the balancer, when you need per-caller work. Path routing alone does not need a gateway product.
4. Non-HTTP traffic, fixed IPs, or very high connection rates: an L4 balancer in front of everything, which is also how the L7 tier itself scales.

Managed gateways come with limits you design around. Amazon API Gateway’s REST APIs, for example, give up on a backend after 29 seconds by default and cap payloads at 10 MB (AWS quotas, as of October 2026; since June 2024 the timeout can be raised for Regional and private REST APIs). A 40-second report export behind it fails every time, which is the right outcome: long work belongs in a job that returns 202 Accepted and a status URL, not in an open HTTP request through several hops.

See it in action

One process that plays each role in turn, so you can watch the roles stack: as a reverse proxy it forwards to one upstream and sets X-Forwarded-For; as a load balancer it round-robins over two copies and marks a failed one down for 10 s; as an API gateway it checks an API key, limits each key to 3 calls in 10 s, and routes by path. The gateway branch runs first, then the balancing loop all roles share.

Python

front_door.py

# runnable version: examples/proxy-lb-gateway/python/front_door.py
def do_GET(self) -> None:
    # Reverse proxy: forward to a backend, and say who the real client was. We overwrite any
    # X-Forwarded-For the client sent: this box is the trust boundary.
    headers = {"X-Forwarded-For": self.client_address[0]}
    pool = ROUTES["/orders"]

    if self.role == "gateway":
        user = API_KEYS.get(self.headers.get("X-Api-Key", ""))
        if user is None:
            return self.reply(401, {"error": "missing or unknown API key"})
        if not self.allow(user):
            return self.reply(429, {"error": f"limit is {LIMIT} per {WINDOW_S:.0f} s"})
        headers["X-User"] = user  # services trust this header, not the key
        prefix = next((p for p in ROUTES if self.path.startswith(p)), None)
        if prefix is None:
            return self.reply(404, {"error": f"no route for {self.path}"})
        pool = ROUTES[prefix]

    if self.role == "proxy":
        pool = pool[:1]  # a plain reverse proxy has one upstream

    # Load balancer: try each backend once, in round-robin order, skipping ones marked down.
    for _ in range(len(pool)):
        with self.lock:
            backend = pool[type(self).next_index % len(pool)]
            type(self).next_index += 1
        if self.down_until.get(backend, 0) > time.monotonic():
            continue
        try:
            return self.forward(backend, headers)
        except OSError:
            self.down_until[backend] = time.monotonic() + DOWN_FOR_S
            self.log(f"{backend} failed, marked down for {DOWN_FOR_S:.0f} s")
    self.reply(502, {"error": "no healthy backend"})

JavaScript

front-door.mjs

// runnable version: examples/proxy-lb-gateway/js/front-door.mjs
http.createServer(async (req, res) => {
  // Reverse proxy: forward to a backend, and say who the real client was. We overwrite any
  // X-Forwarded-For the client sent: this box is the trust boundary.
  const headers = { 'x-forwarded-for': req.socket.remoteAddress.replace('::ffff:', '') };
  let pool = ROUTES['/orders'];

  if (role === 'gateway') {
    const user = API_KEYS[req.headers['x-api-key'] ?? ''];
    if (!user) return reply(res, 401, { error: 'missing or unknown API key' });
    if (!allow(windows, user)) return reply(res, 429, { error: `limit is ${LIMIT} per 10 s` });
    headers['x-user'] = user; // services trust this header, not the key
    const prefix = Object.keys(ROUTES).find((p) => req.url.startsWith(p));
    if (!prefix) return reply(res, 404, { error: `no route for ${req.url}` });
    pool = ROUTES[prefix];
  }

  if (role === 'proxy') pool = pool.slice(0, 1); // a plain reverse proxy has one upstream

  // Load balancer: try each backend once, in round-robin order, skipping ones marked down.
  for (let i = 0; i < pool.length; i++) {
    const port = pool[nextIndex++ % pool.length];
    if ((downUntil.get(port) ?? 0) > Date.now()) continue;
    try {
      return await forward(port, req.url, headers, res);
    } catch {
      downUntil.set(port, Date.now() + DOWN_FOR_MS);
      console.log(`    [${role}] 127.0.0.1:${port} failed, marked down for 10 s`);
    }
  }
  reply(res, 502, { error: 'no healthy backend' });
});

Here is what it printed, trimmed to the interesting lines (Node prints the same, as JSON):

python front_door.py (output trimmed)

1. Reverse proxy: one upstream; the client's own X-Forwarded-For is replaced
  GET /orders/42   200  {'served_by': 'orders-1', 'x_forwarded_for': '127.0.0.1'}  (sent {'X-Forwarded-For': '6.6.6.6'})
2. Load balancer: round robin over two copies, then orders-2 dies
  GET /orders/0    200  {'served_by': 'orders-1', ...}
  GET /orders/1    200  {'served_by': 'orders-2', ...}
    [lb] 127.0.0.1:9002 failed, marked down for 10 s
  GET /orders/5    200  {'served_by': 'orders-1', ...}
3. API gateway: identity, a per-key limit, and routing by path
  GET /orders/1    401  {'error': 'missing or unknown API key'}
  GET /orders/3    429  {'error': 'limit is 3 per 10 s'}  (sent {'X-Api-Key': 'key-asha'})
  GET /users/ravi  200  {'served_by': 'users-1', ..., 'x_user': 'ravi'}  (sent {'X-Api-Key': 'key-ravi'})
  GET /admin       404  {'error': 'no route for /admin'}

Three things to notice. In part 1, the client claimed to be 6.6.6.6, and the backend saw 127.0.0.1 anyway: the proxy wrote the header itself instead of passing on what the client sent. In part 2, the request that landed on the dead orders-2 still succeeded, because the balancer retried it on orders-1 within the same request; the client never knew. In part 3, the services received x_user, never the API key itself. They trust the gateway, not the caller.

The demo is deliberately small. It has no TLS, no active health checks and no timeouts on the client side, and its rate limiter is a fixed window that the rate limiting page explains how to improve. Do not put it in production; put nginx there. Here are the same three roles in one nginx configuration, trimmed to the parts that matter (the full file and a test script are in the examples folder):

nginx.conf (trimmed)

upstream orders {                          # load balancer: a pool, round robin by default
    server 127.0.0.1:9001 max_fails=1 fail_timeout=10s;
    server 127.0.0.1:9002 max_fails=1 fail_timeout=10s;
}
map $http_x_api_key $api_user {            # gateway: API key -> caller
    default "";  "key-asha" asha;  "key-ravi" ravi;
}
limit_req_zone $api_user zone=per_user:1m rate=18r/m;

server {
    listen 127.0.0.1:8080;
    proxy_set_header X-Forwarded-For $remote_addr;   # reverse proxy: overwrite, don't append
    proxy_set_header X-User $api_user;

    location /orders/ {
        if ($api_user = "") { return 401; }
        limit_req zone=per_user burst=2 nodelay;
        proxy_pass http://orders;
    }
}

Full code, the nginx test script and run instructions: examples/proxy-lb-gateway/.

How real stacks combine them

Enlarge figure

A common production stack. The edge ends TLS and absorbs attacks, the layer 4 balancer gives fixed addresses and spreads connections over the gateways, and the gateways check callers and balance over each service's copies. Every arrow is a separate connection.

Each tier does what it is best at: the edge (a CDN, a reverse proxy in many cities) ends TLS near users and absorbs attacks; the L4 balancer gives stable addresses and spreads connections over several gateway copies; the gateways decide about callers and balance requests over each service’s copies. North-south traffic (from outside) takes this path; east-west traffic (service to service) usually skips the gateway and goes through internal L7 balancing or a service mesh, a small proxy such as Envoy next to every service.

Published systems follow the same pattern. Netflix runs its edge gateway, Zuul, behind cloud load balancers, and moved it from a thread per connection to non-blocking I/O because the front door is where open connections pile up when backends slow down or devices retry (Netflix TechBlog, 2016). Cloudflare layers Unimog (L4) under Pingora, its Rust HTTP proxy that replaced nginx in 2022 and serves over a trillion requests a day with about a third of the CPU and memory (Cloudflare, 2022). In Kubernetes, the community ingress-nginx controller was retired in March 2026 and the project points to the Gateway API instead (Kubernetes blog, as of October 2026).

What every extra hop costs

Timeouts must shrink as you go inward, and only one hop should retry. Every hop has a timeout for the hop behind it, and the service does not stop working when its caller gives up. If the gateway gives up after 1 s on a service that needs 1.2 s, and retries, and the client gives up after 1.5 s and also retries, one click turns into several executions of the same expensive request, most of them for nobody.

The count multiplies rather than adds. A hop that makes one attempt plus two retries sends up to 3 attempts to the hop behind it. If the client makes 3 attempts and each one reaches a gateway that makes 3 of its own, the service can run the request 3 × 3 = 9 times; add a third hop that retries the same way and it is 27. Every retry at an outer hop restarts the whole chain inside it:

SimulationOne click, two hops, a timeout and retries at each

PlayStepReset

What to watch for. With the defaults, the service needs about 1.2 s, the gateway gives up after 1 s, and the client after 1.5 s. Each hop retries twice, so one click becomes up to nine service executions, almost all of them amber, and the user still gets an error about half the time. Now set retries to 0, the gateway timeout to 2,800 ms and the client timeout to 3,000 ms: one execution per click and more than 80% of clicks answered. The failures left are the calls that really are slow, and each costs the service one execution instead of up to nine.

The sim’s defaults average about six service executions per click with over 90% of the service’s work wasted, and the user still sees an error about half the time. Under real overload it is worse than the sim shows, because the extra executions make the service slower still, which causes more timeouts: the retry storm that Google’s SRE book describes as a main driver of cascading failures (Google SRE book). The rules that follow: the client waits longest and each hop inward waits a little less; retry at one layer (usually the outermost that knows whether the request is safe to repeat), with a budget; and pass a deadline inward so the service can stop work nobody will read.

Safe to repeat means idempotent: running the request twice leaves the same result as running it once. Reading an order or setting its status to “shipped” is idempotent; a POST /payments that charges ₹500 is not, and two runs charge ₹1,000 unless the request carries an idempotency key the service checks. A retry budget caps retries as a share of traffic, for example at most 10% extra requests, so retries can cover the odd failure but cannot multiply load during an outage.

Idle connections must be closed from the outside first. Opening a connection costs a round trip for the TCP handshake and another for TLS 1.3 before the first byte of a request moves. Keep-alive avoids paying that on every request: after a response, both sides leave the connection open for the next one, and a proxy keeps a pool of such connections to each backend. An open connection holds memory and a file descriptor on both ends, so each side also has an idle timeout: close the connection if nothing has been sent on it for that long. The two sides set their timeouts independently, and a close is a message that takes time to arrive.

If the backend closes an idle connection before the balancer does, the balancer can pick that connection for a new request in the same instant the backend’s close is on its way. The request reaches a socket that is already closed, and the client gets a 502 that never appears in the app’s logs. AWS’s Application Load Balancer keeps idle connections for 60 s by default, while Node.js’s HTTP server closes them after 5 s and Gunicorn after 2 s (as of October 2026). With those defaults, every pooled connection that sits idle for 5 s is closed by Node, and each close is a chance for that race. AWS’s guidance is to set the application’s keep-alive timeout longer than the load balancer’s idle timeout (ALB docs), so the balancer always closes first and never sends on a connection it closed itself.

Health checks must mean “I can do my job”. The L4 balancer checks the gateways; the gateways check the services. A gateway health check that returns 200 while the gateway cannot reach any service keeps traffic flowing into a dead end.

Common confusions

“They are three products, so we need one of each.” They are roles. List the jobs (end TLS, spread load, decide about callers), then pick the fewest boxes that cover them. Every extra box is a hop, a timeout and a failure point.

“X-Forwarded-For is the client’s address.” Any client can send its own X-Forwarded-For, and a proxy that appends to it passes the lie along. Only entries added by proxies you control are trustworthy: overwrite the header at your outermost proxy (as the demo does), or read the list from the right and take the first address that is not one of your proxies (MDN). The standard form is Forwarded (RFC 7239), with the same trust rule. Behind an L4 balancer there is no header to add; the address arrives only if the balancer preserves it or uses the PROXY protocol.

A spoofable header used for security

Rate limits, allow-lists and fraud checks keyed on a client-supplied X-Forwarded-For can be bypassed by anyone who sets the header. Limits keyed on the socket address behind a proxy lump every user into one client. Decide where your trust boundary is and read the address only from there.

“The gateway checks auth, so services can trust every request.” Only requests that came through the gateway. If a service is reachable any other way (another service, a debug port, a loose network rule), anyone can send X-User: admin. Lock services so only the gateway can reach them (network policy or mutual TLS), or pass a signed token the service verifies.

“The gateway is a good place for business logic.” It sees every request, so it is tempting. But then a shared box owned by a platform team holds logic owned by product teams, and every change is a change to the company’s front door. ThoughtWorks put “overambitious API gateways” on Hold for this reason: keep gateways to generic work such as authentication and rate limits.

Every feature in the front door runs on every request

On 2 July 2019 a new firewall rule with a backtracking regular expression went out to Cloudflare’s edge proxies worldwide, used all the CPU on the machines handling HTTP, and caused 502 errors for about 27 minutes (Cloudflare postmortem). Anything in a box every request passes through shares fate with every request: roll it out gradually, and keep its work cheap and bounded.

“Sticky sessions are a feature to turn on.” They send a client to the same backend every time, which papers over per-user state kept in a server’s memory. They make load uneven (the L4 problem, by choice) and lose the state when that server dies anyway. Keep session state in a shared store, and keep stickiness for cases that need it, such as WebSocket connections.

Interview corner

“Reverse proxy vs load balancer vs API gateway?” Strong answers start with roles and the L4/L7 line: what each box can read, and per-connection versus per-request decisions. Then: one product can play all three. Follow-up: “When would you run separate products?” An L4 tier for fixed addresses and to scale the L7 tier; a dedicated gateway when there are external developers, plans and quotas, or a separate team owning the front door.

“Your gRPC service behind a Kubernetes Service has some pods at 90% CPU and some idle. Why?” kube-proxy balances connections, and each gRPC client holds one long-lived HTTP/2 connection. Fixes: an L7 proxy or mesh that balances per request, client-side balancing over all pod addresses (a headless Service), or forcing connections to be recycled periodically. Follow-up: “Which algorithm would you use?” Least outstanding requests or power of two choices, so slow pods get less.

“Isn’t the gateway a single point of failure and a bottleneck?” Not if it runs as several stateless copies behind an L4 balancer across zones, with shared state (rate-limit counters) in a store like Redis and a deliberate fail-open or fail-closed choice when that store is down. It becomes a bottleneck when it holds slow logic, so keep it to cheap checks and timeouts shorter than the client’s.

“Where does authentication happen with a gateway?” The gateway authenticates (who is this, is the token valid) and rejects early. Services still authorize (may this caller do this), because only they know the rules. Identity travels inward as a signed token, and services are not reachable around the gateway.

Check yourself

1. You move a gRPC service from 4 to 12 pods behind an L4 balancer to fix latency. Nothing improves. Why, and what do you change?

Each client holds one long-lived connection, and the L4 balancer assigned each connection to one pod when it opened. New pods get no existing connections, so they sit idle. Balance per request with an L7 proxy or mesh, or balance on the client over all pod addresses.

2. Six identical backends behind round robin; one gets a noisy neighbour and becomes 3× slower. p50 latency is fine but p99 is seconds. Why does round robin cause this, and what fixes it without a health check?

Round robin keeps sending the slow backend one sixth of the traffic, more than it can serve, so a queue builds there and every request that lands on it waits. Least outstanding requests (or power of two choices) sends new requests to backends with fewer in flight; the slow one accumulates in-flight requests and so receives fewer, which needs no health check.

3. The gateway times out a backend after 30 s and retries twice; the mobile app times out after 10 s and retries three times. The backend slows to 12 s. What happens?

Every app attempt times out at 10 s while the gateway is still waiting, so the app retries and starts another gateway request, which may itself retry. Requests multiply at the slow backend (up to 4 app attempts, each starting gateway work that can run up to 3 backend calls) and nearly all of that work is for callers that already gave up, which slows it further. Make timeouts shrink inward (the gateway’s under the app’s), retry at one layer with a budget, and pass deadlines inward.

4. After moving behind a cloud load balancer, your per-IP rate limiter blocks almost every user at the same time. Why, and what is the fix?

Every connection now comes from the balancer’s addresses, so all users share a few “clients”. Key the limiter on the real address from X-Forwarded-For or Forwarded, taking the rightmost entry that is not one of your own proxies and only when the connection came from your balancer; better, limit at the edge or gateway, which knows the caller’s identity, not only their address.

Recap

• A reverse proxy handles HTTP for your servers: TLS, buffering, caching, rewriting.
• A load balancer spreads work over copies: per connection at L4, per request at L7. Long-lived HTTP/2 and gRPC connections need L7.
• An API gateway is a reverse proxy that decides about the caller: keys, limits, routes, quotas.
• Round robin is blind to slow backends; least outstanding requests reacts to them.
• They are roles: one product can play all three. Add roles to boxes you have before adding boxes.
• Every hop needs a timeout shorter than the one outside it, and only one hop should retry.

Where to go next

• Reverse proxiescoming soon

  Buffering, TLS termination, caching and the headers a proxy must get right.

  Traffic & edge
• Load Balancing→

  The picking rules in depth: why least loaded herds on stale data, why least request goes blind behind many sidecars, subsetting, zones and outlier ejection.

  Traffic & edge
• API gatewayscoming soon

  Auth, per-caller limits, routing and backends for frontends.

  Traffic & edge
• Latency vs Throughput→

  Why a backend queue explodes near capacity, why one shared queue beats round robin, and Little's law for sizing pools.

  Foundations
• Timeoutscoming soon

  Deadlines that shrink inward, and how to pass them between services.

  Resilience
• Retries, exponential backoff and jittercoming soon

  Retry budgets and jitter, so one slow service does not become a retry storm.

  Resilience

Further reading

Specs & official docs

• RFC 9110, section 3.7 — proxies, gateways and tunnels, and why “gateway” first meant reverse proxy.
• RFC 7239, Forwarded and MDN: X-Forwarded-For — the client address behind proxies, and which entries to trust.
• RFC 9113, HTTP/2 — streams and multiplexing, and the recommended minimum of 100 concurrent streams.
• HAProxy: the PROXY protocol — the client address through a layer 4 hop.
• nginx upstream module and proxy module — round robin, passive health checks, buffering.
• Envoy: load balancers — round robin, least request with power of two choices, ring hash.
• AWS: Network Load Balancer and ALB attributes — L4 and L7 managed balancers, and the idle timeout.
• Amazon API Gateway quotas and the 2024 timeout change — the 29-second limit.
• Kubernetes Gateway API and the ingress-nginx retirement — the current front door of a cluster.
• Kong Gateway docs — a gateway built from nginx plus plugins.

Engineering blogs & papers

• Kubernetes: gRPC load balancing without tears (2018) — why L4 balancing fails for gRPC, and the fixes.
• Google: Maglev (NSDI 2016) — software L4 balancing with consistent hashing.
• Cloudflare: Unimog (2020) and Pingora (2022) — L4 and L7 layered inside each data centre.
• Cloudflare: the July 2, 2019 outage — one rule, every request.
• Netflix: Zuul 2 (2016) — an edge gateway under connection pressure.
• Google SRE book: Addressing cascading failures — retries, deadlines and overload.

Patterns

• microservices.io: API gateway and Sam Newman: backends for frontends.
• ThoughtWorks Radar: overambitious API gateways — why domain logic stays out of the gateway.