Proxies, load balancers and connection management

Forwarded and X-Forwarded headers, keep-alive and connection pooling, timeouts and retries, idempotency keys, and how CDNs and reverse proxies rewrite traffic.

What an intermediary changes

Once a request passes through anything — CDN, load balancer, service mesh, corporate proxy — the connection the server sees is no longer the connection the client made. The client IP, the scheme and sometimes the host only survive as headers.

GET /api/me HTTP/1.1
Host: app.example.com
Forwarded: for=203.0.113.7;proto=https;host=app.example.com
X-Forwarded-For: 203.0.113.7, 10.0.0.5
X-Forwarded-Proto: https
X-Forwarded-Host: app.example.com
X-Request-Id: 01J8Y7Q2ZK3M4N5P6R7S8T9V0W

• A reverse proxy terminates TLS, so the application sees plain HTTP on a private network. Rate limiting and audit logs must use the forwarded client address, not the proxy's.
• Strip any of these headers arriving from the public internet before adding your own, or a client can forge its own IP and defeat rate limiting.
• A load balancer with sticky sessions pins a client to one backend; that is a workaround for stateful servers, not a design.

Keep-alive, pooling and timeouts

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 82
Connection: keep-alive
Keep-Alive: timeout=5, max=100

// a sized pool with timeouts and one retry for idempotent calls only
const agent = new http.Agent({
  keepAlive: true,
  maxSockets: 64,
  maxFreeSockets: 16,
  timeout: 10_000
});

async function get(path, { retries = 1 } = {}) {
  for (let attempt = 0; ; attempt++) {
    try {
      const res = await fetch(BASE + path, { agent, signal: AbortSignal.timeout(8_000) });
      if (res.status >= 500 && attempt < retries) continue;
      return res;
    } catch (err) {
      if (attempt >= retries) throw err;
      await new Promise(r => setTimeout(r, 100 * 2 ** attempt + Math.random() * 100));
    }
  }
}

• HTTP/1.1 allows one outstanding request per connection, so concurrency equals pool size; HTTP/2 multiplexes streams over one connection and needs a much smaller pool.
• Reuse matters: a new TLS handshake per request can dominate the latency of an internal call. Keep-alive plus pooling is usually the single biggest win.
• Retry only idempotent requests by default, cap the attempts, and add jitter so a recovering service is not hit by every client at the same instant.
• Always honour Retry-After from a 429 or 503 instead of applying your own backoff.

What CDNs and proxies rewrite

location /api/ {
    proxy_pass http://app_upstream;
    proxy_http_version 1.1;
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Request-Id      $request_id;

    proxy_connect_timeout 3s;
    proxy_send_timeout    30s;
    proxy_read_timeout    30s;
    proxy_next_upstream   error timeout http_502 http_503;
}

• A CDN may cache a response you expected to be dynamic. If the origin sends no cache directives, the edge applies its own defaults — be explicit.
• Some proxies buffer request bodies before forwarding, which breaks streaming uploads and disables early rejection of oversized payloads.
• Proxies may normalise paths and decode percent-escapes, so a route that depends on exact encoding can behave differently behind one.
• If you use proxy_next_upstream, only idempotent methods are safe to replay by default; a POST retried at the proxy layer can create two orders.

FAQ

Related

Message anatomy and framing HTTP API design in practice

Last refreshed 2026-09-18.