Scaling: load balancing, autoscaling and stateless design

Horizontal versus vertical scaling, sessions and shared state, target-tracking autoscaling, draining connections, and load testing before launch day.

Horizontal, vertical and stateless

The order that works: make the application stateless, add a cache, move slow work to a queue, and only then add instances. Scaling a slow application horizontally multiplies the slowness and the cost.

State that breaks horizontal scaling

  sessions in a process's memory        -> move to a shared store or a signed cookie
  uploaded files on local disk           -> move to object storage
  in-process cache as the source of truth -> use it as a cache, not as a store
  a scheduled job in the web process      -> one scheduler, or a distributed lock
  local temp files between requests       -> no assumption survives with two instances

Balancing and autoscaling

# target-tracking autoscaling: keep one metric near a target
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300     # ignore a brief dip before removing capacity
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

• Three instances is a better minimum than one: a single instance is a single point of failure, and two means losing one halves capacity.
• stabilizationWindowSeconds prevents flapping. Without it a workload that oscillates between 60 and 80 percent utilisation adds and removes instances continuously.
• Scale on the metric that reflects the bottleneck. If the database is the limit, scaling the web tier makes the database slower.
• Set a maximum that the database can actually serve. Autoscaling into a connection limit is a self-inflicted outage.

# graceful removal: stop sending new requests, then wait for in-flight ones
# in nginx, the equivalent is a slow shutdown with keepalive drained
upstream app {
    server 10.0.1.11:3000 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:3000 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

Draining, caches and load testing

1. On shutdown, mark the instance unhealthy so the balancer stops routing to it.
2. Wait one or two health-check intervals, then stop accepting new connections.
3. Let in-flight requests finish, up to a bounded grace period.
4. Close the process. A shutdown that takes longer than the grace period gets killed mid-request.
5. On start-up, do not mark ready until the pool is warm and the first request would succeed.

const server = app.listen(3000);

let shuttingDown = false;
process.on("SIGTERM", async () => {
  shuttingDown = true;
  health.markNotReady();               // fail readiness immediately

  await sleep(2000);                   // let the balancer notice
  server.close(async () => {           // stop accepting new connections
    await db.end();
    process.exit(0);
  });

  setTimeout(() => process.exit(1), 15000).unref();   // hard bound
});

FAQ

Related

Monitoring, uptime and log management Containers on a budget: Docker and managed container hosts

Last refreshed 2026-09-18.