System Design: Load Balancing

Introduction

Load balancing is how a system survives traffic and recovers from failure. Session stickiness, health checks, and failover policies are the reliability backbone — not optional extras at scale.

← Back to System Design overview

Why load balance?

A single app server has a ceiling — CPU, memory, connections. Multiple instances behind a load balancer give you:

• Horizontal scale — add servers under load
• High availability — route around unhealthy nodes
• Zero-downtime deploys — drain connections from old instances

Architecture sketch

                    ┌─────────────┐
  Clients ────────► │ Load        │
                    │ Balancer    │
                    └──────┬──────┘
           ┌───────────────┼───────────────┐
           ▼               ▼               ▼
        ┌──────┐       ┌──────┐       ┌──────┐
        │ API-1│       │ API-2│       │ API-3│
        └──────┘       └──────┘       └──────┘

Balancing algorithms

Interactive playground

Send requests, switch algorithms, and click a server to simulate a health-check failure.

Health checks

Load balancers probe backends periodically:

• HTTP GET /health — returns 200 when app + dependencies are ready
• TCP check — port open only; weaker signal
• Graceful drain — stop new traffic, finish in-flight requests before shutdown

// Minimal health endpoint
export async function GET() {
  const dbOk = await pingDatabase()
  if (!dbOk) return Response.json({ status: 'degraded' }, { status: 503 })
  return Response.json({ status: 'ok' })
}

Failover behavior

When a server fails health checks:

1. Balancer stops sending new requests
2. In-flight requests complete or timeout
3. Sticky clients may need re-login if sessions were local

Back to index