Build Your Own Load Balancer — a design session, not a reading session

Every real design review starts here, and most rushed ones skip it. Before you draw a single box, you need to be able to say precisely what this load balancer must do, and under what numbers it must keep doing it.

This is the decision from the concepts document, but now you have to actually commit to one for your build, and justify it against the requirements you just wrote.

Now the real design work starts: decompose the load balancer into components with clear, single responsibilities. This is the piece most self-taught designs get wrong — treating "load balancer" as one monolithic blob instead of a small system of cooperating parts.

                          +-------------------+
                          |   Control Plane    |  <- register/deregister backends,
                          |   API (mgmt)        |     update weights, trigger drain
                          +---------+---------+
                                    |
                                    v
   Client                 +-------------------+        +------------------+
   requests -----------> |     Listener /     |<------>|  Backend Registry |
   (data plane)           |   Request Handler   |        |  (in-memory view) |
                          +---------+---------+        +------------------+
                                    |                              ^
                                    v                              |
                          +-------------------+        +------------------+
                          |  Selector / LB     |------->|  Health Checker   |
                          |  Algorithm Engine   |        |  (background loop) |
                          +---------+---------+        +------------------+
                                    |
                                    v
                          +-------------------+
                          |   Backend Pool     |
                          |   (actual servers)  |
                          +-------------------+

Cutting across all of the above: Metrics/Observability Exporter, Config Store (durable)

"Build the APIs" is not a checkbox — it's where vague components become concrete contracts. Design the control-plane API surface: who calls it, with what payload, and what it guarantees.

Important distinction: you are not sizing for request traffic (the LB doesn't store requests). You're sizing for control-plane / config metadata — the backend registry, health history, and policy config. This is usually a small, low-write dataset — knowing that, and being able to say why, is itself the signal of good judgment here.

TABLE backends
  backend_id (PK), pool_id (FK), host, port, weight, state (PENDING/ACTIVE/DRAINING/REMOVED),
  registered_at, last_health_check_at, consecutive_failures, consecutive_successes

TABLE pools
  pool_id (PK), name, algorithm (round_robin/least_conn/weighted), health_check_policy_id, created_at

TABLE health_check_policy
  policy_id (PK), interval_ms, timeout_ms, failure_threshold, success_threshold, check_type (tcp/http/deep)

-- NOT stored here: per-request logs, request bodies, traffic data (that's the metrics/observability pipeline's job)

"Weighted least connections" as a phrase is not a design. Here's where you actually specify the data structure and the exact steps the Selector runs on every request.

struct BackendState {
  backend_id, weight, active_connections, health_status
}

// In-memory, per LB node:
backend_pool: Map<backend_id, BackendState>   // updated by Health Checker + connection tracking

function select_backend(pool):
  healthy = [b for b in pool if b.health_status == ACTIVE]
  if healthy is empty: return OVERLOAD_OR_503
  // score = active_connections / weight  (lower score = more headroom relative to capacity)
  best = min(healthy, key = lambda b: b.active_connections / b.weight)
  return best

on_request_start(backend):
  backend.active_connections += 1

on_request_end(backend):
  backend.active_connections -= 1

You already know health checks can lie (main notes segment 5). Now design the actual subsystem: what runs, on what schedule, and how it talks to the rest of your components.

every interval_ms (per pool's health_check_policy):
  for backend in pool.backends:
    result = run_check(backend, check_type, timeout_ms)   // TCP / HTTP / deep
    if result == success:
      backend.consecutive_successes += 1
      backend.consecutive_failures = 0
      if backend.state == UNHEALTHY and consecutive_successes >= success_threshold:
        backend.state = ACTIVE   // re-admit
    else:
      backend.consecutive_failures += 1
      backend.consecutive_successes = 0
      if backend.state == ACTIVE and consecutive_failures >= failure_threshold:
        backend.state = UNHEALTHY   // remove from routing eligibility

You now have PENDING, ACTIVE, UNHEALTHY, DRAINING, REMOVED scattered across earlier segments. Pull them into one explicit state machine — this is what an interviewer or a code reviewer will actually ask you to draw.

        register()                 N consecutive
           |                        successful checks
           v                              |
       PENDING  ------------------------> ACTIVE  <---------------+
                                            |   \                  |
                          N consecutive     |    \  drain()        | N consecutive
                          failed checks     |     \  or DELETE     | successful checks
                                            v      v                |
                                        UNHEALTHY  DRAINING ---------+
                                            |          |
                                  drain()   |          | in-flight conns = 0
                                  or DELETE |          | OR drain_timeout elapsed
                                            v          v
                                         REMOVED <-- REMOVED

Your LB is now a real system with its own state. The final design question: how does it scale out without becoming the very bottleneck it exists to prevent?

        Clients
           |
           v
  +-----------------------------+
  |  Entry (VIP / DNS / ECMP)    |   <- segment 2 of the concepts doc: LB's own availability problem
  +-----------------------------+
     |        |        |        |
     v        v        v        v
   LB1       LB2      LB3      LB4     <- each: own in-memory copy of registry + health view
     \        |        |        /
      \       |        |       /
       +------+--------+------+
              |
              v
       Config Store (durable, low-write)   <- source of truth, polled/pushed periodically
       Health results: local per node OR shared cache (Redis) if strict consistency needed

Last step. No new concepts — just synthesis. This is what you'd draw on a whiteboard in the last five minutes of an interview to prove the pieces actually cohere into one system.

DATA PLANE (per request):
Client --TLS--> Listener --> Selector (reads in-memory Registry) --> chosen Backend
                                  ^                        |
                                  |                (on_request_start / on_request_end
                          Health Checker              update active_connections)
                        (background loop,
                         updates Registry's
                         health_status)

CONTROL PLANE (config changes):
Operator/Pipeline --> Control-Plane API --> validate --> write --> Config Store (durable)
                                                                        |
                                                            periodic poll/push
                                                                        v
                                                        each LB node's local Registry copy

FAILURE PATHS baked in:
- No healthy backend -> 503 (never crash)
- Backend exceeds failure_threshold -> UNHEALTHY (excluded from Selector)
- DELETE/scale-down -> DRAINING -> REMOVED (bounded by drain_timeout)
- Config Store unreachable -> LB nodes keep serving on last-known-good cache