A revision session built the way the design actually unfolds — not "which algorithm" first, but the fifteen decisions that come before and after it. Plus a real deployment section and a leadership decision-making section rooted in the Wealth & Demat platform. Click a question to think first — reveal only when you're ready to check yourself.
The first decision — what level does the LB operate at
Most people jump straight to "round robin or least connections?" That's the wrong first question. The real first question is: does this load balancer understand TCP/UDP only, or does it understand HTTP? That single choice decides almost everything downstream — what it can route on, what fairness even means, and what it's allowed to inspect.
Layer 4 LB Layer 7 LB
----------- -----------
works at TCP/UDP level works at HTTP/gRPC/app level
forwards connections/packets reads host, path, headers, cookies, method
blind to request content can make per-request decisions
Client --TCP conn--> LB --TCP conn--> Backend Client --HTTP req--> LB (reads host/path/headers) --> Backend
Layer 4 thinks in
Layer 7 thinks in
Connections and packets
Requests and their content
Source/dest IP, port, sequence
Host, path, headers, cookies, method
Fast, cheap, low-level fairness
Rich, content-aware, slightly heavier
Fair at the connection level
Fair at the actual request level
Q1One TCP connection carries 10,000 HTTP requests. A Layer 4 LB sent that connection to Server 1. Is the load still balanced fairly?reveal
No — and this is the core trap of Layer 4. It balances connections, not requests. If Client A opens one connection and fires 10,000 requests down it, all 10,000 can land on Server 1 alone while Servers 2 and 3 sit idle. The LB did exactly what it was designed to do; the "unfairness" is a structural property of the layer it operates at, not a bug.
Q2If Layer 4 has this blind spot, why would anyone still choose it?reveal
Because it's cheaper, faster, and simpler — it doesn't need to parse application protocols, terminate TLS, or hold any request-level state. For raw throughput, protocol-agnostic traffic (not just HTTP), or as a first line in front of an L7 layer, L4 is often exactly right. The choice isn't "L4 is worse" — it's "what am I actually trying to balance: connections or requests?"
Q3Can a system use both L4 and L7 together, or is it one or the other?reveal
Both, layered — very common in practice. An L4 layer often sits at the very edge (fast, cheap, absorbs connection-level load and DDoS-style volume), and hands traffic to an L7 layer that does the smart, content-aware routing. Treating "L4 or L7" as a single platform-wide choice is usually the wrong framing; the real design question is which layer owns which responsibility.
Q4Why does this "level" decision have to happen before anything else in the design?reveal
Because it constrains every later decision — what the LB can see (headers? cookies? just IP/port?), what fairness even means, whether content-based routing is possible at all, and whether TLS termination is even a meaningful question. Picking an algorithm before picking a level is designing the interior of a house before deciding whether it has a foundation.
offset 01
Traffic entry — the first availability problem
Before any backend is ever chosen, the client has to reach a live load balancer endpoint. That's a distinct problem from backend load balancing — it's load balancer availability itself, and it's usually solved with a completely different toolset: DNS, Anycast, VIP, or ECMP.
Client -> DNS resolves service.com -> LB address -> one LB node accepts traffic
Models:
1. DNS-based load balancing (coarse, TTL/caching-limited failover)
2. Anycast IP (same IP announced from many locations, network routes to nearest)
3. Virtual IP (VIP) (floating IP moves to a healthy LB node)
4. ECMP + multiple LB nodes (router spreads flows across several LB nodes)
Q5Why is "how traffic first enters the system" treated as a separate design problem from "how the LB picks a backend"?reveal
Because they fail differently. Backend selection assumes a healthy LB is already receiving traffic. Traffic entry is about what happens when the LB node itself is unreachable or dead — a problem one layer earlier. If the entry point isn't highly available, the smartest backend-selection logic behind it never even gets a chance to run.
Q6Why is DNS often called a "coarse" load balancer, and where does that coarseness bite?reveal
DNS can only hand out an IP; it has no idea which downstream server is actually healthy at request time. Worse, resolvers and clients cache the answer for the TTL, so if an LB node dies, clients keep trying its dead IP until the TTL expires — failover speed is bounded by caching behavior you don't fully control.
Q7Why do VIP or Anycast models generally fail over faster and more transparently than DNS?reveal
Because the failover happens at the network/routing layer, not in a client-visible, cacheable record. A VIP can move to a healthy node in seconds without the client's IP ever changing; Anycast routes to whichever healthy location is topologically nearest. Neither depends on a client honoring a TTL.
Q8If you had to pick one sentence to defend "traffic entry is the first availability problem" to a skeptical architect, what would it be?reveal
"An unreachable load balancer and an unreachable backend look identical to the customer — so the availability of the entry point deserves the same design rigor as the availability of the backend pool it protects."
offset 02
Backend discovery — who is the source of truth?
A load balancer needs a live, accurate backend pool. The interesting question isn't "does it have a list" — it's who owns that list, how fresh is it, and what happens when it's wrong?
Static config: easy, but stale the moment a server is added/removed
Dynamic discovery: harder, but reflects reality
+------------------+ backend list +----------------------+
| Load Balancer | <-------------- | registry / control |
+------------------+ +----------------------+
config file? service registry?
control plane? self-registration?
Q9A backend appears in the registry but isn't actually ready to serve traffic yet (still warming up). Should the LB send it requests?reveal
Not by default. Registry membership answers "does this backend exist," not "is this backend ready right now." That's exactly why health checking has to sit on top of discovery, not be replaced by it — a discovered backend and a traffic-eligible backend are two different states.
Q10If the registry itself is stale — say it hasn't updated in 30 seconds — can the LB still make safe choices?reveal
Only partially. It can keep routing to the backends it already knows about (assuming their health checks still pass), but it can't discover brand-new capacity or safely detect a backend that vanished without deregistering. This is why LB nodes are usually designed to tolerate eventual consistency rather than block on a perfectly fresh registry.
Q11Why is self-registration by servers considered more "realistic" than static config, but also riskier?reveal
It's realistic because capacity changes constantly in modern deployments (autoscaling, rolling deploys) and a human editing a config file can't keep up. It's riskier because a buggy or crashing server might fail to deregister itself on the way out, leaving a ghost entry the LB keeps trying to use until a health check catches it.
Q12Static config vs dynamic discovery — is this really an either/or choice for a mature platform?reveal
Rarely. Many production systems start with static config for stability-critical or low-churn services and move to dynamic discovery where scale/churn justifies the added complexity. The senior question isn't "which one" — it's "which services actually need dynamic discovery, and which ones are just adding operational complexity for no real benefit?"
offset 03
Backend selection — what are we actually balancing?
Picking "the next server" sounds simple. The deeper question is: what quantity are we trying to keep even — request count, active connections, latency, CPU, or in-flight work? Different answers produce different algorithms, and each algorithm is only fair under certain assumptions.
Q13Why can round robin become unfair even though it's mathematically the most "even" distribution?reveal
Round robin assumes every request costs roughly the same amount of work. In real systems, requests vary wildly — a light GET and a heavy report-generation POST get treated identically, so a server that happens to draw several heavy requests in a row can be overloaded while its round-robin "fair share" was technically respected.
Q14Least-connections sounds strictly better than round robin. When does it actually mislead the LB?reveal
When connection count doesn't correlate with actual load — e.g. a server holding many idle, long-lived WebSocket connections looks "busy" by connection count while being nearly idle on CPU, so the LB avoids a server that actually has plenty of headroom.
Q15Why can latency-aware routing become dangerous if it reacts too aggressively?reveal
Because it can create oscillation: the LB sees Server A is momentarily fast, floods it with traffic, A slows down under the new load, the LB swings everything to Server B, B slows down too — a herd effect where the "smart" policy destabilizes the very thing it was trying to optimize. Some damping/randomness is usually needed to prevent this.
Q16What's the one-sentence senior framing for "which algorithm should we use"?reveal
The choice isn't about which algorithm is "best" in the abstract — it's about naming the resource that actually constrains your backends (CPU? connections? memory? external dependency slots?) and choosing the algorithm whose signal tracks that resource, while keeping enough randomness/damping to avoid instability.
offset 04
Health checking — the real heart, and the part that lies
Without health checks, an LB is blind. But the harder truth is: health checks can lie — a "200 OK" from /health and a real user request failing are two different questions, answered by two different code paths.
LB health checker
+--> S1 : /health -> 200 OK (healthy)
+--> S2 : /health -> 500 (unhealthy, removed)
+--> S3 : timeout (unhealthy, removed)
Types: 1. TCP check (can a connection even open?)
2. HTTP check (does GET /health return 200?)
3. Deep check (app + DB + dependencies also OK?)
Q17A health endpoint returns 200 OK, but real user requests are still failing. How is that possible, and what does it mean for what "healthy" should measure?reveal
Very possible — a shallow health check only proves the process is alive and the port is open, not that the actual request path (auth, DB, downstream dependency) works. "Healthy" needs a precise definition: process alive, port open, app responsive, dependencies working, and acceptable latency are five different claims, and a shallow check only proves the first two.
Q18If a deep health check includes a DB call, can the health check itself become a liability?reveal
Yes — a deep check that hits the DB on every poll, from every LB node, at whatever polling interval is configured, adds real load. Under DB stress, the health check can itself contribute to the very degradation it's supposed to detect, or worse, start failing for reasons unrelated to whether the app can actually serve traffic.
Q19Why does "how many failed checks before removal, how many successes before re-adding" matter more than it sounds like it should?reveal
Because these two thresholds control flapping. Too sensitive (remove after 1 failure, re-add after 1 success) and a backend can bounce in and out of the pool on transient blips, causing routing instability. Too lax and a genuinely broken backend keeps receiving traffic for too long. The right values depend on how expensive a false removal is versus how expensive a missed detection is.
Q20Should health check depth be uniform across all services, or does it vary?reveal
It should vary deliberately. A stateless, low-risk service might be fine with a shallow HTTP check. A payment or settlement path, where "looks alive but can't actually complete a transaction" is expensive to discover late, usually justifies a deeper (but rate-limited and cached) check — the depth should track the cost of being wrong, not be a platform-wide default nobody revisited.
offset 05
Connection handling — the LB is managing connections, not just routing requests
An LB is rarely "stateless plumbing." It usually terminates the client's connection and opens its own to the backend — which raises questions about TLS, connection reuse, and, at HTTP/2 and gRPC, whether balancing should happen per connection or per stream.
Proxy model: Client <--conn1--> LB <--conn2--> Backend
Pass-through: LB forwards at a lower layer, less application awareness
HTTP/2 / gRPC: one connection can carry MANY concurrent streams —
balancing per-connection can silently become "balancing per multiplexed bundle of requests"
Q21Should the LB terminate the client's TCP/TLS connection, or pass traffic through unmodified? Why does this matter architecturally?reveal
Terminating gives the LB full visibility (it can inspect HTTP, do TLS offload, inject headers) but makes the LB a real endpoint in the security and failure path. Pass-through keeps the LB simpler and preserves end-to-end encryption to the backend, but sacrifices content-based routing entirely. This is one of the largest architectural forks in the whole design — it decides whether the LB is "infrastructure" or effectively a proxy/gateway.
Q22If the LB terminates TLS, where does the trust boundary move to?reveal
It moves from "client to backend" to "client to LB" plus "LB to backend" as two separately-trusted hops. Anything the backend now believes about the client (identity, IP, headers) is only as trustworthy as the LB's own security — the backend has effectively delegated part of its trust decision to the load balancer.
Q23One client keeps one long-lived connection open. Does that unfairly pin it to one backend forever?reveal
It can — if the LB reuses backend connections and never re-evaluates placement for the life of that connection, a single long-lived client (a persistent WebSocket, a streaming session) becomes structurally stuck on one backend regardless of that backend's later load, unless the design explicitly accounts for connection lifetime in its fairness model.
Q24At HTTP/2 or gRPC, why is "balance per connection or per stream" described as a very important modern question?reveal
Because multiplexing breaks the old assumption that one connection ≈ one unit of work. One HTTP/2 connection can carry hundreds of concurrent streams; balancing "per connection" can silently recreate the exact same connection-vs-request unfairness problem seen with Layer 4 and a single heavy TCP session — just one abstraction layer higher.
Once the LB understands HTTP, it can route on host, path, headers, and method — turning one edge entry point into a traffic router across many service pools. That power comes with a real architectural boundary question.
Host: api.shop.com Path: /checkout -> Checkout service pool
Host: static.shop.com Path: /images/* -> Static service pool
if host=api.shop.com and path starts /users -> user service
if host=api.shop.com and path starts /orders -> order service
Q25A bad routing rule is pushed. What's the actual blast radius compared to a bad backend-selection weight?reveal
Much larger. A bad weight might overload one backend pool. A bad routing rule can misroute or blackhole traffic entirely — e.g. sending /orders traffic to the user-service pool, which will simply 404 or error every request. Routing rules need the same change-control discipline as code deploys, not casual config edits.
Q26Where's the line between "the LB routes traffic" and "the LB has become an API gateway"?reveal
The line is whether it starts doing things beyond routing — rewriting headers/URLs, enforcing auth, applying business-specific policy, transforming payloads. Once it does, it's no longer "just infrastructure"; it's taken on gateway responsibilities, which changes who owns it, how it's tested, and how carefully its changes need to be reviewed.
Q27Why might a platform deliberately choose to keep the LB "dumb" (routing only) even though it's technically capable of more?reveal
Because concentrating more logic into the LB concentrates more risk into a single, shared, hard-to-test-in-isolation component that every request passes through. Keeping the LB narrowly scoped to routing, and pushing auth/transformation to a purpose-built gateway layer, is often a deliberate blast-radius decision, not a missed opportunity.
offset 07
Session stickiness — useful, but it's fairness you're trading away
Stickiness sends the same user to the same backend repeatedly — good for local cache, in-memory session, or warm context. But every sticky rule is a small, deliberate sacrifice of load fairness, and that trade needs to be scoped, not assumed.
User A --> S2
User A --> S2 (again)
User A --> S2 (again)
Methods: cookie-based affinity | client-IP hash | session-id hash | custom key hash
hash(cookie=session123) -> S3
Q28Why should stickiness usually be optional and route-specific rather than a platform-wide default?reveal
Because different routes need different things — a stateless read API gains nothing from stickiness and only inherits its downside (hotspot risk), while a route backed by in-memory session state genuinely needs it. Applying it everywhere "just in case" quietly degrades fairness platform-wide for no benefit on most of that traffic.
Q29A "hot" user with unusually heavy traffic is stuck on one backend by stickiness. What happens, and whose problem is it?reveal
That single backend absorbs a disproportionate share of that user's load while siblings stay comparatively idle — a self-inflicted hotspot. It's a joint problem: the LB config chose stickiness, but the backend design is what made losing that fairness expensive (i.e., it assumed local state that made stickiness necessary in the first place).
Q30The sticky backend for a user dies. What must happen for this to be handled safely?reveal
The LB must remap the user to a healthy backend (stickiness can never override health), and the application layer must be able to rebuild or recover the lost local state — cache warm-up, session reload, reconnect logic. If the app assumed stickiness was permanent and never built a recovery path, this becomes a hard outage instead of a brief blip.
offset 08
Failure handling — where the LB stops being a splitter and becomes a resilience component
Failures can happen at six different points in the path — client-to-LB, the LB node itself, LB-to-backend, the backend app, a backend dependency, or the control plane. Each demands a different answer, and "just retry" is not automatically the right one.
Case 1 — backend dies suddenly: LB thought S2 healthy, sends request, S2 crashes -> request fails. Retry elsewhere?
Case 2 — backend is slow, not dead: S2 responds in 15s, health check still says "alive." Keep sending traffic?
Case 3 — LB node itself dies: active connections on that node — what happens to them?
Case 4 — partial overload: S1 CPU 95%, S2 40%, S3 35% — react before S1 fully fails, or wait?
Q31Why is "just retry on another backend" not automatically safe?reveal
Because a retry re-sends a request that may have already partially executed on the failed backend. For a GET, that's usually harmless. For a POST — payment, order placement, redemption — a blind retry without idempotency guarantees can create a duplicate. Retry safety is a property of the request semantics, not a universal LB feature to just "turn on."
Q32A backend is slow (15s) but its health check still says "alive." Should the LB keep sending it traffic?reveal
This is exactly why health alone is insufficient — a good LB also watches performance signals (latency, error rate) alongside binary health, and should gradually reduce traffic to a slow-but-technically-alive backend rather than treating "healthy" as a single all-or-nothing gate.
Q33An LB node itself crashes. What happens to the active connections it was holding?reveal
Those connections are lost unless the client or protocol can reconnect and the new LB node (or entry mechanism) picks it back up cleanly — which is exactly why traffic-entry availability (segment 2) and failure handling aren't separate concerns; an LB node failure is a failure at both layers simultaneously.
Q34S1 is at 95% CPU but hasn't failed yet. Should the LB react now, or wait for an actual failure?reveal
The stronger design reacts before full failure — reducing traffic share to S1 as its load signal rises is cheaper than waiting for it to fail outright and then recovering. Waiting for hard failure treats the LB as purely reactive; watching leading indicators makes it preventive.
offset 09
Draining, scaling the LB itself, and overload protection
Three problems that all show up during real operations: removing a server without dropping active work, scaling the LB layer without turning it into a bottleneck, and protecting the backend when incoming demand simply exceeds capacity.
Draining: ACTIVE -> DRAINING (no new traffic, old traffic finishes) -> REMOVED
Scaling the LB:
Clients -> [ LB1 LB2 LB3 LB4 ] -> Backend Service Pool
LB nodes stay mostly independent, pull shared config from a control plane,
tolerate small eventual inconsistency rather than coordinate on every request.
Overload protection:
Incoming = 100k rps, backend safe capacity = 60k rps
LB must not blindly forward all 100k -> queue briefly / shed load / rate limit / return 503
Q35Why is graceful draining ("no new traffic, let old traffic finish") a deployment-safety feature, not just a nice-to-have?reveal
Without it, removing a server for a deploy means killing in-flight requests mid-transaction — for something like an order or allotment write, that can leave partial state. Draining converts a hard cutoff into a controlled wind-down, which is often the difference between a clean deploy and a data-consistency incident.
Q36Why should LB nodes avoid coordinating with each other on every single request as they scale out?reveal
Because tight coordination adds latency to the hot path and turns the LB layer itself into a bottleneck and a single point of contention — the exact thing it exists to prevent for the backend. It's usually better for LB nodes to make slightly different routing decisions for a short window (tolerating small inconsistency) than to force synchronized agreement on every request.
Q37If backend capacity is 60k rps and traffic is 100k rps, why is "distribute it evenly across backends" not a real solution?reveal
Because even perfectly even distribution still sends 40k rps more than the pool can safely absorb — fairness doesn't create capacity. The only real answers are to shed/queue/rate-limit some of the demand, or add capacity; balancing an impossible total more evenly just spreads the failure instead of preventing it.
Q38Why does overload protection sometimes matter more than smart backend selection?reveal
Because a perfectly balanced overload still crashes the whole pool — evenly distributing 100k rps across backends built for 60k just makes every backend fail together instead of one failing first. Detecting saturation early (queue depth, latency, error rate) and shedding load in a controlled way protects the system as a whole, which a purely "fair" algorithm doesn't guarantee on its own.
offset 10
Observability, and rethinking "healthy vs unhealthy" as real states
A binary healthy/unhealthy view is often too simplistic. And without observability, an LB is a black box — health status alone can hide real performance degradation.
Q39Why is a DEGRADED state more useful than a strict binary healthy/unhealthy model?reveal
A backend that's "still working, but with higher latency or a partial dependency failure" is a real, common state that binary health can't express — it forces a false choice between full traffic and zero traffic. DEGRADED lets the LB send it reduced traffic, which is both safer and gives a live signal to watch for recovery, almost like an automatic canary.
Q40Can "health status alone" hide a real performance degradation? How?reveal
Yes — a backend can be technically "healthy" (process up, port open, even 200 on /health) while its p95 latency has quietly tripled. Health checks answer "is it alive," not "is it performing acceptably" — that gap is exactly why per-backend latency histograms and tail-latency metrics matter as much as the pass/fail health signal.
Q41Of the listed metrics — selection counts, error counts, tail latency, queue depth, active streams, retry counts — which one most often gets skipped, and why does that matter?reveal
Retry counts is the common blind spot. A rising retry rate is often the earliest signal that something's degrading — before error rate or latency alarms even fire — because retries are the LB quietly compensating for failures the dashboard hasn't flagged yet. Skipping it means finding out about a problem only after the compensation stops working.
offset 11
Security and the trust boundary
A real LB often sits at the exact seam between the public internet and the private backend network — which means every decision it makes about what to trust and forward becomes a decision the entire backend inherits.
Q42The LB injects the client's real IP into a header for the backend to use. What has to be true for the backend to trust that header safely?reveal
The backend must be reachable only through the LB — if a client (or another internal service) can reach the backend directly and set that same header itself, the header becomes spoofable and the backend is trusting an unverified claim. The header is only trustworthy if the network topology actually enforces the LB as the sole path in.
Q43Why does "how much the LB does" (WAF, auth, header rewriting) become a security architecture decision, not just a convenience one?reveal
Because every responsibility the LB takes on expands the set of things the entire platform is trusting a single shared component to get right. An auth bug in the LB now affects every service behind it at once — concentrating trust logic in one place is powerful, but it also concentrates the blast radius of getting it wrong.
Q44What's the real decision hiding inside "should the LB stay focused on transport/routing, or take on security-boundary responsibilities"?reveal
It's really asking: who is accountable for the trust boundary — a dedicated, deliberately-designed security/gateway layer, or an LB that accumulated security responsibilities incrementally because it was "already there"? The second path is common and usually unintentional, which is exactly why it deserves a deliberate review rather than being assumed.
The shape of load balancing behind mutual fund orders, IPO allotment, NSDL demat sync, PolicyBazaar insurance journeys, and LAMF — the numbers to have ready when asked "have you actually thought about this in production?"
Edge layer
L4 + L7 layered
Services behind LB
80+ microservices
Health check depth
Mixed shallow + deep
Stickiness
Selective route-specific
TLS termination
At LB edge boundary
Overload posture
Rate-limit + 503 shed
Flow
Path / dependency
LB-relevant risk
Design lever
MF redemption request
gateway → order-svc → NAV-svc → payment → ledger
Retry after timeout could double-submit
Idempotency key before retry policy
IPO allotment sync
mofsl-webhook-adapter → allotment-svc
Heavy webhook burst on allotment day
Overload protection / queueing at edge
Instant demat (NSDL)
onboarding-svc → nsdl-adapter → kyc-svc
NSDL slow-but-alive not caught by shallow check
Deep health + latency-aware routing
Insurance journey
policy-gateway → policybazaar-adapter
Trust boundary at 3rd-party adapter
Header verification, WAF at edge
LAMF (loan against MF)
loan-svc → valuation-svc → lien-marking-svc
Session state pinning during valuation
Scoped stickiness, safe rebalance on failure
Real issues worth designing for deliberately
One institutional client's single connection hot-spots a backend
Root cause: L4 balances connections, not requests. Fix: move heavy, request-bursty clients to an L7-aware pool, or cap requests per connection.
NSDL adapter shows "healthy" but real calls are timing out
Root cause: shallow TCP/HTTP health check, not a deep dependency check. Fix: combine health status with live latency/error-rate signal, not health alone.
Deploy drops in-flight allotment writes
Root cause: no draining state before removal. Fix: ACTIVE → DRAINING → REMOVED lifecycle enforced in the deploy pipeline, not left to chance.
Retry on POST creates duplicate redemption
Root cause: retry policy applied uniformly regardless of request semantics. Fix: idempotency keys mandatory before retry is ever enabled on a write path.
Allotment-day burst exceeds backend safe capacity
Root cause: no explicit overload posture, LB just forwards everything. Fix: defined safe-capacity ceiling per pool with queue/shed/503 behavior above it.
offset 13
Rapid fire — say the "why" out loud, then move on
No hidden answers here — if you can answer each in one breath, you're ready. If you stall on one, that's your revision signal.
Why does Layer 4 fairness break down when one connection carries thousands of requests?
Why is traffic-entry availability a separate design problem from backend load balancing?
Why does registry membership not automatically mean "ready for traffic"?
Why can least-connections mislead the LB for long-lived, low-CPU connections?
Why can latency-aware routing cause oscillation instead of preventing it?
Why can a health check return 200 while real requests still fail?
Why is "should the LB terminate TLS" one of the biggest architectural forks in the whole design?
Why does balancing per HTTP/2 connection risk recreating the L4 connection-fairness problem?
Why should stickiness be optional and route-specific, not global?
Why is retry unsafe by default on a POST/payment path?
Why doesn't perfectly even distribution solve an overload problem?
Why is a DEGRADED state more useful than a binary healthy/unhealthy model?
Why does trusting a client-IP header from the LB require a network-level guarantee, not just convention?
offset 14
Leadership perspectives — same incident, four minds
Seven situations. Each one is read differently by a developer, an architect, a CTO, and a director. Click to reveal how the same incident produces four different decisions — and the one "invisible question" that separates leading the system from just fixing it.
L1During the MF redemption cutoff window, one backend is pinned at 95% CPU while two siblings sit near idle. Cookie-based stickiness is routing a small set of high-frequency users to that one server.reveal
The stickiness rule was added months earlier for a caching benefit that a later refactor made unnecessary — nobody removed the rule once its original reason disappeared. It was quietly creating a hotspot on the platform's single most time-sensitive daily window.
Developer
Adds capacity to that one backend to absorb the pinned load — treats the symptom, and the same class of user stays pinned to whichever server eventually becomes "the sticky one."
Architect
Verifies whether the original caching reason for stickiness still holds; if the refactor removed the need, the fix is removing the rule, not tuning around it.
CTO
Notes there's no process that revisits stickiness rules after the systems they were designed around change — configuration decisions outlive their justification silently.
Director
Asks how many other routing rules across the 80+ microservices were added for a reason that no longer applies, and whether any audit trail even exists to check.
The invisible leadership question: Do we review why a routing rule exists before we tune around its symptoms, or does every hotspot get treated as a capacity problem instead of a stale-decision problem?
Learning: Stickiness is a fairness trade made for a specific reason. When the reason disappears, the trade should be re-evaluated — but nothing forces that re-evaluation automatically, which is exactly how "temporary" config becomes permanent risk.
L2The NSDL adapter's health check is a shallow TCP probe. NSDL's own API degrades to 8-second responses for two hours — the adapter shows "healthy" the entire time, and demat onboarding failures spike.reveal
Support escalated a wave of onboarding complaints before engineering even looked at the LB dashboard — the health panel showed every backend green the whole time, because "green" only ever meant "port is open."
Developer
Adds a synthetic call to NSDL inside the health check — closes this specific gap, but risks the health check itself becoming an extra load source on NSDL.
Architect
Separates "is the adapter alive" (shallow, cheap, frequent) from "is the adapter's real dependency performing" (latency/error-rate signal fed from actual traffic, not a synthetic probe) — avoiding extra load on NSDL while still catching degradation.
CTO
Realizes every external-vendor adapter on the platform (NSDL, MOFSL, PolicyBazaar) was built with the same shallow-check assumption — this isn't a one-adapter bug, it's a pattern across every third-party integration.
Director
Asks whether "healthy" has ever been explicitly defined per integration, or whether every team quietly assumed "port open" was good enough until an incident proved otherwise.
The invisible leadership question: For every backend on our health dashboard, do we actually know what "healthy" is measuring — or are we trusting a green light that only ever proved the process didn't crash?
Learning: A health check that only proves aliveness gives false confidence precisely when it matters most — during a slow degradation, not a clean crash. "Healthy" needs a definition that matches what a real user request actually depends on.
L3A deploy removes a backend from the pool mid-traffic. Several in-flight IPO allotment writes are cut off mid-transaction, leaving partial records that need manual reconciliation.reveal
The deployment tooling removed the pod the instant the new version was ready, with no drain window — a pattern that had worked fine for stateless read endpoints and was never revisited for write-heavy allotment flows.
Developer
Manually reconciles the affected allotment records — fixes this incident, not the next deploy.
Architect
Introduces an explicit ACTIVE → DRAINING → REMOVED state into the deploy pipeline, so no backend is pulled while it still has in-flight work.
CTO
Points out the deploy tooling was copied from a stateless-service template and never adapted for write-heavy financial flows — a platform-standard gap, not a one-team mistake.
Director
Asks which other write-heavy flows (redemption, lien-marking, settlement) share the same deploy template and are exposed to the identical risk right now.
The invisible leadership question: Does every write-critical flow have draining enforced by the pipeline, or does it depend on whether that particular team remembered to configure it?
Learning: Draining isn't a load-balancer feature you either "have" or "don't" — it's a guarantee that has to be enforced per flow, deliberately, especially where in-flight work has real consequences if cut short.
L4One large institutional client integrates via a single persistent TCP connection and fires thousands of order requests through it. The Layer 4 LB sent that whole connection to one backend — which is now visibly overloaded while three siblings idle.reveal
On paper, the LB's connection-level distribution was perfectly fair — every connection got exactly the same treatment. In practice, one connection carried orders of magnitude more work than the others, and fairness at the wrong unit of measurement produced a real hotspot.
Developer
Adds capacity to the overloaded backend — buys headroom, doesn't fix the structural mismatch for the next large client.
Architect
Moves high-volume institutional integrations to an L7-aware pool that can balance per request, or caps requests-per-connection to force natural redistribution.
CTO
Recognizes this as a foreseeable consequence of choosing L4 for a client class whose traffic pattern was known in advance — a design review gap, not a surprise failure.
Director
Asks whether institutional/high-volume clients should have a documented integration pattern (connection limits, pooled connections) instead of being onboarded the same way as a retail app.
The invisible leadership question: Did we choose Layer 4 because it was the right fit for this traffic shape, or because it's what we already had — and did anyone check that assumption against how our largest clients actually connect?
Learning: "Fair" only means something once you've named the unit being balanced. A perfectly fair distribution of connections can still be a deeply unfair distribution of work — the two are only the same thing under an assumption (uniform request volume per connection) that heavy clients routinely break.
L5On IPO allotment day, incoming traffic hits 100k rps against a backend pool safely rated for 60k rps. The LB forwards everything it receives — the pool starts failing under load within minutes.reveal
Every prior load test had used realistic-but-not-peak traffic, so the LB had never actually been asked to make a shed-or-forward decision before — it simply didn't have that behavior built in, because nobody had needed it yet.
Developer
Scales the backend pool up reactively during the incident — buys time this once, doesn't change what happens next allotment day if demand grows again.
Architect
Defines an explicit safe-capacity ceiling per pool and adds queuing/rate-limiting/503 shedding above it, so the LB actively protects the backend instead of passively forwarding demand.
CTO
Points out that overload protection was treated as "something we'll add later" — a known gap that was accepted as tolerable risk until the first day that actually tested it.
Director
Asks which other known-but-deprioritized gaps are quietly waiting for their own "first day that tests them," and whether allotment-day-scale events should trigger a pre-emptive capacity and shedding review every time.
The invisible leadership question: Do we know, in advance, the exact number where our platform stops being "under load" and starts being "in overload" — and does the LB actually act on that number, or just report it after the fact?
Learning: Perfectly balanced overload is still overload. Fair distribution has no answer to "there simply isn't enough capacity for this demand" — only explicit shedding, queuing, or rate-limiting does, and it has to be designed in before the peak day, not during it.
L6The LB terminates TLS and injects the client's real IP into a header. A downstream service uses that header for fraud scoring — but a security review finds the backend network also allows direct access, bypassing the LB entirely.reveal
The header was designed correctly. The network wasn't. Anyone who could reach the backend directly — a misconfigured internal service, a compromised host on the same subnet — could set that same header to any value they wanted, and the fraud-scoring logic would trust it completely.
Developer
Adds validation logic in the backend to sanity-check the header — a partial mitigation that doesn't remove the underlying direct-access path.
Architect
Closes the direct-access path at the network layer so the LB is the only route in — the header is only as trustworthy as the topology that enforces it.
CTO
Flags this as a trust-boundary design gap, not a header bug — the LB was assumed to be the sole entry point, but that assumption was never actually enforced or verified.
Director
Asks how many other backend services carry the same unverified assumption — "the LB is the only way in" — without network policy actually guaranteeing it.
The invisible leadership question: When we say the LB is our trust boundary, is that a network guarantee we've verified, or a convention we're hoping nobody breaks?
Learning: A header injected by the LB is only trustworthy if the LB is provably the only path to the backend. Trust boundaries described in an architecture diagram and trust boundaries enforced by network policy are two different things, and only the second one actually protects anything.
L7Retry-on-failure is enabled platform-wide "for resilience." A backend times out mid-redemption after the write actually succeeded; the LB retries on another backend, which processes the redemption a second time.reveal
The retry setting had been enabled years earlier for a handful of read-only endpoints and was never scoped down as new write-heavy services were added behind the same LB — "resilience by default" quietly became "duplicate risk by default" for every POST added since.
Developer
Disables retry for the redemption endpoint specifically — fixes this one path, leaves every other write endpoint on the same risky default.
Architect
Makes retry-safety an explicit, per-endpoint declaration tied to idempotency support — retry is opt-in for writes, not opt-out, reversing the default that caused this.
CTO
Realizes retry policy was never reviewed as new endpoints were added to the platform — a governance gap where a resilience feature and a correctness risk were never distinguished from each other.
Director
Asks how many customer-facing financial write paths across the 80+ microservices currently inherit this same platform-wide retry default without anyone having explicitly reviewed it for that path.
The invisible leadership question — the one that separates leaders: Is "retry on failure" a resilience decision we made deliberately for each write path, or a convenience default from years ago that nobody has revisited since — and would we actually know the difference if a regulator asked us to prove it?
Learning: Retry is not free resilience — it's a correctness trade that's only safe once idempotency is guaranteed. A default that was safe for the reads it was designed for silently becomes dangerous the moment it's inherited by writes nobody re-evaluated it against.
The pattern underneath all seven: developers ask "how do I fix this specific request/backend," architects ask "what structural design choice caused this and what's the durable fix," CTOs ask "what governance or platform-standard gap let this happen across more than one place," and directors ask "what's the business or compliance exposure, and who owns the decision to close it." A leadership role means holding all four at once — and knowing which one should drive the call, each time.