A revision session built only on what makes Kafka different from a disappearing-message queue like SQS — plus a real deployment section and a leadership decision-making section. Click a question to think first — reveal only when you're ready to check yourself.
SQS: Producer → Queue → Consumer → Delete. Kafka: Producer → Append to Log → Consumers read by offset. Everything below falls out of this one sentence.
Partition 0 log: [0][1][2][3][4][5]
Consumer A -> currently at offset 3
Consumer B -> currently at offset 1
(the message at offset 1 is NOT gone just because B read it)
Q1If messages aren't deleted on consume, what does "consumption" actually mean in Kafka?reveal
"Consuming" just means advancing your own offset pointer — it's a read, not a removal. Consumption is a per-consumer-group bookkeeping act, not a broker-side state change on the message itself.
Q2Who owns "progress" — the broker or the consumer?reveal
The consumer (group) owns progress, logically. The broker just stores committed offsets on request. This is a power shift: Kafka trusts the consumer with correctness responsibility that SQS used to absorb itself.
Q3Since the same data can be reread by anyone, is Kafka closer to a queue or a database log?reveal
Closer to a database commit log that happens to support pub/sub semantics on top. A queue implies single consumption; a log implies durable, independently-replayable history.
Q4Why does this "log" model make event replay possible at all?reveal
Because reading never mutates or removes the underlying record. Replay is just rewinding an offset pointer. In a delete-after-ack queue, once it's gone, there's nothing to rewind to.
offset 01
Log & partition fundamentals
The primary object isn't "a queue with messages" — it's Topic → Partitions → ordered append-only logs. Ordering is a per-partition guarantee, not a per-topic one.
Topic: orders
Partition 0: [0][1][2][3][4]
Partition 1: [0][1][2][3]
Partition 2: [0][1][2][3][4][5]
-> each partition is independently ordered; no global order across partitions
Q5When someone says "Kafka scales," what is actually being scaled?reveal
Mostly partition count — it determines both write parallelism across brokers and read parallelism across a consumer group.
Q6If strict ordering matters for a given key, do you want more partitions or fewer?reveal
Partition count isn't the lever — consistent key-to-partition mapping is. More partitions is fine for throughput as long as the partitioner keeps a key's events together.
Q7Why does partitioning create a tradeoff between throughput and ordering?reveal
Throughput comes from spreading writes/reads across many independent partitions in parallel. Ordering only holds within a partition — the more you split for throughput, the more you fragment any "global" order.
Q8If you increase partition count on a live topic, what happens to existing key-ordering assumptions?reveal
They can break. Partition assignment is typically hash(key) % partition_count — change the denominator and a key can map to a different partition going forward, splitting its history across two logs.
offset 02
No visibility timeout — offsets replace receipt handles
SQS hides an in-flight message and reveals it again on timeout. Kafka has no such concept — a consumer just reads and, separately, commits a position.
Consumer fetches offsets 100..104
processes 100..102
commits offset = 103 // means "next time, start at 103"
// NOT "broker deleted 100..102"
Q9With no visibility timeout, how does Kafka recover after a consumer crash mid-processing?reveal
The consumer group simply resumes from the last committed offset for that partition, possibly on a different instance via rebalance.
Q10Consumer crashes after processing a batch but before committing the offset. What's the outcome?reveal
Duplicate processing on restart — the same records get reprocessed since the commit never landed. This is why Kafka pushes idempotency onto the consumer.
Q11What conceptually "replaces" visibility timeout thinking in Kafka?reveal
Three things together: offset commit timing, consumer group rebalancing, and replay from last committed position.
Q12Is offset commit timing more dangerous early or late — and why?reveal
Commit too early risks logical data loss. Commit too late risks duplicates. Most systems bias toward "too late" and handle duplicates with idempotency, since loss is usually worse.
offset 03
Replay as a first-class feature, retention as the lifecycle owner
SQS lifecycle: enqueue → receive → ack/delete → gone. Kafka lifecycle: append → retained by policy → eventually removed. Consumer ack never removes anything.
Topic log: [0][1][2][3][4][5][6][7][8]
Consumer A starts at 0
Consumer B starts at 5
Consumer C replays all of it next week -- all valid, all independent
Q13A new service is built six months from now. Should it be able to read historical events?reveal
Yes, if retention still covers that window — this turns the topic into a source of historical truth, not just a transient work handoff.
Q14Who decides how long a message lives in Kafka — the consumer's ack, or something else?reveal
The retention policy — completely independent of whether any consumer has read it.
Q15What's the real tradeoff of setting very long retention?reveal
More replay power and safety for slow/late consumers, at the cost of storage and operational burden.
Q16If replay is a normal, expected operation, what does that force onto every consumer's design?reveal
Consumers must be idempotent by design — processing the same event twice must not corrupt state.
offset 04
Consumer groups & rebalancing
Parallelism inside a consumer group is bounded by partition count — this single constraint drives most of Kafka's capacity planning.
Topic: 3 partitions -- Group G1: C1->P0, C2->P1, C3->P2
C2 dies -> rebalance -> C1->P0,P1 C3->P2
Q17A topic has 3 partitions, the consumer group has 10 consumers. How many are actually doing work?reveal
Only 3 — one consumer per partition, max. The other 7 sit idle for that topic.
Q18One partition is a hot spot. Does adding more consumers fix it?reveal
No — a single partition is consumed by exactly one consumer at a time. You'd need to re-partition or fix key distribution.
Q19Why does rebalancing become a genuine operational concern in high-throughput systems?reveal
Partition ownership reassignment can pause consumption across the whole group while it settles — at high throughput, even a short pause means a real backlog spike.
Q20Is aggressively autoscaling consumers up and down always a good idea in Kafka?reveal
Not necessarily — every scale event can trigger a rebalance, which has a real cost.
offset 05
Producer semantics & the replicated log
A producer doesn't send "to a queue" — it sends to a topic-partition leader, which replicates to followers. Leader election is what makes the ordering guarantee trustworthy.
Producer -> choose partition by key -> batch -> send to Leader(P0) -> Follower A, Follower B replicate
Q21If a producer retries a failed send, can ordering break?reveal
Yes — unless you enable idempotent producer settings, which use sequence numbers to keep in-flight retries ordered.
Q22Why is producer batching far more central to Kafka's performance than in a naive queue model?reveal
The underlying storage is an append-only sequential write — batching plays directly to that strength.
Q23Why is leader election so critical to Kafka's correctness story?reveal
The leader is the single authority for write order on that partition. A wrong or slow election risks unavailability or split-brain writes.
offset 06
Ordering guarantees & stream-native fit
Strong order exists within a partition. Kafka's real superpower is that one event stream can independently feed many downstream systems.
Topic -> Consumer A builds a search index
-> Consumer B updates analytics
-> Consumer C builds a cache
-> Consumer D archives to a data lake (all read the same log, independently)
Q24You need strict ordering per customer account. What's the actual mechanism that guarantees it?reveal
Route every event for that account to the same partition, typically keyed on account ID.
Q25Should "global order across all events in a topic" ever be assumed?reveal
Almost never. Different partitions are consumed independently and can drift arbitrarily.
Q26In SQS, one message is "work for one consumer." In Kafka, one event can feed many independent systems. What changes in your design thinking?reveal
The topic becomes shared infrastructure — you design event schemas for many unknown future consumers, raising the investment needed in schema evolution.
offset 07
Compaction & the hardest correctness questions
Compaction retains only the latest value per key while staying log-shaped. Since the broker doesn't manage delete/visibility for you, correctness lives in how you sequence business logic against offset commits.
Q27Why would a log ever want compaction instead of keeping every event forever?reveal
Because for some use cases, only the latest state per key matters. Compaction gives you that without abandoning replay or broker-managed storage.
Q28How is a compacted Kafka topic different from a plain database table with the same shape?reveal
It's still log-oriented and replayable — a table gives you the current row; a compacted topic gives you the current row and a way to stream how it got there.
Q29Where should offset commit happen relative to the business-logic side effect?reveal
Generally after the side effect is durable — commit-after-process gives at-least-once. The strongest answer is a transactional tie between commit and write, but that's heavier infra.
Q30DB write succeeds, offset commit fails right after. What's the failure mode — and the fix?reveal
On restart, the consumer replays the same record — a duplicate. Fix: make the write idempotent so replaying it is harmless.
offset 08
When Kafka is the wrong tool
The most senior answer isn't "Kafka is better" — it's knowing exactly where it costs more than it gives.
Reach for SQS-style queue when
Reach for Kafka when
Simple work dispatch, one consumer per job
Durable event log many systems will read
Delete-after-processing is the natural model
Replay / rebuild-state is a real requirement
Invisibility timeout maps cleanly to the workload
Ordering per key/partition matters
You want the broker to own lifecycle for you
You need stream processing / CDC / audit trail
Q31Your workload is "do this job once, then forget it." Does Kafka add unnecessary complexity?reveal
Usually yes — you'd pay for partition planning, offset management, and retention tuning for guarantees you never needed.
Q32Your workload: the same event must feed 5 downstream systems, and new ones will appear later. Is SQS too weak?reveal
Yes — SQS's delete-after-ack model fights fan-out to independent readers. This is Kafka's home turf.
offset 09
Our deployment — Wealth / Mutual Fund Platform
This is the shape of the Kafka footprint behind mutual fund order flow, NSDL/MOFSL reconciliation, and demat holdings sync — the numbers to have ready when asked "have you actually run this in production?"
Roughly 6–7 producer services feed the wealth pipeline. Peak load hits during the 2:45–3:00 PM NAV cutoff window, where order + NAV traffic spikes to roughly 3–3.5k events/sec against a steady-state of 400–600/sec.
Real errors we've actually hit
Consumer lag spike at NAV cutoff
redemption-calc-svc fell 30–60s behind during the 2:45 PM burst. Fix: increased partitions on mf.nav.updates from 6→12 and tuned max.poll.records down.
Rebalance storm on rolling deploy
Deploying a new consumer version restarted all pods near-simultaneously, causing a ~20s processing pause. Fix: static group membership + staggered rollout.
Under-replicated partitions during broker patching
ISR briefly dropped below min.insync.replicas on a couple of partitions. Fix: enforced rolling patch windows confirming ISR = RF before the next broker.
Hot partition from PAN-based keying
Early keying by folio prefix clustered heavily on a few large distributor accounts. Fix: re-keyed by folio ID + partition count bump, with a brief dual-write migration.
Schema registry incompatible change
A backward-incompatible field removal broke an old consumer version still deployed elsewhere. Fix: enforced BACKWARD compatibility mode as a CI gate.
The line worth saying in an interview: "We treat partition count and retention as capacity-planning decisions taken at design time, not runtime knobs — most incidents came from changing one of those two under load rather than Kafka misbehaving."
offset 10
Rapid fire — say the "why" out loud, then move on
No hidden answers here — if you can answer each in one breath, you're ready. If you stall on one, that's your revision signal.
Why does Kafka not need a visibility timeout?
Why offsets instead of receipt handles and delete?
Why is a topic-partition log stronger than a disappearing queue for replay?
Why is retention policy more central than consumer ack?
Why is partition count both a scaling and an ordering decision?
Why does consumer group rebalance become a real operational issue?
Why is commit timing one of the deepest correctness questions in Kafka?
Why is Kafka naturally good at many independent consumers on one stream?
Why is Kafka's throughput model aligned with append-only, sequential-disk design?
Why is Kafka not just "a faster queue"?
offset 11
Leadership perspectives — same incident, four minds
Seven situations. Each one is read differently by a developer, an architect, a CTO, and a director. Click to reveal how the same incident produces four different decisions — and the one "invisible question" that separates leading the system from just fixing it.
L1A backfill job replays 6 hours of mutual fund orders into production at 11 PM. By 2 AM, 4,000 customers get duplicate confirmations.reveal
The replay was intentional — data engineering wanted to recalculate NAV figures and assumed "replaying order events will just resync everything." They didn't realize downstream notification and ledger systems would treat replayed events as brand new orders.
Developer
Adds deduplication logic to their consumer — but doesn't ask why the platform allowed an uncontrolled replay in the first place.
Architect
Weighs idempotency in every consumer (expensive, must be right everywhere) against a platform-level replay guard — e.g. an immutable commit log with a replay-audit header, built once.
CTO
Asks whether this is a governance failure (restrict who can trigger replays, require approval) or a capability gap (train teams on eventual consistency) — the answer decides whether you build tools or processes.
Director
Sees the deeper shift: the moment you allow replays of financial transactions, Kafka has quietly become a system of record — with all the audit and compliance burden that implies, not just a stream.
The invisible leadership question: If we replay events, can we prove which downstream systems actually re-processed them and which deduplicated correctly? If we can't prove it, should replays be allowed at all outside a sandbox?
Learning: A message queue lets you forget the past. A log lets you replay it — but replay is only safe if every downstream consumer is idempotent by design, not by accident. The day someone replays events "just to fix the numbers" is the day you find out which consumers were never really idempotent.
L2A new large distributor onboards. Their fund IDs hash to one partition, which now carries 40% of all traffic. Quotes go stale by market open.reveal
You scale the redemption calculator from 2 to 8 instances to catch up. Now 8 instances process orders from different partitions in different orders — nightly reconciliation shows micro-inconsistencies in trade sequencing (amounts are correct, order isn't).
Developer
Tries to add global ordering logic in code — chasing a guarantee Kafka was never designed to give.
Architect
Realizes the key should be customer ID, not fund ID, if per-customer redemption order matters — which means re-partitioning, a genuinely risky migration.
CTO
Notes that nightly batch reconciliation caught this 18 hours too late — real-time consistency checks were missing, not just the fix.
Director
Asks whether this is a scaling problem or a fundamental bet that Kafka's eventual consistency model doesn't actually fit this workload — redemptions that depend on order may need a different consistency model entirely.
The invisible leadership question: If we re-key by customer_id for ordering, we lose parallelism per customer. What's the actual cost of that tradeoff at our current and projected scale?
Learning: Scaling consumer count doesn't scale correctness — it only scales throughput within whatever ordering unit your key already defines. Before adding instances, ask what "order" is supposed to mean here, and whether your partition key actually protects it.
L3Every Friday deploy, 3 of 5 settlement-service pods restart, triggering a rebalance and a 15-second processing pause during market hours.reveal
Static membership fixes the pause. Three months later, compliance asks "how many times did settlement pause in the last 90 days, and for how long?" You never tracked it — there's no historical data to answer with.
Developer
Adds logging for rebalance events going forward — solves it operationally, not strategically.
Architect
Asks whether rebalance pause is a symptom to minimize or a permanent feature of the system that deserves first-class metrics either way.
CTO
Recognizes observability was bolted on (JMX scraping) instead of built in — the difference between ~40 hours of platform investment now versus crisis debugging later.
Director
Sees that the moment compliance asks a question about Kafka's internal behavior, the system has crossed from "message queue" to "system of record" — which demands audit trails and SLA reporting from day one.
The invisible leadership question: If a regulator asks tomorrow for every rebalance event in the last 6 months, can we produce it in 30 minutes — or does it take 30 days of engineering?
Learning: You can't retroactively measure what you never instrumented. The gap between "it happens" and "here's exactly how often, for how long, with what impact" is the gap between running Kafka and being accountable for it.
L4A new field is added to an IPO allotment schema. An old producer version still writes the old shape. The notification dispatcher crashes on the mismatch.reveal
After the fix, you realize 18 consumer groups read 40 topics with no central record of who depends on which field. One team ignores unknown fields, another crashes on missing ones, a third never documented their dependency at all.
Developer
Makes their own consumer resilient to unknown/missing fields — good practice, doesn't fix the ecosystem.
Architect
Distinguishes schema registry as validation (checks syntax) versus governance (tracks dependencies, prevents breaking changes) — the second is harder and far more valuable.
CTO
Decides between centralizing schema approval (slow, safe) and self-service with mandatory compatibility mode + audit trail (faster, still safe enough).
Director
Questions whether the org has built a platform that scales to 18 teams, or just scaled the number of failure points — and whether some topics should be split by ownership instead of shared.
The invisible leadership question: If I want to deprecate a field used by 5 consumers across 3 teams, what's my actual process? If the answer is "ask around and hope," there's no platform yet.
Learning: Schema registry enforces syntax; it doesn't enforce ownership. At scale, the real risk isn't a malformed message — it's an undocumented dependency nobody remembers exists until it breaks.
L5A compliance audit asks for full historical holdings per customer. Your compacted topic only keeps latest state — building an audit log would nearly double storage cost.reveal
Designing the fix, you realize you actually need three separate stores — compacted operational state, a time-series audit log, and a fast compliance-query read model. The "cost saving" of compaction has tripled the total footprint.
Developer
Adds a second topic and writes to both — but now their service is coupled to a three-topic architecture.
Architect
Realizes Kafka alone can't serve three different jobs well — operational state, audit history, and indexed compliance queries may need three different tools, not three Kafka topics.
CTO
Points out the real gap: no data classification system existed to flag "this data needs compliance queries" before the compaction decision was made.
Director
Names the conflation directly: "Kafka is good at streams" got quietly generalized to "Kafka is good at everything." The real question is what job the data needs to do — (a) feed real-time portfolio updates → Kafka fits; (b) serve compliance queries → needs an indexed database, not a log; (c) keep an audit trail → Kafka or S3 both work. Once you separate the jobs, you pick the right tool per job, even if that means three tools instead of one.
The invisible leadership question: When I design a new data entity, what's my framework for which store gets which piece — and am I choosing Kafka because it fits, or because it's the default?
Learning: "Everything goes to Kafka" is a habit, not an architecture. The moment a requirement (compliance, ad-hoc query, transactional consistency) doesn't match what a log is good at, the honest fix is a second tool — not a cleverer use of the first one.
L6A promotional offer drives 8x normal order volume in 4 hours. GC pauses spike, consumers lag 200k+ messages, and scaling the cluster takes 2 manual hours.reveal
The reporting-pipeline consumer also lagged for 6 hours, feeding stale data into a pricing decision — nobody flagged the staleness. The resulting mis-priced orders cost far more ($15k) than the extra broker capacity would have ($2k).
Developer
Optimizes consumer throughput — but the real failure was surge visibility, not consumer speed.
Architect
Weighs consistent scaling (always over-provisioned, stable, costs more) against elastic scaling (cheaper on average, chaotic under surprise load).
CTO
Notes there was no surge strategy — no threshold-based alerting, no traffic-shedding plan, no coordination with the teams (like pricing) that consume downstream data.
Director
Reframes it as a cost comparison that was never actually measured: a few thousand a year in spare broker capacity versus tens of thousands in bad decisions made on stale data during an incident.
The invisible leadership question: Do we know the full cost of failure for every component in this pipeline? Without that number, "keep it as cheap as possible" is a guess, not a decision.
Learning: Under-provisioning looks like savings until the surge comes. The real cost of spare capacity is only assessable against the cost of what happens without it — and most orgs never actually calculate the second number, so the first one always looks "too expensive."
L7A regulator asks for the exact order of all redemption requests for one customer on one date. Three consumer groups process independently — you can't prove a single global order.reveal
Digging deeper, you find corrections are stored as new events, not updates — so "order history" is really a tree of corrections. Reconstructing "the order as it stood at 3 PM" requires applying corrections in the right sequence, which you can't fully guarantee across services.
Developer
Adds a version number to each order — helpful, but doesn't solve the underlying mismatch between Kafka's model and what's being asked of it.
Architect
Has to choose: corrections update the original (needs mutable state — not Kafka's strength) or create new events (needs full event-sourcing reconstruction logic — complex but Kafka-native).
CTO
Points out the org has an event stream, not a financial transaction model — the two look similar but only one guarantees immutable timestamps, sequence numbers, and traceability back to a user action.
Director
Names the real question: did we ask compliance/legal what they actually needed, or did we assume Kafka could handle it? Regulators don't care what tool you used — they care whether you can prove timing, order, authorship, and tamper-resistance.
The invisible leadership question — the one that separates leaders: If my business domain requires proving audit-grade transaction history — timing, authorship, order, and tamper-resistance — am I running a log with financial features bolted on, or a ledger with Kafka underneath it? And did I choose that architecture deliberately, up front, or am I discovering it for the first time under regulatory pressure?
Learning: An event stream tells you "something happened." A financial system has to tell you "this exact thing happened, at this exact moment, in this exact order, by this exact actor, and it cannot be altered." Kafka can be the storage layer for that — but only if someone deliberately designed the transaction model on top of it, rather than assuming the log alone was enough.
The pattern underneath all seven: developers ask "how do I make my code work," architects ask "what design choice caused this and what's the durable fix," CTOs ask "what governance or observability gap let this happen," and directors ask "what's the business cost, and who owns this decision." A leadership role means holding all four at once — and knowing which one should drive the call, each time.