Distributed Transactions: correctness, not commitment
A revision session built around one idea — distributed transactions are not "many databases acting like one," they're workflow coordination plus failure handling plus retry safety plus compensation. Plus a real deployment section and a leadership decision-making section. Click a question to think first — reveal only when you're ready to check yourself.
The mental model — 8 core problems, one root cause
The trap: thinking a distributed transaction is "how do I make many DBs commit as one DB." The real question is how business correctness survives when a workflow spans many services and something fails halfway through.
Place Order
|
+--> Order Service (create order)
+--> Payment Service (charge payment)
+--> Inventory Service (reserve stock)
+--> Shipping Service (create shipment)
User expects: "all-or-nothing"
System reality: each step is a separate commit, in a separate
failure domain, at a separate point in time
The whole discipline exists to answer one question:
what happens when step 2 succeeds and step 3 fails?
Q1Why can't one database transaction naturally span many independently-owned services?reveal
Because each service owns its own database and its own commit boundary — there's no single lock manager or transaction log spanning all of them. A local ACID transaction assumes one engine controls commit/rollback for everything inside it; microservices deliberately give up that single point of control in exchange for independent scaling and ownership.
Q2If each service owns its own database, who has global commit authority?reveal
Nobody, by default — and that's the design problem, not a bug to patch. You either introduce a coordinator that fakes global authority (2PC) or you give up global commit entirely and coordinate through local commits plus compensation (saga). Pretending a natural global commit authority exists is where most naive designs go wrong.
Q3If the network fails midway through a multi-service workflow, can all participants still agree on the same truth?reveal
Not automatically. A response can be lost after the remote side effect already happened — so the caller and the callee can genuinely disagree about whether a step succeeded. Reconciliation and idempotent retries exist specifically because "ask again and get the same answer" isn't guaranteed for free.
Q4Is the real problem a data transaction, or a business workflow?reveal
Almost always a business workflow. Framing it as "how do I make the data transactional" pulls you toward 2PC and locking. Framing it as "how does the business outcome stay correct" pulls you toward saga, compensation, and accepting that some steps will temporarily disagree before converging.
offset 01
Data atomicity vs. business atomicity
These are not the same thing, and conflating them is the single most common design mistake in this space.
Data atomicity : all low-level updates commit together
Business atomicity : the user perceives one consistent outcome
Order created + Payment charged + Shipment delayed 2 min -> often fine
Payment charged + Order missing + Inventory not reserved -> never fine
Q5Does every business workflow really need strict all-or-nothing across every service it touches?reveal
No — and assuming so is what pushes teams toward expensive, over-coordinated designs. The right question is which specific invariant must never be violated (e.g. "payment charged implies order exists") versus which steps can lag briefly (e.g. a shipping label printing two minutes late).
Q6Which parts of a workflow should be strongly consistent, and which can be eventually consistent?reveal
Anything touching money movement or an irreversible external commitment (payment capture, allotment, lien marking) usually needs the strongest guarantees you can afford. Anything downstream and reversible or cosmetic (notifications, recommendation refresh, analytics) is a natural fit for eventual consistency.
Q7Is "send confirmation email" really part of the core transaction boundary?reveal
Usually no. If it fails, the business outcome is unaffected and it can simply be retried later — including it inside the strict transaction boundary only adds fragility (now a flaky email provider can block or roll back a financial workflow) without adding real correctness value.
Q8Should a distributed transaction boundary usually be smaller than the full end-user-visible workflow?reveal
Often yes. The strict, must-never-be-wrong core (order + payment + inventory) is usually much smaller than everything the user experiences as "placing an order." Drawing the boundary too wide is how notification retries end up able to block a financial commit.
offset 02
Four patterns that cover almost everything
Nearly every real design is some combination of these four — the skill is picking the right one per boundary, not picking one for the whole system.
Need strong all-or-nothing? -> maybe 2PC
Need a scalable microservice workflow? -> saga
Need reliable event publication -> transactional outbox
from a local DB transaction?
Need crash-safe recovery at scale? -> idempotency + reconciliation
Q9Why does 2PC sound attractive on a whiteboard, and why does it usually get rejected in real microservice systems?reveal
Attractive because it gives real atomicity semantics — a genuine all-commit-or-all-rollback guarantee. Rejected in practice because it's blocking (participants hold locks while waiting on the coordinator), tightly coupled (every participant must be online and responsive), and reduces overall system availability to the availability of the least-reliable participant.
Q10What is the deepest philosophical difference between 2PC and saga?reveal
2PC tries to prevent partial commit from ever being visible. Saga accepts that partial progress will be visible and temporarily inconsistent, and instead invests in compensating actions to converge back to a correct state afterward. One avoids the mess; the other cleans it up.
Q11Is outbox a replacement for saga, or does it usually sit underneath one?reveal
It sits underneath. Outbox solves a narrower problem — "how do I reliably tell the rest of the system that my local commit happened" — while saga solves the broader problem of coordinating the whole multi-step workflow. Most production sagas are built on top of outbox-published events.
Q12Can one company legitimately use more than one of these four patterns across different parts of its platform?reveal
Yes, and it usually should. A tightly-coupled internal ledger transfer might justify stronger coordination, while the broader order → payment → shipping → notification flow is a textbook saga. Treating "our architecture" as one pattern applied everywhere is itself a design smell.
offset 03
Two-Phase Commit — the classic idea, and why it hurts at scale
The coordinator asks "can everyone commit?" before telling anyone to actually commit. Simple in theory; expensive in practice.
Coordinator
| |
prepare prepare
v v
Service A Service B
\\ /
yes / no
All YES -> Coordinator sends COMMIT to all
Any NO -> Coordinator sends ROLLBACK to all
Failure mode: coordinator crashes after some participants
already said YES and are now blocked, holding locks,
waiting for a COMMIT that may never arrive.
Q13If the coordinator crashes after some participants have already voted YES and prepared, what state are they in?reveal
They're blocked — holding their prepared-transaction locks, unable to unilaterally commit or roll back, waiting for a coordinator decision that may not arrive for a long time (or ever, if the coordinator's log is lost). This blocking window is 2PC's most cited real-world failure mode.
Q14Why does 2PC reduce availability even when nothing has actually failed yet?reveal
Because every participant must hold resources (locks, prepared state) for the duration of the round-trip to the slowest participant — a single slow or unreachable service throttles the whole transaction's throughput and latency, even if it would have eventually succeeded.
Q15Should internet-scale microservice systems rely heavily on 2PC as their default coordination model?reveal
Usually not. The tight coupling and blocking behavior work against exactly what microservices are meant to buy you — independent deployability and availability. 2PC tends to survive only in narrower, tightly-scoped contexts (e.g. a single database engine's own distributed commit, or a small number of co-located, highly-available participants).
Q16Is there a middle ground between full 2PC and full saga?reveal
Yes — patterns like the Try-Confirm/Cancel (TCC) model borrow 2PC's "reserve first, confirm later" shape but with shorter hold times and application-level compensation instead of database-level locking, trading some of 2PC's strength for saga's flexibility.
offset 04
Saga & compensation — the part people design too casually
Saying "we'll use saga" is easy. A saga is only as strong as its compensations — and compensation is not the same thing as rollback.
Forward: Create Order -> Reserve Inventory -> Charge Payment -> Ship
|
FAILS
|
Compensate: Cancel Order <- Release Inventory <- Refund Payment
Compensation != undo. A refund is not "as if the charge never
happened" — it's a new, separate, reversing action.
Q17Is compensation the same thing as rollback? If not, why does the difference matter?reveal
No, and the difference is the whole point. A database rollback erases a change as if it never happened, atomically and invisibly. A compensation is a new forward action (a refund, a cancellation notice, a re-stocking) that happened to run after a failure — it has its own side effects, its own failure modes, and it can itself fail or be delayed.
Q18Can every business step realistically be compensated? What about an email already sent, or a bank charge already processed?reveal
No. Some effects are only approximately reversible (a refund is close to but not identical to "never charged") and some are not reversible at all (an SMS already delivered, a regulator already notified). Good saga design identifies these upfront rather than discovering them during an incident.
Q19Should reversible steps or irreversible steps happen earlier in a saga?reveal
Reversible steps first, irreversible steps as late as possible. If you reserve inventory (reversible) before charging payment (harder to cleanly reverse), a late failure only needs the cheap, safe compensation. Ordering by reversibility is one of the highest-leverage saga design decisions available.
Q20Are compensating actions themselves required to be idempotent?reveal
Absolutely. A compensation triggered twice by a retried failure event can otherwise over-compensate — refunding a customer twice, releasing inventory twice. Compensations need the same idempotency-key discipline as forward actions, not less.
offset 05
Orchestration vs. choreography
Both implement the same saga concept — the difference is where the workflow logic lives, and that choice has real operational consequences.
Orchestration Choreography
-------------- ------------
Saga Orchestrator OrderCreated event
| | | -> Inventory reserves
v v v InventoryReserved event
Order Inv. Payment -> Payment charges
PaymentSucceeded event
+ centralized visibility -> Shipping starts
- orchestrator becomes a + loose coupling
complexity hub - workflow is invisible
until you trace events
Q21At small-to-medium scale, which model is usually easier to reason about, and why?reveal
Usually orchestration — the workflow logic lives in one place, so "what happens next" and "what state is this order in" have a single, readable answer instead of being scattered across every service's event handlers.
Q22Can event-driven choreography become "invisible spaghetti" as the ecosystem grows?reveal
Yes — this is its most common failure mode at scale. Each service only knows its own reaction to an event, not the whole flow. Understanding end-to-end behavior means mentally reconstructing a chain across many independently-deployed services, which becomes genuinely hard once a dozen+ services participate.
Q23If an on-call engineer asks "where is this order stuck?", which model tends to give a clearer answer faster?reveal
Usually orchestration — the orchestrator's state store can directly answer "this order is at step 3, waiting on payment confirmation." In choreography you'd need to reconstruct the same answer by tracing which events were published and consumed, across logs from several services.
Q24Does choosing orchestration eliminate the coupling problem, or just relocate it?reveal
It relocates it. Instead of services being implicitly coupled through shared event contracts, they become explicitly coupled to the orchestrator's workflow definition. That's often a good trade for visibility — but the orchestrator itself now becomes a single, critical piece of shared infrastructure that needs its own reliability investment.
offset 06
Transactional outbox & idempotency — the unglamorous foundation
Neither of these looks exciting on an architecture diagram, and both are the difference between a saga that actually converges and one that silently corrupts state.
Bad flow: update local DB -> publish message
(crash between the two steps loses the event forever)
Outbox: in ONE local DB transaction ->
1. update business row
2. insert row into outbox table
3. commit
-> a separate relay reads the outbox and publishes reliably
Idempotency: same idempotency key -> same business effect,
no matter how many times the request is retried
Q25Why is the outbox pattern considered foundational rather than optional?reveal
Because it closes the single most common silent-data-loss gap in event-driven systems: a service's local DB commit succeeding while the corresponding event never gets published due to a crash in between the two separate actions. Downstream services then never learn the state actually changed.
Q26Does the outbox pattern give you exactly-once event delivery automatically?reveal
No. It guarantees the event will eventually be published at least once — reliable publication, not deduplicated delivery. Consumers still need their own idempotency handling for the (rare but real) case of the same event arriving twice.
Q27What makes a good idempotency key, and why does the choice matter so much?reveal
It needs to uniquely identify this specific business intent, not just this specific HTTP call — often something like order_id + saga_step or a caller-supplied payment_intent_id. A key that's too broad (e.g. just order_id for every step) collapses distinct actions together; too narrow (e.g. a fresh UUID per retry) defeats the purpose entirely.
Q28Should every step in a saga have its own idempotency model, or can one key cover the whole workflow?reveal
Each step needs its own. "Charge payment" and "reserve inventory" are separate side effects with separate retry risk — a single workflow-level key can't distinguish "step 2 was retried" from "step 3 was retried," so each step's own idempotency key is what actually prevents duplicate effects at the point where they'd occur.
offset 07
State machine, source of truth, and the timeout trap
The deepest lesson in this entire topic, and the one most engineers learn the hard way: a timeout means unknown, not failed.
PENDING -> ORDER_CREATED -> INVENTORY_RESERVED -> PAYMENT_CHARGED
-> SHIPPING_CREATED -> COMPLETED
On failure:
PAYMENT_FAILED -> INVENTORY_RELEASED -> ORDER_CANCELLED
Payment request times out. Did the charge happen?
-> UNKNOWN. Not "failed." Do not blindly retry.
Do not blindly cancel. Reconcile first.
Q29Where should the authoritative workflow state actually live — the orchestrator, the event log, or each service's own state?reveal
In an orchestrated saga, the orchestrator's state store is the natural single source of truth for workflow progress — each participant service remains authoritative only for its own local business state (e.g. "is this payment captured"), not for the workflow's overall position.
Q30If the order service says COMPLETED but the payment service says CHARGED_PENDING_CONFIRMATION, who is right?reveal
This disagreement is itself the signal — neither local view is automatically "right." It means the workflow state machine and a participant's local state have drifted, which is exactly the class of problem an explicit state machine plus reconciliation is designed to detect and resolve, rather than something that should ever require a human to just "pick one."
Q31Can a timeout be safely treated as a failure?reveal
Not safely, no. A timeout only tells you the response didn't arrive in time — the remote side effect may have already completed. Treating timeout as failure and immediately compensating can cause a double-charge-then-refund, or worse, a refund for a charge that then still lands. This is arguably the single deepest lesson in distributed transaction design.
Q32Can event history alone reconstruct the truth after a partial failure, without an explicit state machine?reveal
Often yes, if the events were designed well — but only if every state transition was actually published as an event with enough detail to replay. Many systems discover too late that their event log has gaps (steps that changed local state without emitting anything), making full reconstruction impossible exactly when it's needed most.
offset 08
2PC vs. saga, external side effects, and the exactly-once illusion
The senior answer isn't "saga is always better" — it's knowing exactly which guarantee each model buys you, and being honest about what "exactly-once" actually means across a network.
Reach for 2PC-style coordination when
Reach for saga when
Small number of tightly-coupled, highly-available participants
Many independently-owned, independently-deployed services
Blocking briefly is acceptable for a stronger guarantee
Availability and scalability matter more than instant consistency
All participants can realistically join a prepare/commit protocol
Some participants are external systems that can't join any protocol
Temporary inconsistency is genuinely unacceptable
Temporary inconsistency is acceptable if it converges reliably
Q33Can an external payment gateway or a partner API cleanly join your saga's rollback/compensation model?reveal
Usually no — it's a black box outside your control, with its own retry semantics and its own idea of idempotency (or lack of it). External calls need to be wrapped defensively with your own idempotency keys and treated as a source of "unknown" outcomes requiring reconciliation, not as a trusted saga participant.
Q34Is reconciliation a sign that the architecture failed, or a mature and necessary part of the design?reveal
A mature and necessary part of the design, particularly for money movement and external integrations. Not every inconsistency can be resolved instantly and safely — marking a workflow PENDING_RECONCILIATION and checking external truth asynchronously is often the only correct response to a genuinely unknown outcome.
Q35Should a distributed transaction design ever promise true exactly-once semantics across service and network boundaries?reveal
No — true exactly-once across networked services, brokers, and external systems is extremely difficult to guarantee and dangerous to promise casually. The realistic, honest target is at-least-once delivery plus idempotent handling, which together produce an effectively-once business outcome without overclaiming a guarantee the infrastructure can't actually deliver.
Q36Which failures should trigger automatic compensation, and which should route to reconciliation or manual review instead?reveal
A known, confirmed failure (e.g. inventory service explicitly returned "out of stock") is safe to compensate automatically. An unknown outcome (e.g. a timeout, an ambiguous external response) should route to reconciliation rather than automatic compensation, precisely because automatic compensation on an unknown outcome risks double-acting on something that may have already succeeded.
The shape of saga coordination behind mutual fund order flow, IPO allotment, NSDL demat sync, PolicyBazaar insurance journeys, and loans — the numbers to have ready when asked "have you actually run this in production?"
Lien-marking done before disbursal specifically because it's the reversible step
The deliberate design choice worth naming out loud in the LAMF flow: lien-marking happens before disbursal — not because of team convention, but because lien-marking is reversible and disbursal effectively is not. That's the "reversible steps first" principle applied directly to a real regulated workflow.
Real-shaped issues this design has to account for
Payment timeout treated as failure, triggering a premature compensation
An early version cancelled the order and released inventory the moment a payment call timed out — sometimes while the charge had actually gone through downstream. Fix: timeout now routes to a PENDING_RECONCILIATION state that polls the payment provider's status API before any compensation runs.
MOFSL webhook redelivery causing a duplicate allotment update
A retried webhook (the same allotment event delivered twice) briefly double-applied an allotment status change. Fix: idempotency key derived from MOFSL's own event id, with the allotment-svc rejecting any event id it has already processed.
NSDL account-creation step treated as if it were compensable
An early saga design assumed a "cancel account" compensation could cleanly undo a completed NSDL account-open step. In practice this required a manual closure process at NSDL's end. Fix: account creation was deliberately moved to be the last, most-irreversible step in the saga, after every reversible pre-check had already passed.
Outbox relay lag silently delaying downstream saga steps
A backlog in the outbox relay meant "payment charged" events sometimes reached the shipping step 40+ seconds late, with no visible error anywhere. Fix: outbox relay lag became a first-class monitored metric with alerting, not just a log line nobody watched.
Compensation retried on an already-compensated order
A retried failure event triggered a second inventory-release compensation for an order that had already been released, briefly inflating available stock. Fix: compensations now check current saga state before acting, and carry their own idempotency key.
The line worth saying in an interview: "We treat reversibility ordering and reconciliation as design-time decisions for every saga step — most of our transaction-correctness incidents came from treating a timeout as a failure, not from the coordination pattern itself misbehaving."
offset 10
Rapid fire — say the "why" out loud, then move on
No hidden answers here — if you can answer each in one breath, you're ready. If you stall on one, that's your revision signal.
Why is "make many services act like one DB" the wrong framing from the start?
Why is compensation not the same as rollback?
Why should irreversible steps happen as late as possible in a saga?
Why is a timeout "unknown," not "failed"?
Why does outbox not give you exactly-once delivery by itself?
Why must compensating actions be idempotent, not just forward actions?
Why does 2PC reduce availability even before anything actually fails?
Why can choreography become invisible spaghetti at scale, when orchestration doesn't as easily?
Why can an external payment gateway rarely join your saga/rollback model directly?
Why is "effectively-once business outcome" a more honest target than "exactly-once"?
offset 11
Leadership perspectives — same incident, five minds
Seven situations. Each one is read differently by a developer, a solutions architect, a CTO, a director, and a product owner. Click to reveal how the same incident produces five different decisions — and the one "invisible question" that separates leading the system from just fixing it.
L1A mutual fund redemption payment call times out. The saga automatically compensates — cancels the order and releases inventory. Two minutes later, the bank confirms the charge actually succeeded.reveal
The customer is now charged for an order that shows as cancelled in their app. Support has to manually reconcile a refund, and the incident report reveals the saga treated "timeout" as equivalent to "failed" — the exact ambiguity the design was supposed to account for.
Developer
Adds a longer timeout before triggering compensation — reduces the odds but doesn't remove the fundamental ambiguity of "did it succeed or not."
Solutions Architect
Redesigns the payment step so a timeout routes to a PENDING_RECONCILIATION state that queries the bank's status API before any compensation is allowed to run.
CTO
Flags this as a platform-wide pattern risk — asks how many other sagas across the 80+ services treat timeout as failure by default, not just this one flow.
Director
Weighs the cost of building reconciliation infrastructure once, platform-wide, against the recurring cost of manual refund investigations every time this pattern repeats.
Product Owner
Pushes back on scope: asks whether customers should ever see a "cancelled" status before the system is actually certain, or whether the UI itself should show "processing" during any ambiguous window.
The invisible leadership question: Does our system ever show a customer a confident status ("cancelled," "confirmed") while the backend genuinely doesn't know the truth yet?
Learning: Treating a timeout as a failure isn't a coding mistake — it's a design decision made under the false assumption that "no response" means "no." The fix has to happen at the state-machine level, not just by tuning a timeout value.
L2A retried MOFSL webhook applies the same IPO allotment update twice. A customer's allotment status briefly flips from "allotted" to "not allotted" and back.reveal
No money moved, but the customer saw the flicker in the app and raised a support ticket questioning whether their allotment was real. Investigation shows the webhook consumer had no idempotency check — any redelivery just reapplied the full update.
Developer
Adds a check against MOFSL's event id before reapplying any update — closes this specific gap directly.
Solutions Architect
Generalizes the fix into a standard: every external webhook consumer across the platform must dedupe on the provider's own event id, not assume single delivery.
CTO
Notes this is the second external-webhook idempotency gap found this quarter — asks whether webhook handling needs a shared library instead of being reimplemented per integration.
Director
Frames it as a build-once decision: a shared idempotent-webhook-ingestion component pays for itself the moment a third integration avoids the same bug.
Product Owner
Asks what it cost in customer trust for even a no-financial-impact flicker to be visible — and whether the allotment status screen should debounce visible updates rather than reflecting every backend write immediately.
The invisible leadership question: How many of our external integrations assume "the provider only sends this once" without ever verifying that assumption?
Learning: A duplicate event with zero financial impact can still cost real trust if it's visible to the customer. Idempotency isn't only a correctness concern — it's also a perceived-reliability concern.
L3A team designs an early NSDL onboarding saga assuming account creation can be "rolled back" like any other step, and puts it in the middle of the workflow.reveal
The first production failure after account creation revealed there was no clean compensation available — NSDL required a manual closure process that took days, during which the customer's onboarding state was stuck in limbo.
Developer
Writes a "best-effort" compensation handler that calls NSDL's closure API — helps somewhat, but doesn't remove the multi-day gap.
Solutions Architect
Redesigns the saga so account creation moves to the very last step, after every reversible pre-check (KYC, eligibility, document validation) has already passed — minimizing the odds of ever needing that compensation at all.
CTO
Points out the deeper gap: nobody had classified which steps across the platform's sagas are truly irreversible before this incident forced the question.
Director
Asks for a one-time audit: for every saga in the wealth platform, which step is the "point of no return," and is it positioned last?
Product Owner
Asks how the customer experiences the multi-day limbo state today, and whether the onboarding UI should explicitly communicate "this step can't be undone once started" before triggering it.
The invisible leadership question: Have we ever explicitly listed which steps in each of our sagas are irreversible, or are we discovering that ordering only after an incident?
Learning: "Reversible steps first" isn't just a nice principle — it's a concrete design step that requires someone to explicitly classify every action in the saga before deciding the order. Skipping that classification is how irreversible steps end up in the middle of a workflow by accident.
L4The outbox relay for the payment service falls 40 seconds behind during a traffic spike. Downstream shipping steps silently lag with no alert firing.reveal
Customers didn't lose money or data, but several orders sat in "processing" for far longer than normal, and the team only noticed because a customer complained — the relay lag itself had no monitoring attached to it.
Developer
Scales up the relay's throughput — fixes this specific spike, not the blind spot that let it go unnoticed.
Solutions Architect
Adds outbox lag (oldest unpublished event age) as a first-class metric with an explicit alert threshold, treating publication delay as an operational signal, not just an internal implementation detail.
CTO
Notes that outbox adoption solved the "lost event" problem but the team never built the matching observability for "delayed event" — a half-finished investment.
Director
Asks whether outbox relay health should be part of the platform's core SLOs, given how many customer-facing flows now depend on it silently.
Product Owner
Asks whether "processing" is an honest status to show customers during this lag, or whether the product should surface a more specific "payment confirmed, preparing shipment" state so a delay doesn't read as something being wrong.
The invisible leadership question: Do we monitor the plumbing that makes our sagas work, or only the sagas themselves?
Learning: Outbox fixes a correctness problem (lost events) but introduces a new operational surface (relay lag) that needs its own monitoring. Solving reliability at one layer can quietly create a new, unwatched layer underneath it.
L5A compensation event gets retried after a transient failure and fires twice — releasing the same reserved inventory a second time, briefly inflating available stock.reveal
The inflated stock count let a handful of new orders through that shouldn't have been accepted, since the real inventory was already committed elsewhere — a second, downstream problem caused entirely by an un-idempotent compensation.
Developer
Adds a check-before-release guard in the inventory-release handler — closes this specific hole.
Solutions Architect
Establishes a platform rule: every compensation handler must be idempotent by design, with the same rigor applied to forward saga steps — no exceptions for "it's just a rollback."
CTO
Points out the team's idempotency review process only ever covered forward actions — compensations were treated as automatically safe, which this incident disproves.
Director
Asks for the idempotency checklist used in design reviews to be updated to explicitly require compensation coverage, not just forward-path coverage.
Product Owner
Asks how many of the orders let through by the inflated stock count now need to be manually resolved, and whether customers affected by that downstream mess should be proactively notified rather than waiting for them to notice.
The invisible leadership question: When we say "this action needs to be idempotent," do we mean every action in the saga, or only the ones that move money?
Learning: Compensation is not automatically safe just because it's "undoing" something. It's a forward action like any other, with its own retry risk — and needs to be reviewed with exactly the same idempotency discipline as the step it's compensating for.
L6A choreographed insurance-journey flow has grown to eight services reacting to each other's events. No single person can currently draw the full flow from memory.reveal
A recent bug took three days to trace because understanding "what happens when a policy sync fails" required reading event handlers in five separate codebases, none of which referenced the others directly.
Developer
Documents the current event flow in a wiki diagram — helpful today, but drifts out of date the next time someone adds a handler.
Solutions Architect
Weighs migrating this specific flow to an orchestrated model against keeping choreography but investing in distributed tracing across the event chain to make the flow observable without changing its shape.
CTO
Asks whether choreography was chosen deliberately for this flow originally, or whether it just accumulated one event handler at a time without anyone deciding on the coordination model up front.
Director
Names the real cost: three engineer-days to trace one bug is a recurring tax, and asks what threshold (number of participating services) should trigger an automatic architecture review.
Product Owner
Asks how this invisibility affects the team's ability to commit to reliable delivery timelines for new insurance features, since nobody can confidently say how a change will ripple through eight reactive services.
The invisible leadership question: At what point does choreography's flexibility stop being an asset and start being a debugging tax nobody budgeted for?
Learning: Choreography doesn't fail loudly — it fails by slowly becoming unreadable, one added event handler at a time, until a routine bug takes days instead of hours to trace. That drift needs a deliberate trigger to catch, not just hindsight.
L7A regulator asks for proof of the exact sequence of events in a specific LAMF disbursal from three weeks ago. The saga's workflow state was overwritten on completion — only the final state remains.reveal
The team can prove the loan was eventually disbursed correctly, but cannot reconstruct the exact order and timing of lien-marking, valuation, and disbursal — the precise causal sequence the regulator is asking for, because the saga's intermediate states were never retained past completion.
Developer
Starts retaining full state-transition history going forward — helps the next request, not this one.
Solutions Architect
Separates "current workflow state" (needed for live operation) from "full transition history" (needed for audit and compliance) as two distinct, deliberately-retained data stores, since the platform had only ever built the first.
CTO
Frames it as a governance gap, not a technical one: nobody defined retention requirements for regulated financial workflows before the saga engine was built.
Director
Asks the sharper question: for every regulated flow (allotment, redemption, disbursal, demat), was transition-history retention ever a deliberate compliance decision, or did it just default to whatever the workflow engine happened to keep?
Product Owner
Weighs the retention buildout against other roadmap priorities — but flags that failing this exact audit request has a cost (regulatory relationship, potential penalty) that likely outranks most features currently ahead of it in the backlog.
The invisible leadership question — the one that separates leaders: If a regulator asks for proof of exact step sequencing on any of our regulated flows tomorrow, do we know how far back we can actually answer — and was that a deliberate decision, or an accident of what the workflow engine happened to keep?
Learning: Workflow state retention is a compliance decision wearing an engineering costume. Keeping only the final state is fine for running the system, and completely inadequate for proving what happened on a regulated transaction weeks later — and that gap has to be designed in before the regulator asks, not after.
The pattern underneath all seven: developers ask "how do I fix this step," solutions architects ask "what coordination or ordering design caused this and what's the durable fix," CTOs ask "what governance or platform-standard gap let this happen," directors ask "what's the business or compliance cost, and who owns this decision," and product owners ask "how does this affect what the customer sees and trusts, and is it worth prioritizing over the next feature." A leadership role means holding all five at once — and knowing which one should drive the call, each time.