Design Your Own: Distributed Transaction Coordinator

1. Start a named saga instance from a caller-supplied definition + business payload.
2. Execute steps in a defined order, invoking each participant service's API.
3. Track per-step and overall saga state durably, queryable at any time.
4. On step failure, run compensations in reverse order for all completed steps.
5. Detect and resolve ambiguous outcomes (timeouts) via reconciliation, not blind retry.
6. Guarantee idempotent step execution and idempotent compensation under retry.
7. Expose saga status to operators/support without requiring log spelunking.

Scope OUT: the coordinator does not guarantee exactly-once delivery to participants, does not own participant-side business logic, and does not attempt cross-saga transactions (one saga can't atomically depend on another's outcome).

Orchestration. The coordinator you're building is, by definition, the central place that owns workflow sequencing — building it as a choreography engine is a contradiction, since choreography's whole premise is no central owner.

Saga with compensation, at-least-once step execution. 2PC is rejected here specifically because your NFR target (2k/s saga starts, 99.95% availability) is incompatible with blocking every participant on a coordinator-driven prepare phase — and MOFSL/NSDL/PolicyBazaar are external systems that categorically cannot join a 2PC protocol at all.

+----------------+     +-------------------+     +------------------+
| Saga Definition|---->| Orchestrator Core |---->| Step Invoker     |
| Store (config) |     | (state machine     |     | (calls Order/    |
+----------------+     |  executor)         |     |  Payment/NSDL...)|
                        +---------+---------+     +------------------+
                                  |
                       +----------+----------+
                       |                     |
              +--------v-------+   +---------v--------+
              | Saga State     |   | Outbox / Event   |
              | Store (durable)|   | Publisher        |
              +----------------+   +------------------+
                                             |
                                  +----------v---------+
                                  | Timeout & Recon-    |
                                  | ciliation Scheduler |
                                  +---------------------+

Because it runs on wall-clock triggers, not saga events — it has to wake up and act even when nothing has happened, which is a fundamentally different execution model (poll/sweep) from the Orchestrator Core's event-reactive model (react to a step callback).

Blast radius. POST /sagas/{id}/compensate gets far stricter scoping than GET /sagas/{id} even though both touch "the same resource" — because one can trigger real refunds and reversals across an entire saga, and the other cannot.

saga_definitions (durable config)
  saga_type, version, step_sequence[], compensation_map

saga_instances (durable, hot-write)
  saga_id, saga_type, business_key, state, current_step,
  created_at, updated_at

saga_step_log (durable, append-only)
  saga_id, step_id, attempt_no, outcome, idempotency_key,
  started_at, completed_at

reconciliation_queue (ephemeral / derived)
  saga_id, reason, next_check_at   -- rebuildable from saga_instances

The trap: the reconciliation_queue is derived from saga_instances (anything past its timeout with no callback) — it must never be treated as the source of truth. Teams that persist it as durable state independently from saga_instances eventually get the two out of sync, and then can't tell which one to trust.

struct SagaInstance {
  saga_id, saga_type, state, current_step_index,
  lease_owner, lease_expires_at
}

function execute_next_step(saga_id):
  saga = load_and_lock(saga_id)          // SELECT ... FOR UPDATE
  if saga.lease_owner != self and not expired(saga.lease):
      return SKIP  // another instance already owns this saga

  acquire_lease(saga, self, ttl=30s)

  if saga.current_step_index >= len(definition.steps):
      saga.state = COMPLETED
      persist(saga)
      return DONE

  step = definition.steps[saga.current_step_index]
  key = idempotency_key(saga_id, step.id)
  result = invoke(step.endpoint, payload, key)   // at-least-once call

  if result == SUCCESS:
      saga.current_step_index += 1
      persist(saga)
  elif result == KNOWN_FAILURE:
      saga.state = COMPENSATING
      persist(saga)
  else:  // TIMEOUT / UNKNOWN
      saga.state = PENDING_RECONCILIATION
      persist(saga)

current_step_index >= len(steps) is not an error — it's the success terminal state, and it must be checked before attempting to invoke a step, not discovered by an out-of-bounds failure. The equivalent exhaustion check on the compensation path (compensation_index < 0) marks COMPENSATED, the failure terminal state.

every 60s:
  sweep = SELECT * FROM saga_instances
          WHERE state = PENDING_RECONCILIATION
          AND next_check_at <= now()

  for saga in sweep:
      truth = query_participant_status(saga.saga_id, saga.current_step)
      if truth == CONFIRMED_SUCCESS:
          saga.current_step_index += 1
          saga.state = IN_PROGRESS
      elif truth == CONFIRMED_FAILURE:
          saga.state = COMPENSATING
      else:  // still unknown
          saga.attempt_count += 1
          saga.next_check_at = now() + backoff(saga.attempt_count)
          if saga.attempt_count > MAX_ATTEMPTS:
              saga.state = MANUAL_REVIEW
              alert(saga)
      persist(saga)

Centralized (a small dedicated pool, not one-per-orchestrator-instance). Reconciliation is low-frequency (60s sweep) and read-heavy against external systems — running it per orchestrator instance means every instance independently polls the same ambiguous sagas, multiplying load on already-struggling external APIs at exactly the moment they're least able to absorb it.

PENDING -> IN_PROGRESS -> COMPLETED
               |
               +--(known failure)--> COMPENSATING -> COMPENSATED
               |
               +--(timeout/unknown)--> PENDING_RECONCILIATION
                                            |
                              +-------------+-------------+
                              |                           |
                       (resolved)                  (exhausted retries)
                              |                           |
                        IN_PROGRESS/                MANUAL_REVIEW
                        COMPENSATING                (human closes loop)

The MAX_ATTEMPTS → MANUAL_REVIEW transition in the reconciliation loop is the guaranteed exit — every other state either completes, fails cleanly, or eventually times its way into reconciliation, and reconciliation itself is bounded by attempt count, not by external system behavior. Idempotency plugs in at exactly three points: the step invocation (segment 4's callback keying), the compensation invocation (segment 5's compensation_map + its own idempotency_key), and the lease acquisition (segment 6 — re-acquiring an expired lease you already held must be a no-op, not a new lease).

Per-saga leasing (segment 6), not a global lock. Each orchestrator instance polls a shard of "sagas with work pending" (partitioned by saga_id hash) and only contends on the lease row for the specific saga it's about to touch — instances working on different sagas never block each other at all.

Saga backlog age — the gap between a step becoming eligible to execute and an instance actually picking it up. When p99 of that gap exceeds the segment-1 step-latency target (800ms), that's your scale-out trigger, not raw CPU or instance count.

CONTROL / CONFIG PATH
Operator --> POST /saga-definitions --> Saga Definition Store

HOT / DATA PATH
order-svc --> POST /sagas --> Orchestrator Core --lease--> Saga State Store
                                     |
                                     v
                              Step Invoker --> payment-svc / nsdl-adapter / ...
                                     ^                        |
                                     |                 POST /step-callback
                                     |________________________|
                                     |
                        (timeout) --> Reconciliation Scheduler
                                            |
                                  query participant status --> resolve or MANUAL_REVIEW
                                     |
                              Outbox / Event Publisher --> downstream services

This one has no reference answer. Walk your FR list from segment 1 line by line against the diagram you just drew — for each FR, point to the exact box and arrow that satisfies it. If any FR doesn't map to something on the page, that's not a documentation gap. That's a component you haven't actually designed yet, no matter how confident the rest of the diagram looks.