A corridor that works in a demo and one that survives production volume are two different systems. The demo moves one payment while you watch it. Production moves thousands an hour while a partner API times out, a webhook arrives twice, and a currency conversion half-finishes at 3am.
Four patterns decide whether the corridor stays up: idempotency, saga orchestration, observability, and reconciliation. Skip one and the corridor runs fine right up until the day it moves money it shouldn’t, or loses money nobody notices.
Before any of this gets built, we write it down. On corridor projects we start with an architecture description: the flow, the failure points, and the reliability properties each hop has to hold. The client’s team works from the same document and reviews it with us, so from day one they own the design, not just the outcome. The four patterns below are the core of what that document covers.
Here’s the shape of a single cross-border transfer, so the failure points are easy to place:
The reserve is a soft hold, so a transfer blocked at compliance is released before any real money moves. After that, every arrow is a call to a system you don’t control, and every box is a place the money can get stuck.
Idempotency: move the money once, however many times you’re told to
An idempotent handler gives the same result whether it runs once or ten times. For a payment webhook, that means the same event can arrive many times and the money still moves once.
Every payment engineer knows duplicates happen. The tell of whether a team has actually run money at volume is which duplicate they design for. At 100,000 payments a day, a one-in-a-million event isn’t a tail risk you can wave off. It happens, and then it happens again next week. Any probability you don’t drive to zero, volume converts into a certainty.
Duplicates are not an edge case. Payment partners deliver webhooks at least once, not exactly once. If your server is slow to return a 200, or the network drops your acknowledgment, the partner assumes you never got the event and sends it again. And the retry windows are long. Stripe retries failed deliveries for up to 3 days. PayPal’s IPN retries up to 15 times over 4 days. Adyen retries, then queues undelivered events for up to 30 days. So a “payout completed” event you already processed can come back tomorrow, or next week.
The arbiter is the database, never application logic. Each operation is keyed by transfer ID plus operation type, so distinct steps of one transfer never collide but a replay of the same step does, and that key is claimed under a unique constraint in the same transaction as the state change. Two copies race; one wins the insert, the other conflicts and does nothing. An inbox table drops replays at ingestion. Redis fronts it as a cache, never as truth.
- Every webhook carries an idempotency key. Use the provider’s stable event ID (Stripe’s evt_…, Adyen’s eventCode + pspReference), or build one from fields that don’t change on retry.
- The atomic part is the whole point. A naive “check if seen, then process, then record” has a race: two copies arrive at the same moment, both read “not seen,” both move money. The unique constraint closes that gap.
- Keep keys longer than the partner’s retry window, plus a margin. With Adyen queuing for 30 days, a 7-day TTL means a late retry looks like a new event. Size the TTL per partner from their documented window; when the window isn’t documented, 30 days of key storage is cheap insurance against a very expensive duplicate.
There’s a second place duplicates come from, and it’s your own infrastructure. We run webhook ingestion through Kafka: the endpoint validates, writes the event to a topic, returns 200 fast, and consumers do the actual money movement. That design creates two hard cases that separate teams who have shipped this from teams who have read about it.
The first is a consumer that does the work and then dies before its Kafka offset commits. Plain network trouble, and it happens. The offset is committed only after the money movement and its key persist together, so on restart those messages replay straight into an already-claimed key and no-op. Nothing doubles, nothing is lost. Set up the other way, with auto-commit before processing, and a crash silently drops payments instead.
The second is the one that actually costs people money. You call a partner, the request times out, and you don’t know whether they moved the funds. A blind retry risks paying twice; a blind reversal risks refunding someone who was in fact paid. Neither is safe, so doing nothing automatic is the correct move. Some partners expose an idempotency key or a status endpoint and you lean on it; many don’t, and the honest truth is that support is inconsistent across a real corridor. When it’s absent, the transfer holds in an unknown-outcome state that only a status call, a partner reference, or the settlement file can clear, never a retry. Pinning down which of the two each partner gives you, before the corridor is live, is the unglamorous work that decides whether that partner’s 3am timeout is a shrug or an incident.
Sagas: finish the transfer, or undo it cleanly
A cross-border transfer isn’t one transaction. It’s a sequence of steps across systems that don’t share a database: reserve, screen for compliance, debit the sender, convert FX, pay out through the partner, settle. A saga makes that whole sequence either complete or roll back to a clean state.
Why it’s hard: there is no ACID transaction spanning your ledger, an FX provider, a compliance service, and a partner bank. Each step is a separate call to a separate system, and any one can fail or time out. If the payout fails after the debit already took the sender’s money, returning an error is not enough. The money left the sender and never reached the recipient. It’s stuck.
We run the flow as an orchestrated saga on Step Functions, and we orchestrate rather than choreograph for one reason: when a step fails, we need one place that knows exactly how far the transfer got and what’s left to undo.
| Orchestration | Choreography | |
|---|---|---|
| How it works | A central coordinator calls each step and tracks state | Each service reacts to the previous one’s events |
| Where the transfer’s state lives | One place, easy to query | Spread across services, harder to see whole |
| Best fit | Money-critical flows with compensation logic | Simpler, loosely-coupled flows |
| The cost you pay | One more component to run | Harder to find a stuck transfer |
The payout step is where a real corridor gets specific. A payout isn’t one thing. Mobile money is asynchronous: you send the request, the partner accepts it, and the final status lands later as a separate callback, so the saga step suspends on a task token and resumes when that callback releases it. Cash pickup can stay pending far longer, until the recipient collects. We deliberately don’t hold a Step Functions execution open for that. Long-running rails close out into their own pending state in the database and resume on callback, because parking tens of thousands of open executions waiting days on a partner is exactly what that tool is bad at. Each rail gets its own completion model and its own timeout, because a payout still legitimately pending on one rail is a stuck transfer on another.
Compensation turns on a distinction that’s easy to skip: transient versus terminal. A partner returning a 500 or timing out is transient, so we retry with backoff, because compensating a payout that was about to succeed is its own incident. A partner rejecting the payout, for insufficient beneficiary KYC or an invalid account, is terminal, and that triggers the rollback. When it fires after the money has already moved, the saga runs in reverse: release the reservation, post a reversing entry against the debit we took after compliance, and unwind the FX.
That FX leg is the part people underestimate. Reversing a debit is bookkeeping. Unwinding an FX conversion means you booked it at one rate and close it out at a rate that’s moved, so the compensation has to carry that slippage explicitly, or your ledger balances while your actual currency position doesn’t. The compensations aren’t uniform either: a synchronous rejection is a clean reverse, but a cash pickup that sat for days and then expired or was cancelled has its own path, because the money may or may not have reached the recipient.
Two dependencies a real implementation can’t skip. Every compensating step is itself idempotent, because the orchestrator retries it, which makes the guarantee from the idempotency section load-bearing under the rollback. And compensation is a network call too: an FX unwind can fail. When it does, we retry it, and a compensation that exhausts its retries lands in the same manual-review state as any other unresolved transfer. A rollback that silently fails is worse than the original error.
The state machine drives all of this, not a person watching a queue. Terminal rejections compensate automatically and never touch a human. The one case we route to a person on purpose is the ambiguous one, a partner timeout where we can’t tell whether the payout landed, which parks for manual resolution against the partner’s status API or the settlement file, never a blind retry or reversal. Deciding which failures the machine unwinds itself and which stop for a human is the real work. Err toward caution and ops does toil that should’ve been automatic; err the other way and the machine confidently reverses a payout the partner actually made.
This is also where the architecture description earns its keep. The saga steps, their compensations, and the terminal states are written down before the code, and the client’s engineers review them with us. When their team later adds a corridor themselves, they extend a documented state machine instead of reverse-engineering our code.
Observability: what actually happened to transfer T?
Observability is being able to reconstruct what happened to one payment after the fact, without re-running it. Which step ran, which partner responded, what each decision was, and where it stopped.
In practice, incidents don’t get found by someone staring at a dashboard. They get found by an alert firing, a runbook pointing at the likely layer, and an on-call escalation. Observability is what makes each of those steps fast instead of a hunt. The two failures we see most on a live corridor are ordinary: a partner hanging, and consumer lag building on a topic. Neither is exotic, and both are the kind of thing that’s a five-minute answer with the right correlation and a multi-hour one without it, the difference between one query on a transfer ID and grepping five services by timestamp and hoping the clocks agree.
What good looks like:
- One correlation ID per transfer, passed through every service and across every partner boundary. For us the correlation ID is the transfer ID itself, present in every service’s logs and carried in Kafka message headers, so a payment that passes through a topic stays traceable rather than disappearing into it. Distributed tracing with OpenTelemetry and W3C trace context does the propagation.
- Consumer lag per topic, alerted on as a first-class metric. Growing lag on the payouts topic is usually the earliest signal something downstream is failing, a partner slowing or a consumer wedged, minutes before the first customer complaint. That’s the alert that starts the runbook, not the customer.
- Structured logs, so every line carries the transfer ID, step, partner, decision, and outcome as fields you can query. Not prose you read line by line.
This corridor ran on Azure and was later migrated to AWS, with AppDynamics and Grafana for tracing, metrics, and dashboards. The vendors matter less than the property they buy: one identifier that survives every service and partner hop, so any single transfer’s full path is one query away.
PII rules, because a payments log is one bad line away from a compliance incident:
- Log references, never payloads. Transfer ID, partner reference, amount, currency, status. Card numbers, account numbers, names, and addresses stay out of logs entirely.
- Mask at the edge, not at review time. Sanitize in the logging layer before the line is written, so no service can leak by accident.
- Scan for leaks anyway. We run automated PII scanning over log sinks to catch anything that slips through, because someone will eventually log a raw partner response during an incident.
- Trace attributes follow the same rule as logs. A span tagged with a card number is a leak with better tooling.
The three signals answer different questions:
| Signal | Answers | Example in a corridor |
|---|---|---|
| Logs | What happened at this step | “Compliance held T for manual review at 03:12” |
| Metrics | How often and how bad, across all transfers | “Partner X payout failures up 4x in the last hour” |
| Traces | The full path of one transfer | “T: reserve ok → compliance ok → debit ok → payout hold → stopped” |
Without this you can’t tell a real incident from noise, and you can’t answer a partner or a regulator asking about one specific payment.
Reconciliation: the money that goes missing quietly
Reconciliation is the daily, per-rail check that the money you think moved matches the money that actually moved. Your internal ledger on one side, the partner or bank statement on the other.
The quiet break is the one worth building the whole system around, because it’s the failure the other three patterns can’t see. Idempotency stops a duplicate at the door. A saga drives a transfer to a terminal state. Observability lets you trace one payment. None of them notices when your ledger says a payout settled and the partner’s statement never confirmed it. No error, no stuck transfer, no duplicate, just books that are internally consistent and quietly wrong. Reconciliation is the only place that surfaces it. The direction of regulation is the same: the UK’s FCA is moving payment firms to mandatory daily internal and external reconciliations from 2026, so daily recon is turning from good practice into a legal floor.
It stays hidden precisely because each piece looks correct on its own. The payout has a reference, a status, a matching ledger entry. Nothing about that single record is wrong. What’s wrong only exists in the aggregate: match totals per day, currency, and partner, and the day’s payout sum comes up short by one transaction’s worth. Now there’s a break to chase that no line-level check flagged, because every line looked fine in isolation. That second pass, totals rather than just references, is what turns a silent shortfall into a ticket.
This is also why we reconcile at two levels. Internal ledger against partner and processor statements catches the wrong or missing individual payout. Ledger against bank settlement files catches drift in the cash position itself, what actually left the account versus what we recorded. A corridor needs both, and the unglamorous half of the job is the data: some partners expose an API, many send a daily file over SFTP, the bank sends settlement files in its own schema, each with its own layout, cutoff, and time zone. Normalizing that zoo into one comparable shape is most of the work before a single line gets matched.
Every unmatched item becomes a break with an owner, a ticket aged against an SLA. Differences inside a defined tolerance auto-resolve, so people spend attention on real breaks, not FX rounding. Cross-border makes this matter more, because settlement lag and multi-currency timing make some breaks temporary: a credit that settles a day later isn’t lost money, so build tolerance windows per rail so real breaks stand out from timing noise. Unattributed money lands in a suspense account until it’s matched, and one rule separates a real ledger from a hopeful one: suspense doesn’t grow. An entry aging there isn’t waiting, it’s unfinished reconciliation, and you chase it down before it becomes the shortfall a regulator asks you to explain.
The four together
These patterns don’t overlap. Each one catches a class of failure the others let through.
| Pattern | The failure it stops | What good looks like |
|---|---|---|
| Idempotency | The same event moves money twice | Key on transfer ID + operation type, atomic claim under a unique constraint, keys outliving the partner’s retry window |
| Saga orchestration | A multi-step transfer half-finishes and gets stuck | A compensation per step, transient vs terminal handling, driven to completed or fully reversed |
| Observability | You can’t tell what happened to a stuck payment | One correlation ID across every service, Kafka hop, and partner; structured logs; traces |
| Reconciliation | Money drifts quietly until it’s a compliance problem | Daily per-rail matching on totals and lines, breaks owned and aged, suspense that never grows |
Architecture gets a corridor live. These four keep it alive once volume turns rare edge cases into daily ones.
One more thing we learned to do on every corridor engagement: treat the architecture description as the working spec and keep it current as the system changes. We build against it, the client’s engineers review changes to it, and by the end of the engagement their team can extend the corridor without us. Some of that can’t be written down in advance. Every partner has its own retry behaviour, its own settlement timing, its own contract, and each needs a real conversation between engineers and business on both sides. So the spec covers the properties that hold everywhere, the four patterns above, and the people cover the parts where partner reality differs. We’ll write up how we run that spec-driven process in a separate post.
We’ve run this at volume. The transfer corridors we built and maintain for WorldRemit/Zepz cover 130+ countries and 70+ currencies, tuned to run 24/7 at over 100,000 payments a day. At that volume a one-in-a-million duplicate is a daily event, which is why none of these four patterns is optional.
Building corridor reliability in-house, or deciding whether to embed a team that has run it at scale? Read how fintechs actually add new payment corridors for the architecture side, see the WorldRemit case, or talk to our payments team.
