Payment infrastructure

Escrowly

When money moves asynchronously, correctness cannot be optional.

WeiBlocks · 2024 to present · Live

11microservicesSplit so one domain failing cannot take the others with it

4engineers ledFrom first architecture decision through to production

Escrowly holds funds between two parties until the terms of a deal are met, which makes it a system where a single lost event or a double applied one is not a bug you can ship past. The money has to be right, and it has to be provably right afterwards.

The platform is split into 11 domain services communicating over Kafka, each owning its own PostgreSQL schema. A double entry ledger records every movement, so the current balance is always reconstructable from history rather than trusted as a stored number.

What I owned

Led a team of four engineers. Owned the architecture end to end: service boundaries, the event delivery model, the ledger, and the wallet infrastructure.

Scale

11 NestJS microservices, PostgreSQL with schema per service isolation, Kafka as the event spine, AWS KMS encrypted wallets.

A database commit and an event publish are two different operations

The naive version of this system writes to the database and then publishes to Kafka. It fails silently. If the service dies between those two operations the state changed and nothing downstream ever hears about it. No retry, no alert, just an inconsistency that surfaces days later as a support ticket about missing money.

The inverse is just as bad. Retry the publish and a consumer can apply the same transfer twice.

I used a transactional outbox: the event is written into the same local transaction as the state change, so the two cannot diverge. A separate relay publishes from the outbox with retries. Consumers apply changes idempotently, keyed so a redelivery is observed rather than re-executed.

Events that exhaust their publish retries land in a dedicated failed event store with an automated alert and a safe re-queue path. That is the part I care about most: a delivery failure becomes a visible, recoverable incident with an owner, instead of a silent divergence nobody is looking for.

What it already handles

Publish fails after commit
Broker unreachable at the moment of publish
COMMIT
PUBLISH X
RELAY
Event delivered late, never lost
Same event delivered twice
At least once delivery doing its job
APPLIED
REDELIVERY
OBSERVED
One movement, not two
Retries exhausted
Sustained downstream outage
RETRIES
FAILED STORE
ALERT
A visible incident with an owner
Internal transfer between users
Peer to peer transfer inside the platform
TRANSFER
LEDGER
CUSTODY
No on chain transaction, no gas fee

Failure modes

FailureHow it is caughtWhat happens next
Failure
Publish fails after the local transaction committed
How it is caught
Outbox row stays unpublished past its threshold
What happens next
Relay retries from the outbox. State and event were committed together, so nothing is lost while it retries.
Failure
Retry delivers the same event twice
How it is caught
Consumer sees an event key it has already applied
What happens next
Idempotent consumers observe the first result again rather than applying a second time.
Failure
Publish retries exhausted
How it is caught
Retry budget spent without an ack
What happens next
Event captured in the failed event store, alert raised, safe re-queue available once the cause is cleared.
Failure
Consumer applies a change the ledger disagrees with
How it is caught
Double entry ledger will not balance
What happens next
The ledger is append only, so the movement is reconstructable and correctable with a compensating entry rather than an in place edit.

Why I chose what I chose

Why a transactional outbox rather than publishing to Kafka directly?
A direct publish after a commit has a window where the state has changed and the event has not been sent. The outbox closes that window by making the event part of the same transaction. It costs a relay process and a table. It buys the guarantee that state and events cannot diverge, which in a payment system is the whole game.
Why idempotent consumers rather than exactly once delivery?
Exactly once across a producer, a broker and multiple consumers is expensive to build and easy to get subtly wrong. At least once delivery plus consumers that are safe to replay is behaviourally equivalent from the outside, at a fraction of the complexity. The correctness lives in the consumer, where it can be tested directly.
Why a schema per service rather than one shared database?
A shared schema turns every service into a potential writer of every table, and the first incident where two services disagree about who owns a row is the one that teaches you why that matters. Schema per service makes ownership explicit and keeps a migration in one domain from being a coordination problem across eleven.
Why a double entry ledger rather than a balance column?
A mutable balance tells you what the number is. A ledger tells you why the number is correct. When a balance is disputed, and in an escrow product it will be, the ledger is the difference between an answer and an apology.
Why one persistent wallet per chain rather than a wallet per transaction?
A wallet per transaction means repeated deposits and repeated gas fees for the user. One persistent wallet per chain at signup lets funds be reused across deals, and internal transfers become ledger movements rather than on chain transactions. Less friction for the user, and a much simpler reconciliation story internally.

What I would reconsider

Eleven services at this stage is more operational surface than the traffic alone justifies. The reason is domain isolation for money movement rather than throughput, and it is worth being precise about that rather than implying a scale story that is not there.
The outbox relay adds a component that has to be running for events to flow. It is a deliberate trade: a delivery delay when the relay is down is recoverable, a lost event is not.
At least once delivery pushes correctness into every consumer. That is a discipline the team has to hold, and it needs to be enforced in review rather than assumed.