Production Software EngineerSystems fail. Products shouldn't.
I build production products where payments, data, scale and failure are all part of the problem, from requirements and architecture through to implementation, infrastructure and recovery.
Payment systems · Data-intensive platforms · Multi-tenant products · Production AI
- 5M+
- events / dayCoinPerps, after dedup and reconciliation
- 12K
- concurrent sessionsFlyverr, at sub 300ms p95
- 11
- microservicesEscrowly's payment platform
- 50K+
- usersKadipay, across five client surfaces
- 5 yearsacross 3 companies
- WeiBlocks2024 to Now
- Kadipay2023 to 2024
- Team4Tech2021 to 2023
So here is one failing, without taking the product down with it.
The same payment three times. Working, then fragile, then broken and recovered.
Act 1 of 3, The happy path
A user submits a payment.
Every system is built this way. If you are looking for payments, scale or anything where correctness matters, this is the difference between an incident and a refund.
I don't design for the happy path.
- A payment succeeds.
- An event gets delivered.
- An exchange stays online.
- A tenant sees only its own data.
Those are the expected cases. Production is what happens when one of them doesn't.
That is where I start engineering.
What I've built
Different products, different constraints, the same discipline: work out what has to stay correct, what is allowed to fail, and where the system needs to recover on its own. Each one opens into the architecture, the failure modes and the trade-offs.
When money moves asynchronously, correctness cannot be optional.
When the upstreams fail, the product still has to serve.
Thousands of tenants change the isolation problem.
A stale payment state is a trust problem.
When the product is trust, the ledger is the product.
AI gets interesting when it has to run without you.
Before I choose the stack, I find the failure modes
I don't start with whether something should be Kafka, or whether it should be microservices. I start with the product requirement, the consistency boundaries, the workload and the ways it can break. The technology follows the constraint.
The order I actually work in
Why I chose what I chose
Not a technology collection. A set of decisions, each made for a reason. The interesting question is never what I used, it is why that was the right constraint to introduce.
Why Kafka on Escrowly, when I avoided it on Kadipay?
Escrowly moves money across eleven services, so event delivery is part of the product's correctness, not a background nicety. It needs durable, replayable, ordered delivery. Kadipay had a Postgres ledger that was already the source of truth for every money-affecting fact, so its bus only had to carry side effects like notifications and dunning. SQS did that with no cluster to operate. Same category of problem, opposite answer, because what had to be guaranteed was different.
Why a transactional outbox rather than publishing after the commit?
Publishing after a commit leaves a window where the money moved and the event never left. On Escrowly that window is a silent inconsistency someone finds days later in a support ticket. The outbox closes it by writing the event inside the same transaction as the state change. It costs a relay process and a table to maintain, and it means events arrive slightly later. I would take that trade every time in a payment system, and probably not in an analytics pipeline.
Why idempotent consumers instead of exactly-once delivery?
Exactly-once across a producer, a broker and several consumers is expensive to build and easy to get subtly wrong. At-least-once plus consumers that are safe to replay is behaviourally identical from the outside at a fraction of the cost. The honest catch is that it pushes correctness into every consumer, so it has to be enforced in review rather than assumed. That is a discipline cost the team carries, not a free win.
Why shared-schema RLS on Flyverr rather than a database per tenant?
Flyverr has thousands of individual creators, not dozens of enterprises. Database per tenant fails on connection pool arithmetic before anything else, and schema per tenant makes every routine migration scale with tenant count. Shared schema with row-level security keeps both manageable. The price is a real one: a single Postgres primary outage hits every tenant at once. That is acceptable for many small similar tenants and would be the wrong call the day one enterprise customer needs its own fault domain.
Why SET LOCAL, and why is that worth its own answer?
Because it is the detail that decides whether the isolation actually holds. At 12K sessions Flyverr needs PgBouncer in transaction pooling mode, and a session-level SET does not reliably survive across pooled transactions. Get it wrong and tenant context outlives the request that set it and is inherited by the next one on that connection. SET LOCAL scopes it to the transaction so it cannot. It is one line, and it is the difference between isolation and a silent cross-tenant leak under load.
Why Kubernetes on CoinPerps when everything else I build runs on Fargate?
Connector scaling on CoinPerps has to react to Kafka consumer lag, which is a custom application metric. That is KEDA territory, and it has no clean equivalent in ECS scaling on CPU or request count. Everywhere else the scaling signals are ordinary, so Fargate gets the same outcome with no control plane to own. Kubernetes here is paid for by a specific requirement, and I would not reach for it without one.
Why did HoldingHands stay a monolith when Kadipay did not?
Kadipay had four client surfaces with different compliance profiles and release cadences, so service boundaries followed real organizational boundaries. HoldingHands had one engineer and eight weeks. Splitting that into services would have bought coordination overhead and paid for nothing. The simplest infrastructure in this portfolio is deliberate: for a non-profit on a tight budget with one engineer, not reaching for Kubernetes is the senior decision rather than the absence of one.
Why a derived ledger balance instead of a counter?
On HoldingHands the pool balance is what members trust the platform with, so the question that matters is not what the number is but why it is correct. A mutable counter can drift from the records underneath it and you cannot tell which is wrong. Deriving it from an append-only ledger means it is always reconstructable and always auditable against Stripe. The cost is that every read of the balance is a sum, which is why that is exactly the query that got cached.
I like difficult systems for the right reasons.
I'm a hands-on software engineer focused on the engineering underneath production products.
My work has increasingly moved toward systems where the difficult questions are harder to hide: distributed payment workflows, high-volume data ingestion, tenant isolation, asynchronous processing and production AI.
I enjoy the point where a feature stops being “just a feature” and starts having consequences for correctness, concurrency, reliability and operations.
I care about architecture, but I also care about the implementation details that make the architecture survive production.
I build the system, reason through the trade-offs, test the failure paths and stay close to what happens after it ships.
Building something that's becoming difficult?
If you're working around payments, high-volume data, multi-tenant infrastructure, event-driven workflows or production AI, I'd be interested in understanding the problem underneath the product.
Answers only from what is written on this site, and names the system it is drawing from. If something is not covered, it says so rather than guess.