Kadipay
A stale payment state is a trust problem.
Kadipay · 2023 to 2024 · Live
Kadipay runs two product lines that are really the same hard problem wearing different clothes: buy now pay later installments at checkout, and recurring subscription billing. Both are about moving money on a schedule, reliably, with failure handling that keeps the books correct.
There are four separate portals on one backend because consumer, merchant, admin and agent are genuinely different bounded contexts, with different permission models, release cadences and even regulatory exposure. Merchant onboarding carries KYC adjacent requirements the consumer app never touches. Splitting at the client tier while sharing backend services avoids both a permission branching monolith and a full duplication of business logic per portal.
Owned the frontend service layer and state management across the multi portal client surface, the integration with the payment and installment services, and end to end testing of both the success and the failure paths.
50K+ users across a React Native app and four distinct web portals, on a shared microservices backend.
Retries, duplicates and missing events are the normal case
In a payment product the interesting states are not the ones where everything works. A user double taps checkout. The processor sends the same webhook twice. The processor never sends the webhook at all. An installment declines. The client shows a payment as pending that actually settled ten minutes ago.
Every one of those is expected traffic, not an exception, so each gets a specific mechanism rather than a generic retry. Duplicate checkout is deduped on a client generated idempotency key at the order service boundary, and the second request returns the original result instead of creating a second plan. Duplicate webhooks hit a unique constraint on the processor event id and become a no op. A webhook that never arrives is caught by a nightly reconciliation job that diffs the processor's own transaction records against the ledger, because relying on webhooks alone for money critical state is a single point of failure.
Dunning is where I think the real judgment shows. A failed installment is not retried blindly, it is retried according to why it failed. Insufficient funds gets a delayed retry, on the theory that balance timing is the likely cause and an immediate retry just burns the processor's retry quota. An expired card is not retried at all until the user updates their method, because no number of attempts will fix it. A generic decline gets standard backoff, with the plan visibly past due to both consumer and merchant the entire time rather than failing quietly.
What it already handles
- Duplicate checkout
Double tap, or a network level retry
One plan, one chargeSUBMIT x2IDEM KEYONE PLAN - Webhook delivered twice
Processor redelivery
One state transitionHOOK x2EVENT IDNO-OP - Webhook never arrives
Delivery lost on the processor side
Webhooks are not the only source of truthSILENCENIGHTLYCORRECTED - Installment declines
Decline reason returned by the processor
Never a silent failureDECLINEDBY REASONPAST DUE - Stale client state
Payment settled after the client cached it
The backend is the source of truth, never the cacheSTALE VIEWINVALIDATEREFETCHED
Failure modes
- Failure
Duplicate checkout submission
How it is caughtClient generated idempotency key per attempt
What happens nextDeduped at the order service boundary. The second request returns the original result and never creates a second plan or charge.
- Failure
Processor webhook delivered twice
How it is caughtProcessor event id already recorded
What happens nextUnique constraint on processed webhook events makes the second delivery a no op.
- Failure
Processor webhook never arrives
How it is caughtNightly reconciliation against the processor's transaction API
What happens nextAny charge or refund the processor has that the ledger does not raises an alert and a corrective entry. Belt and braces against trusting webhooks alone.
- Failure
User cancels while a renewal charge is in flight
How it is caughtOptimistic concurrency on the subscription row
What happens nextThe charge job rechecks status immediately before charging. If it was cancelled in the interim the charge is aborted and any authorization voided.
- Failure
Installment declines
How it is caughtProcessor decline webhook, with the reason code
What happens nextDecline reason aware dunning rather than a blind retry. The plan is visibly past due to consumer and merchant throughout, never silently failing.
- Failure
Partial refund on a multi installment order
How it is caughtRefund requested against an order, not a single charge
What happens nextRefund logic walks paid installments in reverse, refunding most recent first, so the ledger stays balanced rather than issuing one lump refund against an ambiguous order total.
- Failure
Merchant already paid out, consumer then disputes
How it is caughtDispute raised post payout
What happens nextA rolling holdback on each payout exists to fund exactly this, so there is no need to claw back from the merchant after the fact.
What it is held to
Checkout payment processing, p99
Under 2s
Directly tied to cart abandonment. Checkout is the highest value latency path in the system.
Checkout path availability
99.95%
Revenue critical.
Webhook to internal state update
Under 30s from receipt
Bounds how long anyone can see a stale order or payment status.
Admin and agent portal availability
99.9%
Important, but not revenue blocking in real time.
Installment default rate
Tracked, deliberately not an SLO
A business metric about underwriting calibration, reviewed independently of system uptime.
Why I chose what I chose
Why SNS and SQS here rather than Kafka?
Deliberately not Kafka, unlike CoinPerps. The Postgres ledger is already the durable source of truth for every money affecting fact. The bus only has to move side effects, notifications, dunning, cache invalidation, reliably and at least once. It never needed replay or ordered log semantics, and a fintech team this size gets a fully managed queue instead of adding Kafka operations to its plate for guarantees it was not going to use.
Why Redux Toolkit and RTK Query across five client surfaces?
The five surfaces share overlapping data, order state and installment status, and a stale payment succeeded view is a trust problem rather than a cosmetic one. RTK Query gives a normalized cache with automatic invalidation on mutation, so there is one source of truth for that state. Its optimistic update and rollback pattern is used specifically for installment status, so the UI reacts immediately but reverts cleanly when the server disagrees.
Why PostgreSQL for money movement when other data lives elsewhere?
Ledger, order and installment plan state need transactional guarantees. An installment can never be half charged in the data layer. Relational plus row level locking is non negotiable for that slice, whatever the rest of the stack looks like.
Why tokenize at the processor rather than store card data?
It keeps PCI scope at SAQ-A territory instead of full Level 1 infrastructure. The platform never touches a raw card number, only processor tokens. That is a smaller surface to secure and a much smaller surface to be audited on.
What I would reconsider
- Microservices at this size is heavier operationally than the traffic alone needs. The justification is organizational, four client surfaces with different compliance profiles iterating at different speeds, rather than throughput. That distinction is worth stating plainly rather than implying a scale reason that is not there.
- Nightly reconciliation is a pragmatic gap catcher, not a real time guarantee. For a higher volume version of this system the next step is a near real time reconciliation stream, trading simplicity for faster drift detection.
React · React Native · Redux Toolkit · RTK Query · TypeScript · PostgreSQL · AWS SNS/SQS · ECS Fargate · Jest