Multi-tenant marketplace

Flyverr

Thousands of tenants change the isolation problem.

WeiBlocks · 2024 · Live

12Kconcurrent sessionsOn shared infrastructure with database enforced isolation

<300msp95 API latencyHeld while every query is RLS filtered

Flyverr is a marketplace where creators sell digital products and resellers can relist what they bought, under licence terms that pay the original creator a cut on every resale. The engineering weight sits in two places: keeping thousands of tenants isolated on shared infrastructure, and splitting money correctly when a sale has three parties instead of two.

The tenant profile drives everything. This is thousands of individual creators, not dozens of enterprises, and that cardinality is what rules out the isolation models that would otherwise be obvious.

What I owned

Architected and shipped the platform: the tenant isolation model, the RLS policy design, Stripe Connect payments and payouts, and the scaling work behind 12K concurrent sessions.

Scale

12K concurrent sessions at sub 300ms p95 API latency, across thousands of independent seller tenants rather than a handful of large ones.

Transaction pooling quietly breaks session scoped tenant context

Shared schema with row level security is the right isolation model at this tenant count. Database per tenant fails on connection pool arithmetic alone, since thousands of tenants each holding even a small pool exhausts Postgres. Schema per tenant turns every routine migration into an operation whose cost scales with tenant count.

So RLS it is, with a deny by default policy: a query arriving with no tenant context returns zero rows rather than erroring or leaking, which is the safer direction to fail.

The subtle part is what happens at concurrency. 12K sessions cannot each hold a dedicated Postgres connection, so PgBouncer runs in transaction pooling mode. Transaction pooling has a specific interaction with RLS: a session level SET does not reliably survive across pooled transactions, which means tenant context can outlive the request that set it and be inherited by the next request on a reused connection.

The fix is small and load bearing. Tenant context is set with SET LOCAL inside the request's transaction, so it cannot outlive that transaction regardless of how the connection is reused afterwards. Getting this one detail wrong is the difference between isolation and a silent cross tenant data leak under load.

RLS is the hard boundary, but it is not the only one. Every query is also explicitly scoped at the application layer, so a single missed or misconfigured policy is not the only thing standing between one tenant's data and another's.

What it already handles

Cross tenant read attempt
A request scoped to one tenant asks for another's rows
TENANT A
RLS
0 ROWS
Deny by default, enforced by the database
Connection reused by another tenant
PgBouncer hands a pooled connection to the next request
SET LOCAL
COMMIT
NO CARRY
Reuse is safe by construction
Half finished resale split
Charge succeeded, one of two transfers did not
CHARGED
1 OF 2
RECONCILE
Buyer unaffected, split completes late
Resale chain, several hops
A sells to B, B resells to C
A → B → C
ORIGINAL ID
ROYALTY
No chain of chains lookup at payout time
Chargeback after payout
Buyer disputes once royalties already moved
DISPUTE
HOLDBACK
CLAWBACK
Room to claw back without chasing the seller

Failure modes

FailureHow it is caughtWhat happens next
Failure
A new table ships without an RLS policy
How it is caught
CI gate asserting cross tenant queries return empty for every protected table
What happens next
The merge is blocked until the policy exists and the test passes. Isolation is a release gate, not a runtime property trusted by convention.
Failure
Tenant context leaks across pooled connections
How it is caught
Prevented structurally rather than detected
What happens next
SET LOCAL inside the transaction, never a session level SET. This is the single most important PgBouncer and RLS detail in the system.
Failure
One transfer in a resale split fails after the charge succeeded
How it is caught
Reconciliation job scanning for purchases with incomplete transfer state
What happens next
Idempotent retry keyed on purchase and recipient role, escalating to manual review after a bounded number of attempts rather than retrying forever in silence.
Failure
Two buyers race for a limited licence product
How it is caught
Unique constraint and row lock at purchase creation
What happens next
The second charge is never attempted, because the check happens before charging rather than after. The platform never double sells and then refunds one buyer.
Failure
A seller's payout fails on stale bank details
How it is caught
Stripe payout.failed webhook
What happens next
Funds are held on the connected account rather than lost, and the seller is prompted to update details before the next attempt.
Failure
One seller's heavy catalog query slows everyone
How it is caught
Per tenant query latency percentiles
What happens next
Browse and search run against a read replica while checkout writes go to the primary, so one tenant's storefront cannot starve checkout latency for the rest.

What it is held to

API latency, p95
Under 300ms
A direct product requirement at 12K concurrent sessions.
Catalog and browse availability
99.95%
High, but not the payment critical tier.
Checkout path availability
99.99%
Revenue critical, so it is held to a higher bar than browse.
Cross tenant data isolation
Zero tolerance, enforced in CI
Treated as a correctness invariant rather than a best effort security control.
Checkout unavailability during a primary failover
Under 120s
Bounded by the Multi-AZ failover window itself. Anything longer is the platform's highest blast radius failure mode and trips a SEV1.

Why I chose what I chose

Why shared schema with RLS rather than a database per tenant?
Tenant count here is thousands of individual creators. Database per tenant fails on connection pool arithmetic before anything else. Schema per tenant makes routine migrations scale with tenant count. Shared schema with RLS keeps both manageable, at the cost of a bigger blast radius per outage, which I would trade differently the moment an enterprise tenant needed its own fault domain.
Why enforce isolation twice?
RLS is the hard boundary, but relying on one layer for a security property that must never fail is a single point of failure. Application level scoping is defence in depth, so a missed policy is not the only thing between one tenant's data and another's.
Why Stripe Connect rather than an in house ledger and payouts?
Connect takes seller KYC and payout compliance off the platform entirely. For a product onboarding thousands of individual sellers, that compliance surface is a cost with no product differentiation upside. The platform fee comes out of each charge automatically.
How does a three way resale split work when Connect only splits two ways?
The charge lands in the platform balance rather than going direct to a connected account. The platform fee is retained, then two separate transfers go out, one to the reseller and one to the original creator. Both are recorded against the same purchase row so the full split is reconstructable from one record, which matters the first time someone asks where a specific sale actually went.

What I would reconsider

Shared schema RLS means one Postgres primary outage affects every tenant at once. That is an acceptable trade for many small similar tenants and the wrong one the day an enterprise tenant needs an independent SLA. That case would justify carving out database per tenant as an exception rather than changing the model for everyone.
Reconciliation based transfer retry is eventually consistent by design. A royalty payout can lag the sale in the failure case. The buyer facing purchase still succeeds immediately, so only the internal distribution has a bounded delay.