Real-time data infrastructure

CoinPerps

When the upstreams fail, the product still has to serve.

WeiBlocks · 2024 to present · Live

5M+normalized events / dayMeasured after dedup and reconciliation, not raw wire traffic

20+exchange integrationsEach isolated in its own process so one cannot stall another

<500msp99 exchange to clientTail latency, because that is when users are watching hardest

CoinPerps unifies live market data from more than twenty exchanges and half a dozen news sources into one normalized stream. The hard constraint is not throughput, it is that upstream failure is the steady state rather than the exception. Exchanges go down, rate limit, drop connections and send malformed payloads, and the product still has to serve a coherent view of the market.

The number worth explaining is the 5M events a day. That is the count after dedup, sequence reconciliation and coalescing, not raw inbound. Order book delta streams alone run an order of magnitude above trade volume on an active exchange. The pipeline's actual job is compressing that firehose down to the state changes that are worth persisting and pushing, so the filtering ratio is the more interesting fact than the headline number.

What I owned

Designed and built the ingestion, normalization and real-time delivery pipeline end to end: the connector layer, the event spine, the aggregation logic, and the WebSocket fan-out to clients.

Scale

20+ exchange connectors and 6+ news pollers running as independent processes. Around 5M normalized events a day sustained, with 10x to 20x bursts during liquidation cascades and macro prints.

Never trust someone else's clock, and never guess at a gap

Every exchange has its own idea of what a tick looks like and what time it is. Exchanges have been observed sending timestamps seconds out. So every normalized event carries two timestamps: the exchange reported one, which is kept for display and debugging and trusted for nothing, and a server authoritative receive time assigned by our own clock.

Ordering and staleness decisions are made only on the authoritative one. That single decision removes an entire category of cross exchange ordering bugs caused by trusting third party clocks.

The second half is sequence integrity. Where an exchange gives sequence numbers, the connector buffers deltas, fetches a REST snapshot, discards anything at or below the snapshot sequence, and applies the rest in order. If a gap appears later, the book is marked stale and resynced rather than quietly continuing on state that might be wrong. Where an exchange gives no sequence numbers, correctness is weaker by necessity, so staleness is instead bounded by a forced periodic resnapshot.

What it already handles

Sequence gap
Delta sequence jumps by more than one
GAP
RESNAP
BOOK OK
Never continues on a book that might be wrong
Exchange goes dark
Heartbeat timeout, then sustained
SOURCE X
EXCLUDED
STILL SERVING
Degraded in one source, not down as a platform
Malformed payload
Schema validation fails at the boundary
BAD MSG
DEAD LETTER
CONTINUES
One bad message never kills the connection
Volatility burst
Tick rate 10x to 20x normal
QUEUE DEPTH
SHED
TRADES OK
The critical path degrades last
Client reconnects
Network blip on the client side
DROPPED
LAST SEQ
CAUGHT UP
No silent gap from the client's point of view

Failure modes

FailureHow it is caughtWhat happens next
Failure
Exchange WebSocket disconnects
How it is caught
Heartbeat timeout
What happens next
Exponential backoff with jitter. The connector is marked degraded after two missed cycles, and clients see an explicit staleness flag rather than frozen values that look live.
Failure
Exchange rate limits us
How it is caught
HTTP 429 or an explicit throttle message
What happens next
Per exchange token bucket tuned to documented limits. On a 429 the connector reduces its own rate below the threshold rather than retrying straight into the wall.
Failure
Sequence gap in the delta stream
How it is caught
Local sequence tracking sees a jump
What happens next
Book marked stale, snapshot and delta resync runs, and it stays marked until reconciled.
Failure
Malformed payload
How it is caught
Schema validation at the connector boundary
What happens next
Quarantined to a dead letter topic with the raw payload kept for inspection. The connector keeps processing. One bad message never kills the connection.
Failure
Kafka partition unavailable
How it is caught
Producer ack timeout
What happens next
Bounded local buffer with disk backed overflow, then drop oldest with alerting past a hard cap. Consumers resume from their last committed offset, so delivery is at least once and consumers apply idempotently.
Failure
Crossed book, bid above ask
How it is caught
Sanity check in the aggregator
What happens next
Excluded from the composite price rather than allowed to skew it. Kept in raw storage for later analysis.
Failure
Full exchange outage
How it is caught
Connector down status persists
What happens next
That exchange is dropped from composite calculations and marked unavailable to clients. The platform feed continues, which is the literal implementation of the product promise.

What it is held to

Exchange event to client delivery, p99
Under 500ms
Sub second is what makes it feel real time. p99 rather than average because tail latency spikes during volatility, which is exactly when people are watching.
Platform feed availability
99.9%
The composite feed, deliberately decoupled from any single exchange's uptime.
Per connector availability
Deliberately not an SLO
Individual exchange outages are expected and handled by design. Treating them as incidents would be alerting on normal operation.
Silent data loss
Zero tolerance
Every event is either delivered or the affected scope is explicitly marked stale. Silent staleness is treated as the bug class, not lost data.
Gap detection to resync complete
Under 3s, p95
Bounds the window where a client could be looking at a book that is still catching up.

Why I chose what I chose

Why Kafka here, when other systems in my work use SQS?
This pipeline needs ordered, replayable delivery per symbol, with consumer group scaling and enough retention to backfill. Kafka's partition per symbol hash gives ordering where it matters without a global bottleneck. Redis Streams has weaker replay semantics and couples durability to Redis memory behaviour. RabbitMQ is queue shaped rather than log shaped, which makes reprocessing awkward.
Why one process per exchange?
Node's event loop is vulnerable to being blocked by CPU bound JSON parsing during a tick burst. Isolating each exchange in its own process means a slow parse on one cannot delay another, and normalization moves to worker threads when queue depth crosses a watermark. It is also the infrastructure expression of the isolation principle the whole product depends on.
Why ClickHouse for tick history?
The workload is append only, very high write volume, and read as wide analytical queries like a VWAP across exchanges over four hours. That is a columnar store's shape, not a row store's. TimescaleDB was a close second and lost on ingestion ceiling and compression ratio at this event volume.
Why EKS here when everything else I build runs on Fargate?
Connector scaling needs to react to Kafka consumer lag, which is a custom application metric. That is KEDA territory, and KEDA is native to Kubernetes autoscaling in a way that has no clean equivalent on ECS request count or CPU based scaling. The Kubernetes cost is justified by a real requirement here, and it is not the default I reach for elsewhere.
Why is the gateway allowed to run on Spot?
Both the gateway and the connectors already handle abrupt disconnect and resume, because that is what an exchange does to them daily. A Spot interruption is functionally the same event they already recover from, so running them on the cheapest capacity costs almost nothing in extra engineering. The stateful tiers, Kafka and ClickHouse, are bought for stability instead.

What I would reconsider

Node.js for the connectors is a velocity and consistency choice, not a performance first one. It is fine at current scale and it is the first thing I would revisit if per connector tick rate grew an order of magnitude, starting with worker thread offload and then a selective rewrite of the highest volume connectors.
Redis pub/sub as the fan-out mechanism is simple and fast to build but is not purpose built for massive WebSocket fan-out. It is the honest first bottleneck in this design, and the path out is a dedicated pub/sub tier that decouples subscriber count from gateway process count.
At least once delivery rather than exactly once was deliberate. Idempotent consumers make it behaviourally equivalent from the client's side for a market data use case, at a fraction of the engineering cost.