Market-Data Ingestion Platform

What is it?

A multi-repo platform that ingests a high-volume FIX market-data feed and keeps two properties true at once: the path never lets a noisy upstream session exhaust shared capacity, and every message that flowed through it can be reconstructed after the fact. It composes two independent services. The distributed rate limiter sits at the ingestion edge as a shared admission gate; the FIX session writes its raw frames to disk, and fixlog reads those logs back without loading them into RAM to answer “what actually happened” during an incident. Each service keeps its own repository, its own tests, and its own page — the platform is the contract between them, not a monorepo that absorbs them.

Inter-service flow

A frame arrives on the FIX session and the ingestion worker asks the rate limiter for an admission decision keyed by SenderCompID rather than by source IP, so a single misbehaving counterparty cannot starve the others sharing the same window. The limiter evaluates a Redis sliding window — the same Allow primitive each worker instance calls, so horizontal scaling of the ingestion tier does not loosen the global ceiling. Admitted frames are appended to the on-disk FIX session log in their raw wire form and forwarded to downstream consumers; rejected frames are shed at the edge and counted, never silently dropped. The session log is the seam between the two services: the limiter governs admission in real time, while fixlog operates offline against the bytes the worker already wrote, so the audit path adds zero latency to ingestion.

Failure handling and recovery

The two failure domains are deliberately decoupled. If Redis is unreachable, the limiter fails closed for the global counter but open for liveness: each worker falls back to a conservative per-instance local window so ingestion keeps flowing at a reduced, bounded rate instead of stalling on a dependency, and the breach is logged for reconciliation once Redis returns. If an ingestion worker crashes mid-write, the session log can contain a truncated trailing frame — which is exactly the dirty input fixlog is built to tolerate: it carries the partial frame across reads and re-scans it on the next pass instead of treating the truncation as fatal. Recovery after an incident is therefore a read-only operation: an operator points fixlog at the rotated session logs, filters by SenderCompID and message type, and reconstructs the admitted/shed timeline without touching the live path or replaying load against the limiter.

Why a system, not one project

Either repository stands alone: the rate limiter protects any horizontally scaled Go service, and fixlog reads any FIX log regardless of who produced it. What this page captures is the decision to compose them — the keying choice (SenderCompID, not IP), the fail-closed-counter / fail-open-liveness contract, and the on-disk log as the shared seam between a real-time gate and an offline auditor. That composition is the architecture; the member projects keep their own detail pages, and this entry links to them by translationKey so each repository remains the single source of truth for its own internals.