fixlog

What is it?

A command-line tool and interactive TUI that read FIX (Financial Information eXchange) logs the way they actually appear in production: gigabytes of heterogeneous, partly-corrupt messages. It mmaps the file, sniffs the layout (field separator, line prefix, encoding), and tokenizes each message into (tag, &[u8]) slices that borrow straight from the mapped bytes — zero allocation, zero copy from disk to rendered output. A dictionary layer resolves raw tag numbers to field names for FIX 4.4, FIXT.1.1, and FIX 5.0 / SP1 / SP2; a rayon-parallel index and a hot-tag map sit on top for filtering, tailing, and analysis. It ships as one binary, fixlog, built from a ten-crate Cargo workspace.

Context & Challenge

Real FIX logs are hostile to a naive reader on three axes at once. They are large — an order-management or market-data session produces millions of messages and easily reaches several gigabytes, so loading the whole file into a Vec is not an option. They are heterogeneous — every engine and broker writes a different layout: raw QuickFIX logs delimited by SOH (0x01), pipe- or caret-rendered exports, and lines wrapped in timestamp or logback prefixes. And they are dirty — truncated messages, blank lines, and broken checksums are normal, not exceptional. A parser that assumes one separator and one FIX version, loads the file eagerly, and treats a checksum mismatch as fatal will mis-parse most real logs and crash on the first malformed line. The challenge was to process arbitrary-size logs in any of those layouts, without a configuration flag per format and without dying on bad input.

Decisions

I made the format sniffer, not the user, decide the layout: it inspects the head of the file for the separator, line prefix, and line ending. The parser then ignores the prefix entirely and scans for 8=FIX message boundaries with memchr’s SIMD-accelerated search, so variable-length timestamp/logback prefixes “just work” without ever being configured. I kept the parser and the dictionary fully decoupled — the parser emits raw tag numbers and borrowed byte slices and knows nothing about field names; the dictionary resolves them by selecting a version chain from BeginString + ApplVerID. Adding a FIX version is dropping a QuickFIX XML schema into the build and registering one chain; it never touches the parser.

Checksum and body-length mismatches are non-fatal by design: a bad message is still emitted and logged at debug, never panicked on. For speed at scale I built the index with rayon, splitting the buffer into chunks with explicit boundary ownership so the parallel output is bit-identical to the single-threaded one, plus a secondary hot-tag map ((tag, value) → [ordinals]) so equality filters short-circuit instead of scanning every message. I rejected a RoaringBitmap for that map — denser for huge files, but slower to iterate and an extra dependency — in favor of a plain HashMap<(tag, value), Vec<u32>>.

Order consolidation across rotated logs is streaming, not index-driven: it reads each input (plain, .gz, or stdin) in 1 MiB chunks, carries the trailing partial message across reads, coalesces cancel/replace chains with a union-find over ClOrdID → OrigClOrdID (tags 11 → 41), and deduplicates fills by ExecID (tag 17).

Performance

Parsing sustains roughly 1 GiB/s on long market-data messages and ~240–310 MiB/s on shorter order-management logs (the difference is per-message overhead, not bytes). The rayon index build runs 2.4×–5.1× faster than single-threaded, reaching ~1.08 GiB/s on a 40 MiB buffer; buffers under 1 MiB fall back to the single-threaded path because thread dispatch dominates below that. The hot-tag pushdown is the largest single win: an equality filter like 35=D over one million messages dropped from ~477 ms (full scan) to ~156 µs by intersecting pre-indexed ordinal lists — about 3000×. The interactive TUI renders a frame in ~737 µs on a 1M-message log, ~22× under the 16 ms budget for 60 fps. Rewriting the temporal histogram to extract timestamps in parallel with a narrow single-tag scan took it from ~573 ms to ~32 ms (−94%). Consolidating a ~540 MB corpus of rotated order logs produces ~55k aggregated orders in under five seconds. The release profile is tuned for throughput (lto = true, codegen-units = 1).

Lessons

The sharpest lesson was that the same field name can mean two different numbers. OrderConsolidated.avg_px is computed (notional / cum_qty, where notional sums LastQty · LastPx over fills), while OrderEvent.avg_px is the raw wire value of tag 6 — conflating them silently reports the wrong average price. The second was that real-world data breaks tidy algorithms: a textbook union-find on ClOrdID → OrigClOrdID merged unrelated orders because some broker logs use a placeholder anchor (41=NONE); the fix was a guard that makes an order its own root unless its OrigClOrdID was actually observed earlier as a real ClOrdID. I also stopped trusting buffer.len() as “end of data” — the index’s consumed watermark points just past the last fully parsed message, so a producer that flushed half a message gets that tail re-scanned on the next append instead of losing it. If I revisited one thing, it would be the TUI’s :consolidated overlay: it runs synchronously on the foreground thread with no caching, which is fine for one-shot inspection but should be made incremental for live, growing files.

Quick Start

# build & install the `fixlog` binary
cargo install --path crates/fixlog-cli

fixlog sniff fixtures/synthetic/minimal_4.4.log        # what layout is this log?
fixlog parse fixtures/synthetic/minimal_4.4.log --first 5
fixlog tui   fixtures/real/fix44-om.log                # interactive viewer (press ? for help)
fixlog grep  live.log --filter "35=8 AND 55=AAPL" -F   # tail -f, filtered
fixlog orders consolidate logs/*.log logs/*.log.gz     # aggregate fills across rotated logs