Technical note · Software data acquisition

EventStream

Event building over TCP, with every fragment accounted for.

Saksham SharmaOctober 2026C++20 · Linux sockets · CMake

A detector is read out in pieces. Each readout sector sends its share of a collision as a separate fragment, and software downstream has to rebuild the event: on time, exactly once, and without running out of memory when the input outruns it.

End view of a toy barrel detector, drawn in the page. Each of the four readout sectors is one EventStream source; its hits from one collision become one fragment, and the builder needs all four.

Abstract

EventStream simulates the software side of a detector readout. Several sources emit fragments of numbered events over real TCP connections. Readers frame and verify them, a bounded queue carries them to an event builder, and the builder assembles each event from one fragment per source before filtering and recording it. Missing, duplicated, late and reordered fragments are handled explicitly, and every run ends by checking six conservation equations over its counters. The physics is out of scope: the subject is the acquisition software, and how it behaves when input outruns output.

384 / 384: benchmark runs in which all six accounting equations balanced
0 lost: events across the sweep with credit-based admission; 72 runs lost events without it
16.1–18.3 MiB: peak resident memory in every run, overload included
49.3k events/s: accepted at 50k/s offered per source, 4 sources, with p99 latency of 7.3 ms

Credit-based admission

On Off

Events complete: 0
Evicted incomplete: 0
Late fragments dropped: 0
Reader waits: 0

Figure 1. The event builder under overload, simulated in the page. Four sources send as fast as they can; the builder takes three fragments per step. Readers compete unevenly for queue space, so source S0 tends to run ahead. The builder grid shows its ID window of ten events starting at the oldest unfinished one. With admission off, S0 pushes past the window and the builder evicts events still waiting on slower sources, then drops their late fragments. With admission on, a reader holding a fragment beyond the window waits, and every event completes. An illustration of the mechanism in §8, not measured data.

§1

The problem

A detector reads out in pieces. Each front-end source sees part of every collision and sends its piece, a fragment, tagged with the event number. Downstream, software has to put the pieces back together: wait for every source, notice when one never arrives, ignore a piece that arrives twice, and keep memory bounded when the input outruns the output.

EventStream builds that machinery at a small scale so its behaviour can be measured. Source count, event rate, payload size and every fault are set on the command line. Payloads are generated from a seed with SplitMix64, so a run can be replayed exactly and its output compared, event by event, with an independent reference builder written in Python.

Faults are injected deterministically: fragments omitted by a seeded hash (--missing-every), sent twice (--duplicate-every), delayed (--delay-every, --delay-ms), or sent with adjacent event IDs swapped (--reorder).

§2

Architecture

One Linux process hosts the whole pipeline on the loopback interface, which keeps every timestamp on a single monotonic clock. Each source is a thread with its own TCP connection, and each connection has one reader thread. Readers share one bounded queue, and a single builder thread owns correlation, filtering and output.

Backpressure runs from right to left. A full queue blocks the readers, blocked readers stop draining their sockets, and full socket buffers block the sources. No queue entry is ever overwritten. The dashed path is the one control loop the benchmarks made necessary: the builder tells readers how far ahead they may run (§8).

Figure 2. Data path from sources to files. Each source has its own connection and reader, all readers feed one bounded queue, and one builder thread owns everything after it. The dashed accent path is the admission limit the builder publishes to readers, added after the benchmarks in §8.

§3

Wire format

Fragments travel as explicit big-endian fields, never as a raw C++ struct, so padding, alignment and byte order cannot leak onto the wire. The header is 42 bytes. The payload length is validated before anything is allocated (at most 1 MiB and the configured fragment size), and a CRC32 with the IEEE polynomial covers header and payload, matching Python's zlib.crc32.

Readers loop over partial reads. A frame that arrives one byte at a time still decodes; end-of-stream between frames is a clean close; end-of-stream inside a frame, an unknown version, an oversized length or a checksum mismatch is an error that stops the run and is reported.

Figure 3. Frame layout at 15 px per byte (payload drawn shortened). Shaded fields are checked before or after the payload is read: the version and length before any allocation, the checksum after. The sequence field is diagnostic; the builder deduplicates on run, event and source.

§4

Event building

The builder keeps a map from event ID to the fragments received so far, one slot per expected source. Every fragment that reaches it ends in exactly one of the outcomes below, and each outcome has a counter. Figure 1 animates the window and admission rules.

Complete: A fragment from every source is present. The event goes to the filter, which computes CRC32 over the payloads in source order and accepts it when the checksum is divisible by --filter-modulo.
Duplicate: The same event and source arrive again, before or after the event retires. It is counted and discarded; an event can never be emitted twice.
Timeout: An incomplete event expires --timeout-ms after its first fragment reached the builder, and is written to expired.jsonl with the list of missing sources.
Window: The builder retains a numeric window of --max-inflight IDs behind the newest it has seen. Older pending events retire with reason window, and later fragments for them are late drops. This trades unlimited reordering for bounded memory.
Admission: The builder publishes a limit: the oldest pending ID plus the window. A reader holding a fragment at or beyond it waits, so one fast source cannot push slower sources' events out of the window.

Memory is bounded by construction: pending payloads by window × sources × payload size, the queue by its capacity, and retired identities by the same window. Latency is held in 976 fixed histogram counters, never as a per-event list.

§5

Accounting

A fast result can hide dropped work. To rule that out, the executable checks six conservation equations over its own counters at the end of every run, writes the result to metrics.json, and exits with an error if any equation fails. Every fragment and every event ID must be in exactly one bucket.

F_gen=F_omit+F_sent

(1)

F_wire=F_recv

(2)

F_recv=F_uniq+F_dup+F_drop

(3)

F_uniq=S · E_comp+F_exp

(4)

E_gen=E_comp+E_exp+E_unobs

(5)

E_comp=E_acc+E_filt

(6)

Ffragments: generated, omitted by fault injection, sent once, sent on the wire (including duplicates), received, unique, duplicate, dropped as late or invalid, and held by expired events. Eevent IDs: generated, complete, expired while incomplete, never observed, accepted by the filter, and filtered out. Snumber of sources.

All six held in each of the 384 benchmark runs, and in each of the 80 runs of the read-buffer experiment.

§6

What latency measures

Latency starts at the earliest source timestamp in an accepted event, taken before the payload is generated, and ends after the event's JSONL record has been written and the stream flushed to the operating system. Both ends read the same CLOCK_MONOTONIC on the same host, so no clock synchronisation is involved. Durable storage (fsync) is outside the interval and is not claimed.

Samples go into a log-linear histogram: exact below 16 ns, then 16 buckets per power of two. A reported percentile is the upper edge of its bucket, so it overstates the true value by less than 6.25%; the maximum is exact. Only accepted events contribute samples.

Figure 4. Where one latency sample starts and stops. The interval includes time spent waiting in socket buffers and the queue, so under overload it grows with the backlog even when no event is lost.

§7

Results

The sweep varies source count, payload size, offered rate, queue capacity, filter cost and flush batching: 128 combinations, each run three times over 5000 event IDs. Offered load is the configured event rate per source; accepted load is what the filter persisted. Points are medians and bars span the three runs, because repeats on this laptop can differ by a factor of two.

Data table

Figure 5.

Observations

Every run balanced its accounting, and with admission in place no run lost an event: completeness was 100% across all 384.
With a 256-entry queue and no filter cost, paced load is met up to 50k events/s per source with p99 under 10 ms, except four sources with 1 KiB payloads, which reached about 11k events/s.
Unpaced, two 64 B sources reached about 99k events/s while four fell to 11–15k. Four sources, four readers and the builder oversubscribe four logical CPUs; that explanation has not been profiled.
A 100 µs filter limits the single builder thread to roughly 2–6.5k events/s; the backlog then waits in socket buffers and p99 approaches a second.
Peak resident memory stayed between 16.1 and 18.3 MiB in every configuration.

§8

What the benchmarks found

Two defects passed every functional test and only showed up under sustained load. Both are fixed, and each has a regression test that fails on the old code.

Expiry scanned every pending event

Each incoming fragment triggered a scan of the whole pending map to find timed-out events. Under backlog the map fills to the window, so the cost per fragment grew with the backlog, the builder fell further behind, and throughput collapsed. Short runs finished before the backlog formed, which hid it. Indexing pending events by arrival time makes the check O(log n).

2 sources, 50k/s offered, 20 000 IDs	Before	After
Accepted events/s	6.1k – 9.9k	32.8k – 45.8k
p99 latency	834 – 1124 ms	29 – 48 ms

Backpressure was not coordinated across sources

Each connection was backpressured on its own, and readers did not get equal turns at the queue. Under overload one source ran a full window ahead of another, and the builder evicted events still waiting on the slow source. 72 of 384 runs lost events this way with no faults injected; the worst completed 20.5% of its events. Every lost event was counted, so accounting still balanced. Credit-based admission (Figure 1) removed all of the loss, and unpaced throughput rose as well: four 64 B sources with an 8-entry queue went from 5.4k to 15.2k events/s.

Queue capacity 8

Queue capacity 256

Figure 6. Share of event IDs completed, before and after credit-based admission. Four sources, 64 B payloads, no filter cost, flush every record, no injected faults. Median of three runs; bars span the runs.

§9

Optimisations, measured

Storage batching

--flush-every 64 flushes after 64 records or once the oldest unflushed record has waited 5 ms. At 1k events/s per source it raised median p99 latency from 0.9 to 6.6 ms and cut flushes about sixfold. It helped throughput only when unpaced: two 64 B sources with a 256-entry queue went from 99k to 157k events/s. Select Compare: Flush in Figure 5 to see every configuration.

Batched reads

--read-buffer 65536 reads many frames per system call instead of three calls per frame, at the cost of copying each payload out of the buffer. It removed the system calls and lowered throughput, so it stays off by default. 20 000 IDs, 256-entry queue, no filter cost, medians of five runs:

Sources	Payload	Offered	recv per frame	Accepted events/s	CPU µs per frame
2	64 B	50k/s	3.00 → 0.32	49.8k → 49.9k	20.2 → 21.0
2	64 B	unpaced	3.00 → 0.004	54.7k → 31.4k	16.4 → 21.0
2	1 KiB	50k/s	3.00 → 0.02	27.4k → 11.1k	38.7 → 68.6
2	1 KiB	unpaced	3.00 → 0.02	26.7k → 20.0k	35.1 → 42.4
4	64 B	50k/s	3.00 → 0.16	49.5k → 39.7k	14.2 → 14.0
4	64 B	unpaced	3.00 → 0.002	38.9k → 10.5k	13.0 → 32.8
4	1 KiB	50k/s	3.00 → 0.02	17.2k → 13.7k	32.8 → 35.1
4	1 KiB	unpaced	3.01 → 0.02	17.1k → 8.6k	31.8 → 51.4

System calls were not the bottleneck; the single builder thread is. The extra copy and the bursts a full buffer pushes into the queue are both candidates for the slowdown, and neither has been profiled. An SPSC ring buffer, CPU pinning and alternative encodings were not attempted, and no zero-copy claim is made.

§10

Methods and limitations

Host: Intel Core i3-1115G4 (2 cores, 4 threads), 7.6 GiB RAM, WSL2 Ubuntu on kernel 6.18, GCC 15.2 release build, output on ext4.
Sweep: 2 or 4 sources × 64 B or 1 KiB × 1k, 10k, 50k events/s or unpaced × queue 8 or 256 × filter 0 or 100 µs × flush every 1 or 64 records; three runs of 5000 event IDs each, at commit 014460e.
Rates: Measured over the pipeline's wall interval from thread start to full drain. Offered and accepted rates are reported separately, and unpaced means no pacing, never zero load.
Resources: CPU time and peak RSS come from getrusage and cover the whole process. CPU can exceed 100% on several cores.
Validation: C++ unit tests for framing, partial reads, corrupt input, the queue, the builder and the histogram; 15 end-to-end tests including a comparison with an independent Python builder on shuffled, duplicated input; address and undefined-behaviour sanitizer builds.
Limits: One small, shared laptop, so results are host-specific and noisy. Same-host timestamps only; cross-host latency would need clock synchronisation and an uncertainty estimate. Not a model of detector physics or of a production DAQ.