What each break in the stamps is¶

Every step between waveform records in the release, sorted by mechanism: written twice, a SNUADC clock reset, interleaved streams, a pause shared by every device, whole messages, less than a message, or a fraction of a message beyond whole ones. A continuity test with controls shows which classes lose data: shared pauses and most longer SNUADC gaps do; steps under one message, and SNUADC's lone one-message steps, mostly do not.

Background¶

Message sizes and the steps between records found far more backward steps than gaps, and gaps that are mostly not whole messages. A step in the stamps can mean data lost, data written twice or out of order, or a clock correction that moved the stamps and lost nothing. Each leaves a different trace in the stamps and in the samples either side.

Data¶

All 6,388 recordings; every non-zero step between consecutive records of every waveform track (backward steps capped at 3,000 per track, so backward counts are lower bounds; the number of recordings is exact). Controls: 40 random recordings, up to 200 ordinary junctions per track.

Method¶

• For every step: its size in messages and its remainder; whether record k+1 is identical to record k; whether it continues record k−1 better than record k; whether another device has a gap of over 1 s within 1 s; and join_z, the distance of the first sample after the step from a linear extrapolation of the last samples before it, in units of the record's own median one-step prediction error.
• Classes, first match wins. Backward: duplicate (identical record); reset (SNUADC consecutive backward steps that together step back 0.45–0.52 s: one step, or the five records written out of order at a reset, Five records out of order at every SNUADC reset); interleave (continues record k−1, in a track with at least 50 such steps); other backward. Forward: shared (over 1 s, with another device stepping at the same moment); whole (a whole number of messages, within half a sample); sub-message; fractional.
• Controls for join_z. At ordinary junctions (no step) and with one message cut out at the same junctions, in 40 random recordings. If a class loses no data, its share of join_z < 3 should match the first control; if it loses a message, the second.

Results¶

Counts in SNUADC ECG II (recordings in brackets); the other SNUADC channels are alike.

• Backward steps are three things. Duplicates and interleaving are confined to a few dozen recordings (Eight recordings store whole messages twice, In some recordings two unrelated streams are interleaved in one track). Everywhere else SNUADC's backward steps are its step-mode clock resets: 2,546 recordings, either one −0.49 s step or five records stepping 0.12 / 0.08 s that add up to it. About 500 recordings keep a few unexplained backward steps.
• The continuity test separates the classes. At ordinary junctions join_z < 3 in 92–95% of SNUADC steps; with one message cut out, in 12–24%.
• Shared pauses and SNUADC fractional gaps lose data: shares near the cut-out control (SNUADC fractional 35–42%; about 70% of them lose samples).
• SNUADC sub-message steps (2–20 ms, the frequent-mode corrections) and its lone one-message steps run on as if nothing were missing (88–95%). The one-message steps tend to fall where the signal moves to the mid-scale no-signal level.
• BIS sub-message steps run on (93%, like the control): the clock resynchronisations of Gaps and unbroken segments in every waveform lose nothing. BIS fractional gaps (mostly 0.13–0.5 s) sit between the controls (73%): roughly a third lose data.
• Primus AWP gaps of whole messages (80% of them 20–23 s, in steps of one message) run on almost like the control (86% against 88%), yet the test is weak for AWP, whose 0.48 s records are often nearly flat. A breath-level test against CO₂ shows these gaps do lose about 21 s of AWP (The Primus AWP stops for about 21 s every 7,230 s): for AWP, join_z cannot tell.
• Seconds missing over the release in ECG II: about 196,700 s in fractional gaps, 70,200 s in shared pauses, 5,100 s in whole-message steps.

Discussion¶

The stamps alone say what kind of break each step is, and the samples say, on average, whether data were lost. join_z judges a single step only weakly: losing a few samples, or a message on a flat stretch, does not show. Per class, against the controls, it is clear. For a reader of the data: backward steps need reordering (resets, interleaving) or deduplication, not gaps; sub-message steps are clock corrections; shared pauses and most SNUADC gaps beyond one message are real gaps.

Observations it supports¶

Limitations¶

• The controls come from 40 recordings; join_z's power depends on how steep the signal is near the step.
• Losses of a few samples (the SNUADC sub-message steps) are below what join_z can see; Small positive SNUADC gaps are real data loss found from beat intervals that sub-second SNUADC gaps lose data, which this test can neither confirm nor rule out at 2–20 ms.
• Backward steps were capped at 3,000 per track in the scan.

Open questions¶

• The remaining backward steps (next step). About 500 recordings per SNUADC channel and 3,850 for Primus AWP have backward steps none of the classes explains, mostly of a few samples.
• Fit the missing time (next step). For gaps that lose data, fit how many samples are missing (Does the waveform run on?) rather than testing only whether any are.

Code¶

• vitaldb-bench: tools/step_classify.py, tools/step_classify_agg.py, tools/step_join_null.py