Skip to content

vitaldb-bench

The timing and data-quality audit of VitalDB: the scripts behind most studies in this atlas, the clock model and corrected-time reader, the per-case labels, and resumable TimeNet builds with their fidelity checks.
Repository github.com/metrology-atlas/vitaldb-bench
Branch main
Visibility public
Licence MIT
Language Python 3.11+
Related TimeNet VitalDB reader, physiosync, lineaccess, vitaldb-resolution

What it is

vitaldb-bench measures how far every time stamp in the VitalDB release can be trusted, and builds databases that carry what was found as labels. It has two halves.

  • The installed package (vitaldb_bench) builds TimeNet databases from the .vital files in short, resumable chunks, reads tracks back out, checks every stored value and time against the source files, and benchmarks the builds. It also holds the clock model and a corrected-time reader: a view of a build in which every SNUADC and BIS sample carries its corrected time and an error distribution.
  • The analysis scripts (tools/) produced the tables and figures of the timing audit. Each reads the .vital files directly, writes per-case output that stays on the analysis machine, and then aggregates it. Only aggregates and figures are published (results/).

It depends on TimeNet VitalDB reader for builds; its scripts also use physiosync (timing corrections) and lineaccess (pressure-line accesses).

No VitalDB data are in the repository: only code, aggregate results and figures. To reproduce a result, download the release from PhysioNet and point the scripts' --root at it.

Its part in the atlas

Most studies in this atlas were run with its scripts, and the Code section of each study names them. Several atlas pages come straight from it:

The atlas is meant to become the single source of these numbers: the plan is for the package and scripts to read the atlas JSON rather than keep their own copies.

Install

pip install -e .            # the package: numpy, timenet, timenet-connectors
pip install -e ".[tools]"   # plus the analysis scripts' dependencies (scipy, pandas, physiosync, lineaccess, ...)
pytest -q

Each script's docstring gives its commands and says what it reads and writes; python tools/<script>.py --help lists its options.

What is in it

The package
build.py Resumable TimeNet builds in time-boxed chunks, with a choice of how samples are placed in time.
stores.py Reads a track of a case back out of a build, in a common shape for comparison.
fidelity.py Checks every stored track of every built case against the shared reader, value by value and time by time. With the reset reordering on, the reference is reordered the same way.
perf.py Size, read-speed and downstream-equivalence benchmarks of two builds of the same cases.
clock_model.py The clock model: per-track fit tables written beside a build, and a corrected view of a TimeNet build. Each fit records the sample order it was made on.
correct/ Timing correction applied when data are read: clock models (clock.py), error distributions (dist.py) and the reader CorrectedStore (layer.py), which refuses a fit made on a different sample order.
case_clock.py Recovers each case's clock time of day; the date stays hidden.
label_families.py The derived label families (delay groups, line accesses, lab draws, BIS smoothing ...): where each family's rows come from and what its labels mean.
timenet_labels.py Writes the label families into a TimeNet build as record annotations.
timing_report.py Per-track summary of the rates a resync-timed build estimated (aggregates only).

Scripts not tied to one study
pair_timing_budget.py Timing budget for combining two or more time sources: the model the atlas planner implements. Reads no data.
time_source_accuracy_figure.py The original accuracy-against-resolution figure for every kind of time stamp.
timestamp_resolution.py Temporal resolution of every kind of time stamp (aggregates only).
device_map_figure.py The device and timing-relationship diagram used as the system overview.
sync_figures.py Redraws the clock re-synchronisation figures of the drift document from cached per-case rows.
gen_tools_page.py Writes the scripts reference page from each script's docstring.

Folders and documents
tools/ All analysis scripts (about 90). Short provenance helpers are in tools/adhoc/.
results/ Aggregate results and figures only; no per-case data.
scripts/ Shell drivers for time-boxed build chunks.
Scripts reference Every script in tools/ with what it does.
Reproduce the results Which script made which file in results/, and the two-step pattern (per-case pass, then aggregate).
Corrected times Reading a build with corrected times and their uncertainty.
Timing budget Planning a multi-signal analysis (the text behind the planner).
Availability The availability rule: no value may be used before it could have been known.
Timing modes How builds place samples in time (nominal, file, resync) and what each costs.
VitalDB timing guide (PDF) The longer write-up of the timing audit.

Studies that use it

Study Code Summary
Anaesthesia record times are rounded up tools/emr_markers.py Anaesthesia start and end entered on a 5-minute mark sit 90–290 s later, against device markers, than entries typed to the minute. Rounding up to the next 5 minutes fits every one of 11 markers better than rounding to the nearest or down.
Arterial line accesses in the pressure waveform tools/line_access_scan.py, tools/lab_draw_labels.py Flushes, zeroing and blood draws leave clear signatures in the arterial pressure. A detector finds them in the waveform alone (about one access per hour) and matches 77% of lab draws to an access within 15 minutes, against 26% for random times.
Arterial pressure as a check on the PPG correction tools/art_check.py Arterial pressure is sampled by the same converter but has no frame and no sawtooth, so every PPG slip leaves ECG→ABP unchanged. Against it, detected slips are 97–100% real and tier A leaves about 2 ms of drift mid-tooth.
Clock corrections in every file tools/syncscan.py, tools/syncanalyze.py A scan of all 6,388 files for the jumps where the recorder re-synchronises a device's stamps. Each device type has its own correction signature; only SNUADC and BIS correct clearly enough to measure their drift.
Closing loops in the timing graph tools/loop_closure_art.py, tools/clock_edges.py, tools/clock_reconcile.py, tools/clock_reconcile_figs.py Where two independent routes reach the same clock, the offsets around the loop must sum to zero. Three measured loops close within their noise; the BIS loop closes up to a constant latency; the proposed EMR loop is not a loop at all.
Drug effects as a clock check tools/drug_effect_latency.py Infusion pumps, the rapid infuser and the cerebral oximeter share no waveform with the reference. Drug effects that follow infusions at plausible delays (propofol to BIS in about 2 minutes, fluid to CVP in about 30 s) bound their clocks to well within a minute of the recorder.
Gaps and unbroken segments in every waveform tools/signal_events.py, tools/signal_events_collect.py, tools/continuity_figs.py, tools/continuity_sync.py Every gap and every unbroken segment in the 6,388 recordings, per waveform. The curves are not smooth: a cliff at about 18 s in every device marks the recorder's pauses, SNUADC segments stop at 5 hours, Primus segments at about 2.0 hours and BIS segments at about 2.3 hours (each device's correction cycle), and BIS's short gaps grow with the time since the last one (clock resynchronisations, not lost data).
How often each device is corrected tools/clock_correction_periods.py The devices re-synchronise in different ways: SNUADC when its drift reaches about half a second, BIS on a fixed 8,192 s timer, and the Primus by small steps about every two hours that do not keep up with its drift (+104 ppm, measured later by breath matching).
How the BIS index trails its EEG tools/bis_index_delay.py, tools/bis_smoothing_labels.py Modelling the BIS index as a lagged mix of EEG features shows a trailing average starting about 3 s after its EEG window ends: about 16 s long in half the cases (the 15 s smoothing setting) and 32–61 s in most others. Half the response weight is older than 30 s.
How the monitors compute their numbers tools/numeric_intervals.py, tools/numeric_derivation_pilot.py Recomputing each monitor numeric from its own waveform finds a short, simple algorithm for every one (a median, an exponential average or a time mean over 3–30 s) and a delay of 2–9 s behind the newest data it uses. No value is stamped before its data exist.
Identifying the recording setup of every case tools/setup_assign.py, tools/frame_an.py Recognising which equipment made each recording, from its serial ports and, where those fall short, from clock fingerprints, so that per-setup values can be used for every case.
Message sizes and the steps between records tools/record_scan.py, tools/record_sizes_agg.py, tools/record_steps_collect.py, tools/record_steps_figs.py Every record in every waveform: SNUADC writes messages of 100 samples (0.2 s), Primus 30 (0.48 s), BIS 16 (0.125 s), CardioQ no fixed size. Backward steps outnumber gaps by about a hundred to one and come from a few recordings with duplicated or interleaved records. Primus gaps are whole lost messages; most SNUADC and BIS gaps are not.
Numeric delays are fixed within a case, different between cases tools/numeric_delay_variability.py, tools/numeric_delay_labels.py With each numeric's algorithm held fixed, delays were fitted for every case and every 30-minute block. They do not drift within a case, but 40–70% of their variance lies between cases, so a single population delay is wrong by up to 2 s for HR and 1 s for the others. Each case gets its own delay label.
One device delay for the PLETH across the release tools/pat_offset_census.py The stored ECG-to-PLETH delay was measured in 5,536 recordings, to the foot and to the steepest upstroke of each pulse. It has one peak (655 ms to the foot, 715 ms to the upstroke, in the median recording), no clusters by recording setup or SNUADC port, and no shift along the case order: one global device delay, 385 ms with an adopted R-to-foot finger PAT of 270 ms, fits the whole release, as Ruffolo et al. found in their own data.
Paced ECG as an independent clock tools/paced_screen.py, tools/paced_breaks.py Fixed-rate pacing gives a clock the recording chain cannot touch. Timing paced beats on three timelines shows that each SNUADC step-mode correction loses about 18 ms of signal, and that nothing else breaks the sample stream.
Physical units, shared drift and time order tools/syncanalyze.py Device drift rates cluster into units: about 8 SNUADC boards and 8–18 BIS monitors. Part of the drift is shared by both devices of a case and changes between cases: the recorder PC's own rate. Case order carries no time information.
Rare devices against SNUADC tools/rare_device_sync.py, tools/vigilance_window.py CardioQ, EV1000, Vigileo and Vigilance each get a link to the SNUADC timeline: beat matching for the CardioQ, mean arterial pressure alignment for the EV1000 and Vigileo, and a long-window fit for Vigilance's averaged heart rate.
Recorder PC clock steps and recording pauses tools/pc_clock_steps.py A step of the recorder PC clock would move every device's stamps at once. Searching all 6,388 files for such simultaneous jumps found none of 0.5 s or more; what moves everything together is recording pauses.
Records out of order at SNUADC resets, and where the lost time goes tools/clock_model_fit.py, …/adhoc/reset_gapscan.py, …/adhoc/reset_glitch.py At every SNUADC reset the five records after it are written out of order: two continue the old timing and three start the new. Put back in order, the waveform runs on with no samples missing; the 18–20 ms lost at each reset goes missing inside the last old-timing record, and an extra zero sample follows.
Recovering the time of day vitaldb_bench/case_clock.py The release shifts every case to start at 2100-01-01 00:00 UTC, removing the date and the time of day. The time of day (Korean local time) can be recovered to the minute from the admission date and the seconds of the EMR times.
SNUADC against the PC-stamped heart rate tools/numeric_clock_check.py The Solar8000 heart rate is stamped by the recorder on arrival, so the delay from the ECG to it, on the ECG's raw sample count, ramps at the SNUADC clock's rate: −65.3 ppm, against −66.7 ppm from the stamp model. Two independent routes agree.
Shared bursts between the ECG and the EEG tools/shared_artifacts.py Electrocautery and muscle bursts appear in both the SNUADC ECG and the BIS EEG. Cross-correlating their envelopes gives the offset between the two devices' stamps every 10 minutes: about −0.8 s, in a sawtooth that follows both clocks' corrections.
SpO₂ and pulse rate latency from step events tools/spo2_event_pilot.py SpO₂ cannot be recomputed from one plethysmogram, so its timing comes from events that change the PLETH abruptly. SpO₂ lags the recorded PLETH by about 4 s and then smooths over roughly 10–20 s.
The PPG sawtooth and its correction tools/tier_bench.py, tools/ppg_fix_check.py The ECG→PPG delay creeps at 18–52 ppm and jumps up by one 33.3 ms frame at each slip, every 11–31 minutes. A tooth-by-tooth model removes it; where the correction is confident it is essentially exact on injected test data.
The PPG's 30 Hz frame clock tools/frame_an.py ECG and PPG fiducials resolve to one 2 ms sample, but PPG fiducials taken on raw samples lock to a hidden 33.3 ms frame grid. Scanning for it finds the oximeter's frame clock to about 1 ppm, a fixed fingerprint of each recording setup.
The Primus stamp step at each airway-pressure pause tools/primus_pause_steps.py, tools/primus_co2_gaps.py, tools/primus_pause_aware.py, tools/clock_route_figs.py Measured directly at 2,491 pauses in 1,021 recordings: where CO₂ carries on through the AWP pause (74%), the Primus stamps barely step (median −1 ms, p10–p90 −164 to +187 ms, close to the no-pause controls); where CO₂ stops too (26%), they step by up to seconds. No step undoes the drift built up since the last pause.
The Primus ventilator against SNUADC, by breaths tools/vent_sync_pilot.py, tools/primus_drift_probe.py, examples/ppv_by_breath.py Each breath shows in the Primus airway pressure and, through the chest, in SNUADC's central venous pressure. Tracking that lag puts the Primus on the SNUADC timeline to about 55 ms over the whole release, with a line per stretch between its AWP pauses (49 ms in the pilot). At scale it measures the Primus drifting a median +104 ppm against the recorder PC between its AWP pauses, with steps at the pauses that do not reset it.
The SNUADC clock through the Solar8000 numerics, whole release tools/solar_snuadc_clock.py, tools/clock_route_figs.py On the SNUADC's own sample count, the delay from the waveform to the PC-stamped Solar8000 heart rate and mean arterial pressure ramps at the SNUADC's drift against the PC, measured without its stamps or resets. In recordings with whole reset intervals it agrees with them (−63 ppm; median difference +1.5 ppm, 581 recordings). It is too noisy per recording, and in recordings under about 4 h its rate is biased toward zero, so it cannot yet speak for recordings without resets.
The cardiac artefact in the BIS EEG tools/eeg_sync_pilot.py, tools/eeg_cardiac_recheck.py The BIS EEG carries a weak electrical trace of every heartbeat. Too weak for beat-by-beat matching, it shows clearly when averaged on the ECG's R peaks, and gives a second SNUADC–BIS timing reference that agrees with the bursts to about 16 ms.
The heartbeat in the Primus airway signals tools/airway_heartbeat_pilot.py A pilot in 30 recordings for a heartbeat-locked route from the Primus to the SNUADC. Airway pressure shows the heartbeat in 9% of windows only. CO₂ shows a heartbeat-locked trace in 60% of windows, tight to 20 ms, but its lag does not follow the Primus drift, so it cannot time the Primus. Closed as a clock route.
The stamp sawtooth model for SNUADC and BIS tools/sa_corrected.py, tools/clock_model_fit.py Fitting each track's record stamps against its sample count as a drift with corrections (a sawtooth) puts SNUADC and BIS on one timeline without any artefact. Checked against shared bursts, the EEG–ECG timing error falls from about 790 ms on the recorded stamps to 33 ms with one constant per SNUADC mode in the pilot, and 42 ms over the whole release.
Two heart-rate delay groups, one poll apart tools/numeric_delay_labels.py, …/adhoc/hr_group_ports/ Solar8000 heart rate reaches the recorder either about 1.9 s or about 4.2 s after its window ends: one 2 s poll apart. The group belongs to the recording PC, stays fixed through a case, and on some PCs varies from case to case.
Value resolution of every signal tools/signal_events.py, tools/signal_events_agg.py, tools/record_scan.py, tools/numeric_agg.py How finely every waveform and numeric track in the release records its values. The SNUADC waveforms are 10-bit codes, though VitalRecorder's documentation describes the SNUADC as 12-bit; Primus waveforms use about 10 bits of 12; BIS EEG is a 16-bit grid of 0.05 µV. Numerics are 32-bit floats on decimal grids of 1, 0.1, 0.01 or 0.001, one per track; settings sit on coarser grids of 10 or 100.
What each break in the stamps is tools/step_classify.py, tools/step_classify_agg.py, tools/step_join_null.py Every step between waveform records in the release, sorted by mechanism: written twice, a SNUADC clock reset, interleaved streams, a pause shared by every device, whole messages, less than a message, or a fraction of a message beyond whole ones. A continuity test with controls shows which classes lose data: shared pauses and most longer SNUADC gaps do; steps under one message, and SNUADC's lone one-message steps, mostly do not.
What the lab timestamp marks tools/lab_draw_labels.py VitalDB's lab time is a per-specimen timestamp: not the draw, and not the result. Blood gases are stamped 3–12 minutes after the draw, central-lab panels about 8–24 minutes after it, and the result follows a turnaround the release does not record.
When each device's clock is corrected, in every recording tools/correction_schedules.py, tools/step_classify.py, tools/step_classify_agg.py From the steps between records in all 6,388 recordings: the BIS is corrected every 8,192 s and the Primus every 7,230 s, on their own timers (the first correction comes at any time after the device starts); the SNUADC is reset when 0.49 s of drift has built up. With the full rerun of the correction fit, it gives each device's drift against the recorder PC in 4,728 BIS and 2,728 SNUADC recordings, and shows that the Primus's 20–23 s AWP pauses belong to its correction cycle. Every rate here is relative to the recorder PC, whose own clock drifts and is corrected.

Observations it handles

Observation Status Summary
Eight recordings store whole messages twice established In 8 recordings 15–48% of the waveform records are byte-for-byte copies of the record before them, with the same timestamp: case 15, case 364, case 1946, case 3865, case 4857, case 5414, case 5595 and case 5619. SNUADC, Primus and BIS are affected equally, so the recorder, not a device, wrote them twice.
Every case starts at 2100-01-01 00:00 UTC established The release shifts each case to the same start, removing the date and the time of day. Within a case all times keep their spacing.
Five records out of order at every SNUADC reset established At each step-mode reset the five records after it are written in the wrong order: two continue the old timing and three start the new, interleaved. Read in file order, or sorted by stamp, one second of signal is scrambled.
Primus waveform gains are wrong in every recording, for CO₂ or for AWP established Recordings store the Primus waveforms with one of three sets of scale factors. Checked against the Primus's own ETCO₂ and PIP numerics in all 6,388 recordings, CO₂ fits only 0.075 mmHg per code and AWP only 0.1 mbar per code with offset −20, so most recordings overstate CO₂ by a third, and the others overstate AWP by about 3% (950 recordings) or understate it about fourfold (26).
The Primus AWP stops for about 21 s every 7,230 s established In 5,502 recordings the Primus airway-pressure waveform has gaps of 19–23.5 s (whole messages of 0.48 s, mostly 21.12 s) that recur every 7,230 s, the Primus's correction interval; a clock correction follows about 21.6 s after. The CO₂ waveform carries on through 90% of them. The 21 s of AWP are lost, not merely re-stamped.
In 34 files the SNUADC records overlap throughout open In 34 files the SNUADC records step back over their neighbours far more often than resets explain: in 20 of them 4–95% of all record junctions overlap, by up to about 100 ms. The reset-order rule matches there by coincidence.
In some recordings two unrelated streams are interleaved in one track open In 7 recordings 64–95% of the records alternate between two streams: each record continues the one two before it, not the one before, and the two streams do not match each other. SNUADC and BIS in case 2219, case 2379, case 2640, case 4836, case 5327 and case 5617; SNUADC in case 1939. In case 94, case 945, case 1132, case 3195 and case 5700 4–7% of SNUADC records interleave.

Worked examples

Example Side Status Summary
A regression that carries rounding uncertainty through value draft Acute kidney injury after surgery against intraoperative urine output (mL/kg/h), fitted once on the recorded numbers and once per draw from the true-value distributions: the rounding moves 14% of cases across the oliguria threshold, but widens the confidence interval of the cohort's regression by at most 4%.
Cuff pressure against the arterial line both draft 12,238 cuff readings against the arterial line at the cuff's own moment, in 2,283 patients: no bias over the cohort at any pressure, but each patient's cuff reads a constant 10 mmHg (SD) above or below their line, and half the arterial readings below 65 mmHg are not below 65 by cuff.
End-tidal to arterial CO₂, and pulse oximetry against blood gas, at the right moment time draft For 4,281 blood gases, reading the monitor when the blood was drawn rather than at the lab stamp makes changes in end-tidal CO₂ track changes in arterial CO₂ far better (within-patient correlation 0.74 against 0.54), and agreement peaks exactly at the estimated draw. Oximetry shows the same timing (agreement peaks at the draw), but the 100% ceiling hides the size of the oximetry gradient for three gases in four.
Intraoperative fluid balance as a distribution value draft Fluids in minus fluids out, each a heaped charted volume, as a distribution per case: the recording alone makes a balance uncertain by about ±60 mL, and by ±190 mL when the blood loss was written as a round 500 mL.
Minutes of hypotension per patient, from the monitor and from the waveform both draft Minutes with MAP below 65 mmHg in 3,476 recordings, from the monitor's numeric and from the arterial waveform's beats: the two routes to one pressure differ by several mmHg per patient, which moves half of all patients' minutes by 5 or more and a fifth of their labels, and the numeric's averaging hides short dips in every patient.
Oliguria labels for a prediction model value draft One label per patient, oliguric or not, from three rounded numbers: 10% of labels are uncertain and 2.6% are expected to be wrong, which sets a floor under the error any model can be measured to have. A typical model is barely affected; a good one is.
Post-induction hypotension, from the charted start and from the devices time draft The charted anaesthesia start comes a median 14 minutes before the patient is induced; measured from it, "hypotension within 20 minutes of induction" is found in 27% of 2,913 patients, against 51% from the devices' induction, and the label changes for a quarter of patients.
Pulse arrival time against blood pressure, before and after synchronisation time draft In 3,270 recordings, PAT tracks systolic pressure only once each pulse is paired with its own beat: the hourly correlation moves from a mean of −0.04 to −0.21 (88% of recordings negative), reproducing Ruffolo et al. 2025 in a second dataset; paired naively, PAT is a heart-rate signal.
Pulse arrival time in two correction steps time draft The time from the ECG R wave to the pulse at the finger, as stored, after the PLETH's sawtooth is removed, and after its device offset is removed, smoothed by maximum likelihood, with the arterial line as the reference.
Pulse pressure variation gated by ventilator breaths time draft Pulse pressure variation per ventilator breath, in 3,448 recordings: the ventilator's clock sits a median 1.8 s off the arterial line's, which moves half of all breaths' values by a point or more and scatters where in the breath the pulse pressure peaks; a recording's median PPV hardly changes.
Urine output in mL/kg/h as a distribution value draft Intraoperative urine output per kilogram per hour, from three rounded inputs, as a probability distribution per case: one case in ten cannot be placed confidently on either side of 0.5 mL/kg/h.

Mentioned in

References

  • Harris CR, Millman KJ, van der Walt SJ, et al. Array programming with NumPy. Nature. 2020;585(7825):357-362. ↗
  • Virtanen P, Gommers R, Oliphant TE, et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods. 2020;17(3):261-272. ↗
  • McKinney W. Data structures for statistical computing in Python. In: Proceedings of the 9th Python in Science Conference. 2010. p. 56-61. ↗
  • The pandas development team. pandas-dev/pandas: Pandas. Zenodo. ↗
  • Hunter JD. Matplotlib: a 2D graphics environment. Comput Sci Eng. 2007;9(3):90-95. ↗
  • Apache Software Foundation. Apache Arrow (PyArrow). ↗
  • Raasveldt M, Mühleisen H. DuckDB: an embeddable analytical database. In: Proceedings of the 2019 International Conference on Management of Data (SIGMOD '19). ACM; 2019. p. 1981-1984. ↗
  • Makowski D, Pham T, Lau ZJ, Brammer JC, Lespinasse F, Pham H, Schölzel C, Chen SHA. NeuroKit2: a Python toolbox for neurophysiological signal processing. Behav Res Methods. 2021;53(4):1689-1696. ↗
  • Bota P, Silva R, Carreiras C, Fred A, Plácido da Silva H. BioSPPy: a Python toolbox for physiological signal processing. SoftwareX. 2024;26:101712. ↗
  • Kozea / CourtBouillon. CairoSVG. ↗
  • Krekel H, Oliveira B, Pfannschmidt R, Bruynooghe F, Laugher B, Bruhin F. pytest. 2004. ↗