World Sleep Industry ENCYCLOPEDIA

Sleep Data Quality Assessment Services

4.4.3

Definition, intended use, and service boundaries

Sleep data quality assessment services independently examine PSG, HSAT, EEG, EOG, EMG, ECG, oximetry, PPG, actigraphy, radar, mattress sensors, audio, video, wearables, questionnaires, clinical records, labels, and metadata against a predefined use. They fall under WSIC 4.4.3, Sleep Data Services. The question is not whether data are abstractly “good,” but whether their completeness, accuracy, consistency, timeliness, traceability, representativeness, and usability are sufficient for a particular study, algorithm-training project, independent validation, product-monitoring program, or clinical analysis. Permissible missingness, error, label uncertainty, and population bias vary by use, so every quality conclusion needs a purpose, version, and scope.

Assessment is distinct from data cleaning, annotation, and algorithm evaluation. It may detect defects, quantify impact, and recommend repair or downgraded use, but should not modify source data without authority; repaired data require reassessment. Nor can a single composite score prove that a dataset is certified, an algorithm will be effective, or a clinical inference is reliable. The agreement and protocol should freeze the assessment object, source snapshot, target task, critical variables, quality dimensions, thresholds, sampling, and retest rules before results are seen. If the intended use is not known, the report can describe risks but cannot establish fitness for use.

File, structural, and integrity checks

File-level review reconciles inventories, counts, sizes, checksums, readability, archives, names, duplicates, truncation, corruption, and format versions. Structural review tests schema, types, encoding, units, permitted values, primary and foreign keys, timezone, character sets, null representations, ordering, and cross-table references. EDF, EDF+, DICOM, vendor containers, CSV, JSON, and database exports should be inspected with format-aware parsers rather than by extension alone; private fields, annotations, headers, sidecar labels, and reports belong in scope. Transformation into a common data model also requires validation of local-concept mapping, precision changes, unmapped values, and information lost from source fields.

Completeness does not mean that every cell contains a value. The assessment should distinguish not collected, not applicable, device failure, non-wear, parser failure, manual deletion, and a true negative, and show where missingness occurs by population, center, device, period, and processing stage. Duplicate checks should cover exact copies, repeat exports from one participant, cropped or resampled variants, and near-duplicates across sources. Lineage should trace an analytical record back to the source and through acquisition, correction, de-identification, mapping, and merging. Data that cannot be traced may be unsuitable for high-risk use even when values appear plausible. The assessor’s scripts, settings, thresholds, and execution logs also need versioning.

Signal quality, time, and device state

Signal assessment should inspect sampling rate, physical and digital units, range, reference and polarity, filters, resolution, valid duration, flatline, clipping, saturation, disconnects, packet loss, baseline drift, mains or motion artifact, and nonphysiological changes by channel and time window. PSG review also confirms that EEG, EOG, chin and limb EMG, airflow, respiratory effort, SpO₂, position, and ECG channels match the protocol. Wearables and PPG require attention to wear state, low perfusion, skin contact, motion, and firmware. Radar, mattress, audio, and video data may require occupancy, occlusion, automatic gain, compression, and environmental-interference checks. Thresholds depend on device, task, population, and setting; one generic signal-quality index is not ground truth for every modality.

Temporal quality includes start and end, timezone, daylight-saving time, midnight crossing, clock resets, drift, interpolation, and synchronization among devices. Misaligned waveforms, events, questionnaires, medication records, and device logs can remain readable while producing a false causal order or event rate. The assessment should distinguish total recording duration, technically valid duration, wear time, time in bed, and total sleep time and verify the denominator used by derived metrics. Substituting recording time for sleep time can systematically change an event index. After automated rules locate anomalies, source segments should be replayed or checked against device logs and manual records so genuinely rare physiology is not discarded as artifact.

Labels, reference standards, and derived results

Label review begins with the task definition, scoring system and version, epoch duration, event boundaries, temporal tolerance, ignored regions, scorer qualifications, blinding, review, and adjudication. Sleep stages, arousals, respiratory events, limb movements, and other events should be assessed separately so an overall agreement rate does not hide rare or difficult classes. Human scoring contains structured disagreement, and one scorer is not infallible ground truth. Reports may include per-class confusion, inter-rater agreement, event-count difference, onset and offset error, indeterminate rate, and adjudication rate, together with how the reference was formed. Gold cases repeatedly seen by raters may also overstate real-world performance.

Derived outputs should be recomputed independently from underlying records and compared with existing reports, including total sleep time, sleep efficiency, sleep latency, wake after sleep onset, stage proportions, AHI, or other task measures. Units, rounding, denominator, missing-channel handling, overlapping events, boundary rules, and software versions all require examination. Signals and labels need a shared clock, especially after cropping, resampling, or date shifting. Automated pre-labels should retain model version and the trail of human edits and must not be described as an independent human consensus. When systematic label bias is found, the assessor should locate affected scorers, centers, periods, and rule versions rather than merely relabeling a small random sample.

Cohort representativeness, bias, and leakage

Cohort-level assessment reconstructs inclusion, exclusion, refusal, failure, withdrawal, and loss-to-follow-up paths. It describes participants, recorded nights, centers, regions, age, sex, disease spectrum and severity, comorbidity, devices, acquisition environments, and signal quality. Record count cannot replace independent participant count, because frequent contributors with multiple nights may dominate a distribution. Missingness is not always random: severe disease may produce more artifact, home-device failure may correlate with age or digital literacy, and manual quality review may preferentially retain easily interpreted records. Coverage, missingness, and failure should be reported by important subgroup and over time rather than hidden in an overall mean.

For machine learning, the assessment should determine whether multiple nights from one participant, overlapping windows, repeat exports, household members, centers, or device batches cross training, development, and test partitions. Randomly splitting epochs from whole-night records creates participant leakage; pretraining corpora, public mirrors, and derived versions may also cause content contamination. Representativeness is task-specific: adult single-center laboratory PSG does not automatically represent children, older adults, home HSAT, or consumer wearables. An assessment can expose these limitations and potential biases, but statistical weighting cannot manufacture populations, devices, or workflows that were never observed.

Metrics, issue severity, and report delivery

Each metric should state its numerator, denominator, statistical unit, threshold basis, and uncertainty. Examples include readable-file rate, record and channel validity, missingness and duplication, clock drift, anomalous duration, label agreement, event-timing error, linkage success and error, lineage coverage, subgroup coverage, and discrepancy in results recomputed from source data. Metrics should be separated by participant, night, channel, epoch, event, or field rather than mixing denominators. If an overall grade is necessary, weights and blocking conditions should be public and drill down to records and rules; numerous low-risk passes must not cancel a critical defect.

The issue register should contain a unique identifier, evidence, affected scope, severity, detectability, suspected cause, effect on intended use, proposed disposition, and owner. Disposition may be source correction, re-export, relabeling, quarantine, restricted use, treatment as missing, or documented acceptance of residual risk. Unjustified imputation should not conceal acquisition failure. Deliverables generally include an executive summary, machine-readable issue table, sample indexes, reproducible rules or queries, quality profile, lineage map, known limitations, and post-remediation retest. Assessors should distinguish observed facts, inferred causes, and professional judgment and record what could not be examined.

Procurement acceptance, privacy, and continuous monitoring

The purchaser should provide the intended use, critical variables, data dictionary, acquisition protocol, scoring rules, device and firmware inventory, transformation history, and known changes, then calibrate rules on a representative pilot. Provider selection should examine sleep-domain knowledge, source-format parsing, human review, statistical sampling, data engineering, independence, and privacy controls rather than the dashboard alone. The contract should address the data snapshot, approval of thresholds, review of false positives and false negatives, issue ownership, rework, retest, delivery formats, audit rights, and permitted claims. An assessment report is not regulatory approval or a data warranty, and use beyond the tested purpose, version, or population requires renewed judgment.

Quality work may require identifiable signals, free text, and linkage keys, calling for least privilege, a controlled environment, export limits, defined retention, and verifiable deletion. Rare cases and small groups in the report can also create re-identification risk. A one-time audit describes only a snapshot. A mature service continuously observes drift in centers, devices, firmware, schemas, fields, missingness, label rules, and population composition, with baselines, alerts, human confirmation, and retesting after change. Automated quality indexes improve coverage, but final decisions still require clinical meaning and intended-use judgment. When defects cannot be repaired or risk remains unacceptable, the correct outcome may be downgraded use, rejection, or recollection.