Definition, deliverables, and adjacent service boundaries
Sleep algorithm evaluation data operations continuously maintain sleep records, reference labels, partitions, task specifications, and use controls as a data product for testing and comparing algorithms. The activity is classified under WSIC 6.4.3, Data Asset Operations. Sources may include polysomnography, home sleep apnea tests, actigraphy, wearables, bedside or environmental sensors, questionnaires, and follow-up data. Tasks may include sleep-wake classification, sleep staging, arousal and respiratory- or movement-event detection, signal quality, vital signs, snoring, posture, bed occupancy, and risk prediction. The deliverable is not an anonymous bundle of files. It is a defined release of inputs and references with a data dictionary, provenance and representativeness statement, license, quality records, and change history.
This activity differs from the preceding sleep-data licensing matchmaking service and the following sleep-research data-sharing platform. It also differs from selling training data, publishing a benchmark dataset, or performing an algorithm evaluation service. Matchmaking connects rights holders and users, while a research-sharing platform supports broader secondary investigation. Evaluation-data operations govern data for independent testing of locked algorithm versions. An operator may host a hidden test set and execute scoring, but ownership of the data does not make the operator an independent certifier. A rank, composite score, or license does not establish regulatory authorization, clinical benefit, or applicability to every population and device.
Product specification, source rights, and intake acceptance
Each product first freezes its intended use, target population, acquisition setting, task, prediction unit, allowed channels, device and firmware, sampling and preprocessing, label system, scored regions, temporal tolerance, primary metrics, external-data policy, and acceptance criteria. Its catalog describes participant and recorded-night counts, sites, regions, ages, sex, conditions and severity, comorbidities, devices, signal quality, missingness, and important groups that are absent. Data limited to healthy adults, one-center PSG, or one wrist device cannot represent children, older adults, complex sleep disorders, home settings, or other sensors merely because the catalog calls itself big sleep data.
Intake creates a provenance manifest and immutable checksum for every batch. Review covers consent or another lawful basis, ethics and institutional approvals, collectors and transfer chain, research and commercial uses, geographic or cross-border restrictions, redistribution, derived labels, model training, and result publication. De-identification does not eliminate risk: continuous physiological signals, audio or video, exact dates, rare conditions, location, and device identifiers may permit linkage with other information. The operator applies data minimization, tiered access, and repeated residual-risk assessment and contractually prohibits re-identification, participant contact, and out-of-scope use. A de-identified label, registry listing, or possession of a copy is not a substitute for a valid rights chain.
Independent partitioning, contamination control, and coverage maintenance
Training, tuning, and final test data are separated by participant. Multiple nights from one person, overlapping windows, re-exports, pre- and post-de-identification copies, and derivatives of the same household or event cannot cross partitions. Depending on the purpose, external-distribution sets may also be separated by site, device, acquisition period, or population. Randomly dividing epochs or signal fragments places highly correlated observations in development and test sets and inflates performance. Operations records preserve split code, seeds, stratification, inclusion and exclusion reasons, and denominators, while record hashes, signal similarity, and provenance metadata identify exact and near duplicates.
Public sleep cohorts may be mirrored, renamed, cropped, or incorporated into foundation-model pretraining, so different filenames do not demonstrate independence. Participants disclose data encountered during training, tuning, retrieval, pretraining, manual review, and prior submissions. Suspected contamination is marked and handled with a sealed confirmation set or an independent external set. Repeated queries to a hidden set also become a tuning signal; controls therefore include account- and organization-level limits, delayed feedback, a final holdout, and anomalous-submission monitoring. New intake is periodically compared with the current target population, devices, and settings rather than accepting large volumes of clean, easy, or duplicate records simply to increase size.
Reference labels, quality control, and uncertainty
Reference results may come from qualified sleep scorers, multi-expert consensus, synchronized reference instruments, clinical outcomes, or predefined rules, but the source must match the output being evaluated. PSG and HSAT labels document the scoring standard and version, channels and technical specifications, scorer qualifications, blinding, independent review, adjudication, timing tolerances, and indeterminate regions. Human staging and event scoring have inter-scorer disagreement, particularly for N1, transitions, artifacts, and event boundaries; one scorer's annotation is not error-free truth. When machine pre-annotation assists label production, human changes and label lineage remain visible, and agreement with those labels cannot be presented as fully independent validation.
Quality operations cover file readability, channels and units, sampling, clock alignment, montage mapping, signal loss, artifact, label range, event overlap, metadata consistency, duplication, outliers, and case-level review. Ambiguous records may retain scorer distributions, unresolved labels, confidence, or unscored regions instead of deleting every difficult case. Reports separately disclose record completeness, label coverage, class and subgroup counts, failure and abstention denominators, scorer agreement, and revision volume. A statement such as 99 percent quality passed has no meaning without rules, sampling design, and a denominator. Corrections preserve the previous value, reviewer, reason, and impact analysis.
Versioned release, controlled execution, and leakage prevention
Every release assigns versions, hashes, and release notes to raw files, processing code, labels, dictionary, partitions, scorer, and license. Correcting labels, adding cases, changing windows, resampling, replacing device metadata, or modifying metrics may break comparability and must not silently overwrite an earlier release. Notes distinguish documentation-only corrections, changes that permit a justified bridge, and major changes requiring all systems to be rerun, and state whether prior results remain valid. Freeze, withdrawal, and retirement criteria prevent a leaked, overused, or obsolete set from continuing to define performance for current devices and populations.
Sensitive or hidden data are often better evaluated on an isolated server, trusted research environment, or controlled execution at the submitter's site than directly distributed. The environment fixes containers, dependencies, hardware quotas, randomness, timeouts, network access, and output formats and uses least privilege, multi-factor authentication, key management, logging, malicious-code isolation, and timely revocation. Output review prevents per-case predictions, debug logs, gradients, encoded results, or repeated queries from leaking labels or personal information. Operators may not send restricted data to unauthorized cloud services or subcontractors, and users may not evade controls through multiple accounts, manual inspection of hidden cases, or external identity matching.
Scoring use, result publication, and evidentiary limits
Scoring rules correspond to the task and error costs and are frozen before results are inspected. Sleep staging may report macro F1, stage-specific performance, confusion matrices, and night-level structural error. Event detection defines its matching rule and reports misses, false positives, and onset and offset error. Continuous estimates require bias, error distributions, calibration, coverage, and failure-to-output. Statistical analysis treats independent participants as the primary unit, reports sample size and uncertainty, and stratifies by relevant age, sex, condition, severity, site, device, signal quality, and sensing-related factors. Overall accuracy, AUROC, the best subgroup, or successful-output cases alone can conceal imbalance and selective failure.
Every result is bound to the data release, algorithm fingerprint, preprocessing, thresholds, external data, submission count, protocol deviations, and scoring code. A leaderboard compares only compatible releases and tracks; composite-score components, weights, ties, and withdrawal rules are public. Appeals, corrections, conflicts, negative findings, sponsorship, and promotional citations require governance. Offline evaluation data establish how a locked system performed under defined conditions. They do not replace device integration, human workflow, cybersecurity, usability, valid clinical association, clinical validation, or post-deployment monitoring, and research-use data cannot be expanded into diagnostic, treatment, or sleep-improvement claims.
Contracting, continuous operations, and exit handling
Contracts price data rights, expert labeling, cleaning and harmonization, hidden-set hosting, controlled compute, submission allowance, statistical reporting, periodic updates, and incident response separately and assign the roles of data owner, controller, processor, evaluator, and algorithm developer. Service levels cover access approval, availability, recovery, quality-response time, breach notification, rescoring, support, and audit evidence. Revenue sharing or per-submission charges must not encourage weak intake, useless repeat submissions, or suppression of negative results. When an operator also develops algorithms, invests in a participant, or influences procurement, it discloses the conflict and separates personnel, data, and decisions.
At expiry or termination, new submissions stop; accounts, tokens, mirrors, and subcontractor access are revoked; and data, caches, derivatives, and temporary outputs are returned or verifiably deleted as agreed, while only the minimum legally or operationally required audit record remains. Consent withdrawal, a rights dispute, a material labeling error, a security incident, or increased re-identification risk must be traceable to affected releases, users, models, and public results so that access can be suspended and notification, correction, reevaluation, or leaderboard withdrawal can begin. Mature operations are measured not by catalog size or revenue but by whether every result can be traced to a data release, represented people and devices, authorized actions, recorded changes, known limitations, and a safe exit.