World Sleep Industry ENCYCLOPEDIA

Algorithm Evaluation Benchmark Datasets

4.4.3

Definition, role, and classification boundary

An algorithm evaluation benchmark dataset is a data asset that provides controlled inputs, reference results, data partitions, and a scoring protocol for a specified algorithmic task. It is classified under WSIC 4.4.3, Sleep Data Services. Sleep applications may include sleep-wake recognition, sleep staging, arousal and respiratory- or movement-event detection, signal quality, vital-sign estimation, bed occupancy, snore classification, trend prediction, and multimodal understanding. Its central value is enabling comparable and reviewable evaluation under common conditions, not merely accumulating a large number of recordings.

A benchmark differs from a general training set, a product-acceptance sample, regulatory clinical evidence, and an algorithm evaluation service. It may include open training data, a development set, and a hidden test set, but a leading rank demonstrates performance only for the defined data, task, inputs, and metrics. It does not establish suitability for other devices, populations, hospitals, homes, or clinical workflows and does not replace independent external validation. The maintainer should state which question the benchmark answers, which questions it cannot answer, and which public claims its results support.

Task design, target population, and data specification

Development begins by freezing the task and intended use, including the prediction unit, observation window, permitted input channels, output labels, reference standard, scored and ignored regions, temporal tolerance, threshold selection, external-data policy, and real-time or offline constraints. Sleep staging, event detection, and night-level risk prediction cannot share an undefined accuracy task. Tasks with the same name but different channels, sampling rates, age ranges, scoring rules, or postprocessing should not be placed on one leaderboard without qualification. Documentation should cover the data dictionary, channels and units, acquisition devices, filtering and resampling, missing values, time base, eligibility criteria, and quality controls.

The target population and acquisition environment must support the proposed comparison. The benchmark describes participant and recorded-night counts, sites, regions, ages, sex, conditions and severity, devices, signal quality, and important groups that are absent. Data from healthy adults or one site's equipment cannot represent children, older adults, complex sleep disorders, or home wearable use. Selection should not retain only clean and easily scored cases until the task no longer reflects real difficulty. If the purpose is to measure cross-site, cross-device, or temporal drift performance, the corresponding track should be designed in advance rather than introduced after an unfavorable result.

Data splitting, independence, and contamination control

Training, development, and test partitions must be separated by participant. Multiple nights from the same person, overlapping windows, re-exports, and derived versions cannot cross partitions. Randomly dividing epochs from whole-night signals places highly correlated segments in both training and test data and creates optimistic results. Generalization studies also need suitable out-of-distribution partitions by site, device, acquisition period, or population. Split code, random seeds, sample counts, and stratification rules should be versioned, and maintainers should disclose enough for independent audit without exposing sensitive information or hidden answers.

Public data are frequently mirrored, renamed, cropped, or included in pretraining corpora, so duplicate content cannot be identified from filenames alone. A benchmark should use record checksums, segment similarity, participant metadata, and provenance to detect near duplicates and require participants to disclose external data, pretrained models, and any human exposure to test cases. Repeated queries to a hidden set also turn it into a training signal; submission rates should therefore be limited, a sealed final set retained, abnormal submissions monitored, and test data periodically refreshed. When contamination occurs, affected versions and results need flags rather than continuing to describe the old leaderboard as an independent comparison.

Reference standards, label uncertainty, and quality governance

Reference results may come from qualified scorers, independent multi-reader consensus, synchronized reference instruments, clinical outcomes, experimental measurements, or explicit rules. The choice must match the task and its construction must be documented. Human sleep scoring includes inter-reader disagreement, particularly at stage transitions and for N1, artifacts, and borderline events; a single reader is not an infallible truth. Clinical PSG or HSAT labels should identify the scoring system and version, qualifications, blinding, review, adjudication, and temporal tolerance. Device-specific or research-defined labels need separate names so that they are not mistaken for a clinical standard.

Quality checks cover signal completeness, channels and units, clock alignment, label ranges, event order, metadata consistency, duplicates, outliers, and record-level review. Ambiguous samples need not all be deleted; multiple-reader distributions, unresolved labels, confidence weights, or ignored scoring regions can represent uncertainty when specified in the protocol. Maintainers should preserve original and revised labels, errata, quality results, and reasons for change. An incorrect answer should lead to a new version and an explanation of comparability with old results, not a silent change to test labels or selective rescoring of teams.

Metrics, statistical reporting, and end-to-end evaluation

Metrics follow the intended use and cost of error. Sleep staging may report macro F1, stage-specific sensitivity and positive predictive value, agreement measures, a confusion matrix, and night-level errors in total sleep time, sleep efficiency, and stage proportions. Event detection needs an explicit matching rule and reports misses, false positives, false alarms per hour, and onset and offset error. Continuous estimates require an error distribution, calibration, coverage, and no-output rate. Under severe class imbalance, overall accuracy or AUROC can conceal practical failure and should be accompanied by measures suited to rare events.

Reports include sample size and uncertainty intervals and, where relevant, stratify by age, sex, condition, severity, site, device, skin tone or other factors affecting sensor performance, signal quality, and part of the night. A composite score discloses its components, weights, and tie rules and should not permit frequent, easy classes to compensate for high-risk errors. Separate tracks may cover robustness, missing channels, malformed input, latency, memory, energy, abstention, and human-AI collaboration, but each needs its own interpretation. Statistical significance is not clinical importance, and a model score does not replace assessment of safety, usability, and impact in a real workflow.

Evaluation execution, security, versioning, and leaderboard governance

Open downloads support reproducibility, while hidden health data are often better evaluated on a controlled server or in a container. The evaluation environment fixes the runtime image, dependencies, hardware quota, randomness, timeout, network access, output format, and scoring code and retains submission hashes, logs, and results. Hidden labels, source audio or video, and sensitive metadata require least-privilege protection. Output design should prevent per-record predictions from being used to infer answers or identify people. Human inspection of hidden cases, matching identities through external sources, coding special rules for the test set, and evading rate limits with multiple accounts should be prohibited explicitly.

Data, labels, partitions, metrics, scoring software, and leaderboards all require versions. Every change produces release notes, impact analysis, and conditions for recomputation. A main leaderboard should ordinarily compare one version, with cross-version results identified separately. Maintainers need processes for appeals, corrections, withdrawal of results, conflicts of interest, and security incidents and should prevent organizers or partners from receiving asymmetric test information. A long-running benchmark needs refresh and retirement criteria so that years of repeated optimization do not create leaderboard overfitting and obsolete devices or populations do not continue to define the state of the field.

Licensing, procurement use, and future direction

Licenses and data-use agreements should state whether commercial training, foundation-model fine-tuning, input caching, publication of per-record predictions, derivative labels, redistribution, and post-challenge use are allowed and how publications and product marketing may cite a result. Procurement also addresses test-set hosting, data rights, privacy and ethics authority, subcontracting, security, fees, submission quotas, result ownership, and termination. A buyer reviewing a supplier's score should confirm the exact benchmark version, input channels, external data, and number of submissions rather than relying on a screenshot or one rank.

Emerging directions include dynamic refresh sets, cross-device and cross-site stress tests, human-AI evaluation in actual workflows, continual drift sets, federated or secure-sandbox evaluation, and links to use-specific model cards. A larger benchmark is not automatically better; transparent design, independence, representativeness, reference standards, metrics, and governance determine credibility together. When benchmark evidence supports a medical device, clinical study, or patient-management claim, the applicable quality, ethics, and regulatory requirements still apply, with risk-proportionate independent evidence in representative real settings. A leaderboard is not evidence of authorization, therapeutic effect, or universal applicability.