Definition and responsibility boundary
Research data processing is the professional work of turning collected records into inspectable, analysis-ready, and transferable data products according to a protocol and a data-processing plan. It may include intake and inventory, format conversion, coding, cleaning, queries, standardization, signal preprocessing, variable derivation, quality control, dataset freeze, and delivery of code and documentation. It is not the same as primary data collection, statistical consulting, development of an algorithmic product, or ghostwriting a paper. A processor may implement approved rules and report problems, but should not change the research question, primary outcome, analysis population, or causal interpretation after seeing the data on behalf of the investigator.
Responsibilities should be assigned across the data lifecycle. The commissioning organization identifies the research purpose, lawful and ethical basis, consent and use restrictions, authoritative sources, variable definitions, and evidence that must be retained. The processor protects information within its authorization, executes specified rules, records deviations, and supplies reviewable outputs. Automated output, a tidy table, or a high completion rate does not establish that the data are true, the study is unbiased, or the conclusion is valid. Processing also cannot convert estimates from a consumer device into a clinical diagnosis.
Input inventory and processing specification
Before work starts, a processing specification and input manifest should describe relationships among studies, visits, participants, devices, and specimens, together with file formats, fields, units, time zones, encodings, versions, transfer batches, expected counts, and checksums. The parties should identify whether an original record, certified copy, or system export is authoritative, which fields may be corrected, which may only receive an annotation, and when blinded, randomized, or grouped information may be viewed. The first transfer should be quarantined for integrity checks; source files should be preserved read-only and every later transformation should produce a new version.
A data dictionary should define each name, label, type, permissible value, unit, time basis, source, derivation, missing-value code, and applicable population. A lineage map should connect every output to its source file, field, rule, and program version. Transfers from external laboratories, electronic questionnaires, imaging systems, wearables, and clinical systems need an agreed specification so that identical labels with different meanings, changed units, lost leading zeros, or automatic date conversions do not pass unnoticed. Trial runs should include representative normal, boundary, missing, and erroneous cases, and business and methodological owners should approve mappings before batch processing.
Cleaning, queries, and dataset freeze
Quality checks should distinguish structural errors, range or logical conflicts, duplicates, cross-table inconsistency, temporal anomalies, protocol deviations, and observations that are statistically unusual but possibly genuine. Every query should retain its linked record, rule, originator, response, decision, and time; overwriting a value or silently editing a table is not a resolution. Originally missing, not collected, not applicable, device failure, below quantification, manually excluded, and not computable after processing have different meanings and require distinct codes. Imputation, winsorization, smoothing, and outlier deletion are analysis decisions and should not be introduced silently by data-processing staff without authorization in the plan.
Cleaning rules should, where feasible, be specified before outcome groups are inspected, and critical derivations should be checked through independent programming, sampled review, or separate rule tests. A revision must retain the old value, new value, rationale, actor, and time; a bulk correction also needs its affected scope and rollback method. Freezing a dataset is not renaming a file “final.” It requires closure of outstanding queries, confirmation of blinding and approvals, locking of dictionaries and programs, and generation of manifests and checksums. An unfreeze requires authorization, rationale, impact assessment, and a new identifiable version.
Processing sleep and circadian data
Sleep research often aligns polysomnography, electroencephalography, respiration, oximetry, electrocardiography, actigraphy, questionnaires, diaries, environmental measures, and clinical data, making time itself a critical variable. Processing should preserve device and server clocks, time zone, daylight-saving changes, dates spanning midnight, lights-off and lights-on, in-bed and out-of-bed times, sleep opportunity, wear and non-wear intervals, synchronization events, and clock-correction methods. Channel names, derivations or montages, sampling rates, filters, gains, units, sensor placement, devices, and firmware versions belong in the metadata. Resampling, rereferencing, artifact flags, or signal replacement should never overwrite the original waveform.
Sleep stages, respiratory events, arousals, oxygen desaturations, activity-rest intervals, and circadian phase metrics all depend on definitions and rule versions. Derived measures such as total sleep time, sleep efficiency, arousal index, or apnea-hypopnea index should retain the numerator, denominator, valid recording duration, included events, and unscorable intervals, rather than only a summary number. Harmonization across devices or cohorts should preserve original variables and mappings and should identify concepts that are only approximately comparable. Algorithm confidence, human review, and overrides should be recorded separately, and training data must not be mixed with final evaluation data merely for processing convenience.
Pipelines, validation, and reproducibility
Reproducible processing depends on versioned code, configuration, reference tables, packages, runtime environments, and fixed inputs, not an operator’s memory. Each run should create a unique identifier and record input and output checksums, time, executor, parameters, logs, warnings, and failure status. Random procedures should use a recorded fixed seed, and possible differences caused by parallel execution or hardware should be disclosed. Pipeline testing should cover unit rules, end-to-end paths, known answers, boundary conditions, idempotence, and failure recovery. Material changes to software, rules, or upstream data require impact assessment and appropriate revalidation.
Code review should test both scientific meaning and software behavior: whether joins multiply rows, windows leak future information, units are converted twice, dates shift, or exclusions alter the analysis population. Artificial intelligence or automated annotation tools need fixed model and threshold versions, validation for the intended devices, settings, and subgroups, and documented human-review rules; plausible synthetic completion is not an observed value. When proprietary software prevents full reproduction, the delivery should at least retain its version, parameters, export settings, quality logs, independent cross-checks, and remaining limitations.
Privacy, security, and controlled use
Research data processing should follow purpose limitation and data minimization, receiving only fields needed for the task and separating identity linkage files from analytical data. Names and government identifiers are not the only risks: precise dates, addresses, rare diagnoses, location traces, voice, images, genomic information, and continuous physiological waveforms may identify people in combination. De-identification reduces rather than eliminates risk. Consent, ethics approval, contracts, law, and re-identification likelihood should guide deletion, generalization, date shifting, pseudonymization, or controlled access. Restricted data should not be uploaded to public tools or used to train models without authorization.
Access should be role- and project-based, with controlled transfer, encryption, logs, backups, recovery tests, malware protection, and incident response. Personal accounts must not be shared, and temporary copies, caches, download folders, and support tickets require the same control. Cross-institutional, cloud, and subcontracted processing must identify locations, subprocessors, keys, retention, return or destruction, incident notice, and audit rights. If mistransmission, unauthorized access, identity linkage, loss of blinding, or corruption may affect participant rights or research reliability, the team should isolate the issue, preserve evidence, and assess and report it through the established path.
Delivery, acceptance, and long-term maintenance
A complete delivery package commonly includes an immutable source manifest, processed datasets, data dictionaries, mapping and derivation specifications, query and decision logs, quality reports, exclusion and missingness summaries, code, configuration, dependencies or containers, run instructions, test results, checksums, and known issues. Acceptance should reproduce key derivations from prepared cases and reconcile record counts, uniqueness, links, timelines, and expected errors, rather than merely confirm that files open. The investigator remains responsible for approving analysis-ready data and its limitations; a processor must not manufacture tidy results by deleting unfavorable records, suppressing warnings, or repeatedly tuning rules.
At closeout, the parties should specify which versions enter an institutional archive or controlled repository, who maintains readability, access approval, licenses, and dependencies, how long materials remain, and when secure destruction occurs. If source corrections, rule defects, software vulnerabilities, or changed standards emerge later, the organization should be able to identify affected datasets, figures, papers, and downstream systems, issue a change notice, and decide whether reprocessing is needed. Auditability, interpretability, and portability matter more than permanent dependence on a vendor interface. The ultimate value of research data processing is that authorized people can trace a deliverable back to sources and decisions and understand honestly what the data cannot answer.