Description of feature
Summary
Add support for consuming provenance artifacts generated by nf-prov (e.g. Workflow RO-Crates, BCOs, and associated index files) as an input source for nf-core/datasync.
Rather than requiring users to manually generate and maintain transfer manifests, datasync could extract the list of workflow outputs directly from provenance records generated by upstream pipelines and perform the existing integrity checks and transfers.
Motivation
Many Nextflow workflows already produce provenance information describing workflow outputs and associated checksums. However, users of nf-core/datasync currently need to generate separate manifest files to identify which outputs should be transferred.
This results in duplicated metadata and additional maintenance effort, despite the required information already being available in provenance artifacts.
Supporting provenance-based file discovery would simplify data delivery workflows while improving traceability and reducing opportunities for manual error.
Proposed Functionality
nf-core/datasync should optionally accept provenance outputs generated by nf-prov, such as:
Workflow RO-Crates
BioCompute Objects (BCOs)
Associated provenance index files
From these artifacts, datasync would:
- Extract the list of workflow outputs.
- Read available checksums.
- Verify file integrity prior to transfer.
- Continue using existing transfer mechanisms (e.g. rclone).
- Skip files failing checksum validation and report them to the user.
This would act as an alternative input mode to the existing manually generated manifest approach.
Expected Benefits
- Eliminates manual manifest generation for many workflows.
- Reuses provenance information already generated by Nextflow.
- Improves traceability between workflow execution and downstream data delivery.
- Reduces duplication of metadata across pipelines.
- Encourages adoption of workflow provenance standards within the nf-core ecosystem.
Example Use Case
A demultiplexing pipeline produces sequencing outputs and generates provenance records through nf-prov, e.g. a BCO object or a WRROC that lists all files + checksums in the respective specs/manifest.
Instead of creating a separate manifest listing output files, a user can provide the provenance artifact directly to nf-core/datasync. The pipeline extracts the output inventory, verifies checksums, and transfers validated files to the requested destination using the existing rclone-based implementation.
Out of Scope
- New transfer backends
- Changes to existing rclone functionality
- Generation of new provenance artifacts by datasync
The feature is intended solely as an additional mechanism for discovering and validating files to transfer.
Description of feature
Summary
Add support for consuming provenance artifacts generated by nf-prov (e.g. Workflow RO-Crates, BCOs, and associated index files) as an input source for nf-core/datasync.
Rather than requiring users to manually generate and maintain transfer manifests, datasync could extract the list of workflow outputs directly from provenance records generated by upstream pipelines and perform the existing integrity checks and transfers.
Motivation
Many Nextflow workflows already produce provenance information describing workflow outputs and associated checksums. However, users of nf-core/datasync currently need to generate separate manifest files to identify which outputs should be transferred.
This results in duplicated metadata and additional maintenance effort, despite the required information already being available in provenance artifacts.
Supporting provenance-based file discovery would simplify data delivery workflows while improving traceability and reducing opportunities for manual error.
Proposed Functionality
nf-core/datasync should optionally accept provenance outputs generated by nf-prov, such as:
Workflow RO-Crates
BioCompute Objects (BCOs)
Associated provenance index files
From these artifacts, datasync would:
This would act as an alternative input mode to the existing manually generated manifest approach.
Expected Benefits
Example Use Case
A demultiplexing pipeline produces sequencing outputs and generates provenance records through nf-prov, e.g. a BCO object or a WRROC that lists all files + checksums in the respective specs/manifest.
Instead of creating a separate manifest listing output files, a user can provide the provenance artifact directly to nf-core/datasync. The pipeline extracts the output inventory, verifies checksums, and transfers validated files to the requested destination using the existing rclone-based implementation.
Out of Scope
The feature is intended solely as an additional mechanism for discovering and validating files to transfer.