Abstract:
The growing adoption of computational workflows in scientific research requires robust mechanisms for recording and managing provenance information. Distributed and federated Workflow Management Systems complicate provenance capture: execution is divided across different sites, intermediate states are transient, and data are staged and restaged throughout the workflow execution. Building upon the RO-Crate ecosystem, which packages research outputs with their metadata to make them easier to share, reuse, and understand, we extended the Provenance Run Crate to support distributed WMSs contexts by designing the Distributed Step Crate, a per-step provenance fragment encoded within the distributed node itself. These fragments are collated within the Distributed Provenance Crate, resulting in a contiguous provenance record. We further propose an Orchestration Crate to contain multiple workflow provenance records for a combined view of provenance across complex, distributed workflows. The Distributed Provenance Crate is demonstrated by its application to Globus Flows, a distributed workflow engine. The resulting provenance record reproduces the step-level semantics of a reference, centralized Common Workflow Language Provenance Run Crate record, while extending this record with runtime, hardware, and access-control metadata. The record expands to RDF with its published contexts and is queryable with SPARQL, and we characterize the overhead of the approach: storage and generation costs are fixed per step and grow linearly with the number of instrumented steps, while transfer cost is governed by the orchestrating platform. Finally, we identify four conditions for applying the Distributed Step Crate to production distributed WMSs. An open-source reference implementation and example RO-Crates accompany this paper. Together, these results illustrate how provenance can be effectively extended to distributed WMS contexts, enhancing transparency, reproducibility, and collaboration in scientific research.
