Speaker
Description
The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase
of computing and storage resource usage in the coming LHC data taking. It has been designed
to intelligently orchestrate workflow and data management systems, decoupling data
pre-processing, delivery, and primary processing in large scale workflows. It is an experiment-agnostic service that has been deployed to serve data carousel (orchestrating efficient processing of tape-resident data), ML hyperparameter optimization, active learning, and other complex multi-stage workflows defined via DAG, CWL and other descriptions, including a growing number of analysis workflows. We will present the motivation for iDDS, its architecture, use cases and the status of its production use in ATLAS and the Rubin Observatory, together with plans for the future.
| In-person participation | No |
|---|
