Speaker
Description
As the market trends towards larger hard disk drives to reduce operational costs, the throughput per terabyte (TB) is concurrently decreasing. Simultaneously, the shift of data reconstruction to earlier stages of data processing pipelines—up to the experimental triggers—is transforming the usage of storage resources. These resources are now expected to support multiple read cycles, as required to perform statistical analyses and machine learning algorithm training.
The Worldwide LHC Computing Grid (WLCG) sites have observed a significant increase in the bandwidth demand for disk resource access, which is now replacing compute power as the primary bottleneck for data processing and analysis workflows. To address this, solutions involving faster drives, such as SSDs and NVMe technology, are under discussion.
This contribution presents our experiences with tiering and caching strategies in the context of the LHCb experiment at CNAF. Specifically, we discuss the implementation of an NVMe-backed hot-storage pool, provisioned transparently via GPFS and StoRM-WebDAV storage. Additionally, we report on the AI_INFN Platform, which provisions resources for the ICSC - Spoke2 activities, and our attempt leveraging Cinder volumes fronted by NVMe disks for transparent caching.
Although the discussion has not yet matured to the stage of definitive technical solutions, we believe that disk storage represents one of the most pressing challenges for the future of High Energy Physics (HEP) experiments. We aim to bring this issue to the forefront of the community’s attention, encouraging the development of analytics and fostering R&D initiatives.
