Description
The High Rate Analysis platform that has been implemented in the context of the Working Group 5 of ICSC Spoke2 offers a general purpose environment where analyzers can scale up their computations via the python Dask library. This can be done either distributing the workload within the actual Kubernetes cluster resources, or offloading to remote resources. The latter option is enabled by the adoption of interLink, a tool that generalizes the Virtual Kubelet concept and allows to integrate any kind of resource (HTC, HPC, Cloud) capable of managing a container lifecycle as a virtual node inside the Kubernetes cluster. This way, the analyzer can use the usual Dask JupyterLab extension (or directly the dask_kubernetes library [3]) to deploy and monitor a cluster on the remote resources. This InterLink-based solution has been used to integrate ICSC provisioned resources hosted at INFN Naples HTCondor batch system into the High Rate Analysis platform. This poster shows the technical details of this implementation.
[1] https://github.com/interlink-hq/interLink
[2] https://github.com/dask/dask-labextension
[3] https://kubernetes.dask.org/en/latest/operator_kubecluster.html
