Description
Hyperparameter optimization plays a crucial role in achieving high performance and robustness for machine learning models, such those used in complex classification tasks in High Energy Physics (HEP).
In this study, we investigate and experience the usage of $\texttt{Optuna}$, a rather new, modern and scalable optimization tool in the framework of a realistic signal-versus-background classification scenario carried out by applying $\texttt{XGBoost}$ on CMS Open Data.
The chosen classification task consists in extracting the signal associated to the decay mode of
$\mathrm{B}_s \rightarrow \mathrm{J}/\psi(\mu^+\mu^-)~\phi(K^+K^-)$ by means of a gradient boost tree ($\texttt{XGBoost}$) trained on both Monte Carlo simulated signal sample and a background one taken from the data as invariant mass sidebands. The optimization process of $\texttt{XGBoost}$ is guided by $\texttt{Optuna}$ with the aim to maximize the area under the ROC curve (AUC) while applying an overfitting control mechanism, whereas the Punzi Figure of Merit is used for a performant extraction of the signal within $\texttt{XGBoost}$.
This work demonstrates how $\texttt{Optuna}$ is a suitable tool, running on hybrid computing platforms, that enables efficient and effective exploration of the hyperparameter space in commonly used HEP workflows, while providing valuable diagnostics on the automated model optimization.