Development of machine-learning based app for anomaly detection in CMSWEB

This project that we have developed revolves around the idea of an advanced anomaly detection system for the CMSWEB services, which is a vital part of CERN's infrastructure that houses more than two dozen web services like DBS, DAS, CRAB, and WMCore. We aimed to improve these services' performance and reliability by detecting anomalies in real-time with the help of machine learning (ML) and deep learning (DL) models. Several autoencoder-based models like Hybrid CNN-LSTM, Hybrid CNN-GRU, GRU, LSTM, CNN, and a fully connected autencoder were trained by processing the data gathered from the CMS Monit infrastructure. There are multiple thresholds of anomaly detection incorporated in this system based on statistical methods, like mean + 1.5 standard deviations, Median absolute deviation (MAD) and the 95th and 99th percentiles of errors in reconstruction. To detect the deviations in the testing data, the above mentioned thresholds were applied which provided a robust procedure for anomaly detection. The Hybrid CNN-LSTM model demonstrated as a reliable and suitable model for anomaly detection in CMSWEB services by consistently outperforming the other models in regards to root mean squared error (RMSE), Mean Absolute Error (MAE), and R-squared. The platform used to deploy the system was OpenShift, which provided real-time insights, visualizations and alerts that were automated for anomalies related to services. We used hyper-parameter tuning and continuous learning approaches to ensure the adaption of the models to the changing services behaviors. This project holds significant potential for enhancing CERN's operational efficiency by reducing downtime and improving system stability. Future work will focus on improving the monitoring dashboard, expanding anomaly insights, integrating user interaction features for model training, and implementing an expert feedback loop to refine the detection process.