Searching for the triple Higgs boson production and beyond using machine learning at the ATLAS experiment

This thesis presents the first search for triple Higgs boson (HHH) production conducted at the Large Hadron Collider (LHC), using 126 $fb^−1$ of proton-proton collision data at a centre-of-mass energy of $\sqrt{s} = 13$ TeV recorded by the ATLAS detector during Run-2 data-taking. The Standard Model (SM) $HHH$ production is unique in its direct sensitivity to the trilinear and quartic Higgs self-coupling constants, therefore presenting an opportunity to detect deviations from their predicted values. These deviations are characterised by the Higgs self-coupling modifiers - $\kappa_3$ and $\kappa_4$, and they have been simultaneously constrained for the first time at the LHC through the search for $HHH$ production. This signature also provides an avenue to probe Beyond the Standard Model (BSM) theories in the extended Higgs sectors. Three models in particular are targeted, the "Two Real Singlet Model" (TRSM), "a simple model for dark matter and CP violation" (DM-CPV), and generic heavy scalars. The search is conducted in the final state of $b\bar{b}b\bar{b}b\bar{b}$, maximising statistics but at the cost of a large background arising from Quantum Chromodynamics (QCD) processes. The thesis focuses on the methodology developed to search for such a complex final state against this challenging background. The expected (observed) upper limit of the SM $HHH$ signal strength is extracted to be $750 (760)$ times the predicted value at the 95% confidence level. Constraints on the Higgs self-coupling modifiers $\kappa_3$ and $\kappa_4$ are placed simultaneously by scanning over the profile likelihood ratio, assuming SM values for all other coupling constants. At the 95% confidence level, $\kappa_3$ is constrained between $-11−17$, and $\kappa_4$ between $-230−240$ for both the expected and observed limits, while assuming the SM value of the other modifier. For the targeted BSM models, the expected (observed) upper limits on the allowed cross-sections are placed between $46−350$ fb ($48−310$ fb) for the TRSM and DM-CPV, and between $4.7−69$ fb ($5.7−38$ fb) for generic heavy scalars with a narrow decay width, and between $5.2−53$ fb ($6.3−39$ fb) for a wide decay width. This search, especially the simultaneous constraints on the coupling modifiers, sets the foundation for future $HHH$ analyses. Beyond the main adopted method, several alternatives are explored to initiate discussions on new techniques for multi-Higgs analyses. A novel approach to search for new physics is presented, which utilises previously unexplored topologies within dataset-wide graph representations and Graph Neural Networks (GNNs). The method demonstrates the potential to surpass the performance of conventional machine learning methods in the case study of a BSM leptoquark search. The thesis also presents the feasibility of rejecting $bb$-jets arising from small-angle $g \rightarrow b\bar{b}$ splitting, a common background in the ATLAS triggers, using machine learning. With the potential to reduced the readout rate, it could be highly relevant in future data-taking at high luminosity, and improve signal efficiencies for analyses targeting resolved $b$-quarks.