Search for Higgs Boson Pair Production in the $b\bar{b}\tau^+\tau^-$ Final State with the ATLAS Experiment at the LHC

A search for Higgs boson pair (HH) production in final states with two $b$ quarks and two $\tau$ leptons using 139 fb$^{-1}$ of proton-proton collision data at a centre-of-mass energy of 13 TeV recorded by the ATLAS detector at the Large Hadron Collider (LHC) is presented. Searches for HH production are vital for solidifying our understanding of electroweak symmetry breaking in the Standard Model (SM) since they serve as direct probes of the Higgs boson self-coupling. In addition, they provide sensitivity to new phenomena introduced by theories beyond the SM. The search presented in this thesis targets channels with one or more hadronic $\tau$-lepton decays. It relies on effective tau identification algorithms, which are algorithms that distinguish signatures of hadronic $\tau$-lepton decays in the ATLAS detector from other sources, such as quark- or gluon-initiated jets. A novel tau identification algorithm based on recurrent neural networks is introduced. It exploits information from reconstructed charged-particle tracks, topological clusters of calorimeter cell signals, and purposefully constructed discriminating variables for tau identification. The new technique outperforms the method previously employed in the ATLAS experiment. Three different types of Higgs boson pair production are considered in this thesis: non-resonant HH production predicted by the SM (SM HH production); resonant HH production via scalar, narrow-width resonances; and non-resonant HH production with anomalous values of the Higgs boson self-coupling constant $\lambda_{HHH}$. The search for SM HH production targets the gluon-gluon fusion and vector boson fusion production modes. No statistically significant deviation from the background-only hypothesis is observed; therefore, upper limits are set on the SM HH production cross section. The search yields an upper limit of 4.7 times the SM cross section prediction at 95% confidence level (CL). The search for resonant HH production considers intermediate resonances with masses ranging from 251 GeV to 1600 GeV. Over the considered mass range, the largest deviation from the background-only hypothesis is observed for a mass of 1000 GeV with a local (global) significance of 3.1$\sigma$ (2.0$\sigma$). Upper limits at 95% CL are set on the resonant HH production cross section, which range from 20 fb to 900 fb depending on the mass of the resonance. Finally, the search for SM HH production is reinterpreted in the context of anomalous values of the Higgs boson self-coupling constant. Upper limits at 95% CL are set on the non-resonant HH production cross section as a function of the self-coupling modifier $\kappa_\lambda = \lambda_{HHH} / \lambda_{HHH}^{\text{SM}}$, where $\lambda_{HHH}^{\text{SM}}$ is the Higgs boson self-coupling predicted by the SM. The upper limits on the non-resonant HH production cross section are compared to theory predictions, providing constraints on the self-coupling modifier of $-2.4 \leq \kappa_\lambda \leq 9.2$.