From Dark Matter to the Dark Sector: search for new physics in final states with a photon and missing transverse momentum with the ATLAS detector

The Dark Matter (DM) problem is one of the main open questions in physics, and an important hint of the incompleteness of the Standard Model (SM) of elementary particles. Among the several proposed DM models, one of the most credited and explored DM candidate is the WIMP (Weakly Interacting Massive Particle), while an appealing scenario which is gaining growing interest in the last years, predicts the existence of a full "hidden" or Dark Sector, with its own internal gauge structures and particle content, potentially interacting also with the SM through so-called "portal" interactions. This thesis focuses on WIMP and Dark Photon searches in proton proton collisions at $\sqrt{s}$=13 TeV at the Large Hadron Collider (LHC), using the full Run-2 data collected by the ATLAS experiment between 2015 and 2018, for a total integrated luminosity of 139 fb$^{-1}$. Final states involving a photon and missing transverse momentum ($E_\mathrm{T}^{\mathrm{miss}}$) are considered, the latter being a potential signature of DM or Dark Sector particles.$\\$ The first search targets mono-photon final states, with an high energy photon and $E_\mathrm{T}^{\mathrm{miss}}$, and interprets the results in terms of simplified DM models, where weakly interacting DM candidates are pair-produced via an s-channel axial-vector or vector mediator. In the presence of a signal, an excess of event in the $E_\mathrm{T}^{\mathrm{miss}}$ tails with respect to SM expectations would be observed. Purely data-driven estimates are employed to estimate the background from electrons or jets misreconstructed as photons ($e/j\to\gamma$), while appropriate Control Regions (CRs) are defined for the other contributions from processes with true photons. No excess is observed, therefore, exclusion limits on the existence of new physics are set. Depending on the coupling strength between the mediator and SM or DM particles, mediator masses below 920-1470 GeV for DM candidate masses below 280-580 GeV are excluded at 95% CL. These results are extrapolated to limits on the scattering cross-section between DM and nucleons, as a function of the DM mass, highlighting their competitivity with respect to Direct Detection experiments at low DM masses. $\\$ The second part of this work explores the Dark Sector, by looking for signals of Dark Photon ($\gamma_D$) production. The $\gamma_D$ is predicted as the gauge boson of a new U(1) symmetry group, mediating the interactions in the Dark Sector. An interesting production channel consists in the decay of a (SM or BSM) Higgs boson into a photon and a $\gamma_D$, through a loop interactions featuring an additional BSM messenger field, coupled both to the SM and the Dark Sector. A search is performed, for the first time in ATLAS, targeting Higgs bosons produced in association with a Z boson, leptonically decaying into two electrons or two muons, giving rise to a $\ell\ell + \gamma + E_\mathrm{T}^{\mathrm{miss}}$ final state. The analysis sensitivity is driven by the employment of Boosted Decision Trees (BDT), and the background estimation is strongly based on data-driven techniques for $e\to\gamma$ and fake $E_\mathrm{T}^{\mathrm{miss}}$ backgrounds , with a new method developed to estimate the latter. No excess is observed, and upper limits on the branching fraction $BR(H\to\gamma\gamma_D)$ are set, ranging from 2.19% to 2.52% for $\gamma_D$ masses between 0 and 40 GeV. Finally, a reinterpretation of the mono-photon analysis is performed, to provide exclusion limits on the same decay for BSM Higgs with masses between 400 GeV and 3 TeV, in the gluon-gluon fusion and Vector Boson Fusion production modes. The exclusion limits, at 95% CL, on the $BR(H\to\gamma\gamma_D)$ range from 24.2 fb to 1.44 fb, for increasing Higgs masses. These analysis provides competitive results with respect to other LHC searches, and a statistical combination with other ATLAS results is foreseen to gain further improvement.$\\$