High-Speed Data Scouting: Exploring RoCE and 400G Ethernet for CMS Level-1 Trigger Scouting system

The CMS Level-1 Trigger scouting system, planned as part of the HL-LHC upgrade, aims to collect objects reconstructed by the L1 trigger at the LHC bunch crossing rate to perform online physics analyses. As part of this effort, the use of RDMA over Converged Ethernet is being explored to transfer data from the scouting FPGAs to servers, along with the adoption of 400G Ethernet links. A first implementation of RoCEv2 on an AMD Versal adaptive SoC evaluation board is presented. This work aims to extend the physics reach of the CMS experiment allowing for a completely triggerless acquisition system.

The CMS Experiment at the CERN Large Hadron Collider (LHC) relies on a hardware Level-1 Trigger system (L1T) to process collision events at the LHC bunch crossing (BX) rate of 40 MHz, and to select events of interest for further processing. The upgrades of CMS for the upcoming high-luminosity LHC period (HL-LHC) will vastly improve the quality of L1T event reconstruction. This will provide the opportunity for a complementary Data Scouting approach, in which physics analysis is performed on a data stream containing all collision events, but with object quality limited to that of L1T-level reconstruction. In the baseline implementation, the L1T scouting system adopts custom FPGA-based boards to collect trigger-level reconstructed objects for each BX from the L1 Trigger boards and transmit them to a server farm using TCP/IP over 100G Ethernet links, following the model used by the central CMS DAQ system. As part of the L1T scouting project, an alternative architecture is being explored in which TCP/IP is replaced with RDMA over Converged Ethernet (RoCEv2), enabling direct and efficient data transfers into server and GPU memory, thereby reducing latency and minimizing the CPU resources required for networking. In parallel, the use of 400G Ethernet links is also being investigated. Through the adoption of RoCE and 400G Ethernet, the data throughput of the L1T scouting system can be significantly extended, leading to an increase in its physics reach. A first implementation of the RoCEv2 protocol on the AMD Versal adaptive SoC platform is presented, using the VHK158 evaluation board within the CMS L1T scouting system. In this design, the programmable logic (PL) is used for protocol handling and high-speed networking, while the processing system (PS) is employed for control and monitoring. Performance metrics—including bandwidth, latency, and FPGA resource usage—are reported. This development represents a critical step toward a high-throughput scouting system for the HL-LHC, enabling analyses performed on L1 trigger-level objects and aiming to enhance the physics reach of the CMS experiment.

This work has been [partially] funded by the Eric & Wendy Schmidt Fund for Strategic Innovation through the CERN Next Generation Triggers project under grant agreement number SIF-2023-004.