Published August 1, 2024 | Version v1

Enabling Julia code to run at scale with artefact caching

  • 1. ROR icon European Organization for Nuclear Research

Description

The Julia programming language, which has evolved into a mature tool for scientific computing over the past decade, offers high-level capabilities with just-in-time (JIT) compilation and efficient garbage collection. Its performance, comparable to that of C, makes Julia an attractive option for the high-energy physics community. However, Julia's use of intermediate precompiled files to achieve this performance introduces significant startup delays, known as "time to first plot" (TTFP), which would be a severe drawback in distributed systems where each node would have to compile dependencies locally. This project addresses the TTFP issue by leveraging the shared CernVM-FS (CVMFS) file system to distribute files across nodes and Julia to prepare a ready-to-use directory with all necessary precompiled files for selected applications, thereby reducing startup times. We developed and tested a framework for automating the publication of precompiled Julia files to CVMFS and evaluated its impact on performance using two sample applications: the Julia Jet Reconstruction package and the Geant4 wrapper package. Our findings demonstrate that precompiled files stored in CVMFS significantly reduce startup times, with reductions of up to 97% for the Geant4 package. Furthermore, we examined the effects of cross-compilation for various microarchitectures and found that nodes benefit from shared cache files without notable performance degradation due to microarchitecture differences.

Files

Summer_Student_programme_Report-1.pdf

Files (292.7 kB)

Name Size Download all
md5:40eff6e8d0d8ac5046a6f472a69e7b15
292.7 kB Preview Download

Additional details

Identifiers

CDS Report Number
CERN-STUDENTS-Note-2024-011

Linked records