APPLYING PQUANTML TO LARGE ARCHITECTURES: TRANSFORMERS FOR HIGH ENERGY PHYSICS

Zenodo (CERN European Organization for Nuclear Research)

Abstract

High-energy physics experiments such as the Large Hadron Collider (LHC) generate terabytes of measurement data per second. Therefore, online (real-time) trigger algorithms decide whether detector data should be stored for later offline analysis or not. At the Compact Muon Solenoid (CMS) experiment at the LHC, the Level-1 trigger operates at a frequency of 40 MHz, resulting in latency requirements in the microseconds range. Commonly used devices for machine learning such as GPUs cannot be used for such tasks due to these strict latency requirements. Therefore, field-programmable gate arrays (FPGAs) have been used for inference of neural networks in such use cases. In order to use FPGAs despite their limited computational power, models are often compressed before being compiled for FPGA inference. In this project, a transformer architecture model with approximately 1M parameters is compressed using in-training pruning and quantization, and the performance of the model candidates under different compression strategies is evaluated. In particular, it is studied whether the model can be compressed sufficiently that inference on FPGAs can be possible.

Authors 3

1. Dino Cheng Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

2. Roope Niemi

   European Organization for Nuclear Research

   Affiliation as printed

   European Organization for Nuclear Research

3. M. Pierini

   European Organization for Nuclear Research

   Affiliation as printed

   European Organization for Nuclear Research