What Exactly Did the Transformer Learn from Our Physics Data?

Computing and Software for Big Science, vol. 9

Abstract

Abstract Transformer networks excel in scientific applications. We explore two scenarios in ultra-high-energy cosmic ray simulations to examine what these network architectures learn. First, we investigate the trained positional encodings in air showers which are azimuthally symmetric. Second, we visualize the attention values assigned to cosmic particles originating from a galaxy catalog. In both cases, the Transformers learn plausible, physically meaningful features.

Authors 4

1. Martin Erdmann corresponding Aachen Physics Institute 3A

   RWTH Aachen University

   Affiliation as printed

   Physics Institute 3A, RWTH Aachen University, Otto-Blumenthal-Straße, 52074, Aachen, Germany

2. Niklas Langner Aachen Physics Institute 3A

   RWTH Aachen University

   Affiliation as printed

   Physics Institute 3A, RWTH Aachen University, Otto-Blumenthal-Straße, 52074, Aachen, Germany

3. Josina Schulte Aachen Physics Institute 3A

   RWTH Aachen University

   Affiliation as printed

   Physics Institute 3A, RWTH Aachen University, Otto-Blumenthal-Straße, 52074, Aachen, Germany

4. Dominik Wirtz Aachen Physics Institute 3A

   RWTH Aachen University

   Affiliation as printed

   Physics Institute 3A, RWTH Aachen University, Otto-Blumenthal-Straße, 52074, Aachen, Germany