Look everywhere effects in anomaly detection

arXiv (Cornell University)

Abstract

Machine learning-based anomaly detection methods are able to search high-dimensional spaces for hints of new physics with much less theory bias than traditional searches. However, by searching in many directions all at once, the statistical power of these search strategies is diluted by a variant of the look elsewhere effect. We examine this challenge in detail, focusing on weakly supervised methods. We find that training and testing on the same data results in badly miscalibrated $p$-values due to the anomaly detector searching everywhere in the data and overfitting on statistical fluctuations. However, if these $p$-values can be calibrated, they may offer the best sensitivity to anomalies, since this approach uses all of the data. Conversely, training on half of the data and testing on the other half results in perfectly calibrated $p$-values, but at the cost of reduced sensitivity to anomalies. Similarly, regularization methods such as early stopping can help with $p$-value calibration but also possibly at the expense of sensitivity. Finally, we find that k-folding strikes an effective balance between calibration and sensitivity. Our findings are supported by numerical studies with Gaussian random variables as well as from collider physics using the LHC Olympics benchmark anomaly detection dataset.

Authors 3

1. Marie Hein Aachen Institute for Theoretical Particle Physics and Cosmology

   RWTH Aachen University

   Affiliation as printed

   Institute for Theoretical Particle Physics and Cosmology , RWTH Aachen University , D-52056 Aachen , Germany

2. Benjamin Philip Nachman

   Stanford University · SLAC National Accelerator Laboratory

   Affiliation as printed

   Department of Particle Physics and Astrophysics , Stanford University , Stanford , CA 94305 , USA

   Fundamental Physics Directorate , SLAC National Accelerator Laboratory , Menlo Park , CA 94025 , USA

3. David Shih

   Rutgers, The State University of New Jersey · Environmental and Occupational Health Sciences Institute

   Affiliation as printed

   NHETC , Dept. of Physics and Astronomy , Rutgers University , Piscataway , NJ 08854 , USA