Taming numerical imprecision by adapting the KL divergence to negative probabilities

Statistics and Computing, vol. 34

Abstract

Abstract The Kullback-Leibler (KL) divergence is frequently used in data science. For discrete distributions on large state spaces, approximations of probability vectors may result in a few small negative entries, rendering the KL divergence undefined. We address this problem by introducing a parameterized family of substitute divergence measures, the shifted KL (sKL) divergence measures. Our approach is generic and does not increase the computational overhead. We show that the sKL divergence shares important theoretical properties with the KL divergence and discuss how its shift parameters should be chosen. If Gaussian noise is added to a probability vector, we prove that the average sKL divergence converges to the KL divergence for small enough noise. We also show that our method solves the problem of negative entries in an application from computational oncology, the optimization of Mutual Hazard Networks for cancer progression using tensor-train approximations.

Authors 7

1. Simon Pfahler corresponding

   University of Regensburg

   Affiliation as printed

   Department of Physics, University of Regensburg, 93040, Regensburg, Germany

2. Peter Georg

   University of Regensburg

   Affiliation as printed

   Department of Physics, University of Regensburg, 93040, Regensburg, Germany

3. Rudolf Schill

   ETH Zurich

   Affiliation as printed

   Department of Biosystems and Engineering, ETH Zürich, 4056, Basel, Switzerland

4. Maren Klever Aachen Institute for Geometry and Applied Mathematics

   RWTH Aachen University

   Affiliation as printed

   Institute for Geometry and Applied Mathematics, RWTH Aachen, 52062, Aachen, Germany

5. Lars Grasedyck Aachen Institute for Geometry and Applied Mathematics

   RWTH Aachen University

   Affiliation as printed

   Institute for Geometry and Applied Mathematics, RWTH Aachen, 52062, Aachen, Germany

6. Rainer Spang

   University of Regensburg

   Affiliation as printed

   Department of Informatics and Data Science, University of Regensburg, 93040, Regensburg, Germany

7. Tilo Wettig

   University of Regensburg

   Affiliation as printed

   Department of Physics, University of Regensburg, 93040, Regensburg, Germany