Limit Distribution Theory for f-Divergences

IEEE Transactions on Information Theory, vol. 70, pp. 1233–1267

Abstract

$f$-divergences, which quantify discrepancy between probability distributions, are ubiquitous in information theory, machine learning, and statistics. While there are numerous methods for estimating$f$-divergences from data, a limit distribution theory, which quantifies fluctuations of the estimation error, is largely obscure. As limit theorems are pivotal for valid statistical inference, to close this gap, we develop a general methodology for deriving distributional limits for$f$-divergences based on the functional delta method and Hadamard directional differentiability. Focusing on four prominent$f$-divergences—Kullback-Leibler divergence,$\chi ^{2}$divergence, squared Hellinger distance, and total variation distance—we identify sufficient conditions on the population distributions for the existence of distributional limits and characterize the limiting variables. These results are used to derive one- and two-sample limit theorems for Gaussian-smoothed$f$-divergences, both under the null and the alternative. Finally, an application of the limit distribution theory to auditing differential privacy is proposed and analyzed for significance level and power against local alternatives.

Authors 3

1. Sreejith Sreekumar Aachen Institute for Quantum Information

   RWTH Aachen University · Cornell University

   Affiliation as printed

   School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, USA

   Institute for Quantum Information, RWTH Aachen University, Aachen, Germany

2. Ziv Goldfeld

   Cornell University

   Affiliation as printed

   School of Electrical and Computer Engineering and Department of Statistics and Data Science, Cornell University, Ithaca, NY, USA

3. Kengo Kato

   Cornell University

   Affiliation as printed

   School of Electrical and Computer Engineering and Department of Statistics and Data Science, Cornell University, Ithaca, NY, USA