Measuring data drift with the unstable population indicator

Data Science, vol. 7, pp. 1–12

Abstract

Measuring data drift is essential in machine learning applications where model scoring (evaluation) is done on data samples that differ from those used in training. The Kullback-Leibler divergence is a common measure of shifted probability distributions, for which discretized versions are invented to deal with binned or categorical data. We present the Unstable Population Indicator, a robust, flexible and numerically stable, discretized implementation of Jeffrey’s divergence, along with an implementation in a Python package that can deal with continuous, discrete, ordinal and nominal data in a variety of popular data types. We show the numerical and statistical properties in controlled experiments. It is not advised to employ a common cut-off to distinguish stable from unstable populations, but rather to let that cut-off depend on the use case.

Authors 2

1. M. Haas corresponding Aachen Public Health and Primary Care

   University of Amsterdam · Leiden University Medical Center

   Affiliation as printed

   Public Health and Primary Care, Leiden University Medical Center, Albinusdreef 2, The Netherlands and Business Intelligence, University of Amsterdam, Spui 21, 1012WX Amsterdam, The Netherlands

2. Lisette Sibbald

   University of Amsterdam · Tilburg University

   Affiliation as printed

   Department of Methodology and Statistics, Tilburg University, Prof. Cobbenhagenlaan 125, 5037 DB Tilburg, The Netherlands and Department of Cognitive Neuropsychology, Tilburg University, Prof. Cobbenhagenlaan 125, 5037 DB Tilburg, The Netherlands and Business Intelligence, University of Amsterdam, Spui 21, 1012WX Amsterdam, The Netherlands

   Tilburg University, Prof. Cobbenhagenlaan 125, 5037 DB Tilburg, The Netherlands and Business Intelligence,

   University of Amsterdam, Spui 21, 1012WX Amsterdam, The Netherlands