Improving the Output Quality of Official Statistics Based on Machine Learning Algorithms

Journal of Official Statistics, vol. 38, pp. 485–508

Abstract

Abstract National statistical institutes currently investigate how to improve the output quality of official statistics based on machine learning algorithms. A key issue is concept drift, that is, when the joint distribution of independent variables and a dependent (categorical) variable changes over time. Under concept drift, a statistical model requires regular updating to prevent it from becoming biased. However, updating a model asks for additional data, which are not always available. An alternative is to reduce the bias by means of bias correction methods. In the article, we focus on estimating the proportion (base rate) of a category of interest and we compare two popular bias correction methods: the misclassification estimator and the calibration estimator. For prior probability shift (a specific type of concept drift), we investigate the two methods analytically as well as numerically. Our analytical results are expressions for the bias and variance of both methods. As numerical result, we present a decision boundary for the relative performance of the two methods. Our results provide a better understanding of the effect of prior probability shift on output quality. Consequently, we may recommend a novel approach on how to use machine learning algorithms in the context of official statistics.

Authors 4

1. Quinten Meertens

   Centraal Bureau voor de Statistiek

   Affiliation as printed

   Statistics Netherlands , Henri Faasdreef 312, 2492 JP The Hague, the Netherlands

2. Cees G. H. Diks

   University of Amsterdam

   Affiliation as printed

   University of Amsterdam , Center for Nonlinear Dynamics in Economics and Finance , Roetersstraat 11, 1018 WB Amsterdam, the Netherlands

3. H.J. van den Herik Aachen

   Leiden University

   Affiliation as printed

   Leiden University , Niels Bohrweg 1, 2333 CA Leiden the Netherlands

4. Frank W. Takes Aachen

   Leiden University

   Affiliation as printed

   Leiden University , Niels Bohrweg 1, 2333 CA Leiden the Netherlands