Topic specificity: A descriptive metric for algorithm selection and finding the right number of topics

Natural Language Processing Journal, vol. 8, pp. 100082

Abstract

Topic modeling is a prevalent task for discovering the latent structure of a corpus, identifying a set of topics that represent the underlying themes of the documents. Despite its popularity, issues with its evaluation metric, the coherence score, result in two common challenges: algorithm selection and determining the number of topics. To address these two issues, we propose the topic specificity metric, which captures the relative frequency of topic words in the corpus and is used as a proxy for the specificity of a word. In this work, we formulate the metric firstly. Secondly, we demonstrate that algorithms train topics at different specificity levels. This insight can be used to address algorithm selection as it allows users to distinguish and select algorithms with the desired specificity level. Lastly, we show a strictly positive monotonic correlation between the topic specificity and the number of topics for LDA, FLSA-W, NMF and LSI. This correlation can be used to address the selection of the number of topics, as it allows users to adjust the number of topics to their desired level. Moreover, our descriptive metric provides a new perspective to characterize topic models, allowing them to be understood better.

Authors 6

1. Emil Rijcken corresponding

   Utrecht University · Eindhoven University of Technology

   Affiliation as printed

   Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands

   Jheronimus Academy of Data Science, Eindhoven University of Technology , ’s Hertogenbosch, The Netherlands

   Jheronimus Academy of Data Science, Eindhoven University of Technology , 's Hertogenbosch, The Netherlands

2. Kalliopi Zervanou

   Eindhoven University of Technology

   Affiliation as printed

   Jheronimus Academy of Data Science, Eindhoven University of Technology , ’s Hertogenbosch, The Netherlands

   Jheronimus Academy of Data Science, Eindhoven University of Technology , 's Hertogenbosch, The Netherlands

3. Pablo J. Mosteiro

   Utrecht University

   Affiliation as printed

   Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands

   Department of Methodology and Statistics, Utrecht University, Utrecht, The Netherlands

4. Floortje E. Scheepers

   University Medical Center Utrecht

   Affiliation as printed

   Psychiatry, University Medical Center Utrecht, Utrecht, The Netherlands

5. Marco René Spruit Aachen Leiden Institute of Advanced Computer Science Public Health and Primary Care

   Leiden University · Leiden University Medical Center

   Affiliation as printed

   Leiden Institute of Advanced Computer Science, Leiden, The Netherlands

   Public Health & Primary Care, Leiden University Medical Center, Leiden, The Netherlands

6. Uzay Kaymak

   Eindhoven University of Technology

   Affiliation as printed

   Jheronimus Academy of Data Science, Eindhoven University of Technology , ’s Hertogenbosch, The Netherlands

   Jheronimus Academy of Data Science, Eindhoven University of Technology , 's Hertogenbosch, The Netherlands