The Search for Solid Ground in Text as Data: A Systematic Review of Validation Practices and Practical Recommendations for Validation

Communication Methods and Measures, vol. 18, pp. 249–277

Abstract

Communication research frequently applies computational text analysis methods (CTAM) to detect and measure social science constructs. However, the validity of these measures can be difficult to assess. In addition, there are hardly any established standards and little guidance for researchers on how to best validate CTAM. But how do these challenges affect current validation practices in applied research? And what practical recommendations for better validation of text-based measures can we derive? To answer these questions, we conducted a systematic review of current validation practices and qualitative expert interviews. We focused on political communication, a subfield that has arguably played a pioneering role in embracing the application of CTAM in communication research. Our results show that researchers apply a great variety of validation steps, which, however, are rarely selected based on a unified understanding of validity. The qualitative interviews further reinforce this notion, as interviewees bemoan a lack of established guidelines and frameworks for validation. Based on our empirical findings, we then derive practical recommendations to guide researchers on how to approach validation conceptually. Additionally, we provide insight into ongoing research that focuses on applied validation frameworks, designed to provide hands-on guidance to researchers involved in CTAM validation.

Authors 3

1. Lukas Birkenmaier corresponding

   GESIS - Leibniz Institute for the Social Sciences

   Affiliation as printed

   Survey Design and Methodology GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany

2. Clemens M. Lechner

   GESIS - Leibniz Institute for the Social Sciences

   Affiliation as printed

   Survey Design and Methodology GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany

3. Claudia Wagner Aachen Department of Society, Technology and Human Factors

   RWTH Aachen University · GESIS - Leibniz Institute for the Social Sciences · Complexity Science Hub

   Affiliation as printed

   Complexity Science Hub Vienna, Vienna, Austria

   Computational Social Science, GESIS - Leibniz Institute for the Social Sciences, Cologne, Germany

   Department of Society, Technology and Human Factors, RWTH Aachen University, Aachen, Germany