The search for solid ground in text as data: A systematic review and practical recommendations for validation

Abstract

Communication research frequently applies computational text analysis methods (CTAM) to detect and measure social science constructs. However, the validity of these measures can be difficult to assess. In addition, there are hardly any established standards and little guidance for researchers on how to best validate CTAM. But how do these challenges affect current validation practices in applied research? And what practical recommendations for better validation of text-based measures can we derive? To answer these questions, we conducted a systematic review of current validation practices and qualitative expert interviews. We focused on political communication, a subfield that has arguably played a pioneering role in embracing the application of CTAM in communication research. Our results show that researchers apply a great variety of validation steps, which, however, are rarely selected based on a unified understanding of validity. The qualitative interviews further reinforce this notion, as interviewees bemoan a lack of established guidelines and frameworks for validation. Based on our empirical findings, we therefore derive practical recommendations to guide researchers regarding when and how to validate CTAM. Moreover, we provide a preview of emerging validation frameworks that could prove beneficial for researchers working with text as data.

Authors 3

1. Lukas Birkenmaier corresponding Aachen

   RWTH Aachen University · GESIS - Leibniz Institute for the Social Sciences · Complexity Science Hub

   Affiliation as printed

   GESIS -Leibniz Institute for the Social Sciences , Mannheim , Germany

   , GESIS -Leibniz Institute for the Social Sciences, Mannheim,

   , GESIS -Leibniz Institute for the Social Sciences B6 4, 5, 68159 Mannheim, Germany

   Complexity Science Hub Vienna, Austria

   GESIS -Leibniz Institute for the Social Sciences, Mannheim,

   GESIS -Leibniz Institute for the Social Sciences, Mannheim; RWTH Aachen University, Germany;

2. Clemens M. Lechner Aachen

   RWTH Aachen University · GESIS - Leibniz Institute for the Social Sciences · Complexity Science Hub

   Affiliation as printed

   GESIS -Leibniz Institute for the Social Sciences , Mannheim , Germany

   , GESIS -Leibniz Institute for the Social Sciences, Mannheim,

   , GESIS -Leibniz Institute for the Social Sciences B6 4, 5, 68159 Mannheim, Germany

   Complexity Science Hub Vienna, Austria

   GESIS -Leibniz Institute for the Social Sciences, Mannheim,

   GESIS -Leibniz Institute for the Social Sciences, Mannheim; RWTH Aachen University, Germany;

3. Claudia Wagner Aachen

   RWTH Aachen University · GESIS - Leibniz Institute for the Social Sciences · Complexity Science Hub

   Affiliation as printed

   , GESIS -Leibniz Institute for the Social Sciences, Mannheim,

   , GESIS -Leibniz Institute for the Social Sciences B6 4, 5, 68159 Mannheim, Germany

   Complexity Science Hub Vienna, Austria

   GESIS -Leibniz Institute for the Social Sciences, Mannheim,

   GESIS -Leibniz Institute for the Social Sciences, Mannheim; RWTH Aachen University, Germany;