Benchmarking Crisis in Social Media Analytics: A Solution for the Data-Sharing Problem

Social Science Computer Review, vol. 40, pp. 1496–1522

Abstract

Computational social science uses computational and statistical methods in order to evaluate social interaction. The public availability of data sets is thus a necessary precondition for reliable and replicable research. These data allow researchers to benchmark the computational methods they develop, test the generalizability of their findings, and build confidence in their results. When social media data are concerned, data sharing is often restricted for legal or privacy reasons, which makes the comparison of methods and the replicability of research results infeasible. Social media analytics research, consequently, faces an integrity crisis. How is it possible to create trust in computational or statistical analyses, when they cannot be validated by third parties? In this work, we explore this well-known, yet little discussed, problem for social media analytics. We investigate how this problem can be solved by looking at related computational research areas. Moreover, we propose and implement a prototype to address the problem in the form of a new evaluation framework that enables the comparison of algorithms without the need to exchange data directly, while maintaining flexibility for the algorithm design.

Authors 9

1. Dennis Assenmacher corresponding

   Queensland University of Technology · University of Münster

   Affiliation as printed

   Queensland University of Technology (QUT), Brisbane, Australia

   University of Münster, Germany

2. Derek Weber

   Defence Science and Technology Group · Department of Defence · The University of Adelaide

   Affiliation as printed

   Defence Science and Technology Group, Department of Defence, Canberra, Australian Capital Territory, Australia

   University of Adelaide, South Australia, Australia

3. Mike Preuß Aachen

   Leiden University

   Affiliation as printed

   Universiteit Leiden, the Netherlands

4. André Sergej Calero Valdez Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Germany

5. Björn Roß

   University of Edinburgh

   Affiliation as printed

   University of Edinburgh, United Kingdom

6. Stefano Cresci

   Institute of Informatics and Telematics

   Affiliation as printed

   Institute of Informatics and Telematics (IIT-CNR), Pisa, Italy

7. Heike Trautmann

   University of Münster

   Affiliation as printed

   University of Münster, Germany

8. Frank Neumann

   The University of Adelaide

   Affiliation as printed

   University of Adelaide, South Australia, Australia

9. Christian Grimme

   University of Münster

   Affiliation as printed

   University of Münster, Germany