Advancing open science through research assessment reform? Content analysis of CoARA action plans

Scientometrics, vol. 131, pp. 3897–3918

Abstract

Abstract Research assessment reform and open science are frequently presented as aligned reform movements within science, with assessment reform expected to incentivize open science practices. However, empirical evidence of this integration remains limited. We conducted a quantitative content analysis of open science terminology in action plans from Coalition for Advancing Research Assessment (CoARA) signatories—institutions publicly committed to reforming research assessment. Analysing 248 publicly available action plans, we coded mentions of open science terms across values, umbrella, and operational categories. Results reveal skewed distributions of open science terms: most terms appear in fewer than 20% of documents. Values terms substantially outnumber operational terms in top mentions, suggesting organizations currently emphasize aspirational principles over concrete practices. Open Access emerges as the by far the most widely mentioned operational practice, while key activities emphasized in policy frameworks (Open Evaluation, Open Methods, Open Software) receive minimal attention. Overall, our findings suggest open science integration into assessment reform remains in early developmental stages, characterized by values-heavy discourse, and gaps between policy aspirations and organizational commitments. Beyond substantive findings, this study demonstrates how terminological analysis can track science reform movement development and provide data for longitudinal monitoring of open science and its convergence with reforms of research assessment.

Authors 4

1. Alexander Rushforth corresponding Aachen Centre for Science and Technology Studies

   Leiden University

   Affiliation as printed

   Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, Netherlands

2. Nino Gogadze Aachen Centre for Science and Technology Studies

   Leiden University

   Affiliation as printed

   Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, Netherlands

3. Tornike Skhirtladze

   Georgian National University SEU

   Affiliation as printed

   Georgian National University SEU, Tbilisi, Georgia

4. Janne Pölönen

   Federation of Finnish Learned Societies

   Affiliation as printed

   Federation of Finnish Learned Societies, Helsinki, Finland