Bridging the divide between qualitative and quantitative science studies

Quantitative Science Studies, vol. 1, pp. 918–926

Abstract

In January 2019, the Editorial Board of the Journal of Informetrics decided to resign following a series of disagreements with Elsevier. In collaboration with the International Society for Scientometrics and Informetrics (ISSI) and MIT Press, the Editorial Board thereupon launched this journal: Quantitative Science Studies (QSS). The launch of QSS offers an opportunity to rethink the contents and research agenda of the journal, and marks a turn from the focus on “metrics” to science studies. Such a shift, reflected also in the name change, indicates the intention to seek closer connections with colleagues in “qualitative science and technology studies” and take more distance from journals focusing on specialist “metrics” (Milojević & Leydesdorff, 2013).The goal of this special issue is to explore the relations among and promote conversations between quantitative science studies and neighboring fields. To this end, we invited a number of colleagues conducting research relevant to this theme to articulate the relations between their research and QSS, and to formulate challenges and research agendas for synergies between qualitative and quantitative approaches in the broad area of Science and Technology Studies (STS), science-policy analyses, innovation studies, the sociology of science, the science of science, and related domains. Their response generated 11 articles and one letter that provide a rich panorama of views and exciting ideas for building bridges and pursuing research agendas that have the potential to advance our knowledge about science, scientific knowledge production, and the scientific workforce, as well as to promote the responsible and sustainable usage of metrics for evaluation and policy.The idea of a main “divide” between qualitative and quantitative STS originated in relatively recent studies that examined the relationship between qualitative and quantitative STS empirically. Leydesdorff and Van den Besselaar (1997) argued on the basis of aggregated citation relations among journals that three main groups of journals can be distinguished: one more specifically qualitative oriented (e.g., Social Studies of Science), one specifically focusing on quantitative science studies (e.g., Scientometrics), and a third interfacing between quantitatively oriented journals and innovations studies (e.g., Research Policy). From the latter perspective, however, Martin, Nightingale, and Yegros-Yegros (2012, p. 1194) stated thatSTS today is a rather divided community, with quantitative scientometrics and qualitative STS researchers operating largely in isolation from one another, one or two individual exceptions notwithstanding. The qualitative side of STS continues to expand its work on technology (including constructive technology assessment) and innovation, with the original programme of work analysing the social influences on the content of science having diffused into the mainstream and now attracting less interest. At the same time, scientometric research has been moving beyond science into areas previously the domain of traditional sociology (such as innovation and the analysis of social networks within and between organisations), as well as forming links with information science (as reflected, for example, in the recent creation of the Journal of Informetrics).On the basis of studying 136 chapters in both quantitative and qualitative handbooks of science and technology studies, Milojević, Sugimoto, et al. (2014) concluded that “a great divide” has structured STS intellectually. However, these authors added that[o]ne of the interesting findings of this study is the identification of chapters of shared interest across the qualitative and quantitative divide and the nuanced differences when it comes to studying the topics covered in these chapters: technology, gender and policy.The discussion about a divide between qualitative and quantitative STS is by no means new to the field. In December 1987, for example, a workshop was organized by John Irvine, Anthony van Raan, and one of us (Leydesdorff) on “the relations between qualitative theory and scientometric methods in science and technology studies.” This resulted in a special issue of Scientometrics in 1989 containing more than 300 pages (vol. 15, issues 5–6, pp. 333–631).At the workshop, John Irvine and Ben Martin (1989) contributed a paper entitled “International comparisons of scientific performance revisited,” which offered new perspectives on the measurement of national research performance. Michel Callon and his coauthors (Françoise Bastide and Jean-Pierre Courtial) presented the co-word model (Bastide, Courtial, & Callon, 1989), and Anthony van Raan presented a paper (coauthored with Harry Peters) entitled “Dynamics of a scientific field analysed by co-subfield structures” (van Raan & Peters, 1989) These three programs were, among others, elaborated in the decades since. In the introduction to the special issue, Leydesdorff, Irvine, and Van Raan (1989, p. 333) formulated as follows:There is growing recognition of the need to integrate qualitative theorizing in the philosophy, sociology and history of science with the quantitative perspectives provided by scientometric studies. On the one hand, the use of scientometric indicators in policy analysis has stimulated debates on what exactly various indicators employed indicate, given the significant conceptual and technical problems that exist in measurement. On the other hand, the increased availability of large data-bases challenges researchers in the field of science and technology studies (S & TS) to test more rigorously their hypotheses concerning the various aspects of scientific and institutional developments.In a recent handbook of qualitative STS, Wyatt, Milojević, et al. (2017, p. 87) formulated the following evaluation of research efforts bridging the divide:Scientometrics and qualitative approaches within STS share a common origin, even if they have grown apart over the past decades in terms of research practices, norms and standards. Different skills are needed, and the epistemological assumptions are also different. However, both quantitative and qualitative STS have always shared a deep commitment to the empirical study of science and technology, and practitioners of both can be reflexive about their own knowledge production practices.In sum, although there is empirical evidence for a divide between qualitative and quantitative STS, one can also find efforts to bridge this gap over the past decades.Notwithstanding these common interests in bridging the divide, the tensions between qualitative and quantitative science studies have been constitutive of the field. In a review article entitled “Quantitative measures of communication in science: A critical overview,” David Edge (1979, p. 114)—at the time the editor of Social Studies of Science—for example, criticized quantitative science studies in the following strong wording:One is tempted to say that formal communication in science is “the tip of the iceberg,” were it not for two facts: (a) the “tip” is very large, extensive and important; and (b) there is every indication that the “tip” is radically different in kind from what is “below the waterline.” (Perhaps “the soft underbelly of science” might be a more appropriate metaphor!)Edge’s programmatic perspective of “following the actors” was committed to the “strong program” in the sociology of scientific knowledge (Bloor, 1976). In this sociology of scientific knowledge, it is claimed that the content of science can be explained in terms of sociocognitive interests. From this perspective, the sciences can be considered as belief structures attributed to communities. The evidence supporting the claim of truth in science is constructed (Fuller, 2018). These constructs can be deconstructed. However, an analyst cannot then escape from the reflexive conclusion that one’s own knowledge claim is also constructed; all debates and arguments thus tend to become matters of interests and opinion (e.g., Woolgar, 1988).Unlike an anthropological focus on practices, the study of science as a publication structure allows for a more distanced approach. The dynamics of the literature are sometimes very different from that of science as a social process. It seems to us that this “double hermeneutics” in terms of formal and nonformal communications is unavoidable in science studies (Giddens, 1976) because of the dynamics of the literature enabling us to move back and forth between contexts of discovery and justification (e.g., Myers, 1985). The textual layer (the library, the archive, etc.) is structured with reference to disciplines that also operate as selection mechanisms. The practices generate variation and novelty, that is reflected in the texts (Callon, Law, & Rip, 1986; Callon, Courtial, et al., 1983), and the discursive layer has a dynamic of its own (Gilbert, 1977; Mulkay, Potter, & Yearley, 1983).The context of application in research evaluations, technology assessments, and science and technology (S&T) policy analyses has added a third “mode of knowledge production” to the field of STS during the last decades (Gibbons, Limoges, et al., 1994). Both qualitative and quantitative science studies have been challenged by priority programs such as the National Science Foundation’s “Science of Science and Innovation Policies,” now replaced by the program “Science of Science: Discovery, Communication, and Impact” (cf. Husband Fealing, Lane, Marburger III, & Shipp, 2011; Marburger III, 2005). The European Framework and Horizons Programs call on STS from the perspective of applications. Perhaps the pressure of funding agencies on this field has in the meantime become a unifying factor, because one often needs a variety of perspectives in studies with normative objectives and implications. However, these are empirical questions.While the differing contexts can be distinguished analytically, they are interacting in the practices which are under study when “following the actors.” Pickering (1995), for example, proposed the metaphor of a “mangle of practice.” In the so-called “sociology of translations,” heterogeneous networks (representing people, texts, cognitions, funding, and subjects of study (e.g., scallops [Callon, 1986]) are analyzed in terms of translations from one co-word map into another (Callon et al., 1983). Such heterogeneity—including, for example, also “nonhumans”— provides resources for revisions and for changes.It seems to us that this focus on “heterogeneity” at both the substantive and methodological levels is not so different from Merton’s (1948) call for middle-range theories and pluriformity. At the time, Merton (1948) made two points that are still relevant to the issue, as follows:1. “[…] as a matter of plain fact the theorist is not inevitably the lamp lighting the way to new observations. The sequence is often reversed. Nor is it enough to say that research and theory must be married if sociology is to bear legitimate fruit. They must not only exchange solemn vows—they must know how to carry on from there. Their reciprocal roles must be clearly defined.” (p. 515)2. “What we have said does not mean that the piling up of statistics of itself advances theory; it does mean that theoretic interest tends to shift to those areas in which there is an abundance of pertinent statistical data.” (pp. 512f.)We intend this issue as a contribution to the clarification and definition of the reciprocal roles of quantitative and qualitative STS by focusing on research at the edge between the two approaches.The contributions to this special issue have been grouped into four themes with three papers each: (a) describing and questioning the divide between quantitative and qualitative science studies, (b) the use of numbers in decision-making addressing the usage of quantitative results in the context of policy-making and research evaluations, (c) perspective and bridges showcasing three currently very active research topics that attract researchers and scholars from a wide range of science studies fields, and (d) future research programs laying out roadmaps for the types of questions and approaches that can move the field forward.The three contributions in the first section of this collection address the divide from social, textual, and epistemic perspectives, respectively. First, Geoff Bowker contributes a letter entitled “Numbers or no numbers in science studies.” The author narrates his experiences with the chasm that opened between “quals” (“‘ethnomonsters”) and “quants” (“quantheads”) as political battles over hiring decisions erupted between the two camps of a sociology department. During such episodes, the arguments of each side can be ignored by the other on the basis of legitimations other than scholarly ones. Bowker (2020) argues for the importance of recognizing the complementary strengths of different approaches and for avoiding falling into dogmatic controversies.The divide between qualitative and quantitative STS is empirically studied in a paper by Douglas Kang and James Evans entitled “Against method: Exploding the boundary between qualitative and quantitative studies of science.” The authors compare publications in qualitative and quantitative sciences studies journals. The semantic analysis by Kang and Evans (2020) shows that qualitative and quantitative analyses build on opposite normative worlds: Whereas qualitative studies dwell on concepts such as “social,” “theory,” “political,” and “context,” quantitative analyses focus on “performance,” “measure,” and “results.” The authors argue that these literatures have disparate interests (both cognitively and politically) and are written for different audiences. They envisage that the further development of computer technologies will ease the tensions.Whereas the two previous papers described a divide in qualitative and quantitative terms, respectively, Harriet Zuckerman closes this section with a paper entitled “Is ‘the time ripe’ for quantitative research on misconduct in science?,” in which she analyzes the “why” of the problems involved in integrating the two perspectives. The argument runs as follows: If one relies on statistics for making a qualitative argument, one risks making claims on the basis of data that can be deconstructed from other perspectives. Official government statistics, for example, are organized for another objective. Using the case of misconduct in science, Zuckerman (2020) concludes that “a healthy dose of skepticism is in order in evaluating both the findings of current quantitative studies and of proposals for its remediation.”As noted, a third context of applications has become constitutive of STS in terms of resources, relations with clients, and legitimation (Gibbons et al., 1994). STS develops its own discourse by analyzing among other things the discourses in the techno-sciences under study, and by “translating” both these discourses into political and managerial contexts, such as research evaluations, technology assessments, and public debate. The three articles in the second section explore the relationship between quantitative science studies and the use of numbers for decision-making in these other contexts, including relations with industry and governments.Quantification can be used and abused for justification in decision-making processes (Porter, 1996). The development of S&T indicators, however, has also led to controversies about their use. The feedback from policies and ideologies such as New Public Management have directly influenced research agendas in scientometrics through consultancies and funding sources. In their paper, entitled “The impact of J. D. Bernal’s thoughts in the science of science upon China: Implications for quantitative studies of and the contribution of John (e.g., to the of science” and the that quantitative studies of science has first in the also to this in the use of indicators for policy has been in the with New Public Management and policies these indicators and a perspective were by which at the time in the of and (2020) for a on these as a means to a more between qualitative and quantitative STS in other However, there has been in recent over the of the use of S&T indicators in evaluation studies et al., et al., two following contributions on the policy use of S&T indicators on the of indicators in the research and the importance of appropriate of and policy contexts for the use of S&T In their paper entitled quantitative studies of science that policy and the that quantitative analysis have a impact on policies as an evidence they that the evidence has a policy and (2020) further explore the that to a in The study how the and of the studies are in the of scientific results to generate policy how this has both quantitative and qualitative and use the sociology of to compare the of in and research In this paper entitled “Quantitative science studies be with middle-range theories and concepts from the social and (2020) argue that differences in can the of the In the for example, research evaluation is by has a model by The study is as an of how quantitative science studies from data and analyses with middle-range theories and concepts in order to advance our of institutional of national research the we turn to research topics in science studies that have been from more than a perspective and thus for among Kang and Evans have topics are by qualitative approaches (e.g., more related to or by quantitative approaches (e.g., more related to noted, et al. (2014) programs and studies that were in the divide for substantive and and are analyzed as of bridging between contribution by entitled and to us that the of conducting quantitative science studies on data The availability of data is by and a political that this (2020) that is to it has not become more to This is because information has from to data is for The issue links with and argues that research be made to and to than having studies are often by data among other tends to and disciplines with resources & review science, and issues and gender in science and shows that in this from different and methodological to the and of to by focusing on (a) of collaboration and (b) (2020) on empirical from (e.g., & (e.g., & and publication analysis (e.g., et al., evidence in that for article entitled of scientific A shows how a such as “the of knowledge claims can on and be by from From the of and his by building on the of STS and the sociology of scientific knowledge The author a and various empirical to both qualitative and quantitative to study the of knowledge claims and the analysis of focus to move back and forth between the even (2020) thus provides a focus on the that bridges beyond the three papers in the last section of this issue programmatic These as well as Kang & Evans agendas that seek to the methodological of a between and and and in their paper entitled qualitative and science of the of the to a quantitative indicators to for for all the very under to these the research agendas in different The differences that methodological is related to epistemological and interests in quantitative empirical approaches with the of science studies. They argue that methods such as for between qualitative and quantitative approaches in They the of science (Callon et al., with its on heterogeneous as the mainstream and in The authors envisage how analysis and for an with theories in STS, such as second article in this section is paper, entitled as to the study of in science.” (2020) is in methods for the of knowledge claims in the of an in the sociology of a and rich of the between and science studies during the and The use of statistics provides into the of across a large of the literature of knowledge interest is about the of the can also be used for and how or funding agencies knowledge claims & their paper entitled indicators to A for research and are in the of quantitative studies. Their of is the for indicators of in the of the of and the of they to shift from indicators to In the to science as a and to their methods with their of and and (2020) authors to recent debates on the need to quantitative approaches with the of relevant which is relevant in decision-making sum, this collection of articles offers a of the variety of current perspectives on how quantitative science studies are related to qualitative science studies and neighboring fields. communication is specialist communication that needs to be when used in different It seems to us that both qualitative and quantitative perspectives are in To the from Merton the relations between qualitative and quantitative STS not only solemn know how to carry on from These reciprocal roles can then be elaborated in research and The edge between qualitative and quantitative approaches in STS has also been a for our research authors have no funding was for this are to the authors of the papers for their collaboration and to for in the for this theme the as the of are to the for Science and Technology for the of the papers in this special

Authors 3

1. Loet Leydesdorff

   University of Amsterdam

   Affiliation as printed

   Amsterdam School of Communication Research (ASCoR), University of Amsterdam, PO Box 15793, 1001 NG Amsterdam, The Netherlands

2. Ismael Ràfols Aachen Centre for Science and Technology Studies Netherlands & SPRU (Science Policy Research Unit)

   Leiden University · University of Sussex

   Affiliation as printed

   Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, The Netherlands & SPRU (Science Policy Research Unit), University of Sussex, UK

3. Staša Milojević

   Indiana University · Indiana University Bloomington

   Affiliation as printed

   Center for Complex Networks and Systems Research, The Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, USA