-
COARA will not save science from the tyranny of administrative evaluation
Authors:
Alberto Baccini
Abstract:
Could the generalized adoption of the COARA principles make administrative research evaluation socially desirable and solve all its problems? The answer to this question is no. To reach this conclusion, two main arguments are discussed. The first characterises COARA as a form of 'technocracy' perfectly consistent with a neo-liberal view of research. The second consists in the adoption of Philip Ki…
▽ More
Could the generalized adoption of the COARA principles make administrative research evaluation socially desirable and solve all its problems? The answer to this question is no. To reach this conclusion, two main arguments are discussed. The first characterises COARA as a form of 'technocracy' perfectly consistent with a neo-liberal view of research. The second consists in the adoption of Philip Kitcher's idea of well-ordered science. It is argued that administrative evaluation of research, even if correct on the basis of COARA principles, is at odds with the principles of well-ordered science since it cannot escape neither the tyranny of expertise nor the tyranny of ignorance. These two arguments allow to suggest limiting administrative evaluation to the bare minimum (recruitment of researchers and funding of projects), and to focus attention mainly to the fairness of evaluation procedures.
△ Less
Submitted 10 August, 2024;
originally announced August 2024.
-
Is the panel fair? Evaluating panel compositions through network analysis. The case of research assessments in Italy
Authors:
Alberto Baccini,
Cristina Re
Abstract:
In research evaluation, the fair representation of panels is usually defined in terms of observable characteristics of scholars such as gender or affiliations. An empirical strategy is proposed for exploring hidden connections between panellists such that, despite the respect of formal requirements, the panel could be considered alike as unfair with respect to the representation of diversity of re…
▽ More
In research evaluation, the fair representation of panels is usually defined in terms of observable characteristics of scholars such as gender or affiliations. An empirical strategy is proposed for exploring hidden connections between panellists such that, despite the respect of formal requirements, the panel could be considered alike as unfair with respect to the representation of diversity of research approaches and methodologies. The case study regards the three panels selected to evaluate research in economics, statistics and business during the Italian research assessment exercises. The first two panels were appointed directly by the governmental agency responsible for the evaluation, while the third was randomly selected. Hence the third panel can be considered as a control for evaluating about the fairness of the others. The fair representation is explored by comparing the networks of panellists based on their co-authorship relations, the networks based on journals in which they published and the networks based on their affiliated institutions (universities, research centres and newspapers). The results show that the members of the first two panels had connections much higher than the members of the control group. Hence the composition of the first two panels should be considered as unfair, as the results of the research assessments.
△ Less
Submitted 10 October, 2024; v1 submitted 10 May, 2024;
originally announced May 2024.
-
Fine-grained classification of journal articles by relying on multiple layers of information through similarity network fusion: the case of the Cambridge Journal of Economics
Authors:
Alberto Baccini,
Federica Baccini,
Lucio Barabesi,
Martina Cioni,
Eugenio Petrovich,
Daria Pignalosa
Abstract:
In order to explore the suitability of a fine-grained classification of journal articles by exploiting multiple sources of information, articles are organized in a two-layer multiplex. The first layer conveys similarities based on the full-text of articles, and the second similarities based on cited references. The information of the two layers are only weakly associated. The Similarity Network Fu…
▽ More
In order to explore the suitability of a fine-grained classification of journal articles by exploiting multiple sources of information, articles are organized in a two-layer multiplex. The first layer conveys similarities based on the full-text of articles, and the second similarities based on cited references. The information of the two layers are only weakly associated. The Similarity Network Fusion process is adopted to combine the two layers into a new single-layer network. A clustering algorithm is applied to the fused network and the classification of articles is obtained. In order to evaluate its coherence, this classification is compared with the ones obtained by applying the same algorithm to each of two layers. Moreover, the classification obtained for the fused network is also compared with the classifications obtained when the layers of information are integrated using different methods available in literature. In the case of the Cambridge Journal of Economics, Similarity Network Fusion appears to be the best option. Moreover, the achieved classification appears to be fine-grained enough to represent the extreme heterogeneity characterizing the contributions published in the journal.
△ Less
Submitted 5 December, 2023; v1 submitted 28 April, 2023;
originally announced May 2023.
-
Who are the gatekeepers of economics? Geographic diversity, gender composition, and interlocking editorship of journal boards
Authors:
Alberto Baccini,
Cristina Re
Abstract:
This study investigates the role of editorial board members as gatekeepers in science, creating and utilizing a database of 1,516 active economics journals in 2019, which includes more than 44,000 scholars from over 6,000 institutions and 142 countries. The composition of these editorial boards is explored in terms of geographic affiliation, institutional affiliation, and gender. Results highlight…
▽ More
This study investigates the role of editorial board members as gatekeepers in science, creating and utilizing a database of 1,516 active economics journals in 2019, which includes more than 44,000 scholars from over 6,000 institutions and 142 countries. The composition of these editorial boards is explored in terms of geographic affiliation, institutional affiliation, and gender. Results highlight that the academic publishing environment is primarily governed by men affiliated with elite universities in the United States. The study further explores social similarities among journals using a network analysis perspective based on interlocking editorship. Comparison of networks generated by all scholars, editorial leaders, and non-editorial leaders reveals significant structural similarities and associations among clusters of journals. These results indicate that links between pairs of journals tend to be redundant, and this can be interpreted in terms of social and intellectual homophily within each board, and between boards of journals belonging to the same cluster. Finally, the analysis of the most central journals and scholars in the networks suggests that journals probably adopt 'strategic decisions' in the selection of the editorial board members. The documented high concentration of editorial power poses a serious risk to innovative research in economics.
△ Less
Submitted 6 January, 2024; v1 submitted 9 April, 2023;
originally announced April 2023.
-
A global exploratory comparison of country self-citations 1996-2019
Authors:
Alberto Baccini,
Eugenio Petrovich
Abstract:
Self-citations are a key topic in evaluative bibliometrics because they can artificially inflate citation-related performance indicators. Recently, self-citations defined at the largest scale, i.e., country self-citations, have started to attract the attention of researchers and policymakers. According to a recent research, in fact, the anomalous trends in the country self-citation rates of some c…
▽ More
Self-citations are a key topic in evaluative bibliometrics because they can artificially inflate citation-related performance indicators. Recently, self-citations defined at the largest scale, i.e., country self-citations, have started to attract the attention of researchers and policymakers. According to a recent research, in fact, the anomalous trends in the country self-citation rates of some countries, such as Italy, have been induced by the distorting effect of citation metrics-centered science policies. In the present study, we investigate the trends of country self-citations in 50 countries over the world in the period 1996-2019 using Scopus data. Results show that for most countries country self-citations have decreased over time. 12 countries (Colombia, Egypt, Indonesia, Iran, Italy, Malaysia, Pakistan, Romania, Russian Federation, Saudi Arabia, Thailand, and Ukraine), however, exhibit different behavior, with anomalous trends of self-citations. We argue that these anomalies should be attributed to the aggressive science policies adopted by these countries in recent years, which are all characterized by direct or indirect incentives for citations. Our analysis confirms that when bibliometric indicators are integrated into systems of incentives, they are capable of affecting rapidly and visibly the citation behavior of entire countries.
△ Less
Submitted 10 November, 2023; v1 submitted 14 March, 2023;
originally announced March 2023.
-
Normative versus strategic accounts of acknowledgment data: the case of the top-five journals of economics
Authors:
Alberto Baccini,
Eugenio Petrovich
Abstract:
Two alternative accounts can be given of the information contained in the acknowledgments of academic publications. According to the mainstream normative account, the acknowledgments serve to repay debts towards formal or informal collaborators. According to the strategic account, by contrast, the acknowledgments serve to increase the perceived quality of papers by associating the authors with inf…
▽ More
Two alternative accounts can be given of the information contained in the acknowledgments of academic publications. According to the mainstream normative account, the acknowledgments serve to repay debts towards formal or informal collaborators. According to the strategic account, by contrast, the acknowledgments serve to increase the perceived quality of papers by associating the authors with influential scholars. The two accounts are assessed by analyzing the acknowledgments indexed in Web of Science of 1218 articles published in the "top-five journals" of economics for the years 2015-2019. The analysis is focused on six dimensions: (i) the style of acknowledging texts, (ii) the distribution of mentions, (iii) the identity of the most mentioned acknowledgees, (iv) the shares of highly and lowly mentioned acknowledgees, (v) the hierarchy of the acknowledgment network, and (vi) the correlation at a paper level between intellectual similarity, measured by common references, and social similarity, measured by common acknowledges. Results show that the normative and the strategic account should be considered as valid but partial explanations of acknowledging behavior. Hence, acknowledgments should be used with extreme caution for investigating collaboration practices and they should not be used to produce acknowledgments-based metrics of scholars for evaluative purposes.
△ Less
Submitted 11 October, 2021; v1 submitted 27 May, 2021;
originally announced May 2021.
-
Similarity network fusion for scholarly journals
Authors:
Federica Baccini,
Lucio Barabesi,
Alberto Baccini,
Mahdi Khelfaoui,
Yves Gingras
Abstract:
This paper explores intellectual and social proximity among scholarly journals by using network fusion techniques. Similarities among journals are initially represented by means of a three-layer network based on co-citations, common authors and common editors. The information contained in the three layers is then combined by building a fused similarity network. The fusion consists in an unsupervis…
▽ More
This paper explores intellectual and social proximity among scholarly journals by using network fusion techniques. Similarities among journals are initially represented by means of a three-layer network based on co-citations, common authors and common editors. The information contained in the three layers is then combined by building a fused similarity network. The fusion consists in an unsupervised process that exploits the structural properties of the layers. Subsequently, partial distance correlations are adopted for measuring the contribution of each layer to the structure of the fused network. Finally, the community morphology of the fused network is explored by using modularity. In the three fields considered (i.e. economics, information and library sciences and statistics) the major contribution to the structure of the fused network arises from editors. This result suggests that the role of editors as gatekeepers of journals is the most relevant in defining the boundaries of scholarly communities. In information and library sciences and statistics, the clusters of journals reflect sub-field specializations. In economics, clusters of journals appear to be better interpreted in terms of alternative methodological approaches. Thus, the graphs representing the clusters of journals in the fused network are powerful instruments for exploring research fields.
△ Less
Submitted 19 November, 2021; v1 submitted 13 November, 2020;
originally announced November 2020.
-
Intellectual and social similarity among scholarly journals: an exploratory comparison of the networks of editors, authors and co-citations
Authors:
Alberto Baccini,
Lucio Barabesi,
Mahdi Khelfaoui,
Yves Gingras
Abstract:
This paper explores, by using suitable quantitative techniques, to what extent the intellectual proximity among scholarly journals is also a proximity in terms of social communities gathered around the journals. Three fields are considered: statistics, economics and information and library sciences. Co-citation networks (CC) represent the intellectual proximity among journals. The academic communi…
▽ More
This paper explores, by using suitable quantitative techniques, to what extent the intellectual proximity among scholarly journals is also a proximity in terms of social communities gathered around the journals. Three fields are considered: statistics, economics and information and library sciences. Co-citation networks (CC) represent the intellectual proximity among journals. The academic communities around the journals are represented by considering the networks of journals generated by authors writing in more than one journal (interlocking authorship: IA), and the networks generated by scholars sitting in the editorial board of more than one journal (interlocking editorship: IE). For comparing the whole structure of the networks, the dissimilarity matrices are considered. The CC, IE and IA networks appear to be correlated for the three fields. The strongest correlations is between CC and IA for the three fields. Lower and similar correlations are obtained for CC and IE, and for IE and IA. The CC, IE and IA networks are then partitioned in communities. Information and library sciences is the field where communities are more easily detectable, while the most difficult field is economics. The degrees of association among the detected communities show that they are not independent. For all the fields, the strongest association is between CC and IA networks; the minimum level of association is between IE and CC. Overall, these results indicate that the intellectual proximity is also a proximity among authors and among editors of the journals. Thus, the three maps of editorial power, intellectual proximity and authors communities tell similar stories.
△ Less
Submitted 2 September, 2019; v1 submitted 24 August, 2019;
originally announced August 2019.
-
Citation gaming induced by bibliometric evaluation: a country-level comparative analysis
Authors:
Alberto Baccini,
Giuseppe De Nicolao,
Eugenio Petrovich
Abstract:
It is several years since national research evaluation systems around the globe started making use of quantitative indicators to measure the performance of researchers. Nevertheless, the effects on these systems on the behavior of the evaluated researchers are still largely unknown. We attempt to shed light on this topic by investigating how Italian researchers reacted to the introduction in 2011…
▽ More
It is several years since national research evaluation systems around the globe started making use of quantitative indicators to measure the performance of researchers. Nevertheless, the effects on these systems on the behavior of the evaluated researchers are still largely unknown. We attempt to shed light on this topic by investigating how Italian researchers reacted to the introduction in 2011 of national regulations in which key passages of professional careers are governed by bibliometric indicators. A new inwardness measure, able to gauge the degree of scientific self-referentiality of a country, is defined as the proportion of citations coming from the country itself compared to the total number of citations gathered by the country. Compared to the trends of the other G10 countries in the period 2000-2016, Italy's inwardness shows a net increase after the introduction of the new evaluation rules. Indeed, globally and also for a large majority of the research fields, Italy became the European country with the highest inwardness. Possible explanations are proposed and discussed, concluding that the observed trends are strongly suggestive of a generalized strategic use of citations, both in the form of author self-citations and of citation clubs. We argue that the Italian case offers crucial insights on the constitutive effects of evaluation systems. As such, it could become a paradigmatic case in the debate about the use of indicators in science-policy contexts.
△ Less
Submitted 9 April, 2019;
originally announced April 2019.
-
Errors and secret data in the Italian research assessment exercise. A comment to a reply
Authors:
Alberto Baccini,
Giuseppe De Nicolao
Abstract:
Italy adopted a performance-based system for funding universities that is centered on the results of a national research assessment exercise, realized by a governmental agency (ANVUR). ANVUR evaluated papers by using 'a dual system of evaluation', that is by informed peer review or by bibliometrics. In view of validating that system, ANVUR performed an experiment for estimating the agreement betwe…
▽ More
Italy adopted a performance-based system for funding universities that is centered on the results of a national research assessment exercise, realized by a governmental agency (ANVUR). ANVUR evaluated papers by using 'a dual system of evaluation', that is by informed peer review or by bibliometrics. In view of validating that system, ANVUR performed an experiment for estimating the agreement between informed review and bibliometrics. Ancaiani et al. (2015) presents the main results of the experiment. Baccini and De Nicolao (2017) documented in a letter, among other critical issues, that the statistical analysis was not realized on a random sample of articles. A reply to the letter has been published by Research Evaluation (Benedetto et al. 2017). This note highlights that in the reply there are (1) errors in data, (2) problems with 'representativeness' of the sample, (3) unverifiable claims about weights used for calculating kappas, (4) undisclosed averaging procedures; (5) a statement about 'same protocol in all areas' contradicted by official reports. Last but not least: the data used by the authors continue to be undisclosed. A general warning concludes: many recently published papers use data originating from Italian research assessment exercise. These data are not accessible to the scientific community and consequently these papers are not reproducible. They can be hardly considered as containing sound evidence at least until authors or ANVUR disclose the data necessary for replication.
△ Less
Submitted 21 July, 2017;
originally announced July 2017.
-
Do they agree? Bibliometric evaluation vs informed peer review in the Italian research assessment exercise
Authors:
Alberto Baccini,
Giuseppe De Nicolao
Abstract:
During the Italian research assessment exercise, the national agency ANVUR performed an experiment to assess agreement between grades attributed to journal articles by informed peer review (IR) and by bibliometrics. A sample of articles was evaluated by using both methods and agreement was analyzed by weighted Cohen's kappas. ANVUR presented results as indicating an overall 'good' or 'more than ad…
▽ More
During the Italian research assessment exercise, the national agency ANVUR performed an experiment to assess agreement between grades attributed to journal articles by informed peer review (IR) and by bibliometrics. A sample of articles was evaluated by using both methods and agreement was analyzed by weighted Cohen's kappas. ANVUR presented results as indicating an overall 'good' or 'more than adequate' agreement. This paper re-examines the experiment results according to the available statistical guidelines for interpreting kappa values, by showing that the degree of agreement, always in the range 0.09-0.42 has to be interpreted, for all research fields, as unacceptable, poor or, in a few cases, as, at most, fair. The only notable exception, confirmed also by a statistical meta-analysis, was a moderate agreement for economics and statistics (Area 13) and its sub-fields. We show that the experiment protocol adopted in Area 13 was substantially modified with respect to all the other research fields, to the point that results for economics and statistics have to be considered as fatally flawed. The evidence of a poor agreement supports the conclusion that IR and bibliometrics do not produce similar results, and that the adoption of both methods in the Italian research assessment possibly introduced systematic and unknown biases in its final results. The conclusion reached by ANVUR must be reversed: the available evidence does not justify at all the joint use of IR and bibliometrics within the same research assessment exercise.
△ Less
Submitted 24 March, 2016; v1 submitted 1 May, 2015;
originally announced May 2015.
-
Crossing the hurdle: the determinants of individual scientific performance
Authors:
Alberto Baccini,
Lucio Barabesi,
Martina Cioni,
Caterina Pisani
Abstract:
An original cross sectional dataset referring to a medium sized Italian university is implemented in order to analyze the determinants of scientific research production at individual level. The dataset includes 942 permanent researchers of various scientific sectors for a three year time span (2008 - 2010). Three different indicators - based on the number of publications or citations - are conside…
▽ More
An original cross sectional dataset referring to a medium sized Italian university is implemented in order to analyze the determinants of scientific research production at individual level. The dataset includes 942 permanent researchers of various scientific sectors for a three year time span (2008 - 2010). Three different indicators - based on the number of publications or citations - are considered as response variables. The corresponding distributions are highly skewed and display an excess of zero - valued observations. In this setting, the goodness of fit of several Poisson mixture regression models are explored by assuming an extensive set of explanatory variables. As to the personal observable characteristics of the researchers, the results emphasize the age effect and the gender productivity gap, as previously documented by existing studies. Analogously, the analysis confirm that productivity is strongly affected by the publication and citation practices adopted in different scientific disciplines. The empirical evidence on the connection between teaching and research activities suggests that no univocal substitution or complementarity thesis can be claimed: a major teaching load does not affect the odds to be a non-active researcher and does not significantly reduce the number of publications for active researchers. In addition, new evidence emerges on the effect of researchers administrative tasks, which seem to be negatively related with researcher's productivity, and on the composition of departments. Researchers' productivity is apparently enhanced by operating in department filled with more administrative and technical staff, and it is not significantly affected by the composition of the department in terms of senior or junior researchers.
△ Less
Submitted 27 July, 2014; v1 submitted 30 November, 2013;
originally announced December 2013.
-
Statistical inference on the h-index with an application to top-scientist performance
Authors:
Alberto Baccini,
Lucio Barabesi,
Marzia Marcheselli,
Luca Pratelli
Abstract:
Despite the huge amount of literature on h-index, few papers have been devoted to the statistical analysis of h-index when a probabilistic distribution is assumed for citation counts. The present contribution relies on showing the available inferential techniques, by providing the details for proper point and set estimation of the theoretical h-index. Moreover, some issues on simultaneous inferenc…
▽ More
Despite the huge amount of literature on h-index, few papers have been devoted to the statistical analysis of h-index when a probabilistic distribution is assumed for citation counts. The present contribution relies on showing the available inferential techniques, by providing the details for proper point and set estimation of the theoretical h-index. Moreover, some issues on simultaneous inference - aimed to produce suitable scholar comparisons - are carried out. Finally, the analysis of the citation dataset for the Nobel Laureates (in the last five years) and for the Fields medallists (from 2002 onward) is proposed.
△ Less
Submitted 20 May, 2012;
originally announced May 2012.
-
Statistical analysis of the Hirsch Index
Authors:
Luca Pratelli,
Alberto Baccini,
Lucio Barabesi,
Marzia Marcheselli
Abstract:
The Hirsch index (commonly referred to as h-index) is a bibliometric indicator which is widely recognized as effective for measuring the scientific production of a scholar since it summarizes size and impact of the research output. In a formal setting, the h-index is actually an empirical functional of the distribution of the citation counts received by the scholar. Under this approach, the asympt…
▽ More
The Hirsch index (commonly referred to as h-index) is a bibliometric indicator which is widely recognized as effective for measuring the scientific production of a scholar since it summarizes size and impact of the research output. In a formal setting, the h-index is actually an empirical functional of the distribution of the citation counts received by the scholar. Under this approach, the asymptotic theory for the empirical h-index has been recently exploited when the citation counts follow a continuous distribution and, in particular, variance estimation has been considered for the Pareto-type and the Weibull-type distribution families. However, in bibliometric applications, citation counts display a distribution supported by the integers. Thus, we provide general properties for the empirical h-index under the small- and large-sample settings. In addition, we also introduce consistent nonparametric variance estimation, which allows for the implemention of large-sample set estimation for the theoretical h-index.
△ Less
Submitted 14 February, 2011;
originally announced February 2011.
-
Interlocking editorship. A network analysis of the links between economic journals
Authors:
Alberto Baccini,
Lucio Barabesi
Abstract:
The exploratory analysis developed in this paper relies on the hypothesis that each editor possesses some power in the definition of the editorial policy of her journal. Consequently if the same scholar sits on the board of editors of two journals, those journals could have some common elements in their editorial policies. The proximity of the editorial policies of two scientific journals can be a…
▽ More
The exploratory analysis developed in this paper relies on the hypothesis that each editor possesses some power in the definition of the editorial policy of her journal. Consequently if the same scholar sits on the board of editors of two journals, those journals could have some common elements in their editorial policies. The proximity of the editorial policies of two scientific journals can be assessed by the number of common editors sitting on their boards. A database of all editors of ECONLIT journals is used. The structure of the network generated by interlocking editorship is explored by applying the instruments of network analysis. Evidences have been found of a compact network containing different components. This is interpreted as the result of a plurality of perspectives about the appropriate methods for the investigation of problems and the construction of theories within the domain of economics.
△ Less
Submitted 6 February, 2011;
originally announced February 2011.
