Poster Session

How 'Free' is Free Speech in Academia? Effects on Researchers and Their Research

Kris Gulati

ABSTRACT. In 2019, a math professor called Abigail Thompson published an op-ed in the Wall Street Journal comparing mandatory diversity statements for job applicants to mandatory loyalty oaths that her university required seventy years earlier. The reaction was immediate. Hundreds of colleagues from across the country made or signed statements calling her view “dangerous” and petitioning her employer to investigate whether her behavior was consistent with their “institutional policy”. In 2020, another math professor, Andrea Bertozzi, faced penalties for her work on “predictive policing.” An invited lecture to honor Bertozzi’s “contribution to the mathematical sciences” was cancelled because the organizers “do not believe that mathematicians should be collaborating with police departments” (Castelvecchi, 2020). Social sanctions for both ‘extramural’ speech, like that of Thompson, and ‘academic’ speech, like that of Bertozzi, are increasingly common. Some evidence suggests that these social sanctions may have consequences for academic speech and science more broadly. Frey and Stevens (2023) find that the annual number of attempts to suppress or punish scholars’ speech in the U.S. has increased dramatically over time. A recent survey of university faculty showed that 40% of liberal faculty, 56% of moderate faculty, and 72% of conservative faculty fear losing their jobs or reputations for something they say aloud or post online (Honeycutt et al., 2022). A 2023 survey of academics finds that these same kinds of repercussions affect the kinds of research scholars choose to pursue, potentially altering the direction of scientific inquiry (F. S. Union Tech. Rep., 2023). Despite the increasing prevalence of various sanctions against scholars, there has been little research to understand how these incidents may affect the academic enterprise. In this paper, we investigate whether targeted reputation attacks on scholars reduce the scholar’s recognition in their field and reduce the scholar’s productivity in their area of expertise. Ex ante, it is unclear whether researchers’ professional outcomes would be affected at all, and if they are affected—the effect could plausibly be positive or negative. Affected scholars, for instance, may benefit from additional public attention. Media coverage, a condition for inclusion in our data on incidents, has been positively associated with scholars’ professional success (Philips et al., 1991). Finally, negative reviews can increase sales of products (Berger et al., 2010), which may have parallels to academics receiving criticism, but that draws additional media attention. On the other hand, researchers could face professional penalties and ostracism as the result of an incident. We leverage a new dataset, ‘Scholars Under Fire’, collected by the Foundation for Individual Rights and Expression (FIRE). FIRE is a non-partisan, non-profit organization that defends and promotes free expression in the United States, particularly on university campuses. FIRE gathers information on incidents involving scholars in the U.S. who have received calls for sanction because of their protected speech. The organization collects these records by identifying incidents through news reports. Using a difference-in-difference (DiD) approach, we compare the citations of the work published by the affected scholars before the incidents to a control group of papers published in the same journal in the same year as the affected author’s work. Using this design, we find that the work of affected scholars receives approximately 4 percent fewer citations per year as a result of a “cancellation” incident. Our finding is consistent with the hypothesis that affected scholars are penalized somewhat by the scientific community following their involvement in an incident. We then investigate the effect of incidents on scholars’ future publications and the citations of future work. Compared to a control group of scholars with similar pre-incident characteristics, we find that affected researchers publish 19.5 percent less after an incident and their future work receives 14.5 percent fewer citations. We contribute to the literature in several ways. First, The political-science and legal-rights literatures have long histories of studying free speech in the academy. Researchers studying science and innovation are interested in how speech and controversy affect or distort the scientific enterprise. To date, existing work has largely relied on surveys. We make use of the modern applied economists’ toolkits and look at tangible outcomes on scholars’ careers. Second, we contribute to the theory of ‘Mertonian norms’ (Merton, 1973). Prior work on retractions (Azoulay et al., 2015) and sexual harassment (Widmann et al., 2023) find violations of Mertonian norms in the academy. However, retractions and sexual harassment are both clear examples of professional misconduct. Our empirical setting allows us to test this theory for speech that is usually not an example of professional misconduct. References Azoulay, P., Furman, J.L. and Murray, F., 2015. Retractions. Review of Economics and Statistics, 97(5), pp.1118-1136. “Academic freedom survey” (F. S. Union Tech. Rep., 2023) Berger, J., Sorensen, A.T. and Rasmussen, S.J., 2010. Positive effects of negative publicity: When negative reviews increase sales. Marketing science, 29(5), pp.815-827.Castelvecchi, Davide. 2020. Mathematicians urge colleagues to boycott police work in wake of killings. Nature News, June 19 Frey, K. & Stevens, S.T. (2023). Scholars under fire: Attempts to sanction scholars from 2000 to 2022. The Foundation for Individual Rights and Expression. Merton, Robert K., 1973. The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University of Chicago Press. Phillips, D.P., Kanter, E.J., Bednarczyk, B. and Tastad, P.L., 1991. Importance of the lay press in the transmission of medical knowledge to the scientific community. New England journal of medicine, 325(16), pp.1180-1183. Widmann, R., Rose, M.E. and Chugunova, M., 2022. Allegations of Sexual Misconduct, Accused Scientists, and Their Research. Max Planck Institute for Innovation Competition Research Paper, (22-18)

The Effect of Credit Shocks on Sustained Team Collaboration in Patenting

Kyoungah Noh

ABSTRACT. The exponential growth of knowledge presents a paradox: as the collective reservoir expands, knowledge workers increasingly struggle to stay abreast of the latest developments. This predicament, known as the “burden of knowledge”, implies that knowledge workers may disproportionally rely on established concepts over time, particularly as they progress in their careers, which can stifle innovation due to the "knowledge lock-in". Moreover, the reliance on reputation-based signals such as citation counts, reinforced by the Matthew effect, may hinder knowledge workers’ effective engagement with emerging innovation. Recognizing the importance of assimilating new ideas for innovation arises the pivotal question this research seeks to address: How knowledge workers effectively absorb new innovation? Employing a quasi-experimental design, this paper examines how brief, unstructured exposure to new information can drive substantial knowledge acquisition among knowledge workers. This exploration amplifies the power of transient attention in shaping innovation trajectories and provides valuable insights into effective strategies to expedite knowledge absorption.

Charting the course of artificial intelligence policy: formulation, trends and scientific foundations

Bernardo Cabral

ABSTRACT. Artificial Intelligence (AI) encompasses many technologies, including deep learning, machine learning, natural language processing, neural networks, and rule-based systems (Lu, 2019). As the evolution of AI accelerates, it plays an increasing role in scientific research and technological innovation (Liu et al., 2020). The global emergence of AI as a relevant technological force has reshaped how policymakers and stakeholders perceive technological advancements (Lauterbach, 2019). In response to AI's global impact, AI policy has garnered international attention and emerged as a critical component of technology policy, given its potential impacts and its range of applications (Agrawal et al., 2019).

This research delves into the intricate evolution of AI policies over the past decade, particularly focusing on the scientific underpinnings that inform these policies. By leveraging the extensive Overton policy documents database, we mapped the contributions from various funders, organizations, countries, and journals to the policy AI discourse, offering a novel insight into how scientific knowledge influences policy formulation in the realm of AI. The study bridges the gap between AI technological advancements and policy development, highlighting the significant yet complex role scientific research plays in policymaking (Bozeman & Youtie, 2017; Gunn & Mintrom, 2021). It underscores the dynamic interplay between the rapid advancements in AI technologies and their reflections in the legislative and governance frameworks across different nations and regions.

To investigate the evolution of AI policies and the scientific literature influencing these policies, our study employed a comprehensive methodological approach leveraging the Overton policy documents database, which contains over 10 million policy documents from a wide array of sources, including governments, intergovernmental organizations, think tanks, and non-profit organizations. Our analysis began with a targeted search within the Overton database for policy documents containing key AI-related terms. Following the initial search, the results were exported to VantagePoint software for advanced processing and analysis. To deepen our analysis of the scientific foundation of AI policies, we extracted the Digital Object Identifiers (DOIs) of scholarly articles cited within these policy documents. A subsequent search for these DOIs was conducted across the Web of Science (WoS) database to extract comprehensive metadata for these articles.

Our findings reveal a marked increase in AI policy documents post-2018, underscoring a global acknowledgment of both the potential and challenges presented by AI technologies. It also indicates that the United States and the European Union are at the forefront of AI policy document production, reflecting their historical and ongoing contributions to AI innovation and governance. Additionally, China's substantial role in the scientific research cited within 3rd International Conference of Science of Science & Innovation, July 1-3, 2024 National Academy of Sciences, Washington, DC, USA these policy documents highlights its growing influence in the AI domain despite a lower representation in policy documentation compared to its scientific output.

Our examination further identified key funders, organizations, and countries pivotal to the cited scientific research within these policy documents. The United States, for instance, demonstrates a comprehensive funding landscape for AI research through various institutions, highlighting AI's designation as a General Purpose Technology. In contrast, the high funding output from China aligns with its strategic ambitions in AI, underscoring a complex interplay of national priorities and global influence. The analysis also underscored the significant impact of scientific journals in shaping AI policy discourse. Prestigious journals such as Nature, Science, the New England Journal of Medicine, and The Lancet were among the most cited, indicating a preference for high-impact factor research in policy formulation. This preference suggests that policy documents may benefit from incorporating research published in journals with rigorous peer review and broad recognition, though it also prompts consideration of the broader spectrum of scientific work contributing to policy discussions.

Collectively, these findings offer a nuanced understanding of the dynamics at play in the evolution of AI policies. They reveal not only the geographical and institutional sources of influential research but also the thematic focus areas and ethical considerations shaping global AI policy. This discussion contributes to the ongoing dialogue on how best to harness AI's potential while navigating its ethical, societal, and governance challenges.

This research opens avenues for further exploration into the dynamic relationship between science and policy, emphasizing the need for evidence-based policymaking that leverages the full potential of AI technologies while addressing their ethical, societal, and governance challenges. It also lays the groundwork for using the same methodology for other relevant policy topics and understanding their scientific underpinnings. Additionally, we also acknowledge some limitations, including potential biases in the Overton database and the challenges of capturing the full spectrum of influences on policy development.

Intersectionality’s Publication Dilemmas: Innovation, Influence, and Inequality

Claire Daviss

ABSTRACT. Research has shown that groups who have been historically underrepresented in academic research innovate at higher rates than majority groups members, but their contributions often go unrecognized and unrewarded in their disciplines (Hofstra et al. 2020). In this study, we investigate whether this pattern is reflected in the takeup of intersectionality as a theoretical and analytical framework in sociology. Intersectionality, initially developed by Black feminist scholars, has been described as one of the most significant innovations in sociological scholarship in recent decades (Cooper 2015). As defined by Collins (2015), intersectionality is the critical insight that multiple axes of status and power, including gender, race, sexuality, class, and age, are “reciprocally reconstructing phenomena that in turn shape complex social inequalities.” The concept, theory, and methodology of intersectionality offer important theoretical and empirical contributions to the study of inequality and social stratification, as evidenced by decades of scholarship (Carbado et al. 2013, Cho et al. 2013, Collins 2015). Yet there is reason to wonder whether sociology has fully embraced intersectionality. Early analyses of how far intersectionality has traveled suggested that, even a decade after its introduction to academic literature, the theory remained largely concentrated in specialty journals focused on gender and race (Jones et al. 2013). We expand on these analyses by measuring the extent to which intersectionality, as a key innovation in the study of inequality and stratification, has proliferated through sociology’s academic journals from its entrance into the academic literature up to the present. We track the use of intersectionality in journals, since a main purpose of academic journals is to publish theoretical and empirical innovations. We perform a descriptive analysis of four sociology journals: two of the discipline’s heritage journals, American Journal of Sociology (AJS) and American Sociological Review (ASR), and two leading specialist journals focused on gender and race, Gender & Society (G&S) and the Du Bois Review (DBR). We examine the full set of articles published in these journals from 1990 to 2023, a sample of more than 4000 articles, as well as a subset of these articles that are specifically focused on gender and race. We document the appearance of the term “intersectionality” and its derivatives in articles’ main text, abstracts, and keywords. These sections are likely to reflect differences in the readership’s familiarity with and endorsement of key concepts. Intersectionality in abstracts or keywords signals greater audience familiarity, since authors have limited words to explain concepts in abstracts and since keywords are used to help audiences find articles on topics of interest. In our analyses, we find a consistent pattern: intersectionality has been embraced by specialty journals focused on gender and race, but has largely remained outside of highprestige generalist journals, even in these journals’ gender and race-related articles. As previewed in Figure 1, intersectionality has been most heavily represented over time in the research articles published in G&S. G&S began to publish articles that mentioned intersectionality in the main text in the early 1990s, and in recent years, more than 50 percent of its articles mention intersectionality. In the early 2010s, the percent of DBR articles that mentioned intersectionality jumped, driven in large part by a 2013 special issue focused on 3rd International Conference of Science of Science & Innovation, July 1-3, 2024 National Academy of Sciences, Washington, DC, USA the topic (Carbado et al. 2013). In comparison, the proportion of gender-or race-related articles mentioning intersectionality in AJS and ASR has consistently lagged behind that of G&S and DBR. These patterns are especially noticeable in analyses of abstracts and keywords. In additional analyses (not shown), we demonstrate that these specialty journal articles not only mention intersectionality earlier than heritage journal articles focused on gender and race; they also engaged intersectionality theory more deeply. We are also currently collecting data that will allow us to document the demographic makeup of intersectionality scholars from 1990-2023.

The failure to adopt an intersectional framework in these high-prestige journals potentially distorts the field’s understandings of gender, race, class, and other forms of stratification and inequality (Choo and Ferree 2010). Additionally, if intersectionality research is shut out of the discipline’s highest status journals, intersectionality scholars, who are disproportionality women and people of color, may face difficulties obtaining career advancing opportunities, including grants, academic positions, tenure, and more. Until the field develops a deeper understanding of and appreciation for this innovation, intersectionality scholars are likely to face persistent publication dilemmas and the scientific community will not fully understand the patterns and production of social inequality.

Exploring the Causes for Retracted Research in Chemistry and Related Fields

Yulia Sevryugina

ABSTRACT. The scope of this study is to provide an overview to retractions in chemistry and related fields. By examining the patterns and causes behind the manuscripts retractions, we aim to highlight the strategies that can encourage adherence to best publication practices moving forward. Moreover, by studying the diagnosed misconduct through retraction notices we aim to cultivate a culture where scholars are better equipped to recognize fraud as well as to acknowledge and correct unintentional errors in their own works. For this study, we analyzed a total of 1,292 research and review articles retracted between 1 June 2001 and 23 December 2021. Retraction data were provided by the Retraction Watch(RW) database on May 24, 2022.1 Manuscripts were selected based on containing chemistry as a subject. However, only 19% (N = 245) of retracted manuscripts were indexed with a single subject. Remaining manuscripts were sorted by subjects into nine subject groups: 1) Physics & Energy – also includes mathematics and cosmology (N = 278); 2) Crystallography – also includes spectroscopy (N = 270); 3) Materials Science (N = 266); 4) Engineering - also includes chemical engineering, computer science and technology (N =244); 5) Biochemistry – also includes forensics and biology (N = 143); 6) Environmental Sciences – also includes climatology and archaeology (N = 96); 7) Nanotechnology (N = 92); 8) Medicine - also includes neuroscience, public health, and occupational health and safety (N = 63); and 9) Social Sciences – also includes business, art, education, and humanities (N = 13).

In addition to subjects, each manuscript in the RW database is indexed with a combination of retraction reasons (>100),2 which we grouped into five reason groups: 1) Misconduct (N = 756) - falsification or fabrication of data, image, results (fraud); plagiarism of article, image, text, data, concerns/issues with referencing/attribution; duplication of article, image, text, data; and copyright claims. 2) Errors & Concerns (N = 338) - error in data, analyses, methods, results, text, image, unreliable/irreproducible data/results, data not provided. 3) Author issues (N = 72) - concerns or issues about authorship, false or forged authorship or affiliation, author unresponsive, miscommunication issues, ethical violations by author, conflict of interest. 4) Publisher issues (N = 64) - fake peer review, duplicate publication, error by journal/publisher, rogue editor. 5) Organization issues (N = 11) - objection, lack of approval, error/ethical violation, concerns/issues about involvement (all by a third party or company/institution).

Based on the abovementioned classification, our analysis of retracted chemistry manuscripts showed the prevalence of misconduct; and most notably, plagiarism (Fig. 1). Of all subjects, plagiarism was more frequently observed in materials science and physics papers. Interestingly, plagiarizing someone else’s work is less common (N = 210) than self-plagiarism (N = 306); the latter is defined as duplication of data, image, text, or article. Article duplications represented 37% of all plagiarism cases in our dataset (173 out of 468). The majority of fraud cases were associated with crystallography; namely, publications in the Acta Crystallogr., Sect. E (N = 135). As compared to interdisciplinary manuscripts, those with a single subject of chemistry have a higher fraction of author issues and errors unrelated to misconduct (Fig. 2).

The scientific foundations of Brazilian health policy: an analysis of clinical protocols and therapeutic guidelines

Bernardo Cabral

ABSTRACT. Integrating scientific research into public policy is essential to modern governance, particularly in a world with complex challenges. In healthcare, evidence-based decision-making is paramount (Baicker & Chandra, 2017). It ensures that policies are not only grounded in the best available knowledge but are also attuned to evolving healthcare needs (Gabbay et al., 2020). Brazil's National Commission for the Incorporation of Technologies into the Unified Health System (CONITEC) serves as a prime example of this approach.

By leveraging Clinical Protocols and Therapeutic Guidelines (PCDTs, in its Portuguese acronym), CONITEC standardizes and enhances healthcare practices, embodying a model of health policy formulation that is increasingly recognized as vital for effective and evidencebased healthcare governance (Wang et al., 2020). CONITEC's role in guiding Brazil's healthcare policies through PCDTs mirrors a global trend in health governance, where scientific research directly informs public health measures (Vicente et al., 2021).

In healthcare, utilizing scientific literature within clinical guidelines brings to light important questions about the diversity and relevance of research findings (Maaløe et al., 2021). The geographic and institutional origins of scientific research can greatly influence the nature and suitability of healthcare guidelines (Olayemi et al., 2017). For instance, research from highincome countries might not always address the health challenges of low- or middle-income countries (Orangi et al., 2023).

To contribute to the discussion between scientific evidence and public policies, this study undertakes a comprehensive bibliometric analysis of the scientific literature cited in PCDTs since the CONITEC’s inception. The analysis provides insights into how local and global scientific contributions are incorporated into national health policies. It also underscores the increasing demand for transparency and accountability in how healthcare policies are shaped by scientific research.

Employing a comprehensive data pipeline, this study collects, preprocesses, and analyzes the scientific publications cited in the PCDTs approved by the CONITEC. This methodological framework incorporates techniques such as web scraping, pattern recognition, and bibliometric analysis, enabling a thorough examination of the scientific literature's genesis and its integration into health policy. By extracting and transforming bibliographic data from digital PCDT reports for database querying, and conducting extensive searches in renowned databases like Scopus and Web of Science, the study offers a novel approach to understanding the linkage between scientific evidence and policy-making. The analysis reveals a predominant reliance on research originating from the United States and the United Kingdom, despite notable contributions from Brazilian sources. This indicates a significant global influence on Brazil's health policy, with international research playing a crucial role in shaping national healthcare practices. Moreover, the study identifies major health research institutions and pharmaceutical companies, primarily from outside Brazil, as key funding sources for the referenced scientific literature. These findings underscore the global interconnectedness of health research and its implications for national health policy frameworks.

This research underscores the critical interplay between global scientific research and the formulation of health policy in Brazil, highlighting the substantial impact of international scientific literature on the country's healthcare guidelines. By mapping out the flow of scientific knowledge into health policy, the study not only contributes to a deeper understanding of the policy-making process but also emphasizes the need for a more diversified and inclusive approach to incorporating scientific evidence into health policy. In light of these insights, policymakers are encouraged to consider a broader spectrum of scientific research, including local and regional studies, to enhance the relevance and effectiveness of health policies in Brazil.

Decoupling in US-China scientific collaboration

Li Tang

ABSTRACT. This study reveals that, following bilateral reduced international visitation and academic exchange, Sino-American scientific collaboration is positioned at a turning point in a declining course. American international students originating from China have declined by nearly 22%, and American students studying in China plummeted to 1.8% of the number in 2018–2019. US-China mutuality in scientific collaboration has also declined remarkably. At the same time, the concentration of influential research collaborated between the United States and China is consistently greater than both nations’ research outputs. Following the discussion of possible substitutes and the roles of American and Chinese researchers in global basic science and emerging issues, I argue that the two nations are so entwined in scientific collaboration that an adversarial rivalry perspective misses much of reality. In the face of rising uncertainties and global disasters, humanity does not have time to waste on nationalistic competitions. It is time for visionary leadership from both countries to promote intellectual exchange and scientific collaboration to address pressing global challenges.

Reproducibility, Replicability, and Transparency in Research: What 430 Professors Think in Universities across the USA and India

Tatiana Chakravorti and Sai Koneru

ABSTRACT. Extended Abstract Reproducibility and replicability have gained significant attention in scientific discourse, deeply intertwined with questions about scientific processes, policies, and incentives [1, 2]. Initially centered around the social and behavioral sciences, these concerns now span almost all empirical scientific disciplines [3], including artificial intelligence and machine learning [4]. The open science and science of science communities have responded with innovative initiatives aimed at shoring up the entire research workflow, from conception and study design to data collection and analysis, through to publishing [5]. These efforts have already had important individual and institutional impacts, many of which have been well documented [6]. For example, the Special Interest Group on Computer-Human Interaction (SIGCHI) now recommends providing supplementary materials for ACM publications to enhance replicability [7] and some universities have begun to reward researchers whose work aligns with standards of open science and transparency [8].

Despite these promising advances, however, conversations around reproducibility and replicability have predominantly reflected the voices of researchers in the global North and West [9]. This is concerning for several reasons, most primarily because issues of scientific integrity and scientific process are deeply social and contextual. Our work takes an initial step toward the inclusion of cultural perspectives through a comparative study of researchers in the USA and India. India currently ranks third in research output worldwide, following China and the USA. We conducted a survey-based study involving research faculty from universities in the USA and India. We aimed to gather the perspectives of scientists across different research disciplines. The survey asked participants about their familiarity with the reproducibility crisis, their confidence in work published within their fields, and the factors they believe contribute to this high or low-confidence research. Additionally, we asked participants to share the institutional and practical challenges they faced during their research. We reached out to over 8000 research faculty members and received a total of 430 responses.

Our study offers an in-depth analysis of researchers' views on reproducibility, replicability, and open science practices in the USA and India, casting light on both commonalities and disparities. While Western nations have more proactively tackled the replication crisis, Indian researchers are becoming more conscious of these issues and are making strides toward embracing open science. Yet, our findings indicate that its adoption faces hurdles in both countries and across fields. Respondents across contexts highlight the misalignment of prevailing academic incentives, centered around publications, promotions, and funding, as detrimental to engagement with open science practices. They note a lack of appreciation for replication studies in favor of novelty. We note that new incentives have emerged and should be used as stepping stones for substantial extensions. A notable example is the impact of badges. These simple rewards have a real, measurable impact on engagement with open science practices [10]. The next steps might include the inclusion of badges into reputational metrics and promotional practices. Journals and conferences might consider dedicated opportunities to publish replication studies. Discussions of reproducibility and open science centered in the West have not fully appreciated the challenges faced by researchers working at institutions with fewer resources and less social capital. For example, when attempting to reproduce or replicate a published finding, respondents in India believe that getting responses from the paper's authors in the West is more challenging for them.

Our work confirms shortcomings in researchers' approaches to the assessment of confidence in published work. The author's reputation was noted as a meaningful signal of credibility in both countries and across domains. Current ways for researchers to accumulate good or bad reputations, though, are driven by available existing metrics. The ways in which biases with respect to author reputation may be worsened or mitigated by open science practices, e.g., open peer review, is unclear.

Human Capital and Firm’s Innovation Direction

Xizhao Wang

ABSTRACT. Talent is a scarce resource, and when it comes to innovation, only a few exceptional individuals, like Marie Curie and Isaac Newton, possess the extraordinary potential to revolutionize our way of life ([Akcigit et al., 2020]). For organizations, the talent shortage is a critical problem, as highlighted in the recent Mckinsey Report ([Durth et al., 2023]), which states that ”The talent shortage is a critical problem that is only getting worse. Organizations already face a severe shortage of key talent, and 90 percent say they will have a meaningful skills gap in the coming years.”

Human capital plays a pivotal role in driving a firm’s innovation direction. Dr.Spencer Silver, a chemist at 3M, discovered a unique adhesive with a weak bond but strong sticking properties. Capitalizing on this invention, Silver’s colleague, Art Fry, created the iconic Post-it Notes. Silver’s adhesive innovation ultimately reshaped 3M’s innovation trajectory from purely industrial solutions to accessible and user-friendly products that catered to everyday needs. It paved the way for innovative problem-solving, collaboration, and note-taking, instigating transformative shifts in 3M’s innovation directions ([3M, 2023]).

Motivated by the scarcity of talents in organizations and instances where influential indi- viduals significantly impact a organization’s innovation trajectory, this study aims to uncover how losing talented individuals affect the firm’s innovation direction and explore the underly- ing mechanisms. By leveraging the unexpected deaths of individuals as exogenous events for firms, I utilize a difference-in-difference methodology to examine shifts in innovation direction among firms. This involves comparing firms affected by these events with unaffected coun- terparts before and after the occurrence of the deaths. I aim to address key questions: How does the sudden loss of talent affect the firm’s innovation direction? Do senior executives and inventors drive a firm’s innovation direction? And in what ways do they shape the course of innovation within the firm? This paper constructs a comprehensive micro-level sample dataset by utilizing firm data, employee data, and patent data. Additionally, three different measures are introduced to effec- tively quantify changes in the organization’s innovation direction. Previous studies focus on firm’s innovation direction towards some specific technologies ([Acemoglu and Linn, 2004]; [Acemoglu and Finkelstein, 2008]; [Hanlon, 2015]; [Aghion et al., 2016]). In contrast, this pa- per introduces the Wasserstein distance, total variation distance, and cosine distance to depict the systematic change in technology distribution for firms, based on the CPC codes of patents. The key novelty of these three measures lies in their ability to measure the distance of technol- ogy distributions, systematically capturing how far a firm shifts its innovation technology fields from its previous innovation areas within a given time window.

The Development of Facticity—from Preliminary Finding to Accepted Implicit Knowledge

Yuze Sui and Tianyu Du

Research Collaboration Patterns Before and After COVID

Hanzhe Zhang

ABSTRACT. We examine the change in collaboration patterns among economists before and after COVID. We compare economics working papers in common repositories in from 2010 to 2023. Our analysis suggests that the number of authors per paper has increased, the percentage of papers with four or more authors has increased, yet the frequency of inter-institution collaboration has decreased. Some of these patterns persist beyond the COVID years. Our results suggest that while economics research had become more collaborative in the years leading up to COVID, COVID provided an additional significant boost to the depth and breadth of collaboration.

Quantifying citation imbalance in computer science: Gender and conference tier

Kazuki Nakajima

ABSTRACT. Citation practices are often left up to individual researchers, and they typically vary from discipline to discipline. The citation counts of papers often exhibit imbalances associated with author attributes such as country of affiliation and gender. While recent studies quantified citation imbalances in journal papers in a few disciplines [1, 5], the extent of citation imbalances in computer science remains largely unclear. In computer science, conference publications have a higher status than journal publications [2], and the issue of gender imbalance among researchers has been recognized [3]. Here we investigate citation practices in computer science conference papers in terms of the gender of authors and conference tier.

A System-level Analysis of Conference Peer Review

Yichi Zhang

ABSTRACT. The conference peer review process involves three constituencies: authors desire (fast) acceptance in prestigious venues, conferences aim to accept more high-quality and fewer low-quality papers, and reviewers should be spared overwhelming workloads. These objectives often clash, in large part due to the noise inherent in peer review. In response, conferences have experimented with various policies, including adjusting acceptance thresholds and the number of reviews per submission. We investigate how well various policies work, and more importantly, why they do or do not work. We model the conference-author interactions as a Stackelberg game: a prestigious conference commits to an acceptance policy, and authors respond by (re)submitting or not. Using our model, we explain why about 75% of submissions are rejected despite many of them eventually being accepted at prestigious venues with minor improvements, a phenomenon we term the "Resubmission Paradox". We show that conference prestige, reviewer inaccuracy, and author patience encourage authors to (re)submit and thus increase the review workload for a fixed conference quality. For the robustness of these and other results, we consider variants of our model and conduct simulations based on plausible parameters estimated from real data.

Eliciting Honest Information From Authors Using Sequential Review

Yichi Zhang

ABSTRACT. In the setting of conference peer review, the conference aims to accept high-quality papers and reject low-quality papers based on noisy review scores. A recent work proposes the isotonic mechanism, which can elicit the ranking of paper qualities from an author with multiple submissions to help improve the conference's decisions. However, the isotonic mechanism relies on the assumption that the author's utility is both an increasing and a convex function with respect to the review score, which is often violated in peer review settings (e.g.~when authors aim to maximize the number of accepted papers). In this paper, we propose a sequential review mechanism that can truthfully elicit the ranking information from authors while only assuming the agent's utility is increasing with respect to the true quality of her accepted papers. The key idea is to review the papers of an author in a sequence based on the provided ranking and conditioning the review of the next paper on the review scores of the previous papers. Advantages of the sequential review mechanism include 1) eliciting truthful ranking information in a more realistic setting than prior work; 2) improving the quality of accepted papers, reducing the reviewing workload and increasing the average quality of papers being reviewed; 3) incentivizing authors to write fewer papers of higher quality.

Measuring coherence in research projects output through LLMs

Manuel Jesus Cobo Martin

ABSTRACT. In the competitive realm of scientific research, securing funding is crucial for researchers to propel their work forward and enrich the collective knowledge base. However, concerns regarding the alignment of research project objectives with associated paper outcomes raise questions about scientific integrity. This paper proposes a novel methodology leveraging science of science and natural language processing techniques to measure coherence between project objectives and associated papers. By employing a 5-stage process, including corpus retrieval, word embedding computation, and semantic similarity analysis, we aim to identify potential misconduct patterns. Our method, illustrated with examples, facilitates the detection of discrepancies and empowers stakeholders to uphold scientific integrity. Through this research, we aim to foster transparency and ensure research funding is allocated to projects that genuinely contribute to societal progress.

Sharing Solutions without Spilling Secrets: The Role of Technicians in the Diffusion of Knowledge at Innovation Frontiers

Danielle Bovenberg

ABSTRACT. The recombination of knowledge is fundamental to science-driven innovation, yet studies show that innovators often struggle to recognize and exploit opportunities for recombination present in their field. In this paper, I show how potent forms of knowledge recombination can nonetheless occur through the work of the specialized and often overlooked technical occupations that steward the materials, instruments, and techniques that innovators have in common. As stewards of core research tools, these ideal-typical “technicians” shepherd many, diverse projects. This positions them to facilitate knowledge recombination across scientific domains. But little is known about how technicians foster recombinant innovation, and how their contributions differ from those of the scientists traditionally considered in theories of recombination. To examine this question, I draw on two years of ethnographic fieldwork at four facilities that make the tools of nanoscale R&D available to researchers in domains as diverse as semiconductor engineering and biotechnology. I draw on ethnographic observations of technicians’ work and sociological theories of craft knowledge to show the patterned ways in which technicians at these facilities recognized and exploited opportunities for recombination among the projects they supported. I thereby uncover an important and underrecognized set of pathways for recombinant innovation: pathways not anchored in the concepts and commitments of any one discipline or domain, but rather resting on pragmatic craft knowledge of materials and tools that are core to multiple innovation domains.

The Persistent Effect of R&D-related Industrial Policy: Human Capital Accumulation as a Mechanism

Xuelai Li

ABSTRACT. This paper investigates the long-term effect of the Space Race, an iconic R&D-related industrial policy, on local commercial development. The identification strategy leverages US-Soviet rivalry in space technology to isolate R&D spending. Using a novel dataset from the National Archives, I found that the Space Race has a long-term effect on local business formation and private investment. The influx of R&D funding, though initially intended for space exploration, spurred innovation that attracted venture capital, subsequently benefiting the commercial sector. Additionally, the Space Race contributed to the training of numerous skilled personnel who later founded firms and transferred space technology into the commercial market. This implies human capital accumulation as a key factor in the sustained influence of the Space Race on local commercial development.

Building and Characterizing a Collaboration Network of the US National Cancer Institute’s Extramural Workforce

Margaret Gratian and Christine Burgess

ABSTRACT. Scientific collaboration has become increasingly important to accelerate progress in biomedical research [1]. Collaboration can be a key method to address challenges to progress that arise from increased specialization of fields, siloed institutions, fragmented data ecosystems, rapid generation of large and complex datasets, and the growing expenses of conducting research and clinical trials [2]. In the United States, nationwide efforts such as the White House Cancer Moonshot Initiative [3,4] and the National Cancer Plan [5] bring the goal of fostering greater collaboration to the forefront of cancer research. As the United States’ principal agency for cancer research and training and the world’s largest funder of cancer research, the National Cancer Institute (NCI) supports a substantial extramural workforce of investigators through grants[6]. Given the importance of collaboration in cancer research, we seek to understand and characterize collaboration among the NCI-supported cancer research community. In this talk, we will discuss how we conceptualized and developed a collaboration network of NCI’s extramural workforce. We will provide insight into the network analysis methodology we applied and share our initial findings. We will also discuss challenges with measuring collaboration and how we sought to mitigate them in our work. Lastly, we will discuss how one can use this network and accompanying analysis methodology to inform strategies and policies for fostering collaboration. NCI supports its extramural workforce through a variety of grant mechanisms[7]. These mechanisms are divided into groups representing the types of funding NCI provides, such as Research Grants, Career Development Awards, or Research and Training Fellowships. Because we were interested in measuring scientific collaboration, we chose to focus on Principal Investigators (PIs) who were awarded a research-focused grant to build our network. This includes Research Project Grants such as the R01, R37, R56, and U01 and Research Center Grants such as the P50 or U54. Scientific collaboration may take many forms and some aspects of it may be more tangible than others. Critical to the conceptualization of our network was the concept of measurable collaborations. To determine if PIs had collaborated – that is, worked together in an interdependent fashion toward a scientific outcome – we required evidence of the collaboration in the form of co-funding on an NIH base project or co-authorship on a publication. Not all collaborations will result in co-funding or co-authorship. So, while this does not give us a holistic view of collaboration, it does give us a measurable one. We built our network from a PI-perspective, adding PIs as nodes of the network if they had a qualifying research award in Fiscal Year (FY)* 2017 – FY 2022. Edges were added based on the presence of co-funding or co-authorship between two PIs and therefore represent pairwise collaborative events. To better capture trends in collaboration over time, we included co-funding between PIs that occurred between FY 2012 – FY 2023 and co-authorship between PIs that occurred between 2012 – 2023. Our resulting network consists of over 10,000 PIs representing the nodes of our network. We identified co-funding between these PIs on over 6000 unique NIH base projects and co-authorship on over 200,000 publications. These projects and publications culminate in over one million network edges and over 200,000 unique pairwise collaborations between PIs. Once we had developed our network, we applied a variety of network analysis methodologies to characterize the network. We focused on identifying and applying metrics that could tell us about the strength of the network. In other words, could we see evidence of a robust research community among NCI’s extramural workforce? For example, node degree indicates the number of unique collaborators per PI, while the number of connected components provides insight into how interconnected the network is overall. We also conducted various temporal analyses of the data to understand how collaborations shift over time. We explored whether co-authorship is likely to follow co-funding and investigated the role NCI funding plays in sustaining collaborations and promoting new collaborations to form. We identified several challenges with measuring collaboration that must be considered when interpreting the results of an analysis. First, collaborations can take time to produce measurable events – the absence of co-funding or co-authorship does not imply the absence of a collaboration. Second and related to this point, PIs in the network began collaborating at different points in time and therefore reach collaborative milestones (co-authorship or co-funding) at different points in time as a result. Third, when attempting to measure the impact of an intervention on collaboration, it is important to recognize that most interventions are likely introduced in a setting where collaboration is already occurring. Despite these challenges, a thoughtfully constructed dataset representing measurable collaborations can still be a useful tool to understand and characterize collaborations. Our talk will conclude with insights into how we have used our network of extramural PIs to evaluate initiatives such as the Cancer Moonshot and its goal of fostering greater collaboration and how we plan to use our network going forward.

*The United States Fiscal Year begins on October 1st and ends on September 30th. For example, Fiscal Year 2017 began on October 1st, 2016 and ended on September 30th, 2017.

Does Reading the Literature Improve the Ability to Predict the Effects of Interventions?

Robin Na

ABSTRACT. Does academic literature help us make better predictions about the effects of interventions, or are they generating more noise than signals? Despite ongoing scientific endeavors, making effective policy recommendations remains notoriously challenging due to the formidable heterogeneity and contingency we must navigate to predict the outcomes of interventions. This raises the empirical question of which set of academic papers, if any, could actually shift human priors in a direction that enhances predictions under interventions. Motivated by the infeasibility of conducting controlled human-subject experiments of such a nature, we propose a novel method for investigating this fundamental question by using large language models (LLMs) and retrieval-augmented generation (RAG) techniques as proxies for exploring the counterfactual of reading different sets of research papers.

Each version of GPT-4, augmented with a specific set of behavioral research papers, is asked to predict intervention outcomes of cooperation under the high-dimensional, integrative experiment design space of a public goods game (PGG). The design space consists of 211 unique experiment designs with 20 dimensions of varying conditions to map the rich heterogeneity of treatment effects. We demonstrate the LLM’s ability to retrieve academic papers and alter its distribution of predictions for experiment outcomes in directions that are expected based on the documents' contents, going beyond simply retrieving factual information. However, we find no evidence so far that such shifts in distribution contribute to higher accuracy.

Our contributions are threefold. First, we introduce a method for evaluating the potential utility of academic papers based on their contribution in predicting the effects of interventions in highly contingent environments, which can be complementary to other features of papers such as citation, venue, and network metrics. In other words, we study the impact of academic papers through their interactions with intelligent minds, thereby uncovering numerous ways to evaluate the influence of social science on AI, and vice versa. Furthermore, we propose a benchmark for predicting the effects of interventions in highly context-dependent behavioral settings by utilizing an integrative experiment design that reliably maps such heterogeneity. Finally, we ignite several meta-scientific questions. These include the challenges of predicting intervention outcomes under complex context dependencies, the limits of predictability that cannot solely be attributed to model capacity (thus requiring additional experimental data with explicitly constructed design spaces), and the empirical question of how, if at all, human-interpretable explanations are associated with predictive accuracy. To add robustness to our discussions, we are also in the process of eliciting predictions from human participants with varying familiarity with the literature, along with responses from different versions of LLMs. The full results and analyses are expected to be available by the time of the conference presentation.

Cognitive diversity, repeated cooperation, and collective intelligence in dynamic cooperative networks

Jar-Der Luo, Ziqiu Lin, Yuanyi Zhen, Tianyi Zhang, Xin Gao and Youqiang Wang

ABSTRACT. This paper utilizes simulation models and big data to illustrate the emergence of collective intelligence in dynamic cooperative networks within organizational settings. Using an Agent-Based Model, the study examines how members' cognitive diversity influences collective intelligence and investigates how network members' repeated cooperation moderates this relationship. The research reveals two key findings. First, frequent and repeated cooperation among network members positively impacts collective intelligence. Second, an increase in the probability of repeated cooperation positively moderates the influence of cognitive diversity on collective intelligence. To address the simplification of real-world specifics in the simulation, hypotheses are formulated based on the key findings and tested using empirical data from one of China's largest high-tech companies. The fixed-effect panel model confirms the earlier findings. Additionally, further simulations show that high cognitive diversity among employees, along with the probability of repeated cooperation, leads to the emergence of collective intelligence, particularly when they exceed a certain threshold.

Global Policy Strategies in Response to Tech Innovations

Hana Kim

ABSTRACT. This study investigates the strategic policy responses to technological innovations across different global regions, focusing on developed and developing countries, as well as the United States and the European Union. Through a comparative analysis of AI policy documents, I identify key thematic priorities and how geopolitical and economic classifications shape policy directions. Network analysis of policy topics reveals a complex web of interconnected issues, highlighting the nuanced landscape of global policy strategies. My findings underscore the importance of contextualized policy-making in navigating the challenges and opportunities presented by technological advancements.

Towards Identifying WhoFundedIt: An Assessment of Data Quality and Availability for Analyzing Research Sponsor Bias

Jodi Schneider

ABSTRACT. This work in progress reports current progress towards WhoFundedIt, an interface to sponsorship data. The project will examine the availability and limitations of data, the differences between freely licensed and subscription data about funding; and provide an interface for visualizing who funds research on a specific topic.

The effect of AI on innovation in the case of medical diagnosis

Anne Kavalerchik

ABSTRACT. AI for medical diagnosis is an increasingly popular venture in patenting behavior. If im- plemented, it could disrupt conventional ideas about status, authority, and expertise within the profession (Jutel 2009; Freidson 2007). In fact, medical diagnosis is a significant area of in- terest within AI development (see figures 12). Given recent developments and improvements of AI, particularly through the fine-tuning large language models (Bommineni et al. 2023), we can expect to see the increased application of AI to medical diagnostics. Does this mean that AI is spurring further innovation in this area? Or, is it narrowing the field into the particular use cases for which AI is presently more developed (Messeri and Crockett 2024)?

Stealing the Spotlight or Fading into the Background: Retracted Papers and Disruption

Matthew VanEseltine

Duty Calls: Evaluating Academic Incentives for Sustainable Rare Disease Research and Development

Mengyi Sun

ABSTRACT. Rare diseases, individually uncommon yet collectively affecting at least 3.5% of the global population, present a unique challenge to healthcare systems. The Orphan Drug Act of 1983, designed to incentivize the development of treatments for these diseases through market exclusivity and tax incentives, has led to the approval of over 500 orphan drugs. However, this model, predicated on high prices for orphan drugs, becomes financially unsustainable when considering the cumulative impact of over 6,000 known rare diseases on a significant portion of the population and the pharmaceutical companies' misuse of the Orphan Drug Act.

This study evaluates whether the academic reward system is equipped to take on the burden of rare disease R&D, as an alternative to market-driven incentives. Utilizing the PubMed Knowledge Graph, icite, and provenance data in knowledge accumulation in human genetics, we compare the output, impact and careers of researchers on rare versus common diseases.

Our findings indicate that rare disease research significantly contributes to human genetics knowledge,consistently outpacing studies on common disease (Figure.1A). Yet, in recent years, this research is penalized in terms of citations (Figure.1B), particularly in non-US regions without supportive policies like the Rare Disease Act of 2002 (Figure.1B, bottom). Furthermore, researchers focused on rare diseases face higher attrition rates. This citation penalty and attrition rate suggest the current public science incentive system is ill-prepared for the challenges of rare disease R&D.

We conclude by discussing the necessity of reevaluating and adjusting the academic incentive structure to better accommodate and encourage rare disease research. This adjustment is crucial for creating a sustainable model for rare disease R&D that does not solely rely on market incentives, thereby ensuring ongoing innovation and support for affected populations.

A Continuous Embedding Measure of Disruptiveness Reveals Hidden Simultaneous Disruption

Yong-Yeol Ahn

ABSTRACT. The theories of scientific evolution often contrast "disruptive'' contributions from "developing'' ones. Funk and Owen-Smith's disruption index has been pivotal in quantifying this disruptive impact of scientific works, revealing insights into the evolutionary process of science through its empirical correlations with various factors in science. However, the discrete and localized nature of the disruption index poses some limitations (see Fig. a). The index can be extremely sensitive to small changes and not able to capture any information that is not encoded in the immediate vicinity of the focal paper. The index also has a low resolution because the varying degrees of linkage between earlier and subsequent works cannot be quantitatively incorporated. Last, it fails to capture disruption events distributed across multiple works, especially if there exist citations between them.

To address this gap, we develop a novel graph embedding algorithm that can effectively quantify the disruptiveness of scientific papers. Our algorithm is based on the observation that disruptive papers typically exhibit a greater conceptual "distance'' between their prior and future works, as future works less rely on the previous works after a disruption. Then, if each paper is represented with two embedding vectors---one depicting its prior works through the citing context and the other representing future works through the cited context---the disparity between them can capture the reliance of future works on the prior works of the focal paper (see Fig. c). To achieve this, we employ an approach that confines the window direction in the skip-gram algorithm (see Fig. b). This method guides one vector to locate closely with future works linked to the focal paper while steering the other vector toward prior works. Then we define the Embedding Disruptiveness Measure (EDM) index of paper $i$ as $\Delta_i =1-\frac{\vec{f_i}\cdot \vec{p_i}}{\| \vec{f_i}\| \|\vec{p_i}\|}$, which is the cosine distance between the past vector (vector representing cited context, $\vec{p_i}$), and the future vector (vector representing citing context, $\vec{f_i}$) of paper $i$.

Utilizing a dataset comprising more than 24 million papers from the Web of Science, we demonstrate that EDM (denoted as $\Delta$) better captures disruptive contributions than the disruption index (denoted as $D$). It shows a higher resolution with less degeneracy and captures a broader spectrum of information beyond a single citation (see Fig. d and e). Examination of Nobel Prize-winning papers reveals that EDM identifies those papers as highly disruptive, whereas the disruption index has a bimodal distribution (see Fig. f). This bimodal distribution was mainly due to the simultaneous disruptions, which the disruption index is unable to capture because of its discrete feature (see Fig. h). Additionally, the $D$ distribution in the random networks, where the number of citations and references, and citation age gaps are preserved, is also located in the top 10\%, suggesting citation and reference counts, and their age gap can be confounding factors. Conversely, the $\Delta$ values for Nobel Prize papers in the randomized network exhibited significant deviations from those in the original network, suggesting that their ability to capture disruptive quality goes beyond mere citation and reference counts. We further explore the capacity of EDM to identify simultaneous discovery pairs and find that our method can accurately figure out paper pairs contributing to the same discovery, even those overlooked for Nobel Prizes, such as the case of the Higgs Mechanism discovery (see Fig. g).

In summary, our study introduces EDM as a refined tool for quantifying disruptive innovation, offering a more granular and global perspective on scientific impact. By overcoming the limitations of the disruption index, EDM will shed light on unrecognized disruptions and provide a useful tool to reveal simultaneous discoveries.

How the COVID-19 pandemic affected science: collaboration and impact

Satyaki Sikdar and Santo Fortunato

ABSTRACT. The global COVID-19 pandemic changed our daily routines, limiting social interactions and enforcing lockdowns. Extensive research highlights the pivotal role of in-person interaction in fostering the exchange and blending of ideas. For instance, there has been a decline in the weak ties among scholars during the lockdown, which are fundamental for the transmission of novel information. Here, we investigate how the COVID-19 pandemic has affected science, specifically the collaborations between scholars and the impact of the resulting papers.

Disambiguating Scientific Software in Full-Text Publications for Science of Science Studies

Sarah Bratt and Kai Li

ABSTRACT. Scientific software has proliferated as researchers increasingly use large amounts of digital data [1]. However, the science of science studies of software are hindered by a lack of high-quality datasets due to the immature software citation ecosystem [5] and even state-of-the-art software named entity recognition (NER) datasets are incomplete and contain anomalies [8]. These limitations negatively impact tracking the flow of scientific ideas in software-enabled science and impede innovation studies of software breakthroughs [4]. A particularly intractable problem in this milieu is disambiguation of software mentions. The lack of standard ways to cite software in papers means disambiguation methods, then, have to rely on the context of the mention. Disambiguation of software mentions in papers would accelerate science of science research by creating structured datasets that provide valuable insights into the use, development, and impact of software in scientific research [3].

In this study, we disambiguated entities used the Software Mentions dataset from the Chan Zuckerberg Initiative (CZI) [6] and an LLM approach. The rationale for using an LLM is to utilize the pretraining of the LLMs as well as the ability to compare between two texts [7]. To develop a labelled test sataset, we selected mentions of the PRISM software from the dataset (n=500) and performed manual disambiguation of the software mentioned using human annotation (among three faculty). This approach was taken because (1) there are multiple software entities using this name; and (2) the majority of PRISM mentions in the CZI dataset were incorrectly disambiguated (i.e., they were attributed to an incorrect software entity). We used the sentences the scientific articles in which the software name was mentioned to conduct the manual disambiguation to be used for the labelled dataset for evaluation of the model.

Next, we posed this problem as a binary classification problem. The features for the classification were obtained by generating a 500 X 500 similarity matrix for the software mentions. We queried a pre-trained large language model (PaLM) [2] to develop this similarity matrix. This was achieved by querying the LLM to output whether a pair of mentions referred to the same software. The LLM was prompted to provide an output of 1 if it identified the two mentions to be the same, -1 if it identified the mentions to be different, and 0 if it was not sure. After generating the similarity matrix, we calculated the sum of each of the rows as the feature for our binary classification model. The model achieved an area under the curve (AUC) of 0.7 for distinguishing between PRISM software and other types of mentions entirely based on a single context sentence of the mention. The ROC curve for the binary classification is shown in figure 1.

The advantage of the LLMs method is that it can be automated to apply across a large dataset of software mentions. Granted, the preliminary results fall short of the manual disambiguation and it remains to be seen how well it performs across a varying set of software. Because the manual disambiguation was done using both the context sentence and the full text of the papers where necessary, results could be improved by incorporating more contextual information. More informative features could also be used in the future to bolster the performance of the model. Overall, this preliminary study suggests that LLMs hold promise as a novel approach to the disambiguation of software mentions for science of science studies.

Characterizing Academic Publishers’ Name Change Policies for Gender Diverse Authors

Deanna Zarrillo

ABSTRACT. Extended Abstract

In 2019, the Association for Computing Machinery introduced their new post-publication name change policy. Soon after, several large academic journal publishers followed suit. Name change policies facilitate invisible updates to transgender authors’ names in published articles. Before these policies were introduced, trans, non-binary, and gender diverse authors who changed their names were limited to either forfeiting their publication history, or risking being inadvertently outed to their professional networks (Hauck, 2021). Either path could result in damaging consequences to career trajectory. Even with the introduction of more inclusive and privacy centered policies, the process of auditing and updating one’s personal publication history can be a daunting and emotionally exhausting task. To date, a review of the scope and coverage of these policies has not been done. This abstract represents the initial research stages of the author’s dissertation work on the efficacy of author name change policies. Positioned within an ethic of care, this thematic content analysis examines the public facing post-publication name change policies of nine academic journal publishers using open coding techniques. The analysis examines the similarities and differences in both content and format of these policies and reveals how academic publishers use policy to communicate values with authors and balance their needs with the publishing industry’s meticulous ethical and procedural guidelines.

Digitally published policies from nine academic publishers were collected and analyzed. For publisher information, see Figure 1. Publishers were selected based on the availability of a name change policy, their recognizability, and their overall publication output. As is typical in thematic analyses, the primary methodology involved open coding and re-coding for themes in the documents. The iterative coding process resulted in a codebook used for reference and consistency during data collection and as a structural tool for a close reading analysis (Flick, 2018).

Key findings during the coding process help contextualize what publishers understand to be important to their authors and how they use policy to communicate these values. This initial analysis reveals the distribution and frequency of themes and codes in the policies (Figure 1). SAGE has the most comprehensive coverage of relevant topics having at least one occurrence of each code, while Springer Nature and Cambridge University Press have strikingly sparse policies in comparison. Additionally, topics related to the theme of Privacy seem to be highlighted most frequently in policies, while themes of Engagement and Industry Context receive less attention. Finally, the natural discoverability of the name change policy, or the number of clicks from the homepage it took to get to the page containing the policy, on publisher websites varied widely. These findings demonstrate the range and accessibility of information which may be presented to authors who wish to utilize these policies.

[Figure 1. Includes of the publishers in this study, country of headquarters, year policy introduced, and business type. Number in heatmap is the count of the code’s occurrence. Numbers under Page Location capture click count from homepage. ]

Further insights from this data will be gained through deep readings of the documents and compared to the five guiding principles for publishers developed by the Name Change Policy Working Group (2021). Additionally, future work for this dissertation intends to address the efficacy of name change policies, the strategies gender diverse authors use to navigate their identity in academia, and how name changes (irrespective of reason) affect the performance of publications through bibliometric analyses and interviews with key stakeholders.

The context of the larger study is deeply rooted in identity politics and the importance of names and fluid identities over time. The study population straddles two groups facing different but nevertheless interconnected marginalization, underrepresentation, and mistreatment in academic literature (Vincent, 2018). The findings of this analysis illustrate how this marginalization may still occur in how and what publishers choose to communicate through policy, especially as new standards and ethical guidelines are introduced. Through studying academic publisher name change policies, we get a glimpse at how the publishing industry interacts with, learns from, and responds to the intersectional communities it serves. More broadly, we begin to learn about one facet of how science itself is evolving and who gets to have a voice in its future.

References

Flick, U. (2018). An introduction to qualitative research. Sage Publications.

Hauck, K. (2021). Inclusive Author Name Change Policies. Science Editor, 105–107. https://doi.org/10.36591/SE-D-4403-105

Name Change Policy Working Group. Five Guiding Principles and Best Practices. (2021). https://ncpwg.org/resources/principles/

Vincent, B. W. (2018). Studying trans: Recommendations for ethical recruitment and collaboration with transgender participants in academic research. PSYCHOLOGY & SEXUALITY, 9(2), 102–116. https://doi.org/10.1080/19419899.2018.1434558

Measuring Funding Agencies' institutional capacity to address inequities in science

Yohanna Juk, Karen E.F. Pinto, and Bernardo Cabral

ABSTRACT. Fighting inequality in the research ecosystem has gained prominence as a topic of discussion among scientists, universities, editorial bodies, and research institutions. The calls for change in the way science is practiced, funded, and promoted largely emanate from women and social minorities who encounter various challenges in entering and establishing themselves in the academic ecosystem, remaining as underrepresented and underfunded groups. While the issue is acknowledged, solutions vary and encounter various challenges.

In this context, attention must be directed to the crucial role played by funding agencies (FAs) in combating inequities in the scientific environment. FAs have ample room to improve their policies and practices and, encouraged by social movements and scientists, have begun to implement policies that consider the many diversities in science in grant proposal processes. The Global Research Council itself, an organization through which national scientific FAs from various countries gather to discuss cooperation, review practices, and promote guidelines for scientific funding, held a meeting in 2023 in favor of racial and ethnic diversity in research, with a greater commitment to promoting structural changes and impact in this regard. Initiatives implemented by FAs addressing elements of Equity, Diversity, and Inclusion (EDI) were identified by Moody and Aldercoote (2020), Hunt et al. (2017), Juk et al. (2023), demonstrating a concern on the part of these institutions to promote a diverse and inclusive environment. However, it is still a challenge to: i) categorize these interventions; ii) determine their effectiveness; and iii) to consider the institutional capacity of an organization to determine why certain initiatives are effective and why some of them fail in actually promoting EDI in the research environment.

While Brazil has a history of affirmative action for the inclusion of minority groups in federal universities through quota policies, there has been a growing concern about expanding activities and strategies that consider other axes of inequalities. For instance, in 2022 the São Paulo Research Foundation, one of the most important FAs in Brazil, created a programm for matters related to EDI, which provides a set of actions to expand the diversity of students and scientists funded and aims to improve internal processes and remove obstacles associated to gender and ethnicity. Although these initiatives are crucial and generally suitable for the Brazilian context, recent episodes demonstrate that the narrative often fails to translate into effective change. The evaluation and discrimination reported by mother researchers who attempted to include their maternity leave information in their curriculum, aiming for it to be considered in academic performance evaluations for productivity scholarships by the Brazilian National Council for Scientific and Technological Development (CNPq), serve as an example. The effectiveness of this initiative did not materialize, as several evaluations ignored the maternity leave period and emphasized the lack of productivity during this period as a performance problem. This episode suggests that while EDI initiatives and policies in research are proposed, their actual implementation and institutional barriers are often overlooked.

Given the presented context, this study aims to map the range of EDI activities related to different contexts and identity characteristics of Brazilian FAs. The study also seeks to investigate the institutional capacity of the analyzed agencies in actually implementing the EDI initiatives. Our guiding research questions are: How are EDI-related initiatives being institutionalized by FAs in Brazil, and which initiatives have proven effective or less effective, and why?

We contacted two national FAs, the Brazilian National Council for Scientific and Technological Development (CNPq) and the Coordination of Superior Level Staff Improvement (Capes); and the National Council of State Foundations for Research Support, which currently brings together 27 FAs in Brazil to sent questionnaires to investigate their commitment to EDI strategies and to measure their institutional capacity in implementing these initiatives. We also relied on secondary data analysis on the topics of EDI and analysis of strategic plans from FAs. To build up the questionnaire we conducted a literature review on EDI activities implemented by FAs around the world.

To measure the institutional capacities of organizations to develop and implement effective EDI strategies and interventions, we investigated the following aspects and classified them into basic, intermediary, and advanced levels: i) The organizations’ understanding of the subject ii) To have dedicated departments, teams, or calls to address EDI-related issues iii) To implement guidelines, promote and institutionalize knowledge on the subject within their human resources (e.g.: through training among evaluators to eliminate EDI-related bias in peer review processes, grant and project evaluations) iv) To identify institutional barriers and lock-ins while implementing such initiatives v) To conduct a systematic evaluation of EDI initiatives and interventions vi) To promote accountability, such as making diversity data available or establishing a communication channel with grantees to consider suggestions and improvements

This is a research in progress, so we present here only preliminary results from the investigation, relying on secondary data from the analyzed FAs. Despite the growing attention to the matter of EDI, discussions still seem to be focusing on narratives, principles, and best practices. There seems to be a focus on gender initiatives, which indicates that the intersectional approach that considers other diversity axes, such as race, ethnicity, and disabilities, is not present in actual interventions.

Investigating how FAs, a core element of the research system, are addressing this issue can be an important starting point to assess if real change is actually taking place.

References

Hunt, L.; Nielsen, M. W.; Schiebinger, L. (2022). A framework for sex, gender, and diversity analysis in research: funding agencies have ample room to improve their policies. Science, 377(6614), 1492-1495.

Juk, Y., Salles-Filho, S., E F Pinto, K., Spatti, A., Coggo Cristofoletti, E., Araujo Tetzner, G. & Campgnolli, E. (2023). Diversity, equity and inclusion: how are funding agencies addressing inequalities in research [preprint]. 27th International Conference on Science, Technology and Innovation Indicators (STI 2023). https://doi.org/10.55835/6442ffc28264b1bf681c48ce

Moody, J.; Aldercotte, A.. Equality, diversity and inclusion in research and innovation: international review. Available at: https://www.ukri.org/wp-content/uploads/2020/10/UKRI-020920-EDI-EvidenceReviewInternational.pdf.

Evaluating the use of funded research in policies in Brazil: a study of the São Paulo Research Foundation (FAPESP)

Karen E.F. Pinto

ABSTRACT. Given the ongoing social transformations across various scales and domains, such as climate change, poverty, inequalities, and technological advancements, the significance of understanding the relationship between policy and science has never been greater (Aiello et al., 2021). There is a growing concern among funding agencies, universities, and other stakeholders in the research ecosystem regarding how research can better inform, influence, and impact policy, thereby making this topic increasingly relevant. This also implies discussions about methods to assess the impacts of research (Álvarez-Bornstein & Bordons, 2021).

The literature has explored various methods for identifying, assessing, and measuring the flow of scientific knowledge into policies, encompassing standardization and guidelines (Newson et al., 2018). The utilization of bibliometric and altimetric analyses to assess the nexus between research and policies has seen significant development. This approach often employs keyword searches or text mining techniques to detect references and mentions of research, universities, or researchers within policy documents (Vikings & Grant, 2017). Tools and databases have emerged for this purpose, such as Altmetric.com and Overton (Szomszor & Addie, 2022).

In light of this, the research sought to identify the impact of research funded by the São Paulo Research Foundation (FAPESP), one of the main research funding agencies in Brazil, on policies through the exploration of the Overton tool. In addition, our research also aimed to discuss the potentialities and limitations of this approach to accessing the use and impact of research on policies.

To test Overton, we chose to use the DOIs (articles) from projects and scholarships funded by FAPESP. The data were obtained from the Virtual Library of the Agency. The research team obtained 99,637 DOI records from 52,425 funded projects or scholarships for master's, doctoral, and postdoctoral studies, spanning from 1992 to 2023. The articles are distributed as follows: 0.1% in Linguistics and Arts; 0.5% in Applied Social Sciences; 1% Interdisciplinary; 1.7% in Humanities; 10.5% in Engineering; 11.8% in Agricultural Sciences; 19.7% in Health Sciences; 26.1% in Exact and Earth Sciences; and 28.6% in Biological Sciences. After searching for DOIs in Overton, the data were extracted via Application Programming Interface (API) and integrated into an Excel spreadsheet.

Out of the 99,637 DOIs entered as input, it was identified the mention of 2,994 DOIs in documents indexed in the policy database. These DOIs were referenced in 2,025 policy documents, with 1,132 documents from governmental sources, 387 from International Governmental Organizations (IGOs), 305 from think tanks, and 201 from other sources. It's noteworthy that the majority of these policy documents come from IGOs (387), with the most common originating from the USA (370 policy documents) and the European Union (247). Approximately 80 policy documents from Brazilian sources were identified. Despite the bias inherent in the database (language and indexing bias towards countries such as the USA, UK, European Union, etc.), Overton facilitated the identification of novel data, particularly regarding the use of Brazilian research by foreign government agencies, IGOs, and think tanks. There was a prevalence of themes related to the environment and health in the policy documents. This is exemplified by the IGOs that mention FAPESP articles the most, notably the Food and Agriculture Organization of the United Nations (113 policy documents) and the World Health Organization (55 documents). Another interesting indicator is the identification of 168 policy documents whose source is the Guidelines in PubMed Central, a platform indexing documents related to health research. Despite these interesting and novel results, providing a broader view of research utilization in policies, exploring Overton presents some challenges, notably: analytical effort related to understanding the nature of the various types of policy documents (as the database treats them in a comprehensive manner), the nature of the use and impact of research in these documents and the limitations of coverage, with few results coming from Brazil.

References

Aiello, E., Donovan, C., Duque, E., Fabrizio, S., Flecha, R., Holm, P., ... & Reale, E. (2021). Effective strategies that enhance the social impact of social sciences and humanities research. Evidence & Policy, 17(1), 131-146.

Álvarez-Bornstein, B., & Bordons, M. (2021). Is funding related to higher research impact? Exploring its relationship and the mediating role of collaboration in several disciplines. Journal of Informetrics, 15(1), 101102.

Newson, R., King, L., Rychetnik, L., Milat, A., & Bauman, A.E. (2018). Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research. Health Research Policy and Systems, 16(54). https://doi.org/10.1186/s12961-018-0310-4

Szomszor, M., & Adie, E. (2022). Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), pp. 624–650. doi: 10.1162/qss_a_00204

Vilkins, S., & Grant, W. J. (2017). Types of evidence cited in Australian Government publications. Scientometrics, 113(3), 1681-1695. https://doi.org/10.1007/s11192-017-2544-2

Exploring International Collaboration Dynamics through Multilayer Network Analysis

Pei-Ying Chen and Tzu-Kun Hsiao

ABSTRACT. While international collaboration is widely recognized as a key driver of modern scientific growth and has been extensively studied, its internal heterogeneity remains largely underexplored. This oversight is mirrored in the common practice of aggregating heterogeneous ties between countries in most network analyses of international collaboration. Our study addresses this gap by exploring the dynamics of international collaboration through the lens of multilayer networks. This approach allows us to organize countries into distinct layers based on different co-authorship patterns, as determined by the affiliated countries of first and last authors. Focusing on the co-authorship network between Taiwan and its eight New Southbound Policy (NSP) priority partners as a proof of concept, our analysis of layer--layer correlations reveals that collaborations characterized by equal co-authorship and co-affiliation align most closely. In contrast, those led by NSP countries show the greatest similarity to minimal collaborations. This explorative study highlights the importance of distinguishing among co-authorship patterns to fully understand the dynamics of international scientific collaboration.

Effects of Research Paper Promotion via ArXiv and X

Chhandak Bagchi, Eric Malmi, and Przemyslaw Grabowicz

ABSTRACT. In the evolving landscape of scientific publishing, it is important to understand the drivers of high-impact research, to equip scientists with actionable strategies to enhance the reach of their work, and to understand trends in the use modern scientific publishing tools to inform their further development. Here, based on a large dataset of computer science publications, we study trends in the use of early preprint publications and revisions on ArXiv and the use of X (formerly Twitter) for promotion of such papers in the last 10 years. We find that early submission to ArXiv and promotion on X have soared in recent years. Estimating the effect that the use of each of these modern affordances has on the number of citations of scientific publications, we find that in the first 5 years from an initial publication peer-reviewed conference papers submitted early to ArXiv gain on average 21.1 more citations, revised on ArXiv gain 18.4 more citations, and promoted on X gain 44.4 more citations. Our results show that promoting one's work on ArXiv or X has a large impact on the number of citations, as well as the number of influential citations computed by Semantic Scholar, and thereby on the career of researchers. We discuss the far-reaching implications of these findings for future scientific publishing systems and measures of scientific impact.

Knowledge Recombination and Risk

Sotaro Shibayama

ABSTRACT. Science is a risky business by nature. Scientists explore and cultivate uncharted space of knowledge through trials and errors, in which their original ideas are often rejected and expected goals are not fulfilled for various reasons (Franzoni and Stephan, 2023; Veugelers et al., 2022). Such risk and uncertainty tend to be especially high when scientists aim at novel discoveries, which have the potential to open up new avenues and make substantial advancement (Kuhn, 1970). Despite its fundamental role, risk in science has been poorly understood (Franzoni and Stephan, 2023). This study aims to develop a bibliometric approach to quantify the degree of scientific risk in a particular mode of scientific discovery process – recombination. Scientific knowledge is usually generated on the basis of extant knowledge, and combining multiple elements of extant knowledge is an indispensable route to generate new knowledge. Novel discoveries often result from integrating pieces of extant knowledge that used to appear unrelated (Uzzi et al., 2013). Because of such a fundamental role of recombination, it is of scholarly and practical interest to quantify the degree of risk associated in recombination. To quantify risk in the recombination process, we employ machine learning and natural language processing techniques. Drawing on past trajectories of science, we develop a machine learning model that predicts whether a certain pair of knowledge elements will be linked or not in the future. The developed model calculates the probability that the pair of knowledge elements is combined, or put differently, the risk in achieving or failing in recombination. This risk indicator is validated by a questionnaire survey that we carried out, in which scientists self-assessed the anticipated risk of their own projects. Predicting recombination risk. To build a prediction model of recombination risk, we considered a published paper as a knowledge element and that two knowledge elements are combined if a pair of papers are cited together by at least one paper within a certain period. For this exercise we prepared the training and test data with Web of Science (WoS) publication data. We randomly selected 60,000 paper pairs (i.e., successful recombination) that were later co-cited as well as 60,000 paper pairs that were never co-cited in 10 years (i.e., failed recombination). We then converted each paper into a 300-dimensional document vector (for which we developed a word embedding model with WoS text data) and used a pair of document vectors as the input features for predicting whether paper pairs are co-cited or not. We developed link prediction models with several classifiers and assessed the performance of the trained models, finding that the support vector machine (SVM) achieves the highest and satisfactory F1 score of 0.860 among other classifiers. Risk indicator of a paper. Based on this algorithm, we next construct an indicator for the risk of a project. We considered a published paper as a unit of project and its cited references as elements of knowledge that were recombined in this project. Specifically, given a focal paper, we formed all possible combinations of its references (for example, 10 references make 45 pairs). For each reference pair, we computed the risk score based on the above-developed classifier. A focal paper is thus assigned as many risk scores as the nuber of reference pairs, which we aggregated to a single risk indicator of the focal paper. Validation of the risk indicator. To validate this indicator, we carried out a questionnaire survey to ask the corresponding authros of 4,228 randomly selected papers how they had initially perceived the risk of the project, and we collected 378 responses (response rate = 8.9%). The survey inquired into overall risk – "when you started the project, how confident were you that the project would reach the expected finding or any publishable finding at all?" as well as technical risk – "when you started the project, did you anticipate any technical / methodological challenge that could fail your project?" (Fig.1). We then analyzed the correlation between these two survey scores with the bibliometric risk indicators calculated for the respondents' papers. The result suggests that the risk indicator is significantly correlated with overall risk (r = 0.176, p < .001) but not with technical risk (r = .025, p > .1). Risk and novelty. Some previous studies treated risk and novelty interchangeably (Veugelers et al., 2022). While novel research may entail great risk (Franzoni and Stephan, 2021), risk and novelty are not conceptually equivalent. Thus, we examined the relationship between the two concpts using bibliometric indicators and survey measures (Table 1). The result does not show significant correlation between the two concepts, suggesting that risk and novelty are discrete concepts.

The Smart City as a Field of Innovation: Public-Private Data Collaboration and Its Effects on Innovation at Small and Medium-Sized Enterprises in China

Masaru Yarime

ABSTRACT. Data is increasingly considered to be a key component in stimulating innovation. Numerous promising possibilities have been opened up by rapidly emerging, data-intensive technologies, including the Internet of Things and artificial intelligence. The analysis and interpretation of big data are critical in the growth of technology firms in terms of AI training and computing capabilities. Small and medium-sized enterprises, with their limited internal resources, particularly face a serious challenge of implementing innovation, which increasingly requires the effective processing and use of various kinds of data. The smart city provides an important opportunity for creating data-driven innovation. Significant amounts of data are increasingly available from various sources through sophisticated devices and equipment scattered in smart cities. Many smart city projects across the globe provide rich opportunities for SMEs to explore data-driven innovation. China, in particular, has recently been active in collecting and utilizing various kinds of data in smart cities. The availability of and access to data would help private enterprises develop innovative products and services in China, where massive amounts of data resources are collected and maintained by the public sector. There were few empirical studies conducted, however, to examine how data are actually managed and provided in smart cities and how they affect companies’ innovative activities. It remains unclear how public agencies and private enterprises collaborate on data and how that influences the innovation performance of SMEs in China.

This study examines how collaboration between the government and companies in smart cities influenced firms’ innovation performance in China. By focusing on the case of SMEs in the Guangdong province, this study investigates how different types of services offered in government contracts on smart cities have an effect on enterprises’ innovation ability. Here, government contracts on projects related to smart cities are divided into three categories: equipment supply, platform building, and data analysis. Equipment supply includes a broad range of equipment, covering various kinds of electronic devices and machinery components. Platform building mainly refers to helping the government build, upgrade, or operate a platform that stores data related to the smart city. Data analysis is to access and analyze the data collected in the smart city for the government. We examine what impacts are made on the innovation outputs of firms by participating in the different types of projects for smart city development. In particular, we examine the following three hypotheses. H1: The companies that participate in projects related to data analysis will have more improvement in innovation outputs compared with those participating in projects related to platform building. H2: The companies that participate in projects related to platform building will have more improvement in innovation outputs compared with those participating in projects related to equipment supply. H3: Overall, the companies that participated in projects related to smart city development have more improvement in innovation outputs compared with those that did not participate in such projects.

In this study, we adopt propensity score matching (PSM) to create comparable control and treatment groups based on company attributes and then apply them to a generalized difference-in-difference (DID) model separately. It would be possible that those companies that obtained government contracts already possess some internal characteristics that produce innovation performance compared to those that did not obtain contracts. To deal with this issue, PSM is adopted to create control and treatment pairs among SMEs. We used three major indexes to divide control and treatment groups: GDP per capita at the provincial level (National Bureau of Statistics), the industry categorization (Enterprise Information Publicity System), and the registered capital (Tianyancha). Also, we used the outcome variables, namely, software copyright and patents, respectively, to divide the control and treatment groups.

DID is adopted to examine the effects of smart city contracts on the innovation outputs of firms. Regarding contract identification, we classified the government procurement contracts into three types (data analysis, platform building, and equipment supply) based on keyword identification by checking the contents and titles of the procurement documents and contracts. We took the year when the company obtained a government contract for the first time as the treatment year. When measuring the innovation ability of SMEs, we used both patents and software copyrights to evaluate innovation levels. In terms of the independent variable, we identified obtaining a government procurement contract as a treatment. In addition to this, we also added several control variables. GDP per capita at the provincial level is used to represent the level of economic development of the place where the company is located. The better the economic development, the more business opportunities and innovation are expected for the firm. Based on the average national GDP level, the GDP per capita of each province is classified into three large categories: high, low, and medium. We also include the age of the companies as it has been shown that younger firms are more likely to have more innovation activities than older firms. The registered capital is used to represent the size of the firm since the number of employees is not publicly available for many SMEs in China. We also consider the number of associated firms because if a firm belongs to a group of firms, then the parent company often provides human and financial resources to the subsidiaries, which would promote innovation development.

Our analysis showed that overall, the companies that participated in projects related to smart city development have more improvement in innovation outputs compared with those that did not participate in such projects. In particular, the companies that participate in projects related to data analysis have more improvement in innovation outputs compared with those participating in other types of smart city projects. That suggests that contracts on data analysis provide SMEs with access to various kinds of data in smart cities possessed by the government, which contributes to promoting innovative activities by the companies. Government purchases of hardware would promote the use of these products in smart cities. That, however, would not involve any substantive exchange or transfer of data possessed by the government and would not significantly contribute to stimulating innovation at firms. Based on the findings of this study, we would suggest that the government that aims to promote innovation development can focus more on giving more opportunities to SMEs, especially in terms of tasks involving data analysis. These kinds of projects will vastly encourage the enterprises’ innovation in the context of smart city development. Moreover, by improving data sharing and openness, companies will be able to benefit from the data resources available and further improve their innovation performance. In this study, we were not able to use data on R&D at the firm level as a control variable, as in China, many SME firms are not listed and are not required to release company details. Case studies will be helpful in obtaining further details on what kinds of data are available and how companies have access to data in smart city projects, affecting innovation activities and outputs.

The Dimensionality of an Idea Space and the Success of Familiar Ideas

Tian Chan, Yonghoon Lee and Jamie Song

ABSTRACT. Creativity is considered the engine of scientific, economic, and artistic advancement. Yet, numerous accounts noted that creativity alone often does not lead to the success of an idea. While the novelty of an idea may initially attract attention, its success depends on how people can understand, relate to, and become comfortable using it. This conundrum—familiar ideas can be more successful than novel ideas—can be better understood considering how idea spaces are multi-dimensional. As the number of dimensions that need to be compared and evaluated increases, the possible combinations of different dimensions increase exponentially, leading to greater cognitive difficulties in evaluation in high-dimensional spaces. Using data from three contexts, utility patents in the US, new entrepreneurial products shared at Product Hunt website, and Korean popular music, we measure the dimensionality of an idea space (technology classes in patents, product markets in Product Hunt, or music genres in K-pop) by estimating the number of principal dimensions from the variations in the data with principal components analysis. Across all three datasets, we find that novelty generally increases the level of success, but only when the idea space is low-dimensional. When the idea space is high-dimensional, we see a decrease in the level of success with novelty. This suggests that familiar ideas are more successful than novel ideas when idea spaces are high dimensional. The implications of our findings can be examined against the backdrop of the rising concern that creativity is dwindling. We suggest that this may be a natural consequence of the challenges in the evaluation of creative ideas, as idea spaces become more high dimensional over time.

The Gender Gap in Novelty Claims and Its Influence on Scientific Impact

Jina Lee

ABSTRACT. This study examines how gender and the presentation of novelty in research influence scientific impact across disciplines. Controlling for the intrinsic novelty of the research, I find that male first authors in natural sciences are more likely to assert novelty of their works, and this increases subsequent scientific impact of their research. Interestingly, while there is weak evidence supporting a female citation premium in humanities and social sciences (p-value = 0.057), this gap is not mediated by novelty claims. These findings offer a comprehensive view of the mechanisms behind gender disparities in scientific impact and suggest the need for discipline-specific interventions.

A Multiverse Analysis on the effect of Interdisciplinary and Non-conforming Research on scientific impact

Giulio Cantone and Paul Nightingale

ABSTRACT. This study examines two paradigms for assessing the attribute of scientific articles to evade conventional disciplinary categorizations through a Multiverse Analysis. Nonconformity is more consistently associated to scientific impact than interdisciplinarity.

To what extent researchers utilize Wikipedia to acquire new scientific knowledge?

Ramiro H. Gálvez

ABSTRACT. Despite its widespread popularity, limited research has delved into Wikipedia's influence on real-world outcomes. In this study, we investigate Wikipedia's impact on scholarly citations across disciplines amidst the onset of the Covid-19 pandemic. Notably, as research on epidemic evolution surged in early 2020, involving mathematical modeling, including SIR models, non-specialist researchers also engaged in this domain. We propose that Wikipedia served as a key learning source, despite academic guidelines discouraging direct citations. Employing a synthetic control method, we analyze the citation trends of articles referenced in Wikipedia compared to similar non-referenced articles. Our findings reveal a significant positive effect on citations for Wikipedia-referenced articles during 2020, challenging perceptions of its reliability. This study underscores Wikipedia's substantial role in facilitating researchers' acquisition of scientific knowledge, highlighting its nuanced impact beyond conventional citation practices.

The changing landscape of academic leadership: A study of presidents at US R1s

Jianjian Gao

ABSTRACT. This study explores the changing landscape of academic leadership in education, focusing on the demographic characteristics and academic expertise of university presidents in the US from 1950 to 2023. The research reveals an increasing trend in female leadership since 1975, and a rise in the average age of presidents since 1980. The academic expertise of presidents is diverse, with a significant proportion from science and technology, and a fluctuating percentage from law & politics and social science. The findings contribute to our understanding of academic leadership and pave the way to study its influence on university research and development to inform more effective science policy.

An outcome evaluation of NICHD training programs: 2000-2019

Christopher Belter and Amanda Sztein

Does Organization Size Undermine Disruptive Potential

Haochuan Cui and Linzhuo Li

ABSTRACT. When we discuss the social sources of creative destruction (1), there is a common perspective that such innovation typically originates from agile startups rather than from entrenched large corporations. This view is based on the flexibility of startups and their tolerance for risk, enabling them to better explore new technologies. On the other hand, large companies, due to their size and complex internal structures, are generally thought to move more slowly on the path of innovation, and their efforts to protect their existing market share often limit their capacity for disruptive innovation. However, is this view true in reality? How can this view reconcile with situations in the current age of AI that large tech-companies like Google and Meta seem to be playing a bigger role in driving innovations. While previous studies identified the negative effect of team sizes on disruption, few studies have investigated how these findings might scale up to the organizational level, especially considering that many large companies have recently restructured, adopting more flattened and decentralized models for their working units. Therefore, we study millions of company patents from MAG to examine the relationship between organization size and disruptive innovation. Our finding is that disruptive technologies are consistently more likely to occur in large companies across historical periods.

Are Patent Examiners killing disruptive innovation?

Haochuan Cui

ABSTRACT. How do examiners in the patent review system affect disruptive innovation? Although the patent system was established to protect innovation, previous research has shown that patented innovations are less and less disruptive. We systematically analyzed approximately 2 million USPTO patents from 2004 to 2018. Our Findings show that disruptive innovation is detrimental to a patent granted while examiner workload and examiner work experience both have a positive impact on a patent granted. A puzzling finding is that examiners' high-intensity workloads facilitate the approval of disruptive patents, while low-intensity workloads have the opposite effect. Using control effect regression, we confirm the surprising fact that current examiners are killing disruptive patents.

Geographic focus of the book collection in the Library of Congress

Kai Li

ABSTRACT. The Library of Congress (LoC), as one of the world's largest libraries, aims to provide the US Congress with top-quality research, analysis, and consultation to support its legislative and oversight duties. To support this critical mission, the library has developed its collection with a strong national interest in mind. This extended abstract presents preliminary findings from our project to investigate the geographical coverage of LoC book collection and how its changing portfolio mirrors shifted national and foreign policies of the United States.

This research also aims to demonstrate the significant research values of the library collection metadata for the community of science of science and research evaluation. The library metadata is created by heavy manual curation by professional librarians and using various controlled vocabularies, which underlie its strong research and cultural values. We argue that science of science research should pay more attention to this data source to cover more diverse research activities represented in published books, which is particularly important for the domain of social sciences and humanities and knowledge created beyond academia.

Utilizing the LoC's 2019 metadata dump, we acquired 7.78 million metadata records of books in the LOC collection published during 1970-2019 that were fully curated by librarians in the Machine-Readable Cataloging (MARC) format. We extracted all geographic terms in the book subject assigned according to the controlled vocabulary of Library of Congress Subject Headings (LCSH). We further mapped all extracted geographic names to the Geonames dataset (https://www.geonames.org/), a comprehensive dataset that covers global historic and contemporary geographic names. Among 4.53 million book records with any geographic terms, 98.1% of them can be mapped to a standardized country or region name, representing a place around the world that the books discuss. Below we discuss our two core findings based on the data.

From Anglo-American to global archives. Our results show that the LoC collection has been increasingly more diverse in its geographic focus over time. In the 1970s, a vast majority of books are focused on three primary regions: North America (31%), Western Europe (16%), and Northern Europe (12%). However, four decades later, only 22% of the books are focused on North America, which is closely followed by East Asia (15%). Regions such as Eastern Europe, Western Asia, Southern Europe, South Asia, and Southeast Asia have also seen their representations increase from the 1970s to the 2010s.

LoC's collection on countries echoes their historical trajectories. We observe a strong alignment between changes in the presentation of some countries and historical events in these countries. For example, there was a significant increase in share of books about Japan during the 1980s, coinciding with Japan's economic and technological boom and its trade frictions with the U.S. Similarly, for Mainland China, marked increases are observed during the reform and opening-up of the country as well as its entry into the World Trade Organization in the new century.

Our preliminary results demonstrate that meaningful patterns about knowledge production and dissemination represented by books can be identified from library collection metadata. In our next step, we will publish our fully processed and augmented raw LOC collection dataset to make it available to the broader community. At the same time, along the research presented in this abstract, we will also delve into the state-level geographic focus of the LoC's collection on the United States to understand how domestic political shifts influence the acquisition of the library collection. This study marks a significant step in analyzing the Library of Congress's role in representing global and local knowledge and its potential as a barometer for U.S. policy and political discourse.

Club versus Clan? Comparing innovation networks in the US and China

Yuanyi Zhen and Jar-Der Luo

ABSTRACT. National cultures shape the structure of interpersonal relationships across social and economic life. These influence not only the creation and reproduction of social opportunities, but also their consequences for the ideas and artifacts they generate for society. Theorists have argued for a distinction between the formation of Chinese “clan”-like hierarchical structures versus U.S. “club”-like structures, but difficulties in conceptualizing these small-group structural phenomena at scale and assessing their distribution across data in comparable domains has thwarted empirical analysis. Utilizing a fractal conception of micro-social structure embedded within agent models and validated on domains covering science, technology, and culture, here we demonstrate consistent differences between Chinese and U.S. networks that structure the production of innovation. Across innovation domains, Chinese networks manifest shorter path lengths, higher global efficiency, greater average closeness centrality, and lower inequality in degree centrality. U.S. networks, by contrast, manifest greater inequality in degree centrality, less information distribution speed, and more unevenly connected networks. These patterns reflect differences in respect for local authority and have marked implications for collective innovative outcomes. Chinese networks condition more turnover, and more personal mobility across topics, while U.S. networks support greater persistence and a carrying capacity for more ideas.

Influence of Reproducibility on Scientific Impact

Akhil Pandey Akella

ABSTRACT. Authors of scientific works use several measures to signal preliminary evidence for suggesting the reproducibility of their work. Inherently reproducible works are perceived differently because of the ease of effort present in recreating the experiments helping researchers to expand on others work. As a research question, the influence of reproducibility on the scholarly impact of a paper as observed through citations is under studied and we present this study to initiate the discussion. Although there exists a rich body of work assessing the impact of various factors while determining the scholarly impact, there aren't studies that take supplementary information, presence of code, data or methods in reproducing the original works as factors of analysis. Some researchers argue, artifacts connecting code or data may not be the end-all be all for reproducibility. Our objective is to explore these sentiments and provide evidence about the impact of a diverse group of features while predicting scholarly impact of papers.

The concept of reproducibility echoes diverse sentiments, and from a taxonomy standpoint, the formal definition of reproducibility has evolved as a term and a concept. We align with the National Academy of Sciences in defining \textit{reproducibility} as the process of obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis. This definition provides an ideal generalizable standard applicable to large sections of scientific research within the sub-domains of computer science. Consensus on this definition can make it easier to recognize procedures and protocols for validating and verifying scientific claims. The relevance and importance of reproducibility are heightened more than ever, given the current outgrowth of Artificial Intelligence (AI) models into diverse public domains. The modus operandi of scientific workflows in AI has shifted from offering posterior fixes to building a-priori reproducible AI pipelines. Regardless of the complexity, we can observe the push for making models, datasets, and algorithms available in containers, open-source repositories, and webpages. The significance of reproducibility is multifaceted. First, it upholds a standard for sustaining quality in the results and analysis of scholarly works, ensuring that scientific findings are robust, reliable, and unbiased. Second, it enables researchers to innovate and expand on proven findings quickly. Third, in the context of AI, reproducibility addresses essential safety and trust considerations by ensuring accountability within the systems implementing algorithms that make decisions affecting human lives.

We collected scholarly articles from ACM digital library badged as Results Reproduced where the experiments included in a research paper are possible to recreate without the use of author-supplied artifacts, and the primary findings of the publication have been validated and independently verified in a later investigation by a person or group other than the authors. Additionally, we gathered scholarly papers from the same venue that do not have any badge and refer to them as Unbadged articles. Utilizing both these groups of features and collecting the citations of each article as observed on Google scholar from the year 2023 helped us model a scholarly impact prediction problem. We have used a combination of Artifact, Structural, Linguistic, and Miscellaneous to predict the scholarly impact. These features are not conventional to citation prediction but are extremely useful for assessing reproducibility. Our motivation to extend the same attributes was to assess the impact of similar features, for reproducibility assessment on reproducible paper distributions on citations.

The plot from Fig 1.a) represents the feature importance of the best performing regression model. Overall, these findings underscore the importance of transparency, reproducibility, clear communication, and practical contributions in enhancing the scholarly impact of academic papers. They reflect broader trends in the scientific community towards open science and reproducibility, which are key to our interest in addressing the reproducibility crisis in AI.

The impact of the Russian invasion of Ukraine on higher education and scientific institutions in the European borderland

Adam Ploszaj

ABSTRACT. In this paper, we present the results of a study on the impact of the 2022 Russian invasion of Ukraine on the functioning of universities and other research institutions in 9 European countries bordering Ukraine, Russia, or Belarus. We carried out a large representative survey of scientists working in research institutions in Estonia, Finland, Hungary, Latvia, Lithuania, Moldova, Poland, Romania, and Slovakia. The survey questions were developed on the basis of a dozen individual interviews with scholars from the abovementioned countries. This helped us to identify the most important consequences of the invasion and their specific determinants and to develop relevant questions for the questionnaire. Respondents were invited from a random sample of corresponding authors of publications indexed on the Web of Science database. The survey was conducted online and was available in English and 8 other languages used in the 9 countries covered by the study (Estonian, Finnish, Hungarian, Lithuanian, Latvian, Polish, Romanian, Slovak). We collected 3,718 fully completed surveys, with adequate representation of all surveyed countries. About half the respondents answered the open question that requested examples of the impact of the invasion on them and their institutions (the average entry length was 55 words). Survey responses were collected from mid-February 2024 to the second half of March 2024—hence, it is not possible to present detailed quantitative results in this abstract.

The initial interviews and preliminary survey results show the greatest impact on educational functions, with a lesser effect on research and the so-called third role of universities. For a significant number of respondents affiliated with institutions located along a border with Belarus, Russia, or Ukraine, the repercussions of war are complex and multifaceted, with potentially enduring consequences. In brief, the results vary depending on the border, with positive (!) effects in regions contiguous to Ukraine, such as enhanced student interest and increased financial and collaborative opportunities for research. On the other hand, regions adjacent to Russia and Belarus have experienced mostly adverse consequences, such as the suspension of projects and collaborations, a decrease in the number of students, the need to redirect the research interests of particular scholars, and the modification of institutional strategies to cope with these disruptions. The impact of war depends on various factors, including the level of collaboration with Eastern European partners, the dependence of scholars and institutions on such collaborations, the specific scientific disciplines involved, the university’s ability to provide educational opportunities for international students or the strategic orientation of the affected institution.

Several scholars have investigated the impact of the Russian invasion of Ukraine on academic and educational activities. However, most authors have limited the scope of their analysis to the overall effects upon the Ukrainian science sector (Ganguli & Waldinger, 2023), the relocation of universities from occupied territories (Lopatina et al., 2023; Zakharova & Prodanova, 2023), or the prospects for Ukrainian scientists remaining in the country as well as those who emigrate (de Rassenfosse et al., 2023; Fiialka, 2022; Suchikova et al., 2023). However, research on the impact of the Russian invasion of Ukraine on science and higher education beyond Ukraine is scarce. Noteworthy studies relate to aiding scientists from Ukraine (Gharaibeh et al., 2023), the impact of economic sanctions imposed during the war on EU–Russia knowledge flows (Makkonen & Mitze, 2023), the reactions of students outside Ukraine to the conflict, repercussions on academic life (Kapsa et al., 2022), or the universities' responses to the Russian full-scale invasion of Ukraine (Kushnir, 2023; Kushnir et al., 2023).

Our study stands out for its complex research question. We investigate not only the impact of war on the scholarly sector in bordering regions but also its effect on existing inequalities among universities and research institutions in areas adjoining Belarus, Russia, or Ukraine. Many institutions in this region were already facing unfavorable conditions compared to those in more affluent countries in Europe (Benneworth, 2018). Thus, the invasion of Ukraine, in addition to creating new inequalities, may have also, and more importantly, exacerbated those that already existed.

The chief contribution of this study is an empirically based analysis of the impact of armed conflict on the functioning of scientific and higher education institutions located close to hostilities but outside the zone of direct war operations. In a broader sense, this is an analysis of the effects of an external shock on the operation of such institutions. The project results may be valuable for shaping public policy, including research and higher education policy, in response to this specific situation or similar circumstances that might occur in other regions of the world.

References Benneworth, P. (2018). Universities and Regional Economic Development: Engaging with the Periphery. Routledge. de Rassenfosse, G., Murovana, T., & Uhlbach, W.-H. (2023). The effects of war on Ukrainian research. Humanities and Social Sciences Communications, 10(1). Fiialka, S. (2022). "Assessment of war effects on the publishing activity and scientific interests of Ukrainian scholars". Knowledge and Performance Management, 6(1), 27–37. Ganguli, I., & Waldinger, F. (2023). War and Science in Ukraine. In Entrepreneurship and Innovation Policy and the Economy, volume 3. University of Chicago Press. Gharaibeh, F. A., Ahmad, I., & Malkawi, R. (2023). Impact of the Russia-Ukraine War on Education and International Students. 25. Kapsa, I., Lubik-Reczek, N., & Ušiak, J. (2022). War in Ukraine from the Perspective of Polish and Slovak University Students. Athenaeum, 75, 270–287. Kushnir, I. (2023). 'It is more than just education. It's also a peace policy': (Re)imagining the mission of the European Higher Education Area in the context of the Russian invasion of Ukraine. European Educational Research Journal. Kushnir, I., Milani, Z., & Mbah, M. F. (2023). UK higher education sector's response to the war in Ukraine. Journal of Applied Research in Higher Education, Lopatina, H., Tsybuliak, N., Popova, A., Bohdanov, I., & Suchikova, Y. (2023). University without Walls: Experience of Berdyansk State Pedagogical University during the war. Problems and Perspectives in Management, 21(2), 4–14. Makkonen, T., & Mitze, T. (2023). Geopolitical conflicts, sanctions and international knowledge flows: EU–Russia collaboration during the Ukraine crisis. The World Economy, 46(10), 2926–2949. Suchikova, Y., Tsybuliak, N., Lopatina, H., Shevchenko, L., & Popov, A. I. (2023). Science in times of crisis: How does the war affect the efficiency of Ukrainian scientists? Problems and Perspectives in Management, 21(1), 408–424. Zakharova, O., & Prodanova, L. (2023). A university displaced twice: Irreversible and erroneous losses of human capital. Problems and Perspectives in Management, 21(2), 123–132.