To help evaluate the performance of a large interdisciplinary, cross-institutional research project, we addressed a challenging complex of attributes. The five-year “innovative Urban Transitions and Aridregion Hydro-sustainability” (iUTAH) project was initiated in 2012 with support from the US National Science Foundation’s (NSF) “Established Program to Stimulate Competitive Research” (EPSCoR). The iUTAH research proposal mandated a comprehensive research collaboration assessment in Year 5 that would “…analyze the publication and citation patterns of researchers at all Utah institutions to identify the level of interconnectedness before iUTAH is awarded and in year 4 of the award [to] indicate the connections between institutions, disciplines, and individuals that were stimulated by the iUTAH activities.”
In the past, we and others have cast a variety of analytical and visualization tools to help analyze research attributes (Porter et al., 2010). Such measurement tools fall generally in the category of bibliometrics (De Bellis, 2009), with special note of:
Measures of citation diversity to help gauge interdisciplinary research knowledge interchange (Zhang, Rousseau, & Gläzel, 2016), including “Integration scores” (Porter et al., 2007; 2008), Rao-Stirling diversity (Rafols & Meyer, 2010; Stirling, 2007), and Diffusion scores (Carley & Porter, 2012; Garner, Porter, & Newman, 2014) Cross-research domain knowledge interchange (Kwon et al., under review; Porter et al., 2013) Science overlay maps to visually represent the diversity of publication, citation, or citing sub-disciplinary involvement (Carley et al., under review; Leydesdorff et al., 2013; Leydesdorff & Rafols, 2009; Porter & Rafols, 2009; Rafols et al., 2010; Riopelle, Leydesdorff, & Jie, 2014) Social networking analyses (de Nooy, Mrvar, & Batgelj, 2011), to observe collaboration patterns with visualizations (Garner et al., 2012; Leydesdorff, 2008)
Measuring the interdisciplinarity of research outputs is problematic. Wagner et al. (2011) review the state of the art and conclude that indicators of interdisciplinarity are not well-validated. Integration of knowledge into research designs and processes entails cognitive processes, with additional complexities when that research involves a team (
“Diversity” is a key concept on which many measures focus (Stirling, 2007). Stirling’s three-component model that considers variety, balance, and disparity is compelling. It is equivalent to the empirical approach of measuring “Integration” devised in support of evaluation efforts for the US National Academies Keck Futures Initiative (Porter, Roessner, & Heberger, 2008). Rafols (2014) treats various measures of diversity and coherence. Of note are recent efforts to measure variety, balance, and disparity separately (Wang, Thijs, & Glänzel, 2014; Yegros-Yegros et al., 2010; Yegros-Yegros, Rafols, & d’Este, 2015). For this research assessment, the Integration scoring provided a suitable measure of interdisciplinarity.
Integration and Diffusion scoring, and the science overlay maps, use Web of Science Categories (WoSCs) as the basic unit of sub-discipline categorization (Leydesdorff & Bornmann, 2016). There are various approaches, and attendant issues (Glänzel & Schubert, 2003; Klavans & Boyack, 2009, 2016; Rafols & Leydesdorff, 2009), but the WoSCs offer suitable sub-disciplinary granularity to accord with a key National Academies report’s recommendations (National Academies, 2005). In our experience, granularity should match the study’s main objectives. Recently we have tuned journal assignments to focus on research knowledge interchange among Cognitive Sciences, Education Research, and associated Border Fields (Youtie et al., 2017). We have generated informative results that consolidated the 200+ WoSCs into four “meta-disciplines” to assess interest (citations) from the natural sciences to a US NSF social science program (Garner et al., 2013). Or, one can seek much finer grain, as Klavans and Boyack (2017) demonstrate using some 91,000 topics to predict grant funding prospects. For the iUTAH assessment, interest keyed on the extent to which Center participants collaborated across disciplines, for which the WoSCs provided a manageable categorization.
We have applied various research assessment tools in the course of different research assessments of US federal funding programs, particularly to gauge interdisciplinarity. These include Environmental Protection Agency Science to Achieve Results (STAR) projects (Porter et al., 2003), and NSF Research and Evaluation on Education in Science & Engineering (REESE—Porter et al., 2013), Human and Social Dynamics Program (HSD—Garner et al., 2013), and Research Coordination Networks (RCN—Porter, Garner, & Crowl, 2012; Garner et al., 2012).
What one cites in a paper depends on multiple factors, including relevance, disciplinary training, disciplinary norms, and awareness (
Turning to the assessment task at hand, we focus on connections among institutions, disciplines, and individuals that were possibly bolstered by iUTAH activities. iUTAH aims to strengthen science regarding management of the state’s water resources
This paper reports on our efforts to combine and extend analytical and visualization tools to assess iUTAH’s contributions on multiple factors. We want to measure “connections” made across individuals, organizations, and disciplines, with express interest in variations by seniority (academic rank) and gender. We concentrate on research outputs and their impacts (in the form of citations); we do not address process data
We offer this study as a model of applying bibliometric tools to help assess the research outputs of a major center. It illustrates a quasi-experimental study design to provide practical comparisons that help gauge the effects of instituting the center. While one cannot generalize from a single such study, we offer it to suggest possible measurement approaches.
The research assessment (Garner & Porter, 2017) sought to measure changes in collaboration practices and publication patterns catalyzed by iUTAH, as well as their impact, in terms of extent of citation. This paper emphasizes considerations in measuring and depicting the collaboration attributes.
To achieve suitable comparisons, we determined to implement a quasi-experimental design (Campbell & Stanley, 1963). This approach is modeled on randomized experimental designs, adapted to real-world possibilities (
Time—comparing Before to After metrics for iUTAH subjects (and for the comparison groups) Group—benchmarking iUTAH results against two suitable comparison groups.
For the temporal comparisons, we used 2010–2012 as the Before period and 2014–2016 as After. We set aside 2013 as ambiguous with respect to research publications that are apt to reflect participation in the iUTAH project.
Lacking a randomly assigned control group equivalent to iUTAH researchers, we worked to develop reasonable “comparison groups.” Our first comparison group consisted of participants in two Utah-based university centers comparable in terms of environmental science emphases. Comparison Group 1 (CG1) consists of researchers associated with Interdisciplinary/integrated Research Centers (IRCs) in Utah. We chose two research centers from which to draw this sample—the Ecology Center (EC) at Utah State University (USU) and the Global Change & Sustainability Center (GCSC) at the University of Utah (UU)—these being two major research universities involved in iUTAH. EC and GCSC were selected as prominent centers with commensurate (not identical) interests relating to environmental sciences. EC has some 70 participating faculty (with some overlap with iUTAH) and has been in operation since the 1960’s. GCSC is newer and also has some of its faculty participating in iUTAH. Researchers participating in both iUTAH and EC or GCSC could be included in our sampling as iUTAH. We note that the CG1 centers were in operation during the Before period, so that comparison to iUTAH is not fully symmetric.
For our second comparison group (CG2), we sought individual researchers with similar disciplinary ties but not associated with iUTAH or CG1. For purposes of this article, CG2 is of less interest in that substantial collaboration among an arbitrary set of collegial researchers would not be expected to increase markedly in the time periods studied. We note CG2 here for completeness, and draw limited comparisons.
The general hypothesis is that participation in iUTAH increases collaboration and the degree of interdisciplinarity in the research of its members. Testing of the hypotheses of changed publication outputs, citations received, and especially, collaboration patterns centers on comparing the treatment and comparison groups, Before and After. The ideal pattern would show minimal change over time for the comparison group
We selected, in a non-random fashion, but without preconceived bias, a group of 45 tenure-track iUTAH researchers whose involvement spanned the length of the project. For practical purposes, these are nearly all the possible iUTAH participants as the dozen or so others mostly have special attributes (e.g., in faculty status or more limited involvement). We then matched them with random samples of equal size drawn from CG1 and CG2, and stratified according to institution, discipline, rank, and gender. The assessment report to iUTAH Garner, J., & Porter, A.L. (2017). iUTAH Research Assessment, Report to the iUTAH Program, Search Technology, Inc.
In addition to USU and UU, the iUTAH sample also involved researchers at Brigham Young University (BYU) and several Primarily Undergraduate Institutions (PUIs): Utah Valley University; Weber State University; USU branch campuses; and Salt Lake Community College (SLCC). However, CG1 lacks PUI representation because those centers do not involve PUI researchers. Our intent was to do preliminary analyses on the PUIs in iUTAH and in CG2 to compare and determine further analytic strategies regarding this small number of researchers, because we hypothesized that PUI faculty may be more significantly impacted by participation in iUTAH.
To address the assessment objectives, we compiled publication outputs in the form of abstract records gathered from two major research databases—Web of Science (WoS) and Google Scholar (GS). GS searches were conducted separately for each researcher (author) by first looking for a GS profile; if one existed, we extracted each article posted since 2010. If we did not find an author profile, then we searched for articles with that author name and extracted those. This was followed by manual checking to disambiguate author names, facilitated by using author affiliation information. Resulting records were formed into a single data set file by merging all 45 authors’ records into one GS set for each of the three groups (iUTAH, CG1, and CG2). Those files were pruned to remove duplicate records (e.g., co-authored records captured more than once).
WoS data were gathered and consolidated via corresponding processes to yield separate data sets for iUTAH, CG1, and CG2. We then merged the WoS and GS data sets for each research group, cleaned the resulting files by removing redundant information and formatting, and de-duplicated the cleaned files based on record (article) titles, removing the GS records where there was a corresponding WoS one.
Of the 45 faculty members in each group, not everyone published in each period. Total counts for GS analyses are limited to papers not also indexed in WoS. GS publications are diverse—we extracted distinct source titles for 630 of 868 GS records for 2014–2016. Those are somewhat noisy (not as consistent nor as clean as WoS fields of information). Nearly all are scholarly in nature. For instance, the five source titles occurring most frequently in these GS records are
Some general data attributes deserve noting. Average publication and citation values are of interest, but, as expected, those data tend to be highly skewed. Thus, analyses need to proceed with some caution. Publications range as high as 150 for one author. Of the 3,308 records (articles, etc.) in the full data set, 1,484 show no citations (but recall that the 2014–2016 papers have not had a great amount of time in which to be cited); one book has received 1,043 cites (a CG1 researcher co-authored it).
While this article focuses on discerning and depicting collaboration patterns, we first present the basic program research output and impact results to provide essential content (see Table 1). The first row shows how many of the 45 researchers in each group published one or more papers in the Before (2010–2012) and After (2014–2016) time periods. The next three rows show the publication counts—Total (from GS or WoS combined); Total from GS (excluding WoS duplicates); and Total from WoS.
Publication and citation metrics of authors in three groups of environmental researchers from Utah, 2010–2016.2010–2012 2014–2016 Publications iUtah CG1 CG2 iUtah CG1 CG2 # of group authors with papers 45 44 40 45 45 36 Total Records 702 636 293 788 666 295 Total from GS 431 367 166 407 337 165 Total from WoS 271 269 127 381 329 130 Average Times Cited 9.55 12.81 10.02 2.68 3.65 2.60 Average Times Cited GS 6.49 7.33 6.62 1.13 4.06 1.75 Average Times Cited WoS 14.42 20.49 14.46 3.81 5.13 3.52 Median Times Cited 2 3 1 0 0 0 Median Times Cited GS 0 0 0 0 0 0 Median Times Cited WoS 9 9 4 2 2 1 Cites/Year WoS 2.43 3.47 2.35 2.39 3.08 1.92 H-Index 35 42 22 19 22 14 H-Index GS 26 26 15 13 11 10 H-Index WoS 29 36 18 17 20 12 Integration score 0.535 0.457 0.428 0.523 0.489 0.468
Figure 1 plots the totals for each group, Before and After, for the WoS publications (the Total Records pattern is similar). Results support propositions that participating in a research center (iUTAH or one of the CG1 centers—EC or GC):
Attracts more prolific researchers (more papers in the Before period than the individual CG2 researchers) Boosts research productivity (both iUTAH and CG1 groups increase productivity in the After period, but we note that the CG1 centers were already operating in the Before period)
iUTAH participation appears to boost publication in leading journals (i.e., those indexed by WoS) more than does participation in the CG1 centers—i.e., the iUTAH publication rate in the After period exceeds that for CG1.
Citation comparisons are more difficult to interpret (Table 1). Publications in the Before period have much more time in which to accrue citations than do those in the After period. Also, citation data are highly skewed, so averages are influenced heavily by relatively few highly cited papers. Furthermore, citation rates vary by field, so uneven disciplinary concentrations could favor one group over another. Inspection of Table S-1 shows general correspondence between the leading WoSCs of iUTAH and CG1 publications, so that field propensities to citation are likely not to be too different. However, the CG2 WoSC distribution differs more, and the CG2 publication counts are considerably less robust. Too much should not be made of the relative citation rate for CG2.
Disciplinary emphases.Overall Rank Overall # Recs Web of Science Category CG1 CG1 CG2 CG2 iUTAH iUTAH 2010–2012 2014–2016 2010–2012 2014–2016 2010–2012 2014–2016 # Recs Rank # Recs Rank # Recs Rank # Recs Rank # Recs Rank # Recs Rank 1 282 Environmental Sciences 42 2 38 3 0 21 0 24 88 1 114 1 2 210 Geosciences, Multidisciplinary 29 3 48 2 20 1 11 4 40 4 62 4 3 203 Ecology 53 1 50 1 0 21 3 15 39 5 58 5 4 189 Meteorology & Atmospheric Sciences 29 3 32 4 0 21 0 24 51 3 77 2 5 184 Water Resources 13 9 25 7 2 15 1 19 66 2 77 2 6 79 Engineering, Civil 5 15 11 12 8 6 15 2 13 9 27 6 7 77 Multidisciplinary Sciences 14 7 20 8 11 3 8 7 9 12 15 10 8 69 Geography, Physical 22 5 26 6 2 15 1 19 4 15 14 11 9 67 Engineering, Environmental 14 7 16 10 0 21 3 15 15 7 19 7 10 66 Environmental Studies 9 12 29 5 3 14 4 13 11 10 10 13 11 63 Plant Sciences 20 6 17 9 1 18 5 11 10 11 10 13 12 47 Limnology 9 12 4 18 0 21 0 24 18 6 16 9 13 41 Geochemistry & Geophysics 1 21 0 24 6 10 11 4 5 14 18 8 14 37 Geography 10 11 16 10 1 18 1 19 3 16 6 15 15 36 Sociology 6 14 8 14 9 5 4 13 7 13 2 19 16 34 Soil Science 2 19 4 18 1 18 1 19 15 7 11 12 17 32 Energy & Fuels 5 15 10 13 6 10 6 10 1 19 4 16 18 29 Materials Science, Multidisciplinary 0 23 5 17 10 4 11 4 0 22 3 18 19 27 Chemistry, Multidisciplinary 1 21 1 21 15 2 8 7 1 19 1 20 20 25 Paleontology 5 15 6 16 8 6 5 11 0 22 1 20 21 25 Materials Science, Composites 0 23 1 21 7 9 17 1 0 22 0 23 22 25 Construction & Building Technology 2 19 4 18 5 13 13 3 0 22 1 20 23 23 Evolutionary Biology 12 10 7 15 0 21 1 19 3 16 0 23 24 20 Mechanics 4 18 1 21 6 10 3 15 2 18 4 16 25 11 Engineering, Geological 0 23 0 24 8 6 3 15 0 22 0 23 26 10 Engineering, Multidisciplinary 0 23 0 24 2 15 7 9 1 19 0 23
All that said, the general pattern shows CG1 publications to be more heavily cited, both Before and After. iUTAH and CG2 show relatively similar citation profiles. Again, interpretation of relative center participation impacts is not symmetric in that CG1 operated in both the 2010–2012 and 2014–2016 periods whereas iUTAH operated only in the 2014–2016 period.
H-index is one approach to reduce extreme influences of outlier high value items (Hirsch, 2005). It reports the number of publications receiving at least that many citations. Traditionally, the H-Index is applied to individuals; here, we adopt it for groups of individuals. Using this citation metric, CG1 generally leads, with iUTAH second.
Calculating citations on the basis of years since publication provides more comparable Before
We also broke out publication and citation activity by gender and rank (Garner & Porter, 2017), although this is not a focus of this paper. We do, nevertheless, note a few interesting results (see supplemental figures S-7a, b):
iUTAH Assistant Professors published more in the After period, while Full and Associate Professors’ publications held relatively constant; this result supports an aim of the project to stimulate early-career faculty’s research. Within iUTAH, Full Professors’ papers tended to be more interdisciplinary than those of more junior faculty, in both the Before and After periods.
The bottom row of Table 1 presents Integration scores (Porter et al., 2007; Porter, Roessner, & Heberger, 2008). Those reflect the diversity of WoSCs cited by a given paper. A higher Integration score reflects greater 1) Variety (the cited journals being in different WoSCs), 2) Balance (rather than being heavily concentrated in one or a few WoSCs), and 3) Disparity (how distant those cited WoSCs are from each other based on 2015 journal-to-journal cross-citation propensities). An Integration score of 0 would indicate that all of a paper’s references that were indexed by WoS were in a single WoSC; a score approaching 1 would be extremely diverse (i.e., drawing on widespread sources of research knowledge).
iUTAH Integration scores are statistically significantly higher than each comparison group, both for the Before and After periods, at the 0.001 level, by one-tailed
We also sought to discern the variety of fields in which these researchers publish. Science overlay maps (Rafols, Porter, & Leydesdorff, 2010) provide a means to show the distribution of the WoSCs of their publication journals. We generated science overlay maps for each of our three groups, for both periods (available in the Supplemental Materials, along with Table S-1 tallying the number of papers in leading WoSCs by each group, in each period). The most striking observation is that research emphases held quite stable over time.
Figure 3 presents the science overlay map for iUTAH in the After period. These maps locate the 227 WoSCs based on general cross-citation patterns for WoS in 2015. We color code five “macro” domains—Ecology and Environmental Science and Technology; Chemistry and Physics; Engineering and Mathematics; Psychology and Social Sciences; and Biology and Medicine. The nodes showing prominently in the maps are those with most publications by the iUTAH researchers—e.g., “Environmental sciences” is No. 1 for iUTAH in Figure 3. This map demonstrates that iUTAH researchers address not only multiple environmental specialties, but their papers also reach out into a range of scientific, engineering, and social sciences domains. Given the iUTAH Center’s focus is on better managing the state’s water resources, such diversity seems very suitable.
Figure 4 shows the counterpart science overlay map for CG1 in the After period. CG1 publications also present a generally comparable profile of multiple environmental specialties (with more geoscience activity), plus considerable publication in other macro domains. CG2 shows a generally similar makeup, but with some different emphases. Science overlay maps for CG2 appear as Figures S-3 and S-4 in the Supplemental Materials. Table S-1 (also in the Supplemental Materials) presents paper counts by each group for both periods; this allows further exploration into specific WoSC activity. iUTAH, for 2014–2016, shows 58 or more papers in each of five different WoSCs (so, averaging more than one per each of the 45 researchers included)—environmental sciences, multidisciplinary geosciences, ecology, meteorology and atmospheric sciences, and water resources. We feel the tabular and graphical presentations of publication diversity across WoSCs complement each other. The next section delves into the research collaboration patterns underlying these research outputs.
Figures 5 and 6 convey the most compelling results of our research assessment —a marked increase in collaboration attributable to participation in iUTAH activities. These charts contain four quadrants somewhat unequal in size and shape—separated by hand-drawn dashed lines), representing the institutions engaged—UU, USU, BYU, and PUIs. Nodes represent individual researchers, with larger nodes indicating more total publications (Google Scholar and WoS combined) Network maps are made using VantagePoint software [
Figure 5 maps co-authoring among the iUTAH researchers for 2010–2012 (This period precedes iUTAH activities that would affect research publication to any substantial degree). Interconnections across institutions are almost nonexistent.
Contrasting Figures 5 and 6, we note a substantial increase in networking within universities among many of the iUTAH participants (not all). This is pronounced for UU and USU, but less so for BYU and the PUIs. Even more dramatic is the large increase in connections between UU and USU colleagues in the After period.
Table 2 documents what is apparent in comparing Figures 5 and 6—marked increases in collaboration. This table tallies network statistics corresponding to Figures 5 and 6, as well as to Figures S-5, S-6, and S-7 (Supplemental Materials). Most striking is the confirmation of the visual patterns in Figures 5 and 6—iUTAH researchers increase their collaboration to a striking degree from Before to After.
Networking statistics within Utah researcher cohorts.iUTAH CG1 Before After Before After Average degree 1.2 5.422 0.133 0.178 Density 0.027 0.123 0.003 0.004 Total links 27 122 3 4 Links within discipline 12 63 3 3 Links across discipline 15 59 0 1 Links within rank 12 48 Links across rank 15 74 Links within gender 17 63 Links across gender 10 59 Links within university 24 14 Links across university 3 108
This is clear by inspection, with increases running more than 4-fold—note Average Degree (up from 1.2 to 5.4), Density (up from 0.03 to 0.12), and various tallies of links (e.g., total co-authorship links among the iUTAH researchers increase from 27 to 122).
Likewise, Table 2 comparisons between iUTAH After and CG1 After show large differences here, as also shown in comparing Figures 6 and 8. For one measure, total links in iUTAH After are 122
Figures 5 and 6 also color code the five disciplinary groupings within iUTAH. Focusing for a moment just on the USU and UU researchers:
Of 20 USU faculty in our sample, most show multiple links to other researchers; only two show no collaborations with iUTAH colleagues on their publications in the After period. Of 13 UU faculty, about half show multiple links; only two do not show any collaborations with other iUTAH colleagues in the After period. We see heavy interconnections between ‘core’ researchers in UU and USU in the After period, implying a change attributable to participation in the iUTAH project. Interdisciplinary connection density, the USU “core” of heavily interconnected researchers in the After period, includes all five disciplines. The UU “core” taps four of the disciplines. Collectively, this supports an assertion of high interdisciplinary connection, not “silos” of disciplinary sub-groups. The cross-disciplinary interconnection appears much weaker for the four BYU and eight PUI participating researchers.
Figures 7 and 8 provide the counterpart CG1 comparisons. [We choose to omit these analyses for CG2 as there is no reason to anticipate substantial collaboration among those individual researchers randomly assigned to a comparison group.] We see surprisingly little collaboration in CG1, considering that these are researchers associated with integrated environmental research centers. Since those are single university centers, we would not expect much UU-USU linkage. Also, BYU and the PUIs are not engaged in these centers. It is striking, since GCSC and EC were active in both periods, that the paucity of co-authoring sharply contrasts to the After period in which iUTAH was active (Figure 6).
Supplemental Figures S-6a and S-6b display iUTAH researchers by gender; we did not observe remarkable differences in group engagement of men versus women. Figures S-7a and S-7b are counterparts of Figures 5 and 6. They color-code rank instead of discipline. The impression is that USU has been more successful compared to other institutions in engaging their early-career faculty in research collaboration.
Substantively, we describe the publications, and citations to those publications, emanating from the researchers actively involved in the iUTAH project. We are expressly interested in the cross-disciplinary nature of the research, and show that it, indeed, engages many disciplines.
Moreover, we have run several quasi-experimental comparisons to compare those iUTAH research outputs to other environmental center participants (CG1) and to a matched set of individual researchers at Utah universities (CG2). These indicate that the iUTAH cohort participants are highly productive (equivalent to CG1; higher than CG2 researchers) prior to engaging in iUTAH, and that their publication activity appears to be increased by iUTAH involvement. CG1 publications in the 2010–2012 period attract somewhat more citations than the papers of researchers who join iUTAH (this is the Before period) Again we note that the CG1 centers were in operation in the 2010–2012 period, so, in that sense, they are not “Before.”
We note certain limitations in these analyses. This is a quasi-experimental study, so comparisons are somewhat guarded in the absence of randomized control (which is not plausible in such a real case environment). Given the realities of composing an interdisciplinary research organization, such a design is certainly in order. [i.e., researchers are not amenable to being randomly assigned to join research centers or not.] We sought best available comparisons, but have noted limitations; for instance, the CG1 centers were operating during the period Before iUTAH began work. WoS coverage of publications in the After period are not as complete as for the Before period, due to lags in indexing. More serious, as discussed, citations accrued by those publications are more severely truncated for the After period. Such limitations are reasonably compensated by having the cross-group comparisons of iUTAH to CG1 and CG2. Tallying citations is problematic in several regards, some already noted (e.g., field differences). Fixed citation windows offer advantages
Before
One of us noted the apparent disconnect between collaboration and interdisciplinarity, as gauged by Integration scores. Presumably, cross-disciplinary collaboration offers a route to garner insights from the multiple disciplines represented; so Integration scores that measure diversity of references cited in a paper might be expected to increase. For our iUTAH researchers, collaboration within the 45 researchers escalates from Before to After, but Integration scores do not. As shown in the bottom row of Table 1, those scores do not change significantly. However, Integration scores do increase from Before to After for CG1 and CG2.
This lack of an apparent correlation between Integration scores of the papers and extent of within-group collaboration caused us to consider collaboration more broadly. While not the focus of this assessment that seeks to gauge change in within-group (the 45 iUTAH researchers) collaboration, one could examine collaboration generally. We introduce this here briefly to suggest future research potential in analyzing overall, as well as local, collaboration patterns. Table 3 offers some basic comparisons.
Collaborative extent for all co-authoring on the Utah research group papers (indexed in Web of Science).iUTAH Comparison Group 1 Comparison Group 2 Median Mean Median Mean Median Mean No. of Authors on 2010–2012 papers 4 4.53 4 5.18 4 5.31 No. of Authors on 2014–2016 papers 5 7.12 4 6.15 4 5.02 No. of Organizational Affiliations on 2 2.56 2 3.3 2 2.5 2010–2012 papers No. of Organizational Affiliations on 3 3.65 2 3.83 2 2.37 2014–2016 papers
In brief, the After period papers of both center groups (iUTAH and CG1) show more authors per paper and more author affiliations (these are calculated at the organizational level; it was too difficult to measure departmental affiliations uniformly from the WoS records) per paper. The iUTAH increase in organizational affiliations is consistent with the expansion of cross-Utah organizational ties from Before to After (Table 2). Not an issue here, but we did run across articles with “mega” authoring (i.e., hundreds on a paper) in preliminary searches. Were those in one’s data set, they pose assessment challenges in discerning their degree of relationship (say to Center engagement) and statistical oddities.
To sum up, methodologically, we offer a multi-attribute suite of analytical and visual elements to help assess research outputs and impacts. Of particular interest are measures of research collaboration. We believe the combination of representations provides important, complementary perspectives. In particular, we find Figures 5 and 6 effective in capturing research networking attendant to interdisciplinary center activities. Without undue complexity, they graphically show changes from Before to After, and by comparing to counterpart figures for a comparison group, differences between iUTAH and CG1 here. In a presentation to an “All-Hands” meeting of iUTAH participants (July, 2017), they communicated effectively. These figures also show the nature of cross-organization and cross-disciplinary connections made among a group of researchers. [Supplemental figure variants do likewise for gender and ranks.]
The “take-away” from this study for others is a model of multiple measures of research outputs (publication characteristics) and impacts (citation characteristics). These could be adapted to meet assessment needs of other studies for which cross-disciplinarity and research collaboration are vital elements. In our view Figures 5–8, the co-author maps, are most novel—offering a concise way to communicate several facets of collaboration concisely. Perhaps this warrants consideration as a bibliometric assessment case study?