Shadow Program Committee: Designing for Diversity and Equity within Academic Communities

Academics’ roles have expanded in recent years: quality academics are expected to succeed not only in teaching, but also consistently demonstrate their excellence and impact in community engagement, administration, research, and their active engagement as members of research communities [5, 38]. Browning et al. [8] identified seven basic activities that are required to build successful academic careers: (1) having a research doctorate, (2) being mentored, (3) attending conferences, (4) supervising postgraduate students, (5) being part of an active research group, (6) receiving assistance to develop grant applications, and (7) receiving support for staff to develop their research careers. Consequently, for ECRs to have a successful academic career, they must be equipped with the necessary skills to build a research track record [8]. However, to receive these diverse types of support, ECRs and academics require adequate institutional support, which at times can be quite complex given the diverse disciplines, institutional cultures, personal motivations, and personal characteristics that influence the professional development process that ECRs undergo [38]. Given these factors, dynamic, nuanced, and varied approaches of support are critically needed so that ECRs within the post-COVID academic environment can thrive. Institutions need to create environments that allow ECRs to engage and network with successful researchers to develop their research capabilities and familiarize themselves with the ins and outs of research through mentorship [12]. Despite the benefits of engaging with experienced researchers, Shagrir [51] found that there are cases where experienced academics have minimal interest in engaging with ECRs and novice researchers. In light of this, as institutions develop collaborative spaces, there is the need to be aware of the preferences and motivations of both experienced researchers and ECRs so that continued support and collaboration is encouraged.

Donnelly [12] stresses the importance of collegiality and collaboration as a means towards academic professional development. Spaces that promote collaboration and peer learning are therefore encouraged [11, 51]. Building academic networks, communities, and opportunities to receive feedback to achieve meaningful change are useful ways in which collaboration can be fostered [51]. Academic Conferences have the potential to create opportunities for learning, collaborative engagement, and facilitate entry into research communities [53] and thus are a key space for academic development [11].

While academic publishing is dominated by Western research in many fields, an additional critique that is particular to computer science is that conferences, rather than journals, are the main venue for publication, further increasing the barrier to entry. Grudin argues: “in journal-centered fields, conferences represent work in progress toward journal publication. Higher acceptance rates enable participation by researchers from other disciplines, students, and practitioners who do not aim for journal publication” [18]. There is a vicious cycle that is created where those that attend conferences get more critical feedback and support, further exacerbating and entrenching ECRs who are already disadvantaged. Mentors are critical for career guidance, introductions and for obtaining documentation for job applications (e.g., recommendation and reference letters) [27 from [39]]. Like McKay and Buchanan [39] before us, it calls us to challenge the “myth of meritocracy” that is, highly pervasive in academia.

There is an increasing recognition of the need for inter-disciplinary and diverse, geographically distributed teams within HCI [4]. The calls for greater diversity respond to critiques of the field such as HCI’s disregard or misunderstanding of literature from other fields [36], and the fact that the majority of HCI papers published at prominent HCI venues report studies with participants from wealthy, Global North countries [56].

In response to this recognized need, new ideas have emerged to counter the disadvantage experienced by ECRs and HCI researchers who are located in the Global South [48]. For example, researchers have explored the use of digital platforms to promote ICT4D collaboration with researchers from LMICs [42]. The HCI Across Borders (HCIxB) community is growing, with an emphasis on strengthening HCI research quality transnationally and fostering ties between junior and senior researchers [29]. Muller and Fitzpatrick also created a workshop to help ECRs in identifying career pathways in HCI by bringing them into contact with senior researchers and mentorship opportunities [40]. In addition, researchers like Nacke and Wilson offered courses at HCI conferences on how to write and review papers [43, 58], with the aim to teach participants “what reviewers are looking for and how to signpost this information to make papers more attractive to read” [43]. However, these initiatives have not yet proven sufficient to address the challenges outlined above—not least because workshop participation requires attendance at HCI conferences. Our research is thus guided by the question “How do we create an inclusive environment for novice researchers to be part of the reviewing space and provide appropriate training?”

Despite past recommendations for training reviewers as a means of improving the peer-review process and increasing the numbers of qualified reviewers [54], there is still insufficient training for reviewers—especially those in the early stages of their research career [6, 44, 46]. The current processes of recruiting new reviewers are often opaque, as they are selected from existing reviewers’ own networks based on past reviewing and publication experiences in top conferences. This therefore limits opportunities for ECRs who have not built a review and publication track record in prestigious venues, or lack access to these conference networks.

HCIxB 2019 attempted to “recognize and foster research efforts situated in historically underrepresented contexts, inviting members to submit works in progress and iterate towards stronger scholarship” [31]. Participants who were currently in the middle of project implementation were also invited to present their work so that they could solicit actionable feedback from the HCIxB community [31]. In a similar vein, Munteanu et al. organized an ECR symposium to support PhD graduates through a formal mentoring process for strengthening professional networks and providing advice tailored to career challenges [41].

Such approaches are needed to train new reviewers that do not require significant time or financial investment. Academics are already overwhelmed and are stripped for time [16], so we need strategies that are not time-consuming. One such effort has been to bootstrap conferences with SPC programs to facilitate peer-based researcher upskilling through a community-driven approach [37]. SPC programs have been developed alongside other large, international conferences, and the organizers have shared their outcomes and learning. For example, in 2005 a SPC was run with the Sigcomm conference with a total of 42 participants [15]. Similarly, the SOSP conference in 2007 had an SPC for postgraduate students and early career researchers (ECR) [23]. Both SPC programs included participants mostly from Europe and North America. Both of these SPC programs focused on conducting an in-person SPC meeting to discuss the reviewed papers, rather than providing advance training or extensive resources on how to write reviews before the meeting. Through these programs, postgraduate and ECR learned strategies to improve their papers’ acceptance rate.

Another approach was employed at the ICML 2020 conference, with a slightly different motivation. Rather than focusing on training and introducing ECRs to reviewing, this program focused on expanding the reviewer pool to meet the growing demand for qualified reviewers in the machine learning community. The program recruited novice reviewers and employed a competitive selection process to decide who would participate in the program (unlike earlier programs that accepted all interested participants) [54]. Both this program and the Sigcomm 2005 program found some evidence that the quality of reviews by the SPC was equal or higher than the quality of reviews by the “real” PC. For example, at Sigcomm 2005, the length of SPC reviews and number of on-time submissions were higher for the SPC than the PC [15]. At ICML 2020, there were similar findings on review completion and length, and they also found that blinded meta-reviewers rated the quality of the novice reviewers’ reviews as higher on average than those of the usual conference reviewers [54].

However, there are limited instances of such programs, despite their potential to create value for participants, conference organizers, and authors. Previous attempts at training programs, surveyed above, have focused on this topic from the perspective of conference venues rather than that of equity and diversity. In other words, the question has been framed as “how can conferences attract more reviewers?” rather than “how can we enable the participation of overlooked groups within the academic community?”. Most of the training initiatives mentioned above were carried out in-person at Global North conference venues. Others, like that of Stelmakh et al. [54], were only open to junior researchers from five large, top US universities (CMU, MIT, UMD, UC Berkeley, and Stanford).

Our study was approved by the IRB of the first author, and informed consent was obtained from all of the participants. Participant recruitment was conducted in two phases. In the first phase, we recruited students, post-doctoral fellows and ECR to be a part of the SPC. We recruited participants through social media platforms and online academic groups. Initially, 81 participants expressed interest in participating and signed up for the program. However, nine participants dropped out prior to the first orientation session, due to personal or professional commitments, leaving a total of 72 participants. All participants were informed about the research aspect of the program and invited to participate in the study. Participation in the research component was entirely voluntary and they had the option to withdraw at any time without it impacting their experience in the SPC. The benefits for those taking part included a better understanding about the usefulness of the critical reviews they produced as part of the SPC process, serving as formative feedback in their own development as researchers. Over the course of the program, eight participants dropped out as they could not complete the reviews allocated to them. Data from such participants was excluded and not considered for analysis. Overall, 64 participants completed the entire program.

We had a total of 81 participants (Male=45, Female=34, two did not disclose) in the SPC (see Figure 3). They represented 53 different educational and research institutions from 26 countries around the globe. The majority of the participants were in the early stages of academia, with 55 of them being PhD candidates. We did not receive any applications from participants in South America. In addition, we did not capture the country of origin of participants, so countries present may be a more accurate representation of institutions of origin (for instance, there might have been South American scholars present who are based at US institutions who would nevertheless have put down US as their country. Although most participants (n=75) had written one or more academic papers in the past, they had no or minimal experience of formally reviewing papers. Thirty six participants (44%) had never reviewed a paper before. The number of participants who applied to be part of the SPC in comparison to the attendees for the actual conference was also notable. There were 150 total attendees for SUBVERT Conference, thus the SPC had \(53.3\%\) of that figure. It should be noted that relatively few members of the SPC eventually went on to attend SUBVERT Conference as we had many participants who were interested in becoming better reviewers independently of their interest in SUBVERT Conference. Indeed, the chairs found that SPC met a real need for ECR mentorship programs and that was the primary outcome (rather than increased conference attendance).

Our initial discussions with the General Chairs of the conference revealed the need for a safe space for junior researchers from around the world to improve their review-writing ability and better engage with the SUBVERT Conference community. Here, we detail the actions we planned based on our guiding principles.

We conducted Zoom training sessions as part of the SPC training. There were a total of four sessions: (1) two orientation meetings, (2) one PC meeting, and (3) one debrief meeting.

We had various streams of data that we collected for this research. All of the Zoom-based training sessions were recorded with the consent of the participants. In addition, all correspondence with the participants, in the form of e-mails and participation data via Google forms was recorded for analysis. We also captured the content from WhatsApp group conversations. A total of 492 messages were captured across all 4 groups (Avg = 118). These contained instructions about the program, training resources sent to the participants and the queries participants had about reviewing or any of the content shared. In the second phase, we contacted the paper authors to participate in the research process for the program. The authors were required to receive and assess the reviews made by the SPC on their papers and provide feedback on those reviews. 11 out of 20 authors agreed to participate in this research. We conducted interviews with 11 authors to obtain feedback on the quality and usefulness of the SPC reviews. One section of the interview with authors focused on how the SPC reviews compared to the technical program committee (TPC) reviews they had received. These interviews were conducted via Zoom and audio-recorded. In this way, the interactions between authors and SPC members were mediated by the project team. As such any inherent risks in direct contact between the two parties were averted. We transcribed the training sessions and the author interviews. These transcripts were thematically analyzed using inductive thematic analysis by four members of the research team [25]. Additionally, all correspondence with the participants in WhatsApp chats was analyzed to gain insights about participants’ views and reflections on both the training process and their expectations of SPC.

Junior researchers need extra guidance as they navigate typical processes of academia [35, 45]. Our study shows the importance of infrastructures that take into account mentorship from peers and seniors, access to curated resources and curated spaces to explore the boundaries of their discipline.

During the orientation meetings, many participants described their previous experiences of submitting to peer-reviewed conferences, and admitted their ignorance of what happens behind the scenes. Consequently, they ventured into the SPC program in order to learn the basics of reviewing structures (in addition to the technical aspects of writing reviews). For junior researchers, doctoral programs may provide training on critically reading research papers and academic writing but it is not until they “gain practical experience” [P32] that the learning truly takes place. From our own experience as researchers who have published at HCI conferences, we noted that it was not until we started reviewing papers that we improved as paper writers. This sentiment was put forward by our participants as well. “This experience would help me to understand what attributes of a paper make it more effective in communicating the subject matter with its readers. It will also help me to learn how to identify the strengths, weaknesses, and points to improve upon in a paper. These experiences will be a stepping stone for becoming a more focused researcher and better at presenting research works through quality writing” [P18]. Thus, taking part in the SPC was as much about becoming a better reviewer as it was about becoming a better author. Arguably, the two are inseparable.

Our data indicates that the SPC was valued by participants because it exposed them to disciplines outside their direct area of expertise (a concern particularly relevant for computer scientists). “I’m eager to learn more about the scholarship procedures, norms, and expectations in the SUBVERT Conference community... what methods, concepts, and practices are valued and/or missing within the SUBVERT Conference discourse” [P66]. Others, despite having experience with inter-disciplinary venues, wanted to continue their professional development and stated “I hope to sharpen my review skills and gain more exposure to the behind the scenes process of paper reviews. While I have been a reviewer before for conferences like CHI and DIS, I have never participated in a review mentorship program. I want to ensure that I am contributing back to the academic community in a beneficial way” [P64].

The networking potential of the SPC was also highly valued by participants. Participants believed the SPC would be a great opportunity to meet and interact with fellow ECR and to learn from experienced researchers around the world. For example, one participant mentioned “Besides learning and gaining experience, I look forward to enriching networking opportunities and getting exposed to the work being carried out in our research community” [P23]. However, while group-based programs like SPC are valuable, there is still a need for more direct-mentorship approaches. Our participants were based in institutional contexts where these opportunities were not available and wanted “to be mentored in reviewing research and conference papers” [P20]. Another participant bemoaned the general nature of information available about the reviewing processes for CS conferences and asked for “access to distinctive mentors... for critical and timely guidance in the academic review process” [P18]. Moreover, the participants were eager for individual-level feedback from the authors for the reviews they had submitted, but this was not possible due to authors’ privacy. Instead, the SPC organizers attempted to provide mediated feedback from the authors and illustrative examples through the authors’ quotes to help participants identify where they might have done (or not done) these things in their own reviews.

Since the majority of participants had limited experience of formally reviewing papers, many questions were raised throughout the program. For those who had never reviewed papers before, the SPC provided a platform to gain “first-hand experience of reviewing a full paper” [Group B Debrief Session]. For others who had previously reviewed a few papers, they gained the opportunity to learn “the entire process of reviewing for a conference, end to end” [Group A Debrief Session]. This included a deeper understanding of subcommittees and tracks (at conferences such as COMPASS), and how to frame their research to maximize their chances of gaining a fair hearing from reviewers. These discoveries helped to demystify the review process, especially for first time reviewers.

As anticipated, participants did indeed relate to us as organisers who were ECRs, and felt free to ask a range of questions during the training sessions that were unrelated to review-writing. Such questions prompted us to broaden the scope of the SPC discussions to be flexible to participant needs. Through this approach, we also noticed that SPC members felt comfortable in challenging biases within existing institutional mechanisms. For some attendees, who were based in institutions that exclusively targeted top-tier conferences, they were curious to discover more about alternatives and why some conferences are held in high esteem: “I am also curious to understand the difference in the reviewing process of CHI and other conferences. Why do we hear that the quality of CHI reviewing is better than others?” [Group B, WhatsApp].

SPC participants also raised questions about reviewing etiquette across conferences, especially as they are not explicitly differentiated. This can be a difficult terrain to navigate for novice researchers working in an interdisciplinary research area. One participant pointed out that sample reviews from HCI conferences that we provided looked “more detailed than what you would expect from a systems conference” [Group C Orientation Meeting 2]. At first glance, observations such as this seem basic—but they point to the numerous parts of the conferencing experience where the guidance of an experienced researcher is needed. Whenever such queries were raised, researchers in the WhatsApp group with some experience would often share their knowledge or advice. One such instance was when several papers had not followed the anonymous submission guidelines. Several participants brought this to notice, “I got one paper mentioning the author’s name and affiliation... I wonder if it is ethical to review. I mean reviewers might be biased!! What do you think?” [WhatsApp Group A]. In this case reviewers who had previously submitted to this conference shared their views on how to tactfully approach this concern.

SPC participants also learned how to engage with other reviewers to make accept/reject decisions and how to synthesize reviews when writing a meta-review. Participants sought guidance on how to resolve differences in opinion, asking questions like: “As meta-reviewers, can we use the “add comments” feature on HotCRP with the option “hidden from authors” as a way to discuss and deliberate among the reviewers, specifically if there are disagreements in terms of “overall merit” of a paper?” [Group D WhatsApp]. Taking on board the spirit of SPC, that is, a collaborative and constructive approach to review-writing, we noticed many reviewers taking extra efforts to write balanced and carefully-thought out reviews.

We also encountered questions exploring the nature of meta-reviews. For example, one participant asked: “When writing a meta review, should the score assigned to the paper be the average between 2AC and R or an independent choice of the meta reviewer?” [Group B WhatsApp]. Participants wondered about the role that the score can play, and the power that the meta-reviewer’s judgement plays in the peer-review process. Some even went so far as to suggest that the meta-reviewer role could be antithetical to the spirit of peer-review. “The metareview. can be problematic because it’s basically a translation... the authors might lose [out]... because the AC is favoring one idea over the other as... a centralized decision maker. Personally, I think that plays into un-democratizing academia, which makes me feel a little bit uncomfortable.” [Group A Orientation Meeting 2]. In this vein, some participants preferred meta-reviews that played the role of a helpful summary generator and interpreter (i.e., taking the brief comments of reviewers and helping authors understand how to address them). “So I do like the second review that we read, which was breaking out each reviewers’ work separately and just making them more direct in their wording and feedback to the authors”. Perhaps due to the inexperience of the participants with the review process, we did not encounter any discussions around contentious practices in mainstream conferences, for example, varying acceptance rates between subcommittees at a conference, Program Chairs enforcing stronger accept/reject ratings to reduce an imbalance of papers in the middle range.

As we have seen so far, taking part in the review training process not only provided participants with tips and techniques on how to review, but provoked them to engage with and ask questions about the wider conferencing process. This evoked two effects on our participants: (1) gaining a greater appreciation of wider research, (2) rethinking existing reviewing practices

Firstly, they reassessed their identity as researchers and their own writing practices: “reviewing opened my thoughts to the mindset of a reviewer” [Group B Debrief Session]. For some, it was a shift in their understanding of the importance of the reviewer role and a desire to serve in that role in the future: “being conscious that by reviewing a paper I’m playing a role in advancing knowledge in my field will stay with me throughout my academic journey” [Group C, Debrief Session]. In other words, as a result of their experience of being on both sides of the author-reviewer divide, they gained a broader understanding about how academic research operates: “[this experience] provided a better way of looking at research” [Group B Debrief Session]. Furthermore, the SPC training’s emphasis on reviewer mindset caused many participants to reflect on their own approaches to thinking through and writing reviews: “the [resources] helped me understand how to deal with internal biases which often crop up while reading a paper. I will now definitely try to have an internal dialogue to reduce these biases” [Group A, WhatsApp].

Secondly, the SPC proved to be a space for participants to challenge the peer-review process and the way it is practiced in Computer Science conferences. The majority of participants came into the program with little to no prior review knowledge and experience and, perhaps unsurprisingly, spoke frequently of the lack of confidence to review papers. Throughout the program as participants engaged in the different activities that were assigned, they began to grow in their confidence. Such confidence enabled the participants to not only see the program as a space to becoming better reviewers but also a space to identify ways in which the SPC and the overall academic review system could be enhanced. Some participants identified issues with current practices in reviewing, and discussed better approaches. They began to question the different process such as the the PC meeting and how it is conducted; the ranking of final papers that make it through a typical conference program; the selection of reviewers and the approaches for reviewers to engage and discuss papers before writing the final reviews. A dominant critique that came through has been in the area of reviewer self ratings: “This is questionable how people categorize themselves as expert or knowledgeable. They might be an expert on methods, but not expert in the domain. I have seen ‘experts’ that gave one hundred percent wrong feedback on my paper” [Group C Orientation Meeting 2]. A participant who had previous experience of reviewing papers noted how potential reviewers accept and reject papers based on limited information: “When review requests come to me... just reading a 150 word abstract you will probably not know [if you are an expert]. This is actually a problem of the system as well” [Group C Orientation Meeting 2].

Reviewer ratings do play a role in how a review is viewed by a meta reviewer or editor for a journal. However, the process in which such ratings take place have been questioned whether they are useful in the overall review process. At times, reviewers rate themselves as experts and provide poor quality reviews. Alternatively, there are cases where reviewers rate themselves as knowledgeable and do engage more fully with a paper and provide more meaningful feedback. In such cases, important questions arise about how useful the rating is in terms of the final decision on a paper. We need to explore alternative means of helping reviewers rate themselves more accurately or holistically, providing guidance and ways to capture their expertise across multiple dimensions (rather than a single 5-point summary scale). Questions were also raised about where expertise might be high but due to limited time spent, the reviews may not be useful and as such, there might be value in introducing a confidence score, where the review rates the value of their own review in terms of their own understanding. These are some of the concerns that were raised around existing practices by the SPC participants.

We also noticed a growth in participant self-efficacy through their own belief in being able to write good reviews, provided they were given extra time: “It gave me a lot of confidence. I am now able to write a review that I am proud of. I write knowing that I am giving it my best. Thank you again!” [Group C Debrief Meeting]. Participants frequently wished to have multiple rounds of discussions with their fellow reviewers before finalizing their final reviews, confident that through this process they could make sure to give the paper under consideration full coverage as each person took different angles. To facilitate this, many groups of participants asked for extra time, which was granted to them.

As part of the SPC, we introduced a post-review feedback activity that was designed to reduce the author-reviewer feedback loop. As part of this activity, we contacted the authors of the papers that were reviewed and conducted Zoom interviews with them using the reviews as a discussion point. During the interview, we referred back to participant reviews to get their perspective and motivation for particular sections they had contributed. Typically, reviewers receive no or limited feedback on the reviews that they provide to authors. In some reviewing platforms like HotCRP there are possibilities for other reviewers to anonymously rate their peers’ reviews. However, the only feedback that is solicited from authors is during the author rebuttal stage, during which authors have to defend the merit of their work (and as such, is not intended to be feedback on the quality of the reviews received).

This type of emphasis is not present in previous attempts at SPC, because in those attempts, the emphasis was on creating a separate safe space for amateur reviewers, similar to a moot-court experience for budding lawyers. However, we found that by exposing the reviews of amateur reviewers to real-world authors and receiving their live feedback, a rich avenue for formative feedback emerged. Authors indicated the ways in which the reviews were useful or not, and provided suggestions on how reviewers could improve going forward. The feedback from the authors was collated and summarized into core themes that formed part of the reflective discussions that occurred during the debrief meetings at the end of the program. Such formative feedback provided an additional learning opportunity for the SPC participants.

Many authors felt that the quality of the reviews from the SPC and TPC were similar, particularly in the cases when the SPC members were familiar with the authors’ field. For example, Author 7 said: “To be honest, if I hadn’t known that I received the reviews from the Shadow PC, [then] that would have sufficed as feedback. And also in terms of the decision... there is a very good congruence [with the Technical PC]”.

In many cases, authors even pointed out that the SPC reviews highlighted useful areas for improvement that were not identified by the TPC reviewers. “I think people [from the SPC] really put in a lot of effort. Because there are things that I didn’t get comments on from the Technical PC, that I feel are still very strong points that I’m going to pick up” [Author 7]. From our analysis of the reviews, we surmised that this difference could be attributed to the amount of time and scrutiny that SPC members applied to papers. Their increased efforts, focused on fewer papers than members of TPC (who typically have a greater reviewing load), meant that they were able to pick up more areas of improvement at a micro level. This can also be seen in Author 2’s assessment of the two sets of reviews they received: “we found actionable feedback in a number of the reviews... [Shadow PC reviews] were much clearer about, ‘Here is my concern with the paper, here is how I might address that”’ [Author 2]. However, it was not always the case that SPC reviews were on par with TPC reviews. As would be expected from a training program’s first group of trainees working in a cross-disciplinary conference space, many SPC members were assigned papers to review that were outside their disciplinary area of expertise.

Inevitably, this led to SPC members providing general and high-level feedback, not providing commentary on methods or domains within papers that they were not comfortable with or had limited exposure to: “I felt like the [Shadow PC] reviewers were more scratching the surface, not going into depth” [Author 6]. Indeed, as part of the training, we had advised the participants on strategies in such situations (i.e., acknowledging their limited area of expertise and focusing on areas on the paper that they were confident addressing).

The authors we spoke with also commented on the quality of the writing and communication from the reviewers. Reviews framed in a constructive and polite tone were considered highly desirable. For example, Author 2 said, “I feel like a lot of effort and thought went into making the reviews useful and critical, but not overly harsh. Like, yeah, you see lots of memes about ‘Reviewer 2,’ and I didn’t get ‘Reviewer 2’! And I’m grateful for that”. Based on their previous experience of receiving terse and unhelpfully-framed reviews, the present set of reviews were a welcome change. On the other hand, we also heard from one author whose paper was negatively rated by both SPC and TPC reviewers about negative experiences regarding the communication of reviewer feedback: “I thought [one Shadow PC reviewer] took a very aggressive tone... the way they were asking questions was, “How can you not do this? How can you...?” things like that. Whereas [another Shadow PC reviewer], had a very calm but critical stance which they took in the reviews. It definitely helps to have that calmness in the tone” [Author 6].

Experienced PC members often reference the relationships they built during the time they spent on the PC, suggesting that networking is a key outcome of participation in program committees. In addition, earlier SPC programs identified that participants wanted to “put a name to a face” and meet other participants [15]. Even before the pandemic there were barriers to entry to these committees for researchers lacking resources (e.g., time-off from teaching commitments and travel funding). Since 2020, the barriers have been exacerbated as researchers worldwide have been significantly impacted by the restricted opportunities for networking.

When designing online events, we should consider what is valued in their offline counterparts. Anderson & Anderson argue that the most important function of a conference is “the creation of opportunities for informal socialization, entertainment and networking. deal making... bonding and friendship building by members of that community” [3, p. x]. Unfortunately, online academic events struggle to replicate these benefits. Previous work has explored the design of conference spaces for socialization through the simulating venues for informal discussions (e.g., coffee hours) [22], or the use of interactive virtual hangout spaces (e.g., second life) [21, 49]. However, such techniques have had mixed success and uptake [49].

We approached the SPC program as a different means of facilitating networking in and around online conferencing events. Mindful of participants from the Global South, we made four design decisions aimed at supporting ECRs in gaining social capital and relationship-building: (1) splitting participants into four roughly equal cohorts (of approximately 20 individuals), who were then able to build relationships over the duration of the program. (2) participants were distributed across the groups to ensure that no one country or region was over-represented in any group (despite the inconvenience in meeting scheduling that this would have caused). This ensured that participants based in India (where 18 of the ECRs came from) were able to be co-reviewers with participants in Uganda or Iran. (3) participants were also distributed based on their career stage, that is, to ensure an even distribution of PhD candidates and post-docs in each cohort. (4) supporting asynchronous communication (through WhatsApp) was another priority due to the wide range of time-zones participants came from. This is a practice that is also common in other conferences, notably at the PC meeting for HCI conferences.

Furthermore, these design decisions helped to expand what has been done by previous SPC programs and earlier publications on them. Earlier SPCs were conducted in-person and, as a result, most participants were from high-income countries (Europe, US, and Canada) [15, 23], reducing access for ECRs from the Global South and limiting the breadth of networking opportunities. For instance, Feldmann et al. reported that invitees from Asia were unable to participate in the SIGCOMM SPC due to the cost of travel to Germany [15].

There are many new opportunities that we can identify for increasing social capital of junior researchers that have emerged since the widespread adoption of virtual and hybrid conferences. At a fundamental level, we argue that the review process can be redesigned in order to integrate junior researchers into the academic community and the process of peer-review. SPC is but one of many approaches that can be utilized to achieve this aim. Others have tried to pilot approaches such as: (1) introducing transparency: using application forms for applying to be part of a PC (and thus making the process of enlisting in reviewing papers more inclusive) (2) ECR-specific events: building in workshops or colloquiums that are deliberately focused on providing short and confidential mentorship sessions where confidentiality is possible [16]. We can also design new opportunities for peer-mentorship to reduce any challenges around senior researchers being burdened with mentoring activities. For instance, one such approach may involve pairing junior faculty members and recent PhD graduates with first-year PhD candidates.

One area that has been receiving significant attention is that of improving the peer-review process as a means to oversee quality in academic scholarship. Peer-reviewers and the peer-review process play a significant role in academic publishing. However, the peer-review process itself has to be effective in order to uphold the quality and validity of scholarly publications as it seeks to do [12]. The peer-review process continues to be strained as more academics enter the academic market and publications continue to rise. Consequently, there is a growing demand for reviewers to keep up with the growing publication outputs and conference/journal needs [46, 52, 57]. Moreover, it has proven difficult to maintain a pool of reviewers and have them meet deadlines [12]. According to Newman et al., the typical HCI review process “suffers from some recurring problems such as highly unpredictable submission levels, shortage of experienced reviewers, and a complex, tightly scheduled review cycles. Reviewers often encounter these problems in the form of extra papers to review, papers outside their area of expertise, and lack of sufficient time to do the job properly” [44].

Others before us, such as Birman and Schneider, have questioned the assumptions underpinning CS conferences’ reviewing processes and how they lead to reduced impact of researchers’ work. This will require a fundamental rethink: “Force fields are needed to encourage researchers to maximize their impact, but creating these force fields will likely require changing our culture and values” [7]. Grudin similarly argues that although a potential solution is to bring in junior researchers, they may feel more comfortable identifying minor flaws and less comfortable in declaring that work is more or less important [19]. Moreover, the standard expectations on reviewing have many nuances to them which are often not addressed. For instance, while reviewers are often told to identify if a paper is making original contributions, they are not told to identify “innovation for innovation’s sake” or where there are potentially negative consequences to innovation (for instance, if a new technology is introduced among a marginalized group). To identify this distinction requires nuance, which researchers like Sturdee et al. argue is lacking among ECRs and those new to the peer-review process [55].

If, as we have argued in this article so far, we have junior researchers involved more heavily in the conference peer-reviewing cycle and afforded review training opportunities, then we can build processes around diversifying the review load and tackling the time constraints often experienced by reviewers. This is just one example of the several challenges that both established and emerging conferences face: due to the voluntary nature of reviewing commitments made by established and senior researchers, many reviews are not submitted on time and sometimes are not submitted at all. This is a problem that those on program committees have become used to and there is no way to enforce timely submissions by reviewers, beyond standard approaches like sending repeated reminders. Often when senior researchers do not respond and the program committee is scrambling for reviewers, they turn to reviewers they trust and or junior researchers.

One of the main challenges we faced in delivering the SPC was the logistics of managing a conference cycle while the pandemic was happening. Emergency reviewing is always a challenge in peer-reviewing [47]. We faced this issue during SPC as well, exacerbated by the worldwide COVID-19 pandemic. SPC participants in South Asia were significantly affected as the timeline for SPC coincided with a huge humanitarian disaster on the Indian subcontinent. We thus had to make decisions to extend deadlines to account for the emergency situations faced by a number of our SPC participants. Initially, we had created generous timelines that allowed participants to provide reviews at their own pace. Perhaps unsurprisingly, many scheduled time towards the end of the timelines we provided which also coincided with personal health emergencies for many of our participants. While we initially gave a limited extension and asked participants to submit within the timescale, participants mobilized through the WhatsApp platform to request more time.

Despite this temporary challenge, what we have discovered in our endeavor is that junior researchers are highly motivated to take part in reviewing. We can thus design reviewing activities that enable the production of high quality reviews with the assistance of junior research contributions on certain micro-tasks. Let us take a few examples of where they can contribute: (1) they can help with paper allocation. Rather than using automated algorithms that conference submission platforms use, which are often unreliable. The paper allocation decided by junior researchers can then be vetted by PC members before being sent to potential reviewers. (2) They can help with screening papers for desk rejections and checking if the paper scope is relevant to the conference, again making it easier for the PC to have access to information to make informed decisions in less time. (3) Screening individual papers’ literature reviews to write notes on whether the paper has correctly surmised the cited papers’ arguments, and writing recommendations on whether certain papers need to be cited that are currently missing. Their notes can then be helpful to the reviewers. (4) Parsing the methodology of submitted papers to identify whether appropriate procedures have been followed or if there is an incomplete description of ethical requirements for working with that methodology. It is worth adding a caveat that many of these micro-tasks vary in the level of expertise required to carry them out. As such our intention here is to present them as provocations for the conferencing community to respond to these as potential design opportunities.

Halfaker et al. ask the question “Does the community have a conference or does a conference form a community?” [20]. Cabot et al. found that within the top Computer Science conferences, it is rare to find papers by newcomers to the conference (papers where none of the authors had previously published in that conference) [9]. The authors also argued that semi-newcomers, researchers who had never published in the main track but had published in other tracks (e.g., posters and demos) or satellite events, fared much better. There is a general feeling that “newcomers must first learn the community’s particular “culture” (in the widest sense of the word, including its topics of interest, preferred research methods, social behavior, vocabulary, and even writing style) either by simply attending the conference or warming-up publishing in satellite events, before being able to get their papers accepted in the main research track” [9]. However, this is an unnecessary barrier that is (unintentionally) enabled by insiders within the research community. Conference committees risk becoming a gated community, where external reviewers are called upon to fulfill quick review jobs but very few get invited to stay and gain from the community.

Our findings showcase the extent to which ECRs are unaware about the process of reviewing. There were numerous questions raised by the cohort throughout the program that exemplify this issue. Understanding reviewing processes, writing good reviews and meta-reviews, and simply understanding how paper accept/reject decisions are made were some such concerns. Although these may seem trivial to senior researchers, this is certainly not the case with ECRs. Even though 92% of our cohort had experience of writing papers, the process after they had submitted their work was somewhat unknown, as illustrated by our findings.

Postdoctoral researchers are a key part of the conference reviewing cycle. These same individuals, as authors, submit papers to conferences while not understanding the intricacies of the peer-review process. When these same individuals start reviewing papers, there is no feedback mechanism in place with which their reviewing skills can be assessed and improved. It is left to individual researchers to simply get better over time through their own efforts. Power dynamics are also at play between authors and reviewers. The anonymous format of the reviews gives a lot of power to the reviewer, and many have previously called for greater accountability of reviewers.

One of the key questions is what happens after the reviews are provided to the authors. There are many unanswered questions such as: were the reviews useful feedback to the authors, how were the reviews perceived from the authors’ perspective? If we have a pool of motivated ECRs then it is not a stretch to create an activity which is focused on collecting author feedback on the reviews received. This activity can be carried out by the ECRs. This can take the form of a 15-min video call.

With the expertise they have gained through their doctoral studies, ECRs are strategically placed to carry out a debrief with the author. This has some additional advantages as well. Enabling ECRs to network with global academic audiences in their field and conference of choice (indicated by the fact that they are already present at this conference and they have been assigned a paper within their area of expertise). This humanizes the process with the PhD being involved. Once the reviews have been sent back, the PhD can be told by the program committee about the authors’ identity (with the consent of the authors) to organize a call and then anonymously report back the PhD’s observations to other reviewers as well. Through this step, we will also recognize that ECRs have something very valuable to bring. The design of such pathways need to be carefully considered around the issue of maintaining double-blind reviews, particularly if unfavourable review decisions are made (e.g., through use of asynchronous text-based anonymous platforms). The key takeaway for conference designers is that review processes should be re-designed to scaffold in participation of junior researchers to improve efficiency and also to aid researcher development.

There are many models of improving the reviewing skills of junior researchers. Events like SPC are limited attempts at doing this. This SPC program involved a longer and more-intensive level of training for participants, relative to earlier SPC-type programs [15, 23, 54]. However, even with this additional support and training for participants, it is clear that once-off training programs like this are not the most effective strategies. Extant literature shows that for learning to be effective, learning processes need to be embedded within the week-to-week work of the participants. There was also a strong demand for this kind of ongoing development among participants in our SPC program. Previous research has shown that metrics for tracking academic performance improvements can denigrate into prescriptive methods or approaches that can embody top-down micro-management [13]. As such, we need approaches that are embedded and give greater agency to participants.

Another limitation of this kind of once-off approach to preparing reviewers is the time commitment and effort required on the part of the organizers, spread over many months. This limits its sustainability as an approach. We, the authors, were also in privileged positions to be able to do so through the pandemic: none of us had caring responsibilities, and we were all supported by our respective departments in allocating time towards the initiative. We acknowledge that not all researchers are in this position and as such, organizers need to be aware of these considerations. In our case, the SPC organizing team consisted of four members, which was adequate to share the load.

A third limitation pertains to the unequal impacts of the COVID-19 pandemic, the backdrop for our SPC efforts. Studies among researchers have shown that women are authoring less during the pandemic, while men’s productivity has gone up [30; 32] [39]. One potential explanation could be due to uneven caregiving burdens. Conferences being online make them easier to attend, but can also be impacted by expectations (both self-imposed and from peers and supervisors) about having to work a normal day on top of conference attendance ([18] from [39]). In our particular instance, we deliberately ensured the number of real-time events were few (four) and spread out over a period of 5 months. Despite these limitations, for our particular immediate purposes for SUBVERT Conference, this approach worked well for us. We were able to create a thorough training program, and the authors who received feedback from the SPC members were appreciative of the reviews. In future conferences, we will be trying a more embedded approach.

In many professions (e.g., healthcare and education), continuous professional development (CPD) is an expectation on practitioners. However, within academia there is limited attention that is paid to this aspect of our careers. Unfortunately, the lack of emphasis creates a divide between those who are engaged in, and have access to, activities that contribute to our professional development and those who do not. Those who are privileged have ready access to the informal opportunities for CPD that others do not.

For instance, academia is a highly networked sector and has been described as an “incestuous” industry [27]. Job positions, collaboration opportunities, and grant-writing are competitive, and those with connections with other researchers are rewarded with more opportunities and possibilities. A key part of the professional development of researchers in academia is about networking, mentorship and collaboration events. Networking widens the circle of civil society, private sector and research practitioners that an individual is connected with, making it more likely for them to hear about certain opportunities. Mentorship enables a researcher to overcome gaps in their knowledge, as they gain input from someone who is much further along in their academic journey. And collaborative events are spaces deliberately created to kickstart a working relationship between two or more practitioners with mutual interests.

In this article, we have argued that the research community, like other practice communities, needs a more holistic approach to enable the ongoing professional development of its members (particularly ECRs and those based in the Global south). While the arguments we have put forward are radical in nature and can be perceived as another burden to over-committed academics, adapting existing practices presents an opportunity to grow a global community of trained ECRs. Through such efforts we are in a better position to diversify programme committees and promote inclusion in conference communities. Our findings highlight the value of such initiatives in the professional development of ECRs. Through this article, we contribute to the discussion around future of conferences as community building activities.

Indeed, radical changes to the reviewing process are not new to computing conferences. Prior to 1999, in CHI, differing review criteria existed for papers depending on their category. This was abolished in favor of a unified reviewing criteria that assesses all papers on the strength of their contributions rather than importance relative to their particular category/subcommittee [44].