Showing 1–21 of 21 results for author: Brooks, C

Fairness Hub Technical Briefs: Definition and Detection of Distribution Shift

Authors: Nicolas Acevedo, Carmen Cortez, Chris Brooks, Rene Kizilcec, Renzhe Yu

Abstract: Distribution shift is a common situation in machine learning tasks, where the data used for training a model is different from the data the model is applied to in the real world. This issue arises across multiple technical settings: from standard prediction tasks, to time-series forecasting, and to more recent applications of large language models (LLMs). This mismatch can lead to performance redu… ▽ More Distribution shift is a common situation in machine learning tasks, where the data used for training a model is different from the data the model is applied to in the real world. This issue arises across multiple technical settings: from standard prediction tasks, to time-series forecasting, and to more recent applications of large language models (LLMs). This mismatch can lead to performance reductions, and can be related to a multiplicity of factors: sampling issues and non-representative data, changes in the environment or policies, or the emergence of previously unseen scenarios. This brief focuses on the definition and detection of distribution shifts in educational settings. We focus on standard prediction problems, where the task is to learn a model that takes in a series of input (predictors) $X=(x_1,x_2,...,x_m)$ and produces an output $Y=f(X)$. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Comments: Learning Engineering Virtual Institute

"Don't Step on My Toes": Resolving Editing Conflicts in Real-Time Collaboration in Computational Notebooks

Authors: April Yi Wang, Zihan Wu, Christopher Brooks, Steve Oney

Abstract: Real-time collaborative editing in computational notebooks can improve the efficiency of teamwork for data scientists. However, working together through synchronous editing of notebooks introduces new challenges. Data scientists may inadvertently interfere with each others' work by altering the shared codebase and runtime state if they do not set up a social protocol for working together and monit… ▽ More Real-time collaborative editing in computational notebooks can improve the efficiency of teamwork for data scientists. However, working together through synchronous editing of notebooks introduces new challenges. Data scientists may inadvertently interfere with each others' work by altering the shared codebase and runtime state if they do not set up a social protocol for working together and monitoring their collaborators' progress. In this paper, we propose a real-time collaborative editing model for resolving conflict edits in computational notebooks that introduces three levels of edit protection to help collaborators avoid introducing errors to both the program source code and changes to the runtime state. △ Less

Submitted 6 April, 2024; originally announced April 2024.

Probabilistically-sound beam search with masked language models

Authors: Charlie Cowen-Breen, Creston Brooks, Robert Calef, Anna Sappington

Abstract: Beam search with masked language models (MLMs) is challenging in part because joint probability distributions over sequences are not readily available, unlike for autoregressive models. Nevertheless, estimating such distributions has applications in many domains, including protein engineering and ancient text restoration. We present probabilistically-sound methods for beam search with MLMs. First,… ▽ More Beam search with masked language models (MLMs) is challenging in part because joint probability distributions over sequences are not readily available, unlike for autoregressive models. Nevertheless, estimating such distributions has applications in many domains, including protein engineering and ancient text restoration. We present probabilistically-sound methods for beam search with MLMs. First, we clarify the conditions under which it is theoretically sound to perform text infilling with MLMs using standard beam search. When these conditions fail, we provide a probabilistically-sound modification with no additional computational complexity and demonstrate that it is superior to the aforementioned beam search in the expected conditions. We then present empirical results comparing several infilling approaches with MLMs across several domains. △ Less

Submitted 22 February, 2024; originally announced February 2024.

Bridging Learnersourcing and AI: Exploring the Dynamics of Student-AI Collaborative Feedback Generation

Authors: Anjali Singh, Christopher Brooks, Xu Wang, Warren Li, Juho Kim, Deepti Pandey

Abstract: This paper explores the space of optimizing feedback mechanisms in complex domains, such as data science, by combining two prevailing approaches: Artificial Intelligence (AI) and learnersourcing. Towards addressing the challenges posed by each approach, this work compares traditional learnersourcing with an AI-supported approach. We report on the results of a randomized controlled experiment condu… ▽ More This paper explores the space of optimizing feedback mechanisms in complex domains, such as data science, by combining two prevailing approaches: Artificial Intelligence (AI) and learnersourcing. Towards addressing the challenges posed by each approach, this work compares traditional learnersourcing with an AI-supported approach. We report on the results of a randomized controlled experiment conducted with 72 Master's level students in a data visualization course, comparing two conditions: students writing hints independently versus revising hints generated by GPT-4. The study aimed to evaluate the quality of learnersourced hints, examine the impact of student performance on hint quality, gauge learner preference for writing hints with or without AI support, and explore the potential of the student-AI collaborative exercise in fostering critical thinking about LLMs. Based on our findings, we provide insights for designing learnersourcing activities leveraging AI support and optimizing students' learning as they interact with LLMs. △ Less

Submitted 20 November, 2023; originally announced November 2023.

Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation

Authors: Tung Phung, Victor-Alexandru Pădurean, Anjali Singh, Christopher Brooks, José Cambronero, Sumit Gulwani, Adish Singla, Gustavo Soares

Abstract: Generative AI and large language models hold great promise in enhancing programming education by automatically generating individualized feedback for students. We investigate the role of generative AI models in providing human tutor-style programming hints to help students resolve errors in their buggy programs. Recent works have benchmarked state-of-the-art models for various feedback generation… ▽ More Generative AI and large language models hold great promise in enhancing programming education by automatically generating individualized feedback for students. We investigate the role of generative AI models in providing human tutor-style programming hints to help students resolve errors in their buggy programs. Recent works have benchmarked state-of-the-art models for various feedback generation scenarios; however, their overall quality is still inferior to human tutors and not yet ready for real-world deployment. In this paper, we seek to push the limits of generative AI models toward providing high-quality programming hints and develop a novel technique, GPT4Hints-GPT3.5Val. As a first step, our technique leverages GPT-4 as a ``tutor'' model to generate hints -- it boosts the generative quality by using symbolic information of failing test cases and fixes in prompts. As a next step, our technique leverages GPT-3.5, a weaker model, as a ``student'' model to further validate the hint quality -- it performs an automatic quality validation by simulating the potential utility of providing this feedback. We show the efficacy of our technique via extensive evaluation using three real-world datasets of Python programs covering a variety of concepts ranging from basic algorithms to regular expressions and data analysis using pandas library. △ Less

Submitted 21 December, 2023; v1 submitted 5 October, 2023; originally announced October 2023.

Comments: Published in Learning Analytics and Knowledge Conference (LAK) 2024

Fairness Hub Technical Briefs: AUC Gap

Authors: Jinsook Lee, Chris Brooks, Renzhe Yu, Rene Kizilcec

Abstract: To measure bias, we encourage teams to consider using AUC Gap: the absolute difference between the highest and lowest test AUC for subgroups (e.g., gender, race, SES, prior knowledge). It is agnostic to the AI/ML algorithm used and it captures the disparity in model performance for any number of subgroups, which enables non-binary fairness assessments such as for intersectional identity groups. Th… ▽ More To measure bias, we encourage teams to consider using AUC Gap: the absolute difference between the highest and lowest test AUC for subgroups (e.g., gender, race, SES, prior knowledge). It is agnostic to the AI/ML algorithm used and it captures the disparity in model performance for any number of subgroups, which enables non-binary fairness assessments such as for intersectional identity groups. The teams use a wide range of AI/ML models in pursuit of a common goal of doubling math achievement in low-income middle schools. Ensuring that the models, which are trained on datasets collected in many different contexts, do not introduce or amplify biases is important for achieving the goal. We offer here a versatile and easy-to-compute measure of model bias for all the teams in order to create a common benchmark and an analytical basis for sharing what strategies have worked for different teams. △ Less

Submitted 25 September, 2023; v1 submitted 20 September, 2023; originally announced September 2023.

Logion: Machine Learning for Greek Philology

Authors: Charlie Cowen-Breen, Creston Brooks, Johannes Haubold, Barbara Graziosi

Abstract: This paper presents machine-learning methods to address various problems in Greek philology. After training a BERT model on the largest premodern Greek dataset used for this purpose to date, we identify and correct previously undetected errors made by scribes in the process of textual transmission, in what is, to our knowledge, the first successful identification of such errors via machine learnin… ▽ More This paper presents machine-learning methods to address various problems in Greek philology. After training a BERT model on the largest premodern Greek dataset used for this purpose to date, we identify and correct previously undetected errors made by scribes in the process of textual transmission, in what is, to our knowledge, the first successful identification of such errors via machine learning. Additionally, we demonstrate the model's capacity to fill gaps caused by material deterioration of premodern manuscripts and compare the model's performance to that of a domain expert. We find that best performance is achieved when the domain expert is provided with model suggestions for inspiration. With such human-computer collaborations in mind, we explore the model's interpretability and find that certain attention heads appear to encode select grammatical features of premodern Greek. △ Less

Submitted 1 May, 2023; originally announced May 2023.

Comments: 14 pages, 4 figures

ACM Class: I.2.7

Cross-Institutional Transfer Learning for Educational Models: Implications for Model Performance, Fairness, and Equity

Authors: Josh Gardner, Renzhe Yu, Quan Nguyen, Christopher Brooks, Rene Kizilcec

Abstract: Modern machine learning increasingly supports paradigms that are multi-institutional (using data from multiple institutions during training) or cross-institutional (using models from multiple institutions for inference), but the empirical effects of these paradigms are not well understood. This study investigates cross-institutional learning via an empirical case study in higher education. We prop… ▽ More Modern machine learning increasingly supports paradigms that are multi-institutional (using data from multiple institutions during training) or cross-institutional (using models from multiple institutions for inference), but the empirical effects of these paradigms are not well understood. This study investigates cross-institutional learning via an empirical case study in higher education. We propose a framework and metrics for assessing the utility and fairness of student dropout prediction models that are transferred across institutions. We examine the feasibility of cross-institutional transfer under real-world data- and model-sharing constraints, quantifying model biases for intersectional student identities, characterizing potential disparate impact due to these biases, and investigating the impact of various cross-institutional ensembling approaches on fairness and overall model performance. We perform this analysis on data representing over 200,000 enrolled students annually from four universities without sharing training data between institutions. We find that a simple zero-shot cross-institutional transfer procedure can achieve similar performance to locally-trained models for all institutions in our study, without sacrificing model fairness. We also find that stacked ensembling provides no additional benefits to overall performance or fairness compared to either a local model or the zero-shot transfer procedure we tested. We find no evidence of a fairness-accuracy tradeoff across dozens of models and transfer schemes evaluated. Our auditing procedure also highlights the importance of intersectional fairness analysis, revealing performance disparities at the intersection of sensitive identity groups that are concealed under one-dimensional analysis. △ Less

Submitted 1 May, 2023; originally announced May 2023.

Comments: Code to reproduce our experiments is available at https://github.com/educational-technology-collective/cross-institutional-transfer-learning-facct-2023

Journal ref: FAccT 2023

FLiMS: a Fast Lightweight 2-way Merger for Sorting

Authors: Philippos Papaphilippou, Wayne Luk, Chris Brooks

Abstract: In this paper, we present FLiMS, a highly-efficient and simple parallel algorithm for merging two sorted lists residing in banked and/or wide memory. On FPGAs, its implementation uses fewer hardware resources than the state-of-the-art alternatives, due to the reduced number of comparators and elimination of redundant logic found on prior attempts. In combination with the distributed nature of the… ▽ More In this paper, we present FLiMS, a highly-efficient and simple parallel algorithm for merging two sorted lists residing in banked and/or wide memory. On FPGAs, its implementation uses fewer hardware resources than the state-of-the-art alternatives, due to the reduced number of comparators and elimination of redundant logic found on prior attempts. In combination with the distributed nature of the selector stage, a higher performance is achieved for the same amount of parallelism or higher. This is useful in many applications such as in parallel merge trees to achieve high-throughput sorting, where the resource utilisation of the merger is critical for building large trees and internalising the workload for fast computation. Also presented are efficient variations of FLiMS for optimizing throughput for skewed datasets, achieving stable sorting or using fewer dequeue signals. Additionally, FLiMS is shown to perform well as conventional software on modern CPUs supporting single-instruction multiple-data (SIMD) instructions, surpassing the performance of some standard libraries for sorting. △ Less

Submitted 7 March, 2022; v1 submitted 10 December, 2021; originally announced December 2021.

Comments: Accepted for publication in the IEEE Transactions on Computers. Extension of the similarly-named FPT 2018 paper, with additional functionality variations, comparisons and experiments

Visualization of Intended Assistance for Acceptance of Shared Control

Authors: Connor Brooks, Daniel Szafir

Abstract: In shared control, advances in autonomous robotics are applied to help empower a human user in operating a robotic system. While these systems have been shown to improve efficiency and operation success, users are not always accepting of the new control paradigm produced by working with an assistive controller. This mismatch between performance and acceptance can prevent users from taking advantag… ▽ More In shared control, advances in autonomous robotics are applied to help empower a human user in operating a robotic system. While these systems have been shown to improve efficiency and operation success, users are not always accepting of the new control paradigm produced by working with an assistive controller. This mismatch between performance and acceptance can prevent users from taking advantage of the benefits of shared control systems for robotic operation. To address this mismatch, we develop multiple types of visualizations for improving both the legibility and perceived predictability of assistive controllers, then conduct a user study to evaluate the impact that these visualizations have on user acceptance of shared control systems. Our results demonstrate that shared control visualizations must be designed carefully to be effective, with users requiring visualizations that improve both legibility and predictability of the assistive controller in order to voluntarily relinquish control. △ Less

Submitted 24 August, 2020; originally announced August 2020.

Comments: To appear at IROS 2020

Isometric force pillow: using air pressure to quantify involuntary finger flexion in the presence of hypertonia

Authors: Caitlyn E. Seim, Chuzhang Han, Alexis J. Lowber, Claire Brooks, Marie Payne, Maarten G. Lansberg, Kara E. Flavin, Julius P. A. Dewald, Allison M. Okamura

Abstract: Survivors of central nervous system injury commonly present with spastic hypertonia. The affected muscles are hyperexcitable and can display involuntary static muscle tone and an exaggerated stretch reflex. These symptoms affect posture and disrupt activities of daily living. Symptoms are typically measured using subjective manual tests such as the Modified Ashworth Scale; however, more quantitati… ▽ More Survivors of central nervous system injury commonly present with spastic hypertonia. The affected muscles are hyperexcitable and can display involuntary static muscle tone and an exaggerated stretch reflex. These symptoms affect posture and disrupt activities of daily living. Symptoms are typically measured using subjective manual tests such as the Modified Ashworth Scale; however, more quantitative measures are necessary to evaluate potential treatments. The hands are one of the most common targets for intervention, but few investigators attempt to quantify symptoms of spastic hypertonia affecting the fingers. We present the isometric force pillow (IFP) to quantify involuntary grip force. This lightweight, computerized tool provides a holistic measure of finger flexion force and can be used in various orientations for clinical testing and to measure the impact of assistive devices. △ Less

Submitted 21 August, 2020; originally announced August 2020.

ACM Class: H.5.0; J.3; H.1.2; I.3.6

Journal ref: Isometric force pillow: using air pressure to quantify involuntary finger flexion in the presence of hypertonia. In International Conference on NeuroRehabilitation (pp. 835-840). 2020. Springer, Cham

Building Second-Order Mental Models for Human-Robot Interaction

Authors: Connor Brooks, Daniel Szafir

Abstract: The mental models that humans form of other agents---encapsulating human beliefs about agent goals, intentions, capabilities, and more---create an underlying basis for interaction. These mental models have the potential to affect both the human's decision making during the interaction and the human's subjective assessment of the interaction. In this paper, we surveyed existing methods for modeling… ▽ More The mental models that humans form of other agents---encapsulating human beliefs about agent goals, intentions, capabilities, and more---create an underlying basis for interaction. These mental models have the potential to affect both the human's decision making during the interaction and the human's subjective assessment of the interaction. In this paper, we surveyed existing methods for modeling how humans view robots, then identified a potential method for improving these estimates through inferring a human's model of a robot agent directly from their actions. Then, we conducted an online study to collect data in a grid-world environment involving humans moving an avatar past a virtual agent. Through our analysis, we demonstrated that participants' action choices leaked information about their mental models of a virtual agent. We conclude by discussing the implications of these findings and the potential for such a method to improve human-robot interactions. △ Less

Submitted 13 September, 2019; originally announced September 2019.

Report number: AI-HRI/2019/18

Journal ref: 2019 AAAI Fall Symposium Series

Beyond A/B Testing: Sequential Randomization for Developing Interventions in Scaled Digital Learning Environments

Authors: Timothy NeCamp, Josh Gardner, Christopher Brooks

Abstract: Randomized experiments ensure robust causal inference that are critical to effective learning analytics research and practice. However, traditional randomized experiments, like A/B tests, are limiting in large scale digital learning environments. While traditional experiments can accurately compare two treatment options, they are less able to inform how to adapt interventions to continually meet l… ▽ More Randomized experiments ensure robust causal inference that are critical to effective learning analytics research and practice. However, traditional randomized experiments, like A/B tests, are limiting in large scale digital learning environments. While traditional experiments can accurately compare two treatment options, they are less able to inform how to adapt interventions to continually meet learners' diverse needs. In this work, we introduce a trial design for developing adaptive interventions in scaled digital learning environments -- the sequential randomized trial (SRT). With the goal of improving learner experience and developing interventions that benefit all learners at all times, SRTs inform how to sequence, time, and personalize interventions. In this paper, we provide an overview of SRTs, and we illustrate the advantages they hold compared to traditional experiments. We describe a novel SRT run in a large scale data science MOOC. The trial results contextualize how learner engagement can be addressed through inclusive culturally targeted reminder emails. We also provide practical advice for researchers who aim to run their own SRTs to develop adaptive interventions in scaled digital learning environments. △ Less

Submitted 31 January, 2019; v1 submitted 26 October, 2018; originally announced October 2018.

Journal ref: 2019, The 9th International Learning Analytics & Knowledge Conference, Tempe, AZ, USA. ACM, New York, NY, USA

Enabling End-To-End Machine Learning Replicability: A Case Study in Educational Data Mining

Authors: Josh Gardner, Yuming Yang, Ryan Baker, Christopher Brooks

Abstract: The use of machine learning techniques has expanded in education research, driven by the rich data from digital learning environments and institutional data warehouses. However, replication of machine learned models in the domain of the learning sciences is particularly challenging due to a confluence of experimental, methodological, and data barriers. We discuss the challenges of end-to-end machi… ▽ More The use of machine learning techniques has expanded in education research, driven by the rich data from digital learning environments and institutional data warehouses. However, replication of machine learned models in the domain of the learning sciences is particularly challenging due to a confluence of experimental, methodological, and data barriers. We discuss the challenges of end-to-end machine learning replication in this context, and present an open-source software toolkit, the MOOC Replication Framework (MORF), to address them. We demonstrate the use of MORF by conducting a replication at scale, and provide a complete executable container, with unique DOIs documenting the configurations of each individual trial, for replication or future extension at https://github.com/educational-technology-collective/fy2015-replication. This work demonstrates an approach to end-to-end machine learning replication which is relevant to any domain with large, complex or multi-format, privacy-protected data with a consistent schema. △ Less

Submitted 10 July, 2018; v1 submitted 13 June, 2018; originally announced June 2018.

Comments: arXiv admin note: text overlap with arXiv:1801.05236

Dropout Model Evaluation in MOOCs

Authors: Josh Gardner, Christopher Brooks

Abstract: The field of learning analytics needs to adopt a more rigorous approach for predictive model evaluation that matches the complex practice of model-building. In this work, we present a procedure to statistically test hypotheses about model performance which goes beyond the state-of-the-practice in the community to analyze both algorithms and feature extraction methods from raw data. We apply this m… ▽ More The field of learning analytics needs to adopt a more rigorous approach for predictive model evaluation that matches the complex practice of model-building. In this work, we present a procedure to statistically test hypotheses about model performance which goes beyond the state-of-the-practice in the community to analyze both algorithms and feature extraction methods from raw data. We apply this method to a series of algorithms and feature sets derived from a large sample of Massive Open Online Courses (MOOCs). While a complete comparison of all potential modeling approaches is beyond the scope of this paper, we show that this approach reveals a large gap in dropout prediction performance between forum-, assignment-, and clickstream-based feature extraction methods, where the latter is significantly better than the former two, which are in turn indistinguishable from one another. This work has methodological implications for evaluating predictive or AI-based models of student success, and practical implications for the design and targeting of at-risk student models and interventions. △ Less

Submitted 16 February, 2018; originally announced February 2018.

Journal ref: Eighth AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), 2018

Evaluating Predictive Models of Student Success: Closing the Methodological Gap

Authors: Josh Gardner, Christopher Brooks

Abstract: Model evaluation -- the process of making inferences about the performance of predictive models -- is a critical component of predictive modeling research in learning analytics. We survey the state of the practice with respect to model evaluation in learning analytics, which overwhelmingly uses only naive methods for model evaluation or statistical tests which are not appropriate for predictive mo… ▽ More Model evaluation -- the process of making inferences about the performance of predictive models -- is a critical component of predictive modeling research in learning analytics. We survey the state of the practice with respect to model evaluation in learning analytics, which overwhelmingly uses only naive methods for model evaluation or statistical tests which are not appropriate for predictive model evaluation. We conduct a critical comparison of both null hypothesis significance testing (NHST) and a preferred Bayesian method for model evaluation. Finally, we apply three methods -- the na{ï}ve average commonly used in learning analytics, NHST, and Bayesian -- to a predictive modeling experiment on a large set of MOOC data. We compare 96 different predictive models, including different feature sets, statistical modeling algorithms, and tuning hyperparameters for each, using this case study to demonstrate the different experimental conclusions these evaluation techniques provide. △ Less

Submitted 13 June, 2018; v1 submitted 19 January, 2018; originally announced January 2018.

MORF: A Framework for Predictive Modeling and Replication At Scale With Privacy-Restricted MOOC Data

Authors: Josh Gardner, Christopher Brooks, Juan Miguel L. Andres, Ryan Baker

Abstract: Big data repositories from online learning platforms such as Massive Open Online Courses (MOOCs) represent an unprecedented opportunity to advance research on education at scale and impact a global population of learners. To date, such research has been hindered by poor reproducibility and a lack of replication, largely due to three types of barriers: experimental, inferential, and data. We presen… ▽ More Big data repositories from online learning platforms such as Massive Open Online Courses (MOOCs) represent an unprecedented opportunity to advance research on education at scale and impact a global population of learners. To date, such research has been hindered by poor reproducibility and a lack of replication, largely due to three types of barriers: experimental, inferential, and data. We present a novel system for large-scale computational research, the MOOC Replication Framework (MORF), to jointly address these barriers. We discuss MORF's architecture, an open-source platform-as-a-service (PaaS) which includes a simple, flexible software API providing for multiple modes of research (predictive modeling or production rule analysis) integrated with a high-performance computing environment. All experiments conducted on MORF use executable Docker containers which ensure complete reproducibility while allowing for the use of any software or language which can be installed in the linux-based Docker container. Each experimental artifact is assigned a DOI and made publicly available. MORF has the potential to accelerate and democratize research on its massive data repository, which currently includes over 200 MOOCs, as demonstrated by initial research conducted on the platform. We also highlight ways in which MORF represents a solution template to a more general class of problems faced by computational researchers in other domains. △ Less

Submitted 21 August, 2018; v1 submitted 16 January, 2018; originally announced January 2018.

Student Success Prediction in MOOCs

Authors: Josh Gardner, Christopher Brooks

Abstract: Predictive models of student success in Massive Open Online Courses (MOOCs) are a critical component of effective content personalization and adaptive interventions. In this article we review the state of the art in predictive models of student success in MOOCs and present a categorization of MOOC research according to the predictors (features), prediction (outcomes), and underlying theoretical mo… ▽ More Predictive models of student success in Massive Open Online Courses (MOOCs) are a critical component of effective content personalization and adaptive interventions. In this article we review the state of the art in predictive models of student success in MOOCs and present a categorization of MOOC research according to the predictors (features), prediction (outcomes), and underlying theoretical model. We critically survey work across each category, providing data on the raw data source, feature engineering, statistical model, evaluation method, prediction architecture, and other aspects of these experiments. Such a review is particularly useful given the rapid expansion of predictive modeling research in MOOCs since the emergence of major MOOC platforms in 2012. This survey reveals several key methodological gaps, which include extensive filtering of experimental subpopulations, ineffective student model evaluation, and the use of experimental data which would be unavailable for real-world student success prediction and intervention, which is the ultimate goal of such models. Finally, we highlight opportunities for future research, which include temporal modeling, research bridging predictive and explanatory student models, work which contributes to learning theory, and evaluating long-term learner success in MOOCs. △ Less

Submitted 10 April, 2018; v1 submitted 16 November, 2017; originally announced November 2017.

Citing and Reading Behaviours in High-Energy Physics. How a Community Stopped Worrying about Journals and Learned to Love Repositories

Authors: Anne Gentil-Beccot, Salvatore Mele, Travis Brooks

Abstract: Contemporary scholarly discourse follows many alternative routes in addition to the three-century old tradition of publication in peer-reviewed journals. The field of High- Energy Physics (HEP) has explored alternative communication strategies for decades, initially via the mass mailing of paper copies of preliminary manuscripts, then via the inception of the first online repositories and digita… ▽ More Contemporary scholarly discourse follows many alternative routes in addition to the three-century old tradition of publication in peer-reviewed journals. The field of High- Energy Physics (HEP) has explored alternative communication strategies for decades, initially via the mass mailing of paper copies of preliminary manuscripts, then via the inception of the first online repositories and digital libraries. This field is uniquely placed to answer recurrent questions raised by the current trends in scholarly communication: is there an advantage for scientists to make their work available through repositories, often in preliminary form? Is there an advantage to publishing in Open Access journals? Do scientists still read journals or do they use digital repositories? The analysis of citation data demonstrates that free and immediate online dissemination of preprints creates an immense citation advantage in HEP, whereas publication in Open Access journals presents no discernible advantage. In addition, the analysis of clickstreams in the leading digital library of the field shows that HEP scientists seldom read journals, preferring preprints instead. △ Less

Submitted 25 November, 2009; v1 submitted 30 June, 2009; originally announced June 2009.

Comments: Version to be published in Scientometrics

Report number: CERN-OPEN-2009-012, SLAC-PUB-13693

Information Resources in High-Energy Physics: Surveying the Present Landscape and Charting the Future Course

Authors: Anne Gentil-Beccot, Salvatore Mele, Annette Holtkamp, Heath B. O'Connell, Travis C. Brooks

Abstract: Access to previous results is of paramount importance in the scientific process. Recent progress in information management focuses on building e-infrastructures for the optimization of the research workflow, through both policy-driven and user-pulled dynamics. For decades, High-Energy Physics (HEP) has pioneered innovative solutions in the field of information management and dissemination. In li… ▽ More Access to previous results is of paramount importance in the scientific process. Recent progress in information management focuses on building e-infrastructures for the optimization of the research workflow, through both policy-driven and user-pulled dynamics. For decades, High-Energy Physics (HEP) has pioneered innovative solutions in the field of information management and dissemination. In light of a transforming information environment, it is important to assess the current usage of information resources by researchers and HEP provides a unique test-bed for this assessment. A survey of about 10% of practitioners in the field reveals usage trends and information needs. Community-based services, such as the pioneering arXiv and SPIRES systems, largely answer the need of the scientists, with a limited but increasing fraction of younger users relying on Google. Commercial services offered by publishers or database vendors are essentially unused in the field. The survey offers an insight into the most important features that users require to optimize their research workflow. These results inform the future evolution of information management in HEP and, as these researchers are traditionally ``early adopters'' of innovation in scholarly communication, can inspire developments of disciplinary repositories serving other communities. △ Less

Submitted 22 April, 2008; v1 submitted 16 April, 2008; originally announced April 2008.

Comments: Removed internal pdf links broken on some pdf viewers

Report number: CERN-OPEN-2008-010,DESY-08-040,FERMILAB-PUB-08-077-BSS,SLAC-PUB-13199

Journal ref: J.Am.Soc.Inf.Sci.60:150-160,2009

Open Access Publishing in Particle Physics: A Brief Introduction for the non-Expert

Authors: Travis C. Brooks

Abstract: Open Access to particle physics literature does not sound particularly new or exciting, since particle physicists have been reading preprints for decades, and arXiv.org for 15 years. However new movements in Europe are attempting to make the peer-reviewed literature of the field fully Open Access. This is not a new movement, nor is it restricted to this field. However, given the field's history… ▽ More Open Access to particle physics literature does not sound particularly new or exciting, since particle physicists have been reading preprints for decades, and arXiv.org for 15 years. However new movements in Europe are attempting to make the peer-reviewed literature of the field fully Open Access. This is not a new movement, nor is it restricted to this field. However, given the field's history of preprints and eprints, it is well suited to a change to a fully Open Access publishing model. Data shows that 90% of HEP published literature is freely available online, meaning that HEP libraries have little need for expensive journal subscriptions. As libraries begin to cancel journal subscriptions, the peer review process will lose its primary source of funding. Open Access publishing models can potentially address this issue. European physicists and funding agencies are proposing a consortium, SCOAP3, that might solve many of the objections to traditional Open Access publishing models in Particle Physics. These proposed changes should be viewed as a starting point for a serious look at the field's publication model, and are at least worthy of attention, if not adoption. △ Less

Submitted 23 May, 2007; originally announced May 2007.

Report number: SLAC-PUB-12507