The future of civilization as a complex system is the central question that frames the novel, and... more The future of civilization as a complex system is the central question that frames the novel, and a "character" in the form of social, technological, and environmental change. This is "hard" science fiction, relying on plausible near-future possibilities. These include climate change, hackers who manipulate the DNA of pathogens for profit or lulz, and the technological countermeasures designed to contain these epidemics, or Waves.
The story is told from a single voice--that of an artificial intelligence, or "PDA," who must literally earn her Time on the computer or be turned off. The development of the story is intended to probe ethical boundaries and philosophical questions that concern survival of the individual as well as group survival.
The research background for the framing of this novel comes from my paper "Survival Strategies." Another, more comprehensive look at complex system reliability will appear soon. As an example of how this research inspires the novel, see my blog post "Surviving Entropy."
From Amazon.com:
“Assessment on college campuses has a sordid history, and it is fairly simple ... more From Amazon.com:
“Assessment on college campuses has a sordid history, and it is fairly simple to find someone with a traumatic tale to tell. It is wise to respect that that reputation is deserved.”
“How do you modify the inner workings and culture of a massive institution with minimal resources and even less authority (other than GE course approvals), and thousands and thousands of talented people busy doing other things?”
“The road to departmental assessment can seem both dramatic and apocalyptic, especially if one’s departmental ‘centre cannot hold,’ and purpose falls apart. The Department of English and Linguistics is presently on this journey, slouching towards its own revelations of mission and fulfillment of purpose.”
“I have become more optimistic about the potential value of the process, even if some of my initial skepticism remains. This skepticism, however, has been valuable, forcing me to think in more concrete ways about what I do in the classroom.”
As these excerpts show, this is no conventional book about assessment. It presents the unvarnished first-person accounts of fourteen faculty and administrators about how they grappled, and engaged, with assessment and how – despite misgivings and an often-contentious process – they were able to gain the collaboration of their peers as the benefits for student learning became evident.
This is a book for skeptical faculty, for those who have been tasked to spearhead their institution’s call to create a culture of assessment; and, on campuses where assessment has been widely accepted and implemented, for those who now need to ensure this commitment will endure.
For all these audiences, this book offers valuable advice, strategies, models and ideas.
Many resources on implementing general education are available, but few are written to help those... more Many resources on implementing general education are available, but few are written to help those faculty and administrators responsible for general education with its evaluation. This book is a compilation of good practice case studies that are intended to assist faculty and administrators in both two-year and four-year institutions with the evaluation of student learning as it relates to general education. There are several ways in which to evaluate general education, and each case study varies in its approach. How do differences in institutional culture affect the evaluation of general education? Are institutions that lack the capacity or the culture to work across departmental or division lines to identify learning outcomes or the criteria to evaluate those outcomes employing effecting outcomes-based assessment? While not a how-to book on engaging in general education assessment, this book provides the reader with ideas to consider when adapting the evaluation of general education to his or her own institutional culture. Thirteen good practice institutions outline their assessment strategies and illustrate self-reflection that contributes to improved integration of and quality in student learning and development within general education. The case studies are supplemented by an introductory chapter that presents some considerations to address when planning for general education assessment and a concluding chapter that summarizes good practice strategies for application.
The Handbook of Research on Assessment Technologies, Methods, and Applications in Higher Educatio... more The Handbook of Research on Assessment Technologies, Methods, and Applications in Higher Education combines in-depth, multi-disciplinary research in learning assessment to provide a fresh look at its impact on academic life. A significant reference source for practitioners, academicians, and researchers in related fields, this Handbook of Research contains not only technological assessments, but also technologies and assumptions about assessment and learning involving race, cultural diversity, and creativity.
Journal of assessment and institutional effectiveness, Apr 1, 2020
ABSTRACT This study analyzes the reliability of approximately 800,000 college grades from three h... more ABSTRACT This study analyzes the reliability of approximately 800,000 college grades from three higher educational institutions that vary in type and size. Comparisons of intraclass correlation coefficients (ICCs) reveal patterns among institutions and academic disciplines. Results from this study suggest that there are styles of grading associated with academic disciplines. Individual grade assignment ICC is comparable to rubric-derived learning assessments at one institution, and both are arguably too low to be used for decision making at that level. A reliability lift calculation suggests that grade averages over eight (or so) courses per student have enough reliability to be used as outcome measures. We discuss how grade statistics can complement efforts to assess program fairness, rigor, and comparability, as well as assessing the complexity of a curriculum. The R code and statistical notes are included to facilitate use by assessment and institutional research offices.
Journal of Assessment and Institutional Effectiveness
Course grades are not typically used as primary data for assessing learning in reports prepared f... more Course grades are not typically used as primary data for assessing learning in reports prepared for accreditation due to a complicated history. This article encourages readers to reconsider by offering several examples of grade analysis to show how to estimate students’ abilities and course difficulties, and link those to discipline-specific learning. Linear and ordinal regression are used to model rubric rating averages over time, plausibly showing that student ability affects learning development. Another analysis offers a way to estimate marginal learning gains associated with course rigor. The examples give rise to several research questions that can contribute to a body of practical research on grades, student success, student learning, and equity of outcomes.
Intersection: A Journal at the Intersection of Assessment and Learning
Limitations of assessment reporting are analyzed in order to reorient practice toward improving t... more Limitations of assessment reporting are analyzed in order to reorient practice toward improving the equity of student success. Reporting practice is contextualized by scholarship within assessment and accreditation organizations, and the implicit assumptions driving current practice are examined. Two novel findings are presented: an assessment of the number of discrete learning goals that comprise an academic program and a demonstration of how course grades can provide insights into curricular barriers in way that standard reporting does not. New assumptions are proposed that are intended to contribute to the evolution of practice so that equity goals may be achieved.
Handbook of Research on Assessment Technologies, Methods, and Applications in Higher Education
This chapter describes Coker College’s subjective performance assessment program to rate student ... more This chapter describes Coker College’s subjective performance assessment program to rate student thinking and communication skills. It uses a discussion of the epistemology of assessment to motivate an emphasis on direct observation by experts as the basis for “authentic” assessment for complex learning outcomes. Results from Coker College’s experience are given and discussed in the context of this approach. The purpose of this chapter is to give a philosophical framework and practical methods that can help institutions assess liberal arts learning outcomes. Such assessments can provide information crucial to improving programs and pedagogy and form the basis for institutional effectiveness reports to stakeholders.
assessment to continually describe, improve, predict, and ultimately explain the complex construc... more assessment to continually describe, improve, predict, and ultimately explain the complex construct of student learning and development. Overall, this book is an excellent resource for educators across the spectrum of academic departments, student aff airs, and other units that are dedicated to fulfi lling the promise of higher education. Assessment remains a complex and often misunderstood process that deserves further attention. Wehlburg’s book is a helpful addition to an educator’s library, off ering both purposeful and practical ways to match the ideals of assessment with the reality of its implementation.
Structured Abstract • Technique Identification: A new graphical technique is presented for visual... more Structured Abstract • Technique Identification: A new graphical technique is presented for visualizing and assessing inter-rater agreement in discrete ordinal or categorical data, such as rubric ratings. To that aim, a chance-corrected Kappa with two new features is derived. First, it is based on interpreting ratings for each subject as vectors to visualize the data. This is done by creating two-dimensional vectors from a subject-rating summary table, sorting the vectors by their slopes, and plotting them in that order to create a trajectory that displays all the data in context. Second, it presents a graph and accompanying statistics (Kappa, p-value) for each pair of ratings in an organized display so that all useful comparisons of the data are visually displayed and statistically assessed. This information is presented on a logical grid, usually called facets. Kappa is calculated in the usual way, by referencing the actual results with an average of random rating assignments. This...
The interaction of two binary variables, assumed to be empirical observations, has three degrees ... more The interaction of two binary variables, assumed to be empirical observations, has three degrees of freedom when expressed as a matrix of frequencies. Usually, the size of causal influence of one variable on the other is calculated as a single value, as increase in recovery rate for a medical treatment, for example. We examine what is lost in this simplification, and propose using two interface constants to represent positive and negative implications separately. Given certain assumptions about noncausal outcomes, the set of resulting epistemologies is a continuum. We derive a variety of particular measures and contrast them with the one-dimensional index.
The future of civilization as a complex system is the central question that frames the novel, and... more The future of civilization as a complex system is the central question that frames the novel, and a "character" in the form of social, technological, and environmental change. This is "hard" science fiction, relying on plausible near-future possibilities. These include climate change, hackers who manipulate the DNA of pathogens for profit or lulz, and the technological countermeasures designed to contain these epidemics, or Waves.
The story is told from a single voice--that of an artificial intelligence, or "PDA," who must literally earn her Time on the computer or be turned off. The development of the story is intended to probe ethical boundaries and philosophical questions that concern survival of the individual as well as group survival.
The research background for the framing of this novel comes from my paper "Survival Strategies." Another, more comprehensive look at complex system reliability will appear soon. As an example of how this research inspires the novel, see my blog post "Surviving Entropy."
From Amazon.com:
“Assessment on college campuses has a sordid history, and it is fairly simple ... more From Amazon.com:
“Assessment on college campuses has a sordid history, and it is fairly simple to find someone with a traumatic tale to tell. It is wise to respect that that reputation is deserved.”
“How do you modify the inner workings and culture of a massive institution with minimal resources and even less authority (other than GE course approvals), and thousands and thousands of talented people busy doing other things?”
“The road to departmental assessment can seem both dramatic and apocalyptic, especially if one’s departmental ‘centre cannot hold,’ and purpose falls apart. The Department of English and Linguistics is presently on this journey, slouching towards its own revelations of mission and fulfillment of purpose.”
“I have become more optimistic about the potential value of the process, even if some of my initial skepticism remains. This skepticism, however, has been valuable, forcing me to think in more concrete ways about what I do in the classroom.”
As these excerpts show, this is no conventional book about assessment. It presents the unvarnished first-person accounts of fourteen faculty and administrators about how they grappled, and engaged, with assessment and how – despite misgivings and an often-contentious process – they were able to gain the collaboration of their peers as the benefits for student learning became evident.
This is a book for skeptical faculty, for those who have been tasked to spearhead their institution’s call to create a culture of assessment; and, on campuses where assessment has been widely accepted and implemented, for those who now need to ensure this commitment will endure.
For all these audiences, this book offers valuable advice, strategies, models and ideas.
Many resources on implementing general education are available, but few are written to help those... more Many resources on implementing general education are available, but few are written to help those faculty and administrators responsible for general education with its evaluation. This book is a compilation of good practice case studies that are intended to assist faculty and administrators in both two-year and four-year institutions with the evaluation of student learning as it relates to general education. There are several ways in which to evaluate general education, and each case study varies in its approach. How do differences in institutional culture affect the evaluation of general education? Are institutions that lack the capacity or the culture to work across departmental or division lines to identify learning outcomes or the criteria to evaluate those outcomes employing effecting outcomes-based assessment? While not a how-to book on engaging in general education assessment, this book provides the reader with ideas to consider when adapting the evaluation of general education to his or her own institutional culture. Thirteen good practice institutions outline their assessment strategies and illustrate self-reflection that contributes to improved integration of and quality in student learning and development within general education. The case studies are supplemented by an introductory chapter that presents some considerations to address when planning for general education assessment and a concluding chapter that summarizes good practice strategies for application.
The Handbook of Research on Assessment Technologies, Methods, and Applications in Higher Educatio... more The Handbook of Research on Assessment Technologies, Methods, and Applications in Higher Education combines in-depth, multi-disciplinary research in learning assessment to provide a fresh look at its impact on academic life. A significant reference source for practitioners, academicians, and researchers in related fields, this Handbook of Research contains not only technological assessments, but also technologies and assumptions about assessment and learning involving race, cultural diversity, and creativity.
Journal of assessment and institutional effectiveness, Apr 1, 2020
ABSTRACT This study analyzes the reliability of approximately 800,000 college grades from three h... more ABSTRACT This study analyzes the reliability of approximately 800,000 college grades from three higher educational institutions that vary in type and size. Comparisons of intraclass correlation coefficients (ICCs) reveal patterns among institutions and academic disciplines. Results from this study suggest that there are styles of grading associated with academic disciplines. Individual grade assignment ICC is comparable to rubric-derived learning assessments at one institution, and both are arguably too low to be used for decision making at that level. A reliability lift calculation suggests that grade averages over eight (or so) courses per student have enough reliability to be used as outcome measures. We discuss how grade statistics can complement efforts to assess program fairness, rigor, and comparability, as well as assessing the complexity of a curriculum. The R code and statistical notes are included to facilitate use by assessment and institutional research offices.
Journal of Assessment and Institutional Effectiveness
Course grades are not typically used as primary data for assessing learning in reports prepared f... more Course grades are not typically used as primary data for assessing learning in reports prepared for accreditation due to a complicated history. This article encourages readers to reconsider by offering several examples of grade analysis to show how to estimate students’ abilities and course difficulties, and link those to discipline-specific learning. Linear and ordinal regression are used to model rubric rating averages over time, plausibly showing that student ability affects learning development. Another analysis offers a way to estimate marginal learning gains associated with course rigor. The examples give rise to several research questions that can contribute to a body of practical research on grades, student success, student learning, and equity of outcomes.
Intersection: A Journal at the Intersection of Assessment and Learning
Limitations of assessment reporting are analyzed in order to reorient practice toward improving t... more Limitations of assessment reporting are analyzed in order to reorient practice toward improving the equity of student success. Reporting practice is contextualized by scholarship within assessment and accreditation organizations, and the implicit assumptions driving current practice are examined. Two novel findings are presented: an assessment of the number of discrete learning goals that comprise an academic program and a demonstration of how course grades can provide insights into curricular barriers in way that standard reporting does not. New assumptions are proposed that are intended to contribute to the evolution of practice so that equity goals may be achieved.
Handbook of Research on Assessment Technologies, Methods, and Applications in Higher Education
This chapter describes Coker College’s subjective performance assessment program to rate student ... more This chapter describes Coker College’s subjective performance assessment program to rate student thinking and communication skills. It uses a discussion of the epistemology of assessment to motivate an emphasis on direct observation by experts as the basis for “authentic” assessment for complex learning outcomes. Results from Coker College’s experience are given and discussed in the context of this approach. The purpose of this chapter is to give a philosophical framework and practical methods that can help institutions assess liberal arts learning outcomes. Such assessments can provide information crucial to improving programs and pedagogy and form the basis for institutional effectiveness reports to stakeholders.
assessment to continually describe, improve, predict, and ultimately explain the complex construc... more assessment to continually describe, improve, predict, and ultimately explain the complex construct of student learning and development. Overall, this book is an excellent resource for educators across the spectrum of academic departments, student aff airs, and other units that are dedicated to fulfi lling the promise of higher education. Assessment remains a complex and often misunderstood process that deserves further attention. Wehlburg’s book is a helpful addition to an educator’s library, off ering both purposeful and practical ways to match the ideals of assessment with the reality of its implementation.
Structured Abstract • Technique Identification: A new graphical technique is presented for visual... more Structured Abstract • Technique Identification: A new graphical technique is presented for visualizing and assessing inter-rater agreement in discrete ordinal or categorical data, such as rubric ratings. To that aim, a chance-corrected Kappa with two new features is derived. First, it is based on interpreting ratings for each subject as vectors to visualize the data. This is done by creating two-dimensional vectors from a subject-rating summary table, sorting the vectors by their slopes, and plotting them in that order to create a trajectory that displays all the data in context. Second, it presents a graph and accompanying statistics (Kappa, p-value) for each pair of ratings in an organized display so that all useful comparisons of the data are visually displayed and statistically assessed. This information is presented on a logical grid, usually called facets. Kappa is calculated in the usual way, by referencing the actual results with an average of random rating assignments. This...
The interaction of two binary variables, assumed to be empirical observations, has three degrees ... more The interaction of two binary variables, assumed to be empirical observations, has three degrees of freedom when expressed as a matrix of frequencies. Usually, the size of causal influence of one variable on the other is calculated as a single value, as increase in recovery rate for a medical treatment, for example. We examine what is lost in this simplification, and propose using two interface constants to represent positive and negative implications separately. Given certain assumptions about noncausal outcomes, the set of resulting epistemologies is a continuum. We derive a variety of particular measures and contrast them with the one-dimensional index.
This study analyzes binary categorical ratings using binomial mixture models to induce latent rat... more This study analyzes binary categorical ratings using binomial mixture models to induce latent rater accuracy, true class probability, and a correct guess rate. The models make explicit an assumption that knowledge is justified true belief, integrating reliability and validity in a single framework. Bayesian inference is shown to recover parameters with simulated data for general and hierarchical models. An expert rater assumption is shown to entail an extension of the Fleiss inter-rater kappa statistic. The method is illustrated with two real data sets: wine tasting and college student writing ability. The Kappa Paradox is analyzed from this new perspective.
Uploads
The story is told from a single voice--that of an artificial intelligence, or "PDA," who must literally earn her Time on the computer or be turned off. The development of the story is intended to probe ethical boundaries and philosophical questions that concern survival of the individual as well as group survival.
The research background for the framing of this novel comes from my paper "Survival Strategies." Another, more comprehensive look at complex system reliability will appear soon. As an example of how this research inspires the novel, see my blog post "Surviving Entropy."
“Assessment on college campuses has a sordid history, and it is fairly simple to find someone with a traumatic tale to tell. It is wise to respect that that reputation is deserved.”
“How do you modify the inner workings and culture of a massive institution with minimal resources and even less authority (other than GE course approvals), and thousands and thousands of talented people busy doing other things?”
“The road to departmental assessment can seem both dramatic and apocalyptic, especially if one’s departmental ‘centre cannot hold,’ and purpose falls apart. The Department of English and Linguistics is presently on this journey, slouching towards its own revelations of mission and fulfillment of purpose.”
“I have become more optimistic about the potential value of the process, even if some of my initial skepticism remains. This skepticism, however, has been valuable, forcing me to think in more concrete ways about what I do in the classroom.”
As these excerpts show, this is no conventional book about assessment. It presents the unvarnished first-person accounts of fourteen faculty and administrators about how they grappled, and engaged, with assessment and how – despite misgivings and an often-contentious process – they were able to gain the collaboration of their peers as the benefits for student learning became evident.
This is a book for skeptical faculty, for those who have been tasked to spearhead their institution’s call to create a culture of assessment; and, on campuses where assessment has been widely accepted and implemented, for those who now need to ensure this commitment will endure.
For all these audiences, this book offers valuable advice, strategies, models and ideas.
The story is told from a single voice--that of an artificial intelligence, or "PDA," who must literally earn her Time on the computer or be turned off. The development of the story is intended to probe ethical boundaries and philosophical questions that concern survival of the individual as well as group survival.
The research background for the framing of this novel comes from my paper "Survival Strategies." Another, more comprehensive look at complex system reliability will appear soon. As an example of how this research inspires the novel, see my blog post "Surviving Entropy."
“Assessment on college campuses has a sordid history, and it is fairly simple to find someone with a traumatic tale to tell. It is wise to respect that that reputation is deserved.”
“How do you modify the inner workings and culture of a massive institution with minimal resources and even less authority (other than GE course approvals), and thousands and thousands of talented people busy doing other things?”
“The road to departmental assessment can seem both dramatic and apocalyptic, especially if one’s departmental ‘centre cannot hold,’ and purpose falls apart. The Department of English and Linguistics is presently on this journey, slouching towards its own revelations of mission and fulfillment of purpose.”
“I have become more optimistic about the potential value of the process, even if some of my initial skepticism remains. This skepticism, however, has been valuable, forcing me to think in more concrete ways about what I do in the classroom.”
As these excerpts show, this is no conventional book about assessment. It presents the unvarnished first-person accounts of fourteen faculty and administrators about how they grappled, and engaged, with assessment and how – despite misgivings and an often-contentious process – they were able to gain the collaboration of their peers as the benefits for student learning became evident.
This is a book for skeptical faculty, for those who have been tasked to spearhead their institution’s call to create a culture of assessment; and, on campuses where assessment has been widely accepted and implemented, for those who now need to ensure this commitment will endure.
For all these audiences, this book offers valuable advice, strategies, models and ideas.