Journal of research on leadership education, Aug 6, 2013
We interviewed eight principals from Bermuda and Florida about how they identify and manage their... more We interviewed eight principals from Bermuda and Florida about how they identify and manage their most pressing challenges. Their challenges are composed of both adaptive and technical work, requiring leaders to learn to diagnose and manage them. Challenges focused on change and were traced to accountability contexts, yet accountability was not the driving force for all principals. Neither external demands nor principals themselves dictated whether the problem was technical or adaptive; instead, it was the nature of the problem itself. Leadership preparation programs are encouraged to provide a framework to address managing phases of adaptive, technical, and mixed challenges.
Journal of research on leadership education, Apr 1, 2012
An online survey (using forced-choice and open-ended questions) of faculty at two university-situ... more An online survey (using forced-choice and open-ended questions) of faculty at two university-situated degree-granting leadership preparation programs revealed that the faculty describe critical connections for developing leadership capacity: theory-practice nexus, university-based learning and “real-life” experience, and nurturing deeper faculty-to-student and faculty-to-faculty relationships. The faculty perceive that accreditation, certification, and licensure requirements have more influence over curriculum than legislative or local regulation; they voiced concerns about future impacts, although they did not identify current negative repercussions. The faculty reported doing reasonably well addressing contemporary domains of adult learning and development and social-emotional dimensions of leadership. Institutional supports are needed for furthering connections, holistic preparation, and practice.
Annual meeting of the Northeast Educational Research Association, 2023
This paper presents the nine New Core Principles and the five Pillar Practices of Rigorous Test D... more This paper presents the nine New Core Principles and the five Pillar Practices of Rigorous Test Development (RTD), an item-centric principled approach to test development that focuses primarily on item validity (rather than reliability). RTD was inspired by Evidence Centered Design (ECD).
Annual meeting of the Northeast Educational Research Association, 2023
This paper offers a set of principles to apply in item development and item evaluation during the... more This paper offers a set of principles to apply in item development and item evaluation during the test development process that focuses on the substantive content of items and their ability to elicit evidence of the targeted cognition for the range of typical test takers.
Norman Webb’s (2002) Depth of Knowledge is the most commonly used typology of cognitive complexit... more Norman Webb’s (2002) Depth of Knowledge is the most commonly used typology of cognitive complexity across the assessment industry. Unfortunately, it is widely misapplied, inflating the reported DOK levels of test items – a problem that is commonly understood but rarely publicly acknowledged. We lay out common misunderstandings/misapplications of DOK and offer a more robust system for classifying items by cognitive complexity by returning to Webb’s original conceptions and descriptions of DOK. This method of classification recognizes the impact of increased proficiency on reducing cognitive load, thus the fact that cognitive complexity is as much a product of test taker proficiency as the tasks given to test takers. We show the feasibility of this approach with a simple interrater reliability study.
International Journal of Adult Vocational Education and Technology, 2013
This article examines the views of graduate students who are aspiring or practicing school leader... more This article examines the views of graduate students who are aspiring or practicing school leaders and faculty from two university degree granting leadership preparation programs. Drawn from a larger mixed methods study, the authors focus here on survey results that show how these groups rated the effectiveness of 14 potential curricular dimensions drawn from traditional leadership content (e.g., budget/finance, legal compliance) and more recent (contemporary) additions to leadership curricula (e.g., reflective practice, adult learning and development, social-emotional capacity). Both traditional and contemporary areas received high ratings for effectiveness and importance to professional growth and development. Implications of this research point to the joint importance and feasibility of a more integrated approach to leadership education that includes contemporary and traditional dimensions. These finding may have important implications for other settings as well.
The paper presents survey research that examines how educational leadership capacity is developed... more The paper presents survey research that examines how educational leadership capacity is developed by university programs and by practicing school administrators in the U.S. and internationally, and the influence of external mandates. To face adaptive challenges, we must more holistically support leadership development that builds on adult learning principles and the social-emotional dimensions of leadership in our preparation programs and in professional development that occurs at the school site.
ABSTRACT An online survey (using forced-choice and open-ended questions) of faculty at two univer... more ABSTRACT An online survey (using forced-choice and open-ended questions) of faculty at two university-situated degree-granting leadership preparation programs revealed that the faculty describe critical connections for developing leadership capacity: theory-practice nexus, university-based learning and “real-life” experience, and nurturing deeper faculty-to-student and faculty-to-faculty relationships. The faculty perceive that accreditation, certification, and licensure requirements have more influence over curriculum than legislative or local regulation; they voiced concerns about future impacts, although they did not identify current negative repercussions. The faculty reported doing reasonably well addressing contemporary domains of adult learning and development and social-emotional dimensions of leadership. Institutional supports are needed for furthering connections, holistic preparation, and practice.
Although Evidence Centered Design (ECD) has been an important advancement in assessment, it remai... more Although Evidence Centered Design (ECD) has been an important advancement in assessment, it remains focused – as its name implies – on the design phases of test development. We present a framework to complement ECD that focuses on the test production, especially the time- and resource-consuming work of item development. Rigorous Test Development’s (RTD) principles and practices aim to improve the alignment of items with content standards and specifications, and thereby increase the validity of assessments and the inferences based upon them. This approach is particularly important in an age in which standardized test results play an increasing role in high stakes decisions and the assessment community has so little control over the uses to which its products are put.
Annual Meeting of the University Council for Educational Administration,.
This paper presents findings related to different elements of the wide educational community who ... more This paper presents findings related to different elements of the wide educational community who act as accountors for school principals face (i.e., those to whom they are responsible), drawn from an empirical larger study into the multiple accountabilities experienced by public high school principals. This study found 30 different categories of accountors, grouped into seven super-categories. Interpersonal-relationships are a key driver of accountee status. Principals are co-creators of their accountabilities, rather then merely subject to them. The Principle of Reciprocity is shown not to apply to the American high school principalship.
Annual meeting of the University Council for Educational Administration,.
This paper presents the findings related to the vast variety of the multiple accountances that pr... more This paper presents the findings related to the vast variety of the multiple accountances that principals face (i.e., that for which they are responsible), drawn from a larger empirical study into the multiple accountabilities experienced by public high school principals. This study found 45 different categories of accountors, grouped into 11 super-categories. This range of accountances is far more varied than the set of roles usually ascribed to principals would predict, and the most important accountance category was not Academics or even in the Student Outcomes super-category. A new Principle of Variety is discussed.
Annual Meeting of Northeast Educational Research Association
The development of items (i.e., questions) for standardized tests has traditionally been a black ... more The development of items (i.e., questions) for standardized tests has traditionally been a black box, in which standards and assessment targets are turned into items without examination by researchers or establishment of a profession. The new demands of the Common Core State Standards have pressured the assessment industry to develop assessments for skills it has never assessed before. In this paper, we open that black box and examine how principles of organizational learning have been applied to extend ECD (Evidence Centered Design) to item writing and review by focusing on how the GED Testing Service developed innovative new practices and tools to meet these challenges.
In 2014, administration of a much more rigorous GED Test will begin. This new test represents a d... more In 2014, administration of a much more rigorous GED Test will begin. This new test represents a departure from the 2002 Series GED Test, in that it both contains content derived from Common Core skills and will be delivered on computers for the first time. The test includes greater expectations for test-takers’ reading abilities, a focus on higher order thinking skills, and an emphasis on more authentic texts and tasks. The adult basic education community must be ready for these changes, and researchers should prepare for the opportunities that the redesign of the test presents.
This paper presents findings from a theory building study that examines the nature of leadership ... more This paper presents findings from a theory building study that examines the nature of leadership capacities. In light of Elmore’s Theory of Reciprocity and his work on multiple accountabilities, this paper offers a typology for the kinds of capacities that high school principals report needing to fulfill their responsibilities. It includes personal, professional and leadership capacities developed in formal programs, through workplace experience and through personal history. Participants report that internal capacities almost invariably are developed through opportunities for trial, error and failure, and not from formal per-service or in-service training, or even from guidance from mentors. A typology of the types of leadership capacities that principals report being necessary for their effective practice is presented.
Journal of research on leadership education, Aug 6, 2013
We interviewed eight principals from Bermuda and Florida about how they identify and manage their... more We interviewed eight principals from Bermuda and Florida about how they identify and manage their most pressing challenges. Their challenges are composed of both adaptive and technical work, requiring leaders to learn to diagnose and manage them. Challenges focused on change and were traced to accountability contexts, yet accountability was not the driving force for all principals. Neither external demands nor principals themselves dictated whether the problem was technical or adaptive; instead, it was the nature of the problem itself. Leadership preparation programs are encouraged to provide a framework to address managing phases of adaptive, technical, and mixed challenges.
Journal of research on leadership education, Apr 1, 2012
An online survey (using forced-choice and open-ended questions) of faculty at two university-situ... more An online survey (using forced-choice and open-ended questions) of faculty at two university-situated degree-granting leadership preparation programs revealed that the faculty describe critical connections for developing leadership capacity: theory-practice nexus, university-based learning and “real-life” experience, and nurturing deeper faculty-to-student and faculty-to-faculty relationships. The faculty perceive that accreditation, certification, and licensure requirements have more influence over curriculum than legislative or local regulation; they voiced concerns about future impacts, although they did not identify current negative repercussions. The faculty reported doing reasonably well addressing contemporary domains of adult learning and development and social-emotional dimensions of leadership. Institutional supports are needed for furthering connections, holistic preparation, and practice.
Annual meeting of the Northeast Educational Research Association, 2023
This paper presents the nine New Core Principles and the five Pillar Practices of Rigorous Test D... more This paper presents the nine New Core Principles and the five Pillar Practices of Rigorous Test Development (RTD), an item-centric principled approach to test development that focuses primarily on item validity (rather than reliability). RTD was inspired by Evidence Centered Design (ECD).
Annual meeting of the Northeast Educational Research Association, 2023
This paper offers a set of principles to apply in item development and item evaluation during the... more This paper offers a set of principles to apply in item development and item evaluation during the test development process that focuses on the substantive content of items and their ability to elicit evidence of the targeted cognition for the range of typical test takers.
Norman Webb’s (2002) Depth of Knowledge is the most commonly used typology of cognitive complexit... more Norman Webb’s (2002) Depth of Knowledge is the most commonly used typology of cognitive complexity across the assessment industry. Unfortunately, it is widely misapplied, inflating the reported DOK levels of test items – a problem that is commonly understood but rarely publicly acknowledged. We lay out common misunderstandings/misapplications of DOK and offer a more robust system for classifying items by cognitive complexity by returning to Webb’s original conceptions and descriptions of DOK. This method of classification recognizes the impact of increased proficiency on reducing cognitive load, thus the fact that cognitive complexity is as much a product of test taker proficiency as the tasks given to test takers. We show the feasibility of this approach with a simple interrater reliability study.
International Journal of Adult Vocational Education and Technology, 2013
This article examines the views of graduate students who are aspiring or practicing school leader... more This article examines the views of graduate students who are aspiring or practicing school leaders and faculty from two university degree granting leadership preparation programs. Drawn from a larger mixed methods study, the authors focus here on survey results that show how these groups rated the effectiveness of 14 potential curricular dimensions drawn from traditional leadership content (e.g., budget/finance, legal compliance) and more recent (contemporary) additions to leadership curricula (e.g., reflective practice, adult learning and development, social-emotional capacity). Both traditional and contemporary areas received high ratings for effectiveness and importance to professional growth and development. Implications of this research point to the joint importance and feasibility of a more integrated approach to leadership education that includes contemporary and traditional dimensions. These finding may have important implications for other settings as well.
The paper presents survey research that examines how educational leadership capacity is developed... more The paper presents survey research that examines how educational leadership capacity is developed by university programs and by practicing school administrators in the U.S. and internationally, and the influence of external mandates. To face adaptive challenges, we must more holistically support leadership development that builds on adult learning principles and the social-emotional dimensions of leadership in our preparation programs and in professional development that occurs at the school site.
ABSTRACT An online survey (using forced-choice and open-ended questions) of faculty at two univer... more ABSTRACT An online survey (using forced-choice and open-ended questions) of faculty at two university-situated degree-granting leadership preparation programs revealed that the faculty describe critical connections for developing leadership capacity: theory-practice nexus, university-based learning and “real-life” experience, and nurturing deeper faculty-to-student and faculty-to-faculty relationships. The faculty perceive that accreditation, certification, and licensure requirements have more influence over curriculum than legislative or local regulation; they voiced concerns about future impacts, although they did not identify current negative repercussions. The faculty reported doing reasonably well addressing contemporary domains of adult learning and development and social-emotional dimensions of leadership. Institutional supports are needed for furthering connections, holistic preparation, and practice.
Although Evidence Centered Design (ECD) has been an important advancement in assessment, it remai... more Although Evidence Centered Design (ECD) has been an important advancement in assessment, it remains focused – as its name implies – on the design phases of test development. We present a framework to complement ECD that focuses on the test production, especially the time- and resource-consuming work of item development. Rigorous Test Development’s (RTD) principles and practices aim to improve the alignment of items with content standards and specifications, and thereby increase the validity of assessments and the inferences based upon them. This approach is particularly important in an age in which standardized test results play an increasing role in high stakes decisions and the assessment community has so little control over the uses to which its products are put.
Annual Meeting of the University Council for Educational Administration,.
This paper presents findings related to different elements of the wide educational community who ... more This paper presents findings related to different elements of the wide educational community who act as accountors for school principals face (i.e., those to whom they are responsible), drawn from an empirical larger study into the multiple accountabilities experienced by public high school principals. This study found 30 different categories of accountors, grouped into seven super-categories. Interpersonal-relationships are a key driver of accountee status. Principals are co-creators of their accountabilities, rather then merely subject to them. The Principle of Reciprocity is shown not to apply to the American high school principalship.
Annual meeting of the University Council for Educational Administration,.
This paper presents the findings related to the vast variety of the multiple accountances that pr... more This paper presents the findings related to the vast variety of the multiple accountances that principals face (i.e., that for which they are responsible), drawn from a larger empirical study into the multiple accountabilities experienced by public high school principals. This study found 45 different categories of accountors, grouped into 11 super-categories. This range of accountances is far more varied than the set of roles usually ascribed to principals would predict, and the most important accountance category was not Academics or even in the Student Outcomes super-category. A new Principle of Variety is discussed.
Annual Meeting of Northeast Educational Research Association
The development of items (i.e., questions) for standardized tests has traditionally been a black ... more The development of items (i.e., questions) for standardized tests has traditionally been a black box, in which standards and assessment targets are turned into items without examination by researchers or establishment of a profession. The new demands of the Common Core State Standards have pressured the assessment industry to develop assessments for skills it has never assessed before. In this paper, we open that black box and examine how principles of organizational learning have been applied to extend ECD (Evidence Centered Design) to item writing and review by focusing on how the GED Testing Service developed innovative new practices and tools to meet these challenges.
In 2014, administration of a much more rigorous GED Test will begin. This new test represents a d... more In 2014, administration of a much more rigorous GED Test will begin. This new test represents a departure from the 2002 Series GED Test, in that it both contains content derived from Common Core skills and will be delivered on computers for the first time. The test includes greater expectations for test-takers’ reading abilities, a focus on higher order thinking skills, and an emphasis on more authentic texts and tasks. The adult basic education community must be ready for these changes, and researchers should prepare for the opportunities that the redesign of the test presents.
This paper presents findings from a theory building study that examines the nature of leadership ... more This paper presents findings from a theory building study that examines the nature of leadership capacities. In light of Elmore’s Theory of Reciprocity and his work on multiple accountabilities, this paper offers a typology for the kinds of capacities that high school principals report needing to fulfill their responsibilities. It includes personal, professional and leadership capacities developed in formal programs, through workplace experience and through personal history. Participants report that internal capacities almost invariably are developed through opportunities for trial, error and failure, and not from formal per-service or in-service training, or even from guidance from mentors. A typology of the types of leadership capacities that principals report being necessary for their effective practice is presented.
Radical Empathy is a rigorous and disciplined technique used by CDPs (content development profess... more Radical Empathy is a rigorous and disciplined technique used by CDPs (content development professionals) to examine items through the eyes of the range of typical test takers for an assessment. This difficult practice is important to assuring that items elicit evidence of the targeted cognition for the range of typical test takers. It is the heart of the RTD (Rigorous Test Development) Item Alignment Examination procedure.
This typology is not about performance levels that one might report from an assessment. Rather, i... more This typology is not about performance levels that one might report from an assessment. Rather, it is a more theoretical framework about the true range of proficiency that people truly possess. Assessment developers and consumers should be careful to be mindful of the kinds of proficiency claims that assessments can support and the various costs (e.g., financial, testing time) of generating sufficient evidence to support
The RTD Alignment Scale presents five discrete levels of alignment and is applicable to the full ... more The RTD Alignment Scale presents five discrete levels of alignment and is applicable to the full range of item types and classroom activities. Because valid items elicit evidence of the Targeted Cognition for the range of typical test takers, this scale focuses on the quality of the evidence that the item provides. It considers whether test takers who have produced the desired work product have produced strong observable evidence of proficiency with the Targeted Cognition and whether test takers who have fallen short of providing the de-sired work product have produced strong evidence of a lack of proficiency with the Tar-geted Cognition. That is, it is very mindful of the twin problems false positive and false negative evidence.
Uploads
Papers by Alexander M Hoffman