Zhang, D., Albert, D., Hockemeyer, C., Breen, D., Kulcsar, Z., Shorten, G., Aboulafia, A., & Lövquist, E. (2008). Developing competence assessment procedure for spinal anaesthesia. Proceedings of the 21st IEEE International Symposium on Computer-Based Medical Systems, 397-402. Developing Competence Assessment Procedure for Spinal Anaesthesia Dajie Zhang University of Graz Dietrich Albert University of Graz Cord Hockemeyer University of Graz Zsuzsanna Kulcsár Cork University Hospital Erik Lövquist University of Limerick George Shorten Cork University Hospital dajie.zhang@uni-graz.at Dorothy Breen Cork University Hospital Annette Aboulafia University of Limerick the many factors that influence learning and performance of medical procedures. Such determinants include cognitive, motor, communicative, and human (e.g. fatigue, anxiety, and fear) factors [4]. In recent years, most European countries have tried to introduce competence-based training into medical education. These efforts are restricted by the absence of a universally-accepted and valid means of assessing competence in medical procedural skills [5-7]. Evaluation of medical procedural skills entails (i) rating by supervising clinicians during the apprenticeship or the residency, and (ii) assessment based on clinical simulations. The former involves exposing patients to inexperienced trainees, relying on selective or second hand data, or non-validated assessment techniques that are subject to bias (e.g. race or gender) [6]. Although assessments based on clinical simulation offer benefits in terms of reliability and validity, evaluation of the influence of human factors (i.e., anxiety, fatigue, etc.) is compromised by the artificial settings [6,7]. In other domains, competence-based knowledge space theory (CbKST) [8,9] has been successfully applied to facilitate personalised learning and to assess competence. Recognizing the success of the CbKST approach as well as the urgent and great need in the medical domain, a new project MedCAP (“Competence Assessment for Spinal Anaesthesia”, homepage: http://www.medcap.eu) commenced in 2007 [10]. It aims to transfer the innovative CbKST approach to the medical domain in order to develop a valid, reliable and practical CAP for one medical procedural skill, spinal anaesthesia. As performing spinal anaesthesia requires elements of competence in several domains common to many other (and more complex) medical procedures, the principles applied in developing a CAP for spinal anaesthesia could be applied to others. Abstract Traditional approaches of assessment in the medical domain are insufficient for evaluating trainees’ technical skills. Currently, many European medical training bodies are attempting to introduce competence-based training programmes for technical skills as well as other domains (e.g., communication, professional behaviour, clinical cognition). These efforts are limited due to the absence of appropriate assessment tools. Based on Competence-based Knowledge Space Theory (CbKST), a collaborative project MedCAP intends to develop a valid and reliable competence assessment procedure for one important medical skill, spinal anaesthesia. The paper briefly overviews the current states of training and assessment for medical procedural skills, describes the core ideas of CbKST, and introduces the ongoing project that will transfer the innovative approach of CbKST in personalized learning and competence assessment to the medical domain. 1. Introduction Worldwide, medical training is undergoing dramatic changes, moving from a process and structure-based training paradigm towards a competence-based paradigm [1,2]. The former determines learning on the basis of exposure to specified content over a certain period of time, while the latter does so on the basis of competence achievement [3]. Competence-based training necessitates valid and reliable competence assessment procedures (CAPs). However, for most medical procedural skills, no such CAP exists. The challenges in developing such CAPs lie in defining each competence and taking account of 1 2. Training of spinal anaesthesia Laptop Computer With 3D visuals Spinal anaesthesia is a delicate procedure involving the injection of local anaesthetic solution into the fluid surrounding the spinal cord to facilitate lower abdominal or lower limb surgery (Figure 1). By feeling the resistive forces of the needle passing through various tissues, the anaesthetist places the tip of the needle into the correct space without causing damage to surrounding tissues and nerves. As with training of other medical technical skills, students learning spinal anaesthesia are routinely taught manual techniques and necessarily practice the novel skills on hospital patients. Due to the mounting pressures in the clinical and training environment, such as emphasis on operating room efficiency (European Working Time Directive), execution of the Bologna Accord, emphasis on patient safety, cost factors and others, the opportunities for an individual trainees to acquire “hands on” experience in procedural skills has decreased substantially. Epidural Needle Epidural LOR Syringe Patients back Figure 2. Epidural Simulator (with permission) (Design Based Medical Training, http://www.dbmt.eu). As the procedure of spinal anaesthesia relies heavily on tactile cues, learners are required to recognize the characteristic “sensations” in the procedure. In DBMT, a haptic device, PHANTOM® DesktopTM from Sensable Technologies (see: http://www.sensable.com), has been adapted to replicate these sensations. Sense of touch and resistive forces are simulated. The physical make-up of each individual layer of tissue in a human back was modelled. The haptic device has a mechanical arm with five joints, enabling the user to manipulate, interact and feel objects and sensations. The arm has been modified by attaching a spinal needle, thus providing the user with a realistic instrument to hold. The movement of the needle is within a three dimensional space, thereby facilitating easier navigation. Stereoscopic glasses are used to create the illusion of depth on the screen (Figure 3). To implement the 3D model and the force feedback properties of the various tissue layers, the developing software H3DAPI (see: http://www.h3d.org) is used with the extension of VHTK (Volume Haptics Toolkit). Additionally, CT scan images are used to create the 3D model of a human back. With this augmented reality system [14], learners experience the realistic sensations of inserting the spinal needle on the one hand, and monitor what happens under the skin of the patient on the other. Patient variations and levels of difficulty can be built into the system to offer different training challenges. More importantly, the system tracks all the movement by the user during the procedure, thus providing a basis for assessing the procedural skills which is required by the MedCAP project. Figure 1. Demonstration of a spinal anaesthetic injection [11] Computer-based technology (e.g., simulation, webbased learning and virtual reality) has been introduced into medical training purportedly to improve the efficiency, effectiveness and safety of learning and teaching of procedural and other skills [12,13]. Some high-fidelity simulators are available for training and assessment purposes. For example, (i) a commercially available simulator (Figure 2) for epidural anaesthesia (which shares certain characteristics with spinal anaesthesia) developed by MedicVision (partner of MedCAP project, see: http://www.medicvision.com.au) has been successfully marketed in European countries. It has brought expertise in the development of technical training using simulators. (ii) To provide effective and safe training without subjecting patients to risk in spinal anaesthesia, an interactive virtual learning system has been developed during the DBMT project 2 able to solve the addition of two decimals, he should already be able to solve the addition of two integers. As such, problem type q is called the prerequisite or the precedence of problem type p. By correct responses to type p problems, correct answers to type q problems can be surmised. Such a surmise relation (or prerequisite, precedence relation) can be illustrated in a Hasse diagram showed in Figure 4, which consists of five hypothetical problem types a, b, c, d and e. The prerequisite relation between the problem types is indicated by the descending segments. In Figure 4, for instance, problem types a, b, and c are the prerequisites for type e. If a student responds correctly to problems of type e, it is likely that he can also solve problems of type a, b and c. Figure 3. Spinal anaesthesia training system developed by DBMT [14] e 3. Competence-based Knowledge Space Theory (CbKST) d b In MedCAP, the competence assessment procedure will be developed based on CbKST. Traditional evaluation of knowledge marks individual achievements with numerical scores. By nature, a test score offers no cue to what an examinee can do and what he still needs to learn, hence contributes little to further learning and development. When two students score equal on a test, there is no evidence whether they possess the comparable competences. In addition, in a traditional linear test, all examinees are presented with the same set of items in a predefined form. The test score for an individual is obtained based on responses to all the items in the current test, although the items only cover a fraction of the knowledge in the complete domain. This is an inefficient and inaccurate way of assessing ability. Given the intrinsic flaws of the traditional assessments, new approaches are required for evaluating individual competences. CbKST developed by Albert and colleagues [8] is such an approach suitable for adaptively assessing individual competences without numerical representations [cf. 15,16]. It is an extension of the Knowledge Space Theory (KST) [17,18]. The original KST was behaviouristic, judging an individual’s knowledge state via his observable performances (i.e., being able or not able to solve particular problems in a test). Later works of different research groups have extended the KST by analyzing competences entailed in a given knowledge domain, and assigning them to the test problems and learning objects [15,16,19-22]. Both KST and CbKST ground on a basic phenomenon that acquiring some pieces of knowledge normally precede some other pieces of knowledge. A certain type of problems p may be solvable by a student only if another type of problems q has already been mastered by the student. For example, if a student is c a Figure 4. A hypothetical Hasse-diagram illustrating surmise relation of five problems According to CbKST, a knowledge domain can be represented by two structures: (a). A collection of competences that are inherent in a domain. (b). A set of problems that can be solved in the domain given the competences in (a). Both (a) and (b) can be structured based on the surmise relations. Importantly, the number of competences in a domain is finite while the viable problems can be solved may be infinite. Approaching a knowledge domain via (b), a knowledge state refers to a specific subset of problem types in the domain that some individual is capable of solving. The Hasse-diagram in Figure 4 completely defines the feasible knowledge states in this hypothetical mini-domain. Analyzing Figure 4, exactly 10 knowledge states can be induced, forming the set K = [23], {c}, {a, c}, {a, b}, {a, b, c}, {a, b, d}, {a, b, c, e}, {a, b, c, d}, Q}, of which ∅ refers to the empty set, and Q refers to the complete set of {a, b, c, d, e}. The set K is called the knowledge structure of this hypothetical domain. The knowledge structure can be illustrated in a setinclusion diagram consisting of all the feasible states (Figure 5). The structure implies different possible learning paths moving from the naïve knowledge state ∅ to the full mastery of Q. For example, one can start by first mastering a, and successively the other types bĺdĺcĺe (Figure 5). Alternatively, one can also start with c, and proceed to aĺbĺeĺd. Note that Figure 4 and 5 illustrates a domain with merely 5 types of problems. In reality, even for an elementary knowledge 3 domain, the number of knowledge states and of learning paths can become very large [24]. As mentioned before, a knowledge domain can also be identified by its inherent competences. A competence (or skill)1 is defined as a combination of an action and a concept2 (e.g., “state Theorem of Pythagoras” and “apply Theorem of Pythagoras” are two different skills) (see Marte et al. [25] for a discussion of the connection between CbKST and Bloom’s [26] taxonomy of hierarchical classification of educational goals). By comprehensive analysis of a knowledge domain, underlying competences can be identified. Analogous to constructing the knowledge structure, a competence structure can be derived, containing the competence states organized by surmise relations. For example, in the competence structure of spinal anaesthesia, the competence state “performs lumbar puncture” surmises the state “applies knowledge of anatomy to identify the interspace”. The competence states and the knowledge states (i.e., sets of test problems) can be matched mutually. On the one hand, for each type of problem, a particular competence state (or several competence states) is/are sufficient to solve it (Figure 6, left panel). On the other hand, given a particular competence state (involving one or more competences), one or more types of problems can be solved and the corresponding knowledge state can be inferred (Figure 6, right panel). {a, b, c, d, e} {a, b, c, d} {a, b, c, e} {a, b, d} {a, b, c} {a, b} {a, c} {c} {a} ∅ Figure 5. Knowledge structure consistent with the knowledge domain illustrated in Figure 4. The dashed arrows display one of the possible learning path As suggested by Figure 5, learning can take place step by step, one problem type at a time. Specifically, each knowledge state (except Q) has at least one immediate successor state which contains all the same problem types, plus exactly one. The knowledge state {a, b, c} of K, for instance, has the two states {a, b, c, d} and {a, b, c, e} as immediate successors. Problem types d and e are the outer fringe of the state {a, b, c}. It contains exactly the problem types that a particular learner processing knowledge state {a, b, c} should proceed to learn. Conversely, each knowledge state (except ∅) also has at least one predecessor state that contains exactly the same problems, except one. The knowledge state {a, b, c}, for example, has two predecessor states, i.e., {a, b} and {a, c}. Problem types b and c together form the inner fringe of state {a, b, c}, which are the most sophisticated problem types the learner has mastered by far. If the learner has difficulty solving the outer fringe problems, reviewing materials in the inner fringe should normally be recommended. The two fringes are sufficient to specify a particular knowledge state, of which the outer fringe directs progression while the inner fringe monitors the possible retreats. Both are crucial for generating personalized learning paths. To sum up, knowledge in a particular domain can be represented by different types of problems organized by prerequisite relations. An individual’s current knowledge state is identified by the problems he masters in the domain. The collection of the feasible knowledge states forms the knowledge structure. For any given knowledge structure, divergent learning paths are possible, each leading from the naïve knowledge state to the complete mastery of the knowledge domain. Each knowledge state has an outer and an inner fringe. The former directs the learning progression while the latter implies possible reviews. Problem type Competence state(s) Competence state Knowledge state a {1,2,4}, {3,4} {1,2,4} {a,b} b {1,2} {1,2} {b} c {3} {3} {c} d {3,5} {3,5} {c,d} {3,4} {a,c} Figure 6. Illustration of the relationship between competence states and knowledge states. Numbers refer to different competences Consequently, an assessment based on CbKST will not only identify what kinds of problems a learner is able to solve, more importantly, it will reveal an individual’s current competence state underlying his visible behaviour. In a learning situation, skills are pre-assigned to each learning object (e.g., a learning scenario contains tutoring and exercises). A learning object is always defined by its prerequisite skills and the new skill(s) to be learned. Appropriate objects will be suggested to the 1 The terms “competence” and “skill” have been used interchangeably in the literature and are accepted as exchangeable in this paper. 2 cf. [16], where “concept“ and “action“ are not separated. 4 learner in a virtual environment, which adapts to the learner’s current competence state. and offer graphical output (test objects). The Web service will provide the main functionality and will be used to implement the assessment algorithms (Figure 7). 4. Competence assessment procedure for spinal anaesthesia Logging of data After the competence structure has been identified, its induced assessment does not have to exhaust all the problems in a knowledge domain. Instead, based on the surmise relations, the assessment will be much more efficient. At the onset of the assessment, an individual is given a randomly selected item of a certain problem type p, for which he would have about a 50% likelihood of solving it. The likelihood of each problem could be derived, for example, from the average success rate of other comparable peers (e.g., those who study in the same grade) who have been tested with it. If the student responds correctly, the likelihoods of all the knowledge states containing p are increased and, accordingly, the likelihoods of all the states not containing p are decreased. A false response given by the student has the opposite effect: the likelihoods of all the states not containing p are increased, and those of the remaining states decreased. The following test problems are then selected by the same mechanism, based on the updated likelihoods of the states deriving from the individual’s previous responses. In this way, the problem types left to be tested reduce rapidly, and the likelihood of some states gradually increases. The procedure stops when some peak in the likelihood function is reached [27]. The system has now revealed the most likely knowledge state of the individual. The state will then be interpreted by the underlying competences, providing detailed information about what an individual is able to do and what he is ready to learn. To apply CbKST to the medical domain, an essential task is to comprehensively define competence and knowledge structures of the relevant domain. As for spinal anaesthesia, the competence assessment should encompass: medical knowledge; technical ability; communication; patient management skills and other dimensions. Preliminary work carried out at Cork University Hospital (CUH, Partner of MedCAP project) implied that such competence structure exists for spinal anaesthesia [28]. Since November 2007, five partners in Europe (CUH, University of Graz, University of Pecs, Interaction Design Centre and MedicVision Ltd) have jointed to develop a CAP for spinal anaesthesia. The project will comprise a learning management system (LMS) and a Web service. The LMS will provide the interface with the user, accept user input User LMS Interface Assessment Logic Back end web service Competence structure Ontology of competences Figure 7. The CAP system in MedCAP 5. Discussion In order to develop a valid and reliable CAP for spinal anaesthesia, the partnership of MedCAP will comprehensively describe the competences required in the domain, generate algorithms necessary to assess individual performance, implement the CAP in a userfriendly, web-based format and test it in simulated and real clinical settings for construct validity and reliability. Challenges remain in how to (a) comprehensively define the competence structure of the domain; (b) generate corresponding types of test problems in suitable presenting formats; and to (c) determine criteria to classify the possible responses to the problems. The valid and reliable CAP shall be applied in the European medical training bodies, supporting personalized learning and competence-based training as well as improving the safety and efficiency of the medical environment. The principles employed in developing the CAP for spinal anaesthesia could be extrapolated to developing similar assessment tools for other medical procedural skills. Acknowledgement The project MedCAP is funded by European Commission: LDV/LLP/TOI/2007/IRL-513. References [1] [2] [3] 5 R. Aggarwal and A. Darzi, "Technical-skills training in the 21st century," N Engl J Med, vol. 355, pp. 2695-6, 2006. R. K. Reznick and H. MacRae, "Teaching surgical skills -- changes in the wind," N Engl J Med, vol. 355, pp. 2664-9, 2006. C. Carraccio, S. D. Wolfsthal, R. Englander, K. Ferentz, and C. Martin, "Shifting paradigms: from flexner to competencies," Acad Med, vol. 77, pp. 3617, 2002. [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] A. F. Smith, C. Pope, D. Goodwin, and M. Mort, "What defines expertise in regional anaesthesia? An observational analysis of practice," Br J Anaesth, vol. 97, pp. 401-7, 2006. A. J. Byrne and J. D. Greaves, "Assessment instruments used during anaesthetic simulation: review of published studies," Br J Anaesth, vol. 86, pp. 445-50, 2001. M. Srinivasan, J. C. Hwang, D. West, and P. M. Yellowlees, "Assessment of clinical skills using simulator technologies," Acad Psychiatr, vol. 30, pp. 505-15, 2006. R. M. Epstein, "Assessment in medical education," N Engl J Med, vol. 356, pp. 387-96, 2007. D. Albert and J. Lukas Eds., Knowledge Spaces: Theories, Empirical Research, and Applications. Mahwah, NJ: Lawrence Erlbaum, 1999. J. Heller, C. Steiner, C. Hockemeyer, and D. Albert, "Competence-based knowledge structures for personalised learning," Int J e Learn, vol. 5, pp. 7588, 2006. D. Albert, C. Hockemeyer, and G. Shorten, "Competence assessment for spinal anaesthesia," in HCI and Usability for Medicine and Health Care Proceedings of the Third Symposium of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society (USAB 2007) Graz, Austria, A. Holzinger, Ed., Berlin: Springer, 2007, pp. 165-70. U. Dreifaldt, Z. Kulcsar, and P. Gallagher, "Exploring
haptics
as
a
tool
to
improve
training
of
medical
doctors
in
the
procedure
of
spinal
anaesthetics," Eurohaptics 2006, Paris, 3rd-6th of July, 2006. Y. A. Zausig, Y. Bayer, N. Hacke, B. Sinner, W. Zink, C. Grube, and B. M. Graf, "Simulation as an additional tool for investigating the performance of standard operating procedures in anaesthesia," Br J Anaesth, vol. 99, pp. 673-8, 2007. R. S. Haluck and T. M. Krummel, "Computers and virtual reality for surgical education in the 21st century," Arch Surg, vol. 135, pp. 786-92, 2000. E. Lövquist, A. Aboulafia, and Z. Kulcsar, "A medical simulator for teaching spinal anaesthesia," INMED, scientific meeting, Dublin, 7-8 Feb., 2008. I. Düntsch and G. Gediga, "Skills and knowledge structures," Br J Math Stat Psychol, vol. 48, pp. 9-27, 1995. K. Korossy, "Modeling knowledge as competence and performance," in Knowledge Spaces: Theories, Empirical Research Applications, D. Albert and J. Lukas, Eds., Mahwah, NJ: Lawrence Erlbaum Associates, 1999, pp. 103-32. J.-P. Doignon and J.-C. Falmagne, "Spaces for the assessment of knowledge," Int J Man Mach Stud, vol. 23, pp. 175-96, 1985. J.-P. Doignon and J.-C. Falmagne, Knowledge Space. Berlin: Springer, 1999. J.-C. Falmagne, M. Koppen, M. Villano, J.-P. Doignon, and L. Johannesen, "Introduction to [20] [21] [22] [23] [24] [25] [26] [27] [28] 6 knowledge spaces: How to build, test and search them," Psychol Rev, vol. 97, pp. 201-24, 1990. J.-P. Doignon, "Knowledge spaces and skill assignments," in Contributions to mathematical psychology, psychometrics and methodology, G. Fischer and D. Laming, Eds., New York: Springer, 1994, pp. 111-21. D. Albert and T. Held, "Establishing knowledge spaces by systematical problem construction," in Knowledge Structures, D. Albert, Ed., New York: Springer, 1994, pp. 78-112. C. Hockemeyer, O. Colan, V. Wade, and D. Albert, "Applying competence prerequisite structures for eLearning and skill management," J Univers Comput Sci, vol. 9, pp. 1428-36, 2003. M. Lipman, "Teaching students to think reasonably: Some findings of the Philosophy for Children program," Clearing House, vol. 71, pp. 277-80, 1998. J.-C. Falmagne, E. Cosyn, J.-P. Doignon, and N. Thiéry. (2004) The assessment of knowledge, in theory and in practice. [Online]. Available: http://www.aleks.com/about_aleks/Science_Behind_ ALEKS.pdf B. Marte, C. Steiner, J. Heller, and D. Albert, "Activiy and Taxonomy-Based Knowledge Representation Framework (Contribution to the 2nd EleGI conference, Oct. 2006, Barcelona)," International Journal of Knowledge and Learning., In Press. B. S. Bloom, Taxonomy of educational objectives: The classification of educational goals - Handbook 1: cognitive domain. New York, NY: Mckay, 1956. J.-C. Falmagne and J.-P. Doignon, "A class of stochastic procedures for the assessment of knowledge," Br J Math Stat Psychol, vol. 41, pp. 123, 1988. Z. Kulcsar, A. Aboulafia, T. Hall, and G. Shorten, "Determinants of learning to perform spinal anaesthesia - a qualitative study," Eur J Anaesthesiol, in press.