Diagram-based CBA using DATsys and CourseMaster

Diagram-Based CBA using DATsys and CourseMaster Colin Higgins, Pavlos Symeonidis, Athanasios Tsintsifas School of Computer Science and IT, The University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, UK. ltr@cs.nott.ac.uk Abstract Supporting the assessment of an ever-increasing number of students is an error-prone and resource intensive process. Computer Based Assessment (CBA) software aids educators by automating aspects of the assessment of student work. Using CBA benefits pedagogically and practically both students and educators. The Learning Technology Group at the University of Nottingham has been actively researching, developing and using software to automatically assess programming coursework for 14 years. Two of the systems developed, Ceilidh and its successor CourseMaster, are being used by an increasing number of academic institutions. Recent research has resulted in a system for supporting the full lifecycle of free-response CBA that has diagram-based solutions. The system, DATsys, is an authoring environment for developing diagram-based CBA. It has been designed to support the authoring of coursework for most types of diagram notations. Exercises have been developed and tested for circuit diagrams, flowcharts and class diagrams. Future research plans for authoring exercises in many more diagram notations. 1: Introduction – Definitions Learning technology is a widely used term that refers to the application of technology to enhance the learning process. Computer Assisted Assessment (CAA) is a field of learning technology that studies the use of computers to deliver, analyse and mark student coursework. A specialisation of CAA is Computer Based Assessment (CBA). CBA differs from CAA in that with CBA the coursework’s solution is entered on-line. Culwin distinguishes between two types of automated assessment, fixed response and free response [5] also referred to as objective and non-objective respectively [3]. The distinction between the two is based on the nature of response to the CBA exercise. Fixed response systems Proceedings of the International Conference on Computers in Education (ICCE’02) 0-7695-1509-6/02 $17.00 © 2002 IEEE require the learner to choose a solution from a list of available options. Such CBA systems use either: multiplechoice questions, simple text exercises, numeric value exercises, or graphical hotspot exercises. Further categorisations are given by Charman and Elmes [4]. In contrast, free response systems can assess unanticipated solutions. Such systems have been employed for assessing the implementation or design of computer programs [7] and essays [16]. Typically, in free response assessment a qualitative strategy describes the criteria attributing to the evaluation of the exercise. The automatic assessment of diagrams belongs to the free response type of assessment since the solution has potentially infinite solutions with varying degrees of correctness. We define diagram-based CBA as “a type of CBA in which the solution to a required problem or the answer to a required question is drawn by the student using a suitable graphical environment and is assessed automatically according to appropriate marking guidelines”. 2: The problem of developing Diagram-Based CBA exercises In what aspects does diagram-based CBA differ from others? CBA systems, conceptually, consist of a database, software components for authoring, marking, administering, presenting, and submitting coursework, and analogous user interfaces (UI) (figure 1). Three basic types of data exist within the database in CBA systems: • Authored material, such as coursework and marking related data • Class related material, such as student submissions, marks, and user information • Properties for administration and marking The software components are the modules of software that support the requests from the user-interfaces to the database. Typically, these modules contain logic that describes the execution and properties of the processes involved in assessment. The user-interfaces provide the means to use the system and support various options. Figure 1: A high level view of the parts of typical CBA systems The variation between types of assessment does not really affect the database, the software components for setting, submitting and administering coursework, nor much of the UIs. These can be implemented generically. However, in three main areas, diagram-based coursework exhibits differences. These are: • The UI part that allows the entry and editing of the solution to the exercise by the student • The marking mechanism that directs the marking of the student solution and returns appropriate feedback • The process of authoring coursework and the refinement of the generic CBA model to support the full lifecycle of the CBA coursework for summative purposes in a controlled environment Each difference initiates a respective problem area. The first problem area is the building of a graphical editor customised to support the exercise and its domain. Software engineering research has documented methods for the construction of generic diagramming editors over the past 10 years [15], [12], [2], [1]. Most of these methods aim to empower programmers with libraries powerful enough to ease the development of domain specific graphical editors such as circuit, flowchart, and drawing editors. Yet, these libraries are complex and demand significant programming expertise. The effort and resources involved in producing a single “per domain” or “per exercise” editor prohibits their use in CBA. Hence regarding the first problem area the objective has been to design and implement a framework for generic diagram editors in which the creation of a new type of editor is a simple task that does not require programming. The second problem area concerns the marking process. It extends to the description and execution of the marking criteria and the creation of appropriate feedback. Foxley and Zin have described a generic technique to express marking schemes for assessing exercises in programming languages using “Oracles” [9]. Oracles represent the marking criteria that are used to mark an exercise. They are described in a form that uses regular expressions and they express the text matching behaviour Proceedings of the International Conference on Computers in Education (ICCE’02) 0-7695-1509-6/02 $17.00 © 2002 IEEE that can conveniently be used to assess an exercise solution. The objectives in the second problem area have been to investigate whether a similar technique can be used to describe marking schemes for diagram-based coursework. The third problem is concerned with the provision of support for the full lifecycle of diagram-based CBA exercises. For many reasons, including a need to increase scalability, performance, maintainability, extensibility and usability, the original Ceilidh system had to be redesigned and re-implemented [6], [10]. While restructuring Ceilidh into its successor, CourseMaster, the objectives have been to increase software quality and integrate the diagrambased facility with the generic marking mechanism. 3: Design and Implementation Five types of users participate in the assessment process: students, tutors, teachers, developers and administrators. Tutors and administrators are not affected by the type of CBA. In contrast, students, teachers and developers have additional responsibilities. Students draw the solution within an appropriate diagram editor, teachers author new exercises using authoring tools, and developers create elements for new diagram domains. Figure 2: Conceptual plan for diagram-based CBA Figure 2 illustrates the overall scheme. The course developer prepares diagram editor specifications and marking tools for a specific domain. Within the teacher’s environment these are customised further to meet the requirements of the diagram-based CBA exercise. A newly built exercise contains the student diagram editor and marking customisations. The student uses the diagram editor to complete the exercise that is marked via a marking scheme. A marking scheme is a program that specifies the invocation and configuration of appropriate marking tools [11]. Finally, feedback is returned to the student. • The implementation built to support diagram-based CBA consists of: • Two platforms: DATsys and CourseMaster • Two authoring environments: Daidalos and Ariadne Both DATsys and CourseMaster are written exclusively in Java and have been tested under various platforms running Windows, Solaris and Linux. Daidalos and Theseus are applications based on DATsys. Ariadne is also based on DATsys, and additionally, requires parts of CourseMaster. The CourseMaster clients and servers are built on top of CourseMaster and are completely independent of DATsys. • • Oracles to describe the correct output values or ranges of values per test-data Feedback messages for all testing cases. Marking Tools Typography Marking Tool Checks the Typography of a program Typographic Rules, Ranges, Weights, N-Level Feedback Program-Features Marking Tool Checks specific features of a program Oracles for Features, Weights 2-Level Feedback Compilation Marking Tool Compiles a program and checks the output Oracles for CompilationOutput N-Level Feedback Dynamic Test Marking Tool Circuit Simulation Marking Tool Flowchart Marking Tool OO Design Marking Tool Figure 3: Relationship between DATsys parts Figure 3 illustrates a detailed view of the data flow between the implemented parts. Diagram notation specifications are authored by the course developer using Daidalos and are expressed as domain tool libraries and application options. These are used in Ariadne by the course teacher who develops the diagram-based CBA exercise. Using Ariadne, the set of domain tool libraries and application options can be refined to address the intent of specific exercises. In addition, Ariadne produces exercise setting-up and marking files that are used by CourseMaster’s marking server. Theseus uses the tool library and the set of application options to allow the student to draw the exercise solution diagram. Upon submission, CourseMaster’s marking server uses the marking files that have been created in Ariadne to mark and return the marking results with feedback to the student. A comprehensive guide to the marking process is given in [17]and [11]. Figure 4 illustrates a range of programming and diagramming marking tools in relationship to their configuration. Of these, three are new diagram-based marking tools: • CircuitSimMarkingTool: marks logic design based exercises • FlowchartMarkingTool: marks flowchart based exercises • OOMarkingTool: marks object-oriented design based exercises The circuit-simulator marking tool needs as configuration: • Test data that will be fed to the inputs of the circuit for the simulation, Proceedings of the International Conference on Computers in Education (ICCE’02) 0-7695-1509-6/02 $17.00 © 2002 IEEE Marking Tool Configuration Runs a program, provides test-data and checks the output Simulates a circuit, provides test-data, and checks the output Test Data, Oracles for Program Output N-Level Feedback Test Data, Oracles for Circuit Output N-Level Feedback Translates the Flowchart to a program and reuses the Dynamic Marking Tool Dynamic Tool Configuration Checks specific features of the OOD Diagram Oracles for Features, Weights, N-Level Feedback Figure 4: Marking Tools in CourseMaster The flowchart-diagram marking tool, after translating the diagram to a program, reuses the dynamic-test marking tool. Therefore, it needs the configuration of a dynamic-test tool. The object-oriented design marking tool, investigates the student diagram for particular features such as the use of specific relationships between predefined classes and objects, the naming of classes and methods, and various other exercise dependant criteria. 4: Authoring Diagram-Based CBA using DATsys and CourseMaster The authoring of a diagram-based CBA exercise involves the following stages: • Using Daidalos to build a tool library for creating and connecting diagram elements • Using Ariadne to build a CBA exercise (by choosing a subset of Daidalos’ tools for the student tool library, selecting application features, developing the marking scheme, and configuring the marking tools and the CBA exercise) The full lifecycle of CBA exercises involves the following additional stages: • Testing and deploying the exercise through CourseMaster • Running the exercise and marking student solutions • Administering the exercise and evaluating the results Diagram-based CBA exercises are authored with ease once appropriate marking tools have been developed. The task is lengthy, but straightforward and the outcome benefits both students and educators alike. Figure 5 depicts the overall process. Daidalos is used to create tool libraries. Ariadne is then used to author CBA exercises in the domain for which the tools have been created. Daidalos 1. create diagram elements by defining : - graphic view - connectivity - data-model 2. create tools and make a tool library 3 create and make application options CourseMaster Ariadne 4. author diagram-based CBA by defining: - theseus configuration - CM exercise files - Marking files Marking System Marking Tool Marking Command A diagram-based CBA exercise Theseus configuration Tool Library Available Options File Options Undo-Redo Clipboard Options Alignment Options Grouping Options Zooming Options ... ... ... External Tool Options CourseMaster exercise configuration files: - title - question - exercise properties - setup properties - client project properties - solution - skeleton solution - scaling information Marking tool configuration Marking scheme (mark.java) Figure 5: Steps for authoring diagram-based CBA Application options for Theseus must be selected, and liked the uncluttered feel and the speed with which Ariadne is then used to develop the marking scheme and Theseus responded during the drawing of their solution configure the appropriate marking tools and the and the marking of their work. The students also properties of the specific CBA exercise. mentioned that learning Theseus was easy and that the The exercise is then ready to be deployed and tested intuitiveness of the interaction with the diagram induced a through CourseMaster and Theseus. playful state of mind. The second tested exercise type incorporated flowcharts. This exercise requires students to draw a 5: Experience flowchart for comparing three numbers. Although simple, In the academic year 2000-2001, exercises in three this example uses all the nodes of the flowchart diagram different domains were automatically assessed using notation. The students were taught the basics in flowchart DATsys and CourseMaster at the University of diagrams. Being novice programmers, they had to design Nottingham. The exercises were part of a diagramming a simple algorithm. course that taught a range of diagram-based concepts. The flowchart exercise proved to be very popular. The authored exercises domains were: The majority of the students entered their solution directly • Logic design into Theseus. Others got the order of input wrong and produced mixed results, but with the help of • Flowchart design CourseMaster’s feedback, they rectified their mistake to • Object-oriented design complete the exercise. The exercises were authored using Daidalos and Object-oriented design was the third domain for a Ariadne and were deployed via CourseMaster servers. diagram-based CBA exercise. Students were required to The class consisted of 167 first year undergraduate design a hotel management application according to a students majoring in computer science. well-defined specification. This exercise is harder to The two logic design exercises were to draw a simple solve than the circuit or flowchart exercises because of the circuit for an elevator control board and to design a expressiveness of the object-oriented diagram notation. circuit for a switchboard that controls a nuclear facility. The students had to perform critical thinking on which A demonstration was important to familiarise the students object-oriented elements to use and how to connect them. with Theseus, although students were already familiar The object-oriented design exercise has been popular due with CourseMaster. to its clear and well-focused question. The complex These exercises proved to be successful. Once nature of the solution drove a small number of students to students understood the question, they were able to draw the solution on paper first and then to enter it into identify the necessary components and draw the solution. Theseus. Upon closer inspection, the root of the problem The vast majority of the students came up with the correct seemed to be that these students preferred to design on results. Some students experiencing difficulties were paper in general. Overall, diagramming exercises met helped by CourseMaster’s feedback and as a result were with considerable success. able to draw the correct circuit. Students particularly Proceedings of the International Conference on Computers in Education (ICCE’02) 0-7695-1509-6/02 $17.00 © 2002 IEEE 6: Future Work Many other types of exercises in various diagrambased domains can be created using the combination of DATsys and CourseMaster. Figure 6 illustrates 16 types of possible coursework for which tool libraries have been authored. The time spent authoring for these 16 examples was minimal. It took just three hours to make all 16 tool libraries using Daidalos. An additional three hours were needed to draw the diagrams using Theseus. Creating a tool library within Daidalos is sufficient only if the developer is satisfied with the similar-looking types of Theseus editors. A course developer with some programming experience could also extend DATsys with a new package of classes representing a completely new editor. In this case, the concepts modelled within DATsys could be directly reused. Devising a marking strategy and appropriate marking tools is an essential step in authoring the CBA exercise. In some cases, common diagrammatic marking tools can be reused. In others, completely new tools and configurations are necessary. Diagram-based marking tools can be developed to mark a selection of diagram notations that have been illustrated in figure 6. The feature tool can often be used to address any shortcomings in other marking oracles. • Data-flow diagrams could be marked in a similar manner to flowcharts. • Database scheme diagrams could be marked by using a suitable tool that converts the diagram to a database table, runs SQL queries, and tests the output data using oracles. • Network diagrams might be converted into formats understood by various network simulator tools. Such tools can perform a variety of tasks including load balancing distribution examination, data throughput analysis and performance scaling investigation. • Pert diagrams can be marked with the combined use of a pert simulation tool and a feature tool. The simulation tool would calculate and evaluate time dependencies and identify any discrepancies in the students’ solutions. • Medical diagrams (and any other picture-based diagrams) can be assessed by developing a marking tool of the graphical/hot spot category. • Analogue circuit diagrams could be marked with the use of an analogue simulator tool. A converter would 1. Arrows and Cells 2. Data-Flow Diagram 3. Database Schema 4. ERD 5. Structure Diagram 6. Process Diagram 7.Network diagram 8. Pert Diagram 9. Mind Map 10. Medical Diagram 11. Petri Net 12. State Transition Diagram 13. Graph – Tree 14. Chemical Diagram 15. Concept Map 16. Analog Circuit Diagram Figure 5: Various views of Theseus for sixteen notations Proceedings of the International Conference on Computers in Education (ICCE’02) 0-7695-1509-6/02 $17.00 © 2002 IEEE • • need to convert the diagram to a net-list in a format that is understood by an external simulation tool such as Spice [14]. Spice could be used directly or through a Java wrapper such as JSpice [13]. Concept maps can be marked by latent semantic analysis tools similar to Lou’s work on essay-based assessment [8]. A feature tool can then comment on the presence or absence of certain types of concepts. 7: Conclusions Evaluation results on diagram-based domains, such as circuit design and software design, indicate that the automation of the assessment of diagrams can be effective and useful. CourseMaster has been available within academia since 1999, providing support for the marking of programming coursework. The diagram authoring extension, entitled DATsys, was integrated into CourseMaster in May 2000. Together they provide a novel and realistic foundation towards authoring, running, and administering diagram-based CBA. Exercises for three domains have been authored, deployed, marked, administered and evaluated with 167 students at the University of Nottingham. Many more example exercises can be created. Daidalos has libraries for many types of diagrams. With the addition of appropriate marking tools diagram-based CBA can be created for most types of diagrams. Although developing new marking tools is a development task that needs planning, reuse of existing marking tools with minimal configuration modifications is possible. References [1] Beck K., Gamma E., Advanced Design with Patterns in Java, OOPSLA’97, Tutorial No. 30 [2] Brant J., HotDraw, MSc thesis, University of Illinois, Urbana-Champaign, 1995 [3] Bull, J., Update on the National TLTP3 Project: The implementation and evaluation of computer-assisted assessment, Keynote, Proceedings of the 3rd Annual Computer Assessment Conference, Loughborough, UK, 16-17 June, 1999 [4] Charman D., Elmes A., Computer Based Assessment: A guide to good practice, Volume I, University of Plymouth, 1998 [5] Culwin F., Web hosted assessment--possibilities and policy, Proceedings of the 6th annual Conference on the Teaching of Proceedings of the International Conference on Computers in Education (ICCE’02) 0-7695-1509-6/02 $17.00 © 2002 IEEE Computing/3rd Annual ITiCSE Conference on Changing the Delivery of Computer Science Education, 1998, Pages 55–58 [6] Foxley E., Higgins C., Hegazy T., Symeonidis P., Tsintsifas A., The CourseMaster CBA System: Improvements over Ceilidh, 5th Annual Computer Assisted Assessment Conference, Loughborough, UK, 2-4 July, 2001, Pages 189-201 [7] Foxley E., Higgins C., Tsintsifas A, Symeonidis P. Ceilidh, a System for the Automatic Evaluation of Student Programming Work, Proceedings of the 4th International Conference on Computer Based Learning in Science (CBLIS99), University of Twente, Netherlands, July 2-6, 1999, Section I6 [8] Foxley E., Lou B., A Simple Text Automatic Marking System, Artificial Intelligence and Simulation of Behaviour 94 Conference for: Computational Linguistics for Speech and Handwriting Recognition, Workshop in Leeds University, UK, April 12th, 1994 [9] Foxley E., Mohd Zin A. M., The Oracle output recogniser, LTR Report, Computer Science Department, The University of Nottingham, 1993 [10] Higgins C., Hegazy T., Symeonidis P., Tsintsifas A., The CourseMaster CBA System, submitted to the Journal of Education and Information Technologies (official Journal of the IFIP Technical Committee on Education) published by Chapman and Hall [11] Higgins C., Symeonidis P., Tsintsifas A., The Marking Sybsystem for CoureMaster, to be presented at the 7th Annual Conference on Integrating Technology Into Computer Science Education, University of Aarthus, Denmark, June 24-26, 2002 [12] Johnson R., Documenting Frameworks using Patterns, Proceedings of OOPSLA’92, ACM SIG-PLAN Notices, Volume 27, Number 10, Vancouver BC, Canada, October 1992, Pages 63-76 [13] Souder D., Herrington M., Garg P., DeRyke D., JSPICE: A component-based distributed Java front-end for SPICE, Proceedings of the 1998 Workshop on Java for HighPerformance Network Computing, 1998 [14] Vladimirescu A., The SPICE Book, John Wiley, New York, 1994 [15] Vlissides J. Generalized Graphical Object Editing, Ph.D. thesis, Stanford University, 1990 [16] Williams R., Automated essay grading: An evaluation of four conceptual models, In Herrmann A. and Kulski M. (Editors), Expanding Horizons in Teaching and Learning, Proceedings of the 10th Annual Teaching Learning Forum, Perth, Curtin University of Technology, February 7-9, 2001 [17] Symeonidis P., An in-depth review of CourseMaster Marking SubSystem, Technical Report, LTR Group, University of Nottingham, UK, 2001