Fostering Theoretical, Empirical and Tool Specific Knowledge in a Software Testing Learning Scenario

Fostering Theoretical, Empirical and Tool Specific Knowledge in a Software Testing Learning Scenario Ellen F. Barbosa1, Christian M. Adriano1, José C. Maldonado, Ivan L. M. Ricarte, Mário Jino Abstract  Teaching some subjects, as for example Computer Science, concerns the support for a learning process that involves cooperation of theoretical and empirical knowledge with related software tools. The underlying hypothesis is that there is a lack of appropriate support, to students and instructors, to the apprenticeship of specific theories and skills. The current research aims at a more specific learning scenario – Software Testing – addressing of what would be these specific theories and skills. A Software Testing Lab requires three cooperating types of knowledge – theoretical, empirical and tool specific, which together compose the learning and the evaluation process. To investigate such hypothesis, a specific learning scenario is implemented within a platform named CALM. This learning scenario comprises a unit testing activity and the support of an integrated testing tool – PokeTool. With the implemented learning scenario we hope to be in a position to suggest more specific guidelines to be integrated into the platform CALM. It is expected that this experience can also be extended to other software engineering courses. Index Terms  Software Testing Lab, Testing Tool, Educational/Training Environment, Learning Scenario I. INTRODUCTION In a previous experience [1] we investigated the construction of computer supported environment as a solution for problems faced while teaching computer programming. Such problems often relate to mismatch between theory and praxis, as well as lack of specific skills. Our approach was to complement theory and lab experiences with a more flexible solution inspired by the GBS (Goal Based Scenarios) [2]. A result of this effort was twofold: the conception of a component-based platform named CALM (Computer Aided Learning Material) and, consequently, a series of discussions concerning the implementation of learning scenarios under divergent pedagogical approaches. A difficulty faced was the lack of an actual learning context to subsidize technical and pedagogical argumentation. In other words, which would be the specific theories and skills? The first point is the need for an actual learning scenario in which we could apply the educational technology and evaluate its robustness, availability, usability, extensibility and the ease to integrate legacy tools. Robustness is whether technology fulfills learning scenario requirements without crashing. Availability defines the environmental conditions for access, 1. Supported by FAPESP. Contact: Ivan L. M Ricarte, DCA/FEEC/Unicamp, tel. +55-19 778-3771, fax. +55-19 289-1395, ricarte@dca.fee.unicamp.br. for example, an intranet, modem access, and stand-alone operation. Usability evaluates the graphical interface design. The extensibility aims at criticizing design decisions in respect to how easily the platform accommodates architectural changes. The ease to integrate legacy tools is a major point of concern. According to the GBS approach, a learning scenario provides that students perform real world roles. By performing such a role, individuals apply concepts and acquire, as a byproduct, the intended skills. Moreover, tools are a selfdescribing embodiment of ideas and concepts. In other words, they encapsulate information that cannot be easily captured on written material. Most tools allow users (students) to acquire dynamic, runtime information. Besides to embody ideas and concepts, tools are also a means to observe/measure the student’s apprenticeship, acting as a way to improve it. In this sense, tools are essential to the learning process, as well as to its evaluation/improvement [3]. It is then straightforward that a realistic learning scenario requires manipulation and mastering of professional tools. Given that several researchers from software testing area compose our research team, an interesting context was naturally devised, an educational/training-testing environment that could be used in both academia and industry. Our hypotheses issue the appropriate technological and pedagogical harmonization necessary to foster distinct cooperative software testing skills. The remainder of this paper is organized as follows. In Section 2, an overview of software testing is provided. Section 3 briefly presents the platform CALM. Some relevant aspects related to the integration of PokeTool within the platform CALM are discussed in Section 4. Section 5 exposes an exemplification of a software testing learning scenario. In Section 6, our conclusions and further work are presented. II. TEACHING SOFTWARE TESTING Software Engineering can be defined as a discipline that applies sound engineering principles in order to produce high-quality and low-cost software. Teaching Software Engineering deals with a learning process that involves cooperation of theoretical and empirical knowledge with related supporting tools. In other words, the “knowledge” in Software Engineering may be divided in theoretical knowledge, empirical (practical) knowledge and supporting tools. This structure is also applied to software testing, which, in the context of Software Engineering, constitutes one of the most relevant activities to guarantee the quality and the reliability of software under development. Software testing has as objective the identification of notyet-discovered errors. The success of the testing activity depends on the quality of a test set. Since the exhaustive test is, in general, impracticable, some restricting criteria are necessary. Such criteria aim at selecting subsets of the input domain, while preserving the probability of uncovering errors. There are a large number of criteria available to evaluate a test set for a given program against a given specification. Testing criteria are established from three techniques: functional, structural and error-based. In the functional technique, criteria and test requirements are established from the program specification; in the structural technique, we focus on details of a given implementation; in the error-based technique, criteria and test requirements are derived from the knowledge of typical errors occurred during the software development process. Another relevant point related to the quality and productivity of software testing refers to supporting tools. Applying a testing criterion without the support of a tool is an error prone approach. The availability of testing tools makes the conducted test a more satisfactory process and minimizes the errors caused by human intervention. Moreover, some tools enable that basic concepts be applied in realistic situations. Such possibility fosters training situations and promotes exchange of technology between industry and academia. A significant amount of work related to software testing has been carried out: definition and evaluation of testing criteria [4][5][6], supporting tools development [7][8][9] as well as theoretical and empirical studies [10]. Despite its importance in the software development process, the state of the practice and the state of the art present a gap. Mastered concepts and tools in academia and in advanced research centers are not naturally transferred or at least tried out in real software production. This is the case, for instance, for data-flow based testing criteria, introduced by the 70’s. Although a sound, supporting tool is available [7], these criteria are not common place in industry; i.e., its application in industry is very limited. The availability of an adequate learning environment would promote both, the improvement of personnel formation and technology transfer. III. THE PLATFORM CALM CALM is its essence a platform of software components to build web-based learning systems. Following the LTSC (Learning Technology Standard Committee) definition [11], a web-based learning system is: “A learning technology system that uses web-browsers as the primary means of interaction with learners, and the Internet or an intranet as the primary means of communication among its subsystems and with other systems”. The pedagogy orientation is defined by an appropriate implementation for a component named Tutor. Such a component embeds an intelligent tutoring system responsible for recommending lessons. Our provided implementation for the Tutor component produces recommendations by regarding on student-defined learning objectives, learner history, and the available course content. The learning objective is a composition of subjects of interest. While delivering lessons, the Tutor component keeps track of the student already done learning units. A lesson comprises a learning unit, a test, and a set of recommended learning units. Recommended learning units consist of prerequisites for the current learning unit. The Tutor also provides some degrees of freedom, one of that lets the student obtain a learning unit without accomplishing its prerequisites. At any moment the student can interrupt the lesson and request one of the recommended learning units. The interrupted lesson can be resumed later at anytime. The test is not used as a certification process, rather it is thought as a complementary opportunity for inquiring, problem solving, and knowledge bridging. IV. POKETOOL INTEGRATION Our horizon comprises basically the specification, design and implementation of a technological support for a software testing learning scenario. At ICMC/USP, among the disciplines directed to undergraduation students, we have a specific one related to software testing: VV&T – Verification, Validation and Test. It comprises, basically: 1) Theoretical classes, in which the main testing concepts, as well as some of the most relevant techniques and criteria, are presented to the students; and 2) lab classes, which consists on letting students apply acquired concepts by means of testing tools. One of these tools is PokeTool (Potential Uses Criteria Tool for Program Testing) [8], developed at DCA/FEEC/UNICAMP in collaboration with ICMC/USP. The PokeTool supports the testing of Cprograms at unit level. It measures the adequacy of test sets with respect to the control-flow and data-flow based testing criteria (structural technique). Basically, PokeTool performs three main operations: 1. Test session creation: A test session relates to some activities comprising the test. This way, if the test has to be interrupted, the intermediate status can be stored and resumed lately. 2. Test case handling, including execution and inclusion/exclusion of the test cases provided by the tester. 3. Adequacy analysis: Determines the percentage of test requirements for a specific criterion that was already satisfied by the set of test cases. It involves the generation of statistical reports about the conducted tests. PokeTool is available for DOS and UNIX. Its version for DOS has a simple interface, based on menus. The version for UNIX has functional modules that can be called by graphical interface or command line (shell scripts). The graphical interface allows the user to explore and learn the testing concepts as well as to better understand the tool operation. Conversely, to conduct empirical studies, scripts are more adequate once they reduce the amount of interactions required by the graphical interface. The compromise is that scripts require more programming effort, deep understanding of testing concepts, and precise information of the tool-composing modules. In lab courses, we experience using PokeTool with graphical interface and with scripts. Regarding our knowledge about the tool and the gathered experience, we designed and implemented an architecture to integrate PokeTool to the platform CALM. Although PokeTool supports the application of several structural testing criteria, in a first attempt we dealt only with the control-flow criteria: All-Nodes and All-Edges. After all, these criteria represent the most used criteria of the structural technique and some concepts worked on them are the basis to understand other structural criteria. The tool integration was directed by a concern of providing a generic interface between systems (tool and educational/training environment). Such concern recalls on three important issues: access to source codes, need for a uniform human-readable protocol for input and output, and studentsystem interaction. Usually, the access to the legacy source code is neither possible nor desired. Our solution was to deal with the output files that, in general, are produced by the tools. Therefore some effort is spent in decoding output files. The file-based approach imposes that the resulting integration architecture should convey tool specific formatted data. Since the chosen platform was CALM, all integration must be component-based. In short, a component is a black box with a publishable interface for its context and events. The context represent the data exchanged between components; furthermore it may consist of any data type. Some uniformity and standardization of such exchange are desired. The major reason is that any human eventual intervention and discussion about output and input results would be extremely facilitated. Therefore, the envisaged solution was to specify an XML (eXtensible Markup Language) based data type. In the case of software testing, we define a set of XML-tags (e.g., Node, Edge, Requirement, and Criterion). Another point is the decision about the student-system interaction. We defined that interfaces should be completely integrated to the platform CALM. Since scripts vary from one legacy tool to another, any programming specificity should be filtered by a proper graphical interface. Therefore, we developed an applet-based interface to allow the input and output of data. Basically, the integration architecture comprises two software components (Fig. 1). The CALM_Converter component encapsulates data necessary to setup PokeTool. The Poke_Converter component receives such data and activates PokeTool scripts. All output is processed by the Poke_Converter component and sent back to the CALM_Converter component. The CALM_Converter extracts such data and presents them to the student. All data is conveyed in the XML-defined protocol. CALM Interface data stream CALM Converter internet XML XML Poke Converter script file PokeTool Fig. 1. PokeTool and CALM Integration Architecture VI. SOFTWARE TESTING LEARNING SCENARIO The software testing learning scenario aims at providing the student with the basic concepts and steps to conduct a testing activity. Besides familiarizing with the relevant test concepts, students should be able to plan tests, to construct test cases and to analyze/evaluate the testing results. In terms of pedagogical relevance, the concept of testing criteria can be considered as fundamental. Testing criteria, in a simplified view, provide guidance on how to test a given program and when to stop testing. A testing criterion can be used for selecting test cases or checking if a selected test set is adequate on testing the product. There are several testing criteria, each one dealing with a specific set of test requirements and exercising the program in different perspectives. Such diversity of criteria and aspects allied with the objective of establishing an effective, low-cost testing strategy drove the test community to theoretical and empirical studies. Effectiveness, cost and strength are the three most meaningful bases against which testing criteria may be compared. Effectiveness is related to the fault detection capability of a criterion; cost indicates the effort to satisfy a criterion; and strength refers to the difficulty of satisfying a given criterion C2 for a test case set T that already satisfies another criterion C1. Many other concepts are related to testing criteria. To apply a specific criterion, the student must know the test requirements and the basic underlying model on which the criterion is based on. Analyzing test requirements, the student should be able to construct specific test cases against which the program (or their specific functions) will be tested. As mentioned before, in our testing learning scenario, students deal with the All-Nodes and All-Edges criteria. To satisfy the All-Nodes criterion, for example, it is required that all the program statements (nodes) be executed by the test case set at least once. Thus, the test requirements for AllNodes are the program statements. In the same way, the test requirements related to All-Edges are the program’s decision statements, i.e. the transfer of control between nodes (e.g., ifthen, if-then-else, case, while, for, repeat-until). Concerning the underlying model, All-Nodes and All-Edges are based on the control flow graph, i.e., a directed graph whose nodes and edges correspond to the program elements. At first, the student needs to create a test session, i.e., to provide, via CALM interface, the necessary information to proceed the testing activity on PokeTool. The needed information is: name of test session, directory where the source program to be tested is, compilation command and functions (modules) to be tested. After creating a test session, a screen with the test requirements for All-Nodes and All-Edges is presented to the student. At this point, the student has to exercise skills of analyzing test requirements and of constructing test cases based on them. In this perspective, the testing criterion is a method for selecting test cases. Another relevant point comprises the adequacy analysis to the test cases with respect to the criterion. The student should know to interpret the resulting data in order to continue or not the testing activity. The goal is to exercise as much as possible the test requirements. The adequacy of a test case set with respect to All-Nodes, for example, may vary between 0 and 1. An adequacy of 1 represents that 100% of the test requirements for the criterion were matched. If the adequacy obtained is low, the student may project other test cases and require a new evaluation. This process is repeated until a high adequacy value is accomplished. In this case, the testing criterion plays a role of adequacy criterion, providing a coverage measure. The complementary aspects of testing criteria may also be observed in our learning scenario. In general, after satisfying all test requirements of All-Nodes, some of All-Edges test requirements may still remain unsatisfied and new test cases should be constructed in order to fulfil All-Edges requirements. This way, the student is able to compare results of previous attempts. The amount of test cases may measure the cost aspect. The effectiveness may be investigated by using a program that contains some faults; as test requirements are satisfied, the student may observe the ability of the chosen testing criterion to expose faults. Concerning the strength aspect, the student may observe the difficulty to satisfy AllEdges requirements while having satisfied All-Nodes requirements. The student-system interactions of the described learning scenario are presented in Figure 2. this work is a first step on direction to integrate external tools within educational/training environments. As future work we intend: 1) to develop a testing learning scenario using all the criteria supported by PokeTool; and 2) to define a mechanism for allowing the student to use the graphical interface of PokeTool within CALM. Furthermore, we intend to define a mechanism to allow conducting empirical studies via the CALM interface by means of the PokeTool own testing scripts. The batch processing enabled by such scripts are very important for the empirical studies, in which a given sequence of steps must be run several times until results reach some statistical significance. Another relevant point to be considered is to allow other testing tools, supporting other testing criteria, to be integrated with the platform CALM. Such objective aims at establishing a complete learning process for teaching software testing. Moreover, we intend using the tools to measure the student’s apprenticeship, giving a feedback to improve the learning process. Finally, we also plan a case study for the exposed learning scenario. The case study will evaluate the learning scenario and the underlying technology in respect to educational issues and technical concerns such as robustness, extensibility, and availability. REFERENCES [1] C. M. Adriano et al., “Inquiring the Course Paradigm with CALM”, Proceedings of the International Conference of Engineering and Computer Education, August 1999, Rio de Janeiro, RJ, Brazil. [2] R. C. Schank, A. Fano, B. L. Bell, M. Y. Jonea, “The Design of Goal Based Scenarios”, The Journal of the Learning Sciences, vol.. 3(4), pp. 305-345, 1993-94. [3] Computing Research Association, “Evaluating Computer Scientists and Engineers for Promotion and Tenure”, http://www.cra.org/reports/tenure_review.htm, September 1999. [4] J. W. Laski, B. Korel, “A Data Flow Oriented Program Testing Strategy”, IEEE Transactions on Software Engineering, v. 9, n. 3, May 1983. [5] S. C. Ntafos, “On Required Element Testing”, IEEE Transactions on Softaware Engineering, v. 10, n. 6, November 1984. [6] S. Rapps, E. J. Weyuker, “Selecting Software Test Data Using Data Flow Information”, IEEE Transactions on Software Engineering, v. 11, n. 4, pp. 367-375, April 1985. [7] H. Agrawal et al., “Mining System Tests to Aid Software Maintenance”, IEEE Computer, pp. 64-73, July 1998. [8] M. L. Chaim, J. C. Maldonado, M. Jino, E. Y. Nakagawa, “PokeTool: Estado Atual de uma Ferramenta para Teste Estrutural de Software Baseado em Análise de Fluxo de Dados”, Proceedings of the XII Brazilian Symposium of Software Engineering, Maringá, PR, Brazil, pp. 37-45, October 1998 (in Portuguese). [9] M. E. Delamaro, J. C. Maldonado, “Proteum: A Tool for the Assessment of Test Adequacy for C Programs”, Proceedings of the Conference on Performability in Computing Systems (PCS’96), Brunswick, NJ, July 1996. Fig. 2. Student-System Interaction V. CONCLUSION AND FUTURE WORK In this paper we discussed the need to integrate legacy tools into an educational/training environment as a way of promoting skills, empowering students and applying theoretical knowledge. We defined a possible mechanism to integrate the PokeTool (a testing tool) into the platform CALM. Although simple, [10] A. P. Mathur, W. E. Wong, “An Empirical Comparison of Data Flow and Mutation-Based Test Adequacy Criteria”, The Journal of Software Testing, Verification and Reliability, v. 4, n. 1, pp. 9-31, March 1994. [11] Learning Technology Standards Committee, “Working Draft Glossary: February 28th 2000”, http://ltsc.ieee.org/wg3.