Evaluate and Improve GUI Testing Coverage Automatically

Evaluate and Improve GUI Testing Coverage Automatically Alsmadi et al Evaluate and Improve GUI Testing Coverage Automatically Izzat Alsmadi (1) , Samer Samarah (2), Ahmed Soefan(3), and Mohammed AL Zamil (4) Department of Computer Information Systems, Yarmouk University, Jordan E-mails: (1)ialsmadi@yu.edu.jo, (2) samers@yu.edu.jo, (3) ahm79_cs@yahoo.com, (4)mohammed_alzamil@yahoo.com ABSTRACT Evaluating coverage criteria is a core subject in software testing. Coverage can be evaluated based on several concerns such as: code, requirements, syntax, paths, decisions, graph, etc. The focus of this research is in evaluating the Graphical User Interface (GUI) testing coverage. In this research, several GUI model based methods are proposed and evaluated for the goal of automatically testing user interfaces and evaluate the testing coverage. The user interfaces’ components of tested applications are collected while applications are running and this information is used to build a GUI structural model. This model is used as an input for the process of generating test cases automatically and on considering different aspects of GUI coverage such as GUI paths, edges, nodes, or components. Microsoft NModel and SpecExplorer are used as tools for the GUI model verification. This is usually accomplished indirectly through generating test cases from the model and evaluating the model through the effectiveness and coverage of the generated test cases. Keywords: GUI testing, model based verification, User interface verification, User Interface modeling, GUI specification, Formal methods. 1- INTRODUCTION In model based testing, a critical subset or aspect of the system is selected. A model, based on this focus or concern, is then built. Later on, test cases are generated with the goal of covering this aspect of the application. A major example of system aspects that are usually used in model based testing is the system state. In such approach, the possible states of the system are described. Test cases are then generated to cover all – or most- of the possible states and that expose all possible transitions. One of the major goals of most software testing research papers is to improve the coverage in optimizing test case generation and execution. There are several attributes that are used to evaluate the quality of test case generation methods. Examples of those attributes include: performance, accuracy, and coverage. Performance refers to the time it takes to generate, execute and 41 Int.J. of Software Engineering, IJSE Vol.4 No.2 July 2011 verify those test cases. One or more of those activities can be automated. However, in all cases, it is usually acknowledged that an automatic process is faster than a manual process (i.e. that is executed through a human tester). The process of automatically generating test cases from the requirements, the design or the code is a major research focus in this area. There are also several tools and research papers that discuss the automatic execution (and possibly verification) of test cases through tools or testing robots. There are two main adopted techniques for the automatic execution of test cases. Those are: Record/play tools that start by a human executing the test cases and then a robot repeating the process. The second automatic execution technique, that is usually called the object or the data driven model, utilizes using reverse engineering methods to analyze and serialize the actual application dynamically to use this knowledge in generating and executing test cases automatically. A previous work by the author is accomplished in this subject and described in the papers: [1, 2, 3, and 4]. The last stage in testing includes the verification of the results. The results from the test case generation process are compared with the expected results. There are several existed tools for the automatic verification. Examples of those that are used on the unit or structural code testing are: NUnit, JUnit, NAnt. Most programming development and testing tools incorporate such tools within their Integrated Development Environments (IDEs). The second criterion that can be used to evaluate the quality of the generated test cases is the accuracy. Accuracy means the correctness of the inputs and expected outputs of the proposed test cases. Accuracy includes also the correctness of the described test case path, preconditions, post conditions and any possible assumptions. The third criterion, which is the focus of this paper and the majority of software testing papers, is coverage. Coverage or adequacy is a software testing metric that is used to evaluate the relevancy or the effectiveness of the generate test suite accumulatively. There are several types of concerns in the application that can be used to evaluate coverage. Some of those coverage criteria are code based such as: statement, decisions, and paths coverage. Besides code criteria, coverage can be calculated based on requirement or design criteria. It is widely acknowledged that reaching the perfect stage of covering all application aspects in testing is practically impossible. It is also usually acknowledged that even most of coverage aspects cannot be verified or covered 100 %. This is largely due to the large amount of time and resources required to reach such coverage levels. Automatic test case generation, execution and verification are always welcomed choices as they are expected to reduce the time and resources required to accomplish those tasks. The focus of this paper is on proposing automatic methods for generating test cases from the GUI model. Formal methods are used to assist in the process of describing the GUI model formally. This can facilitate the automatic genera42 Evaluate and Improve GUI Testing Coverage Automatically Alsmadi et al tion, execution and verification of the generated test cases. The rest of the paper is organized as the following. First some of the relevant subjects will be introduced. This includes: formal methods in GUI testing, software graph models (GUI, event and control flow graphs), and challenges of using GUI model based testing. The next section introduces selected relevant research papers. As the paper is related to several different aspects of software testing, those selected papers are not comprehensive. This means that the selected papers are not the only or the best in those fields of study. The main section in the paper elaborates our work in developing, analyzing and evaluating model-based GUI test automation techniques to evaluate and improve different coverage aspects. Using Formal methods in GUI testing Formal Methods are "mathematically based techniques for describing system properties" [5]. They are used to verify the correctness of the software requirement or design before reaching the testing stage. The idea is to discover problems early and hence reduce the impact of the possible consequences. Using formal methods is usually avoided due to the extra time it takes to learn and formally define requirements, model etc. This time is critical in the early stages of the development. Another reason for not considering formal methods is due to the continuously evolving environment of the software projects that make taking extra time to verify requirements unjustifiable. Due to its inherent strictness, formal methods have been well accepted when applied to critical systems. Model checking is a technique used to prove, usually automatically or programmatically, that a logical property, P, holds true of a system behavior, B, specified as a Finite State Machine (FSM). Properties are expressed in temporal logic that allows reasoning over the possible execution paths. Formal methods in general and specification and model based testing in particular have been applied to the testing of software low level interfaces (APIs; Application Program Interfaces). However, they are not so commonly applied to the testing of software applications through their GUI. The major goal of model based testing is to improve the specification or the design and hence, indirectly, improve testing. In this project some light weight formal ideas are introduced and developed to be evaluated automatically. Such automatic evaluation may help in popularizing such methods. Designing, developing and testing user interfaces usually takes a considerable amount of project time. Automating all or parts of such activities may help in the project resource optimization. Control Flow Graph (CFG), Event Flow Graph (EFG) and GUI Graph (GUIG) There are many graph models that are used model one or more aspects of a program. In this section, three types will be introduced. Those are: CFG, EFG 43 Int.J. of Software Engineering, IJSE Vol.4 No.2 July 2011 and GIG or GUIG. A CFG is drawn to understand the complexity of the program in terms of its decisions and branches. Figure 1 shows the CFG graph for the following code. int k=0, total=0; int a[] = new int [10]; bool flag=false; R1. while(k < 9 && flag ==false){ total=total+ a[k]; R2. if(total >= 8){ flag=true; else k=k+1;} Console.WriteLine(total); Figure 1 A simple CFG graph CFGs were used to have significant information in structured programming languages where methods have public or global meaning. However, this may not be the case for object oriented languages. Nonetheless, CFG can give us realistic indication of the complexity level of the program. Test coverage or adequacy can use the CFG graph as an important factor (i.e. decision coverage). A CFG of a program is a directed graph G = (N, E, s, e) where N is a set of nodes, E is a set of edges, and s and e are unique entry and exit nodes to the graph. Nodes in the graph represent decision statements (for example an: if, else or while statements). Each decision element involves branching for one or more branches. Typically a CFG should have one distinguished entry and exit nodes. An EFG is usually used to model the interaction between the application components. A GUI EFG can be drawn to represent all possible interactions between GUI components. This is somewhat similar to a state transition diagram where each possible state of the system is described along with all possible transitions and event from these states. A GUIG is a static representation of the GUI. This representation studies all GUI components, their attributes and their hierarchical relations. This is a dif- 44 Evaluate and Improve GUI Testing Coverage Automatically Alsmadi et al ferent graph from the EFG that focuses on GUI components and their interaction with each other. The GUIG can be used to generate event and test sequences without the need to know or consider the exact events as the majority of GUI component events can be predicted. For example, menu items in a menu are usually triggered through mouse clicks on those objects, textboxes are usually expecting text to be entered in their text attributes, buttons are usually accepting mouse-click events, and so on. The advantage of using GUIG over CFG is that getting GUIG is simple in managed codes (i.e. codes of most new object oriented languages) where such information can be serialized through a reverse engineering or reflection process from the code metadata. NModel, that will be described and used later in this paper can be used to draw a graph to represent the possible states in the program. Challenges of using GUI model based testing Applying model based testing techniques for testing user interfaces may face some challenges. 1. State explosion. The first challenge is dealing with the state explosion problem where there will be a large number of possible GUI states in even simple applications. There are several techniques used in the research to deal with GUI states’ explosion. In contribution to this goal, the GUIG model that is adopted in this research is an abstraction model that ignores or excludes several irrelevant events and GUI components’ attributes in order to reduce the number of GUI possible states. States are defined on classes, objects, or systems based on attributes values in which the system behaves differently. Table 1 shows an example of a course section class where the possible states are: new, open, full and canceled. In each one of these states, some behaviors are permissible while others are not. Table 1An example of a class section possible states. State New Open Full Closed Permissible behavior 1. Register new student 2. Cancel section 1. Register new student 2. Deleting a registration 3. Calculating No. of registered students. 1. Deleting a registration 2. Calculating No. of registered students. 1. Activate section. Prohibited behavior 1. Deleting a registration. 2. Calculating No. of registered students. 1. Cancel section 1. Cancel section 2. Register new student 1. Register new student 2. Deleting a registration 3. Calculating No. of registered students. 45 Int.J. of Software Engineering, IJSE Vol.4 No.2 July 2011 2. GUI modeling. There are several alternatives on how to model a user interface from its application or code. Three of those are explained earlier (CFG, EFG and GUIG). One limitation of the majority of GUI models is that they are generated manually which makes the process cumbersome and do not justify its usage on actual testing activities where resources are already fixed and limited. Another limitation is that they don’t take dynamic aspects into consideration. On the other hand, the initial developed model may not seem related to the final developed GUI. Other examples of challenges are that GUI modeling tools are not standards. Each research proposed tool used its own methodology and generate its own file format. The last difficulty in GUI modeling is the difficulty of the verification process. All those issues are considered in this research. The automatic generation of the GUIG from the dynamic or real time version of the application and the XML format of the output of the GUIG modeling maybe used to justify solving the majority of the challenges mentioned in this paragraph (completely or partially). The process of generating the GUIG model is not the subject of this paper as it is described in previous papers of the first author. 3. Model transformation. A GUI design model may not be easily or automatically converted to an implementation model. In this research we didn’t use model transformation to transfer a design model into an implementation one. The GUI implementation or dynamic model is used as an input. An XML file is generated to represent the GUIG model that is used for the process of test case generation, execution and verification. 4. GUI model construction effort. Generating a GUI model manually is time consuming and may justify skipping such stage trading off this effort with its possible advantages. This problem can be bypassed through automation. However, building such model automatically is not a trivial task. The hypothesis that this paper is built on is that using model based formal verifications automatically may improve software testing in terms of time (i.e. automation) and coverage. 2- RELATED WORK The focus of this research is in improving coverage in GUI test automation through formal methods or modeling. This may link this project to papers related to test automation, coverage evaluation criteria, GUI testing and the usage of formal methods in software testing. Beer et al proposed an automatic test case generation method utilizing formal models [6]. The coverage type for these test cases is feasible state-transition coverage. The framework focuses on automating test case generation only. The execution and verification processes were not automated through the proposed system. Similar to our approach, the behavior of the GUI is represented formally in a state transition diagram. However, the framework 46 Evaluate and Improve GUI Testing Coverage Automatically Alsmadi et al used the task oriented scenarios as the input to the formal specification of the GUI. In our model, the input to the whole process is the GUI model that is generated from the application at run time. This is usually the first step required in the automation process and is the input to the test case generator. Belinfante introduced a model based tool for web test automation [7]. Web test automation is a rich field especially in the industry where there are hundreds of tools for the automation of web test case generation, execution and verification. Papers [8 and 9] discussed using one of the open source web test automation tool; Selenium. Selenium is a suite of tools to automate web app testing across many platforms (http://seleniumhq.org/) Belli et al introduce a framework for the automation for testing component based applications with the goal of improving coverage and reducing test costs. [6]. The component model is the input to the behavioral model and then to the test cases. The comparison between the component and the behavioral model is used to evaluate the approach. The proposed test automation is evaluated using WinRunner record/replay test automation tool. Zhao et al proposed automating smoke testing using Rational functional tester and virtualization technology [10]. Smoke testing is a continuous integration test automation method that is usually using in agile software development methodologies. Kwon et al proposed a method and automatic tool for GUI mobile testing [11]. The MoGuT framework, which is explained and evaluated only heuristically in the paper, is proposed to run under Eclipse and generate XML files as an output. Paper [12] is another example of a paper that discussed GUI testing techniques on smart phones. The test model used the Labeled Transition States to describe the application. These special state labels could be transformed into keyword sequences that would check that the tested application is in the correct state. Appasami et al conducted performance analysis of various set of UI Test Automation for Silverlight applications [13]. Silverlight is a development platform for creating interactive user experiences for Web, desktop, and mobile applications. The research evaluates performance of accessing data for the different possible file types usually used in web applications such as text files, XML files, databases, or oracle table. Results showed that XML files showed the best performance. Xiaochunet al proposed a test automation solution on GUI functional testing [14]. Rational Functional Tester (RFT) is used as the test engine. RFT can recognize GUI objects (through the object capture built-in tool) when it touches them and can record their details for further testing. Fu et al proposed an approach to handle a weakness in GUI test automation scripts and automatically inferring types of GUI objects [15]. Scripts are usually rigid when dealing with GUI objects as they will have problems detecting the 47 Int.J. of Software Engineering, IJSE Vol.4 No.2 July 2011 objects if they move or change their color or size. Capture/replay tools are used to extract values of properties of GUI objects, so that these objects can be later identified on GUI screens by matching prerecorded values with the properties of GUI objects that are created at runtime. Bertolini et al proposed four blackbox testing techniques for crashing programs through their GUI interfaces [16]. The proposed techniques attempt to explore the state space of the application through stressing the application with automated interactions). In our paper, random testing methods are proposed to evaluate some GUI attributes such as detecting reachability or dead lock problems. Mathew et al discussed software test automation in Software as a Service (SaaS) applications [17]. SaaS is a software production or delivery method that provides access or use of the software and its functions remotely as a Web-based service. Grechanik et al introduced an approach to maintain and evolve test scripts while the application user interface is evolving without the need for extensive modifications [18]. One of the major challenges in GUI testing is that a relatively small change in the application may cause a large change in the GUI. The approach was intended to detect failures that are caused by the GUI evolution. Triou et al proposed a declarative testing that focuses on the abstraction of the user interface functionalities [19]. Although declarative test cases may not be very sensitive to GUI evolution, however, their high level abstraction nature may not make them applicable in many situations. Qian et al presented an event interaction model and algorithm to generate GUI test cases [20]. The paper used an Event Interaction Structure (EIS) to describe the GUI even interactions and generate test cases. Nonetheless, the paper didn’t elaborate on the different possible types of GUI events and focused on a simple GUI state transitions representation of the GUI. Abdul Rauf et al [21] and Alsmadi [3] proposed using genetic algorithms for the optimization of GUI test case generation and execution. There are many papers in the literature that are also aiming at improving the effectiveness or coverage of test case generation based on genetic algorithms, artificial intelligence, data mining or any type of useful algorithms or techniques. 3- GOALS AND APPROACHES In order to formally verify a software model or design against specifications, we have to clearly and as formally as it is possible describe the specification. We have to have specifications to verify against. How can we describe the GUI specifications in a format that can be verified? 48 Evaluate and Improve GUI Testing Coverage Automatically Alsmadi et al 3-1 GUI GRAPH AND TEST COVERAGE Test coverage is a general term that is used to evaluate one or more concerns in the application and calculate the percentage that the test cases in a test suite are covering. Examples of test coverage types used are: statement, decisions, paths, etc. As we will generate a GUI model that includes the GUI components, their attributes and relations, several coverage concerns can be evaluated from this model. This includes: GUI paths, nodes, edges and components coverage. 3-2 MODEL CHECKERS Model checkers are formal method tools that are used to describe a system or some of its components or properties and verify them based on some aspects or properties. Models do not have to be graphs or diagrams. They can also be formal descriptions of the requirements. Why do we want to formally define a user interface?! What are the things that can be checked and verified automatically using model verification?! The hierarchical nature of most applications’ GUIs makes it possible to define some properties that can be valid for the majority of the time. Here are examples of some of those properties. For the sake of simplifying the research proposal, we will assume that those hypotheses are applied on all tested applications. 1. A GUI application starts from one entry point. This entry point can be fixed or user defined. 2. A GUI application contains a finite number of components. Some of those components are containers i.e. they can be composed of other components. A parent-child relation or whole-part relation is connecting the container with the components it contains. 3. Each GUI component belongs to a class. It has attributes and values for those attributes that can be distinguished by from other components. 4. GUI paths can be formed starting from the GUI entry point, traversing all possible containers and components till reaching a node that is a leaf. A leaf component is a component that is does not call any component. For example, the “close”, and “exit” commands in MS Word are leaf components as they don’t call any further components once they are activated. Formally, a GUI model program can have the following components: • A finite vocabulary of state variables or attributes. Those are the variables that can impact the overall GUI state. For example, components location and size variables are state variables. • A finite vocabulary of action symbols. Actions are events that trigger state transitions. In GUI, components are classified by classes where each component class accepts certain types of actions. Those are the actions can change its state and ultimately the overall GUI state. Actions are atomic and can have 49 Int.J. of Software Engineering, IJSE Vol.4 No.2 July 2011 zero or more arguments. On the components level, a state can define the possible actions that can be triggered. Accordingly, actions may change the component and the GUI state. Figure 2 shows a simple GUI class diagram showing each class main attributes and actions. • An initial state S0. In GUI applications, values of components attributes can change dynamically at run time. As such, the initial state is that loading or design time state which contains initial values for all GUI components. In some cases, even the GUI components themselves can be added or removed at run time. • Guard rules or constraints. Those are preconditions for particular states, transitions or events to occur. Figure 2 A simple GUI class diagram 3-3 MAPPING GUI TO CODE As part of the GUI modeling process, it is important to define a formal procedure to transform the GUI model into code. This is similar in principle to mapping a UML class diagram to the code. In the class diagram case, in most cases, the mapping process is straightforward where each class in the class diagram is mapped directly to a class (or more) in the code. In GUI, only the main containers (e.g. Forms, web pages, etc) are transformed to classes. For example, in a .NET program, the majority of the classes can be divided into two components, code classes and forms. A GUI model or graph is proposed here to represent all containers. An inheritance-like relation is then drawn between a container and its contained components where the container (e.g. form, web page, panel, menu, frame, win50 Evaluate and Improve GUI Testing Coverage Automatically Alsmadi et al dow, etc) is the parent and its components are children. If any of those components is also a component, then it is also a parent for its own components. A tool is build to define such relation formally and generate or serialize GUI components based on this formalism to an XML file (Figure 3). Figure 3 A GUI to XML graph model. The advantage of such transformation is that it is much easier to use this GUI representation in the different processes such as model presentation, verification, test case generation, execution, etc relative to the actual GUI model. 3-4 USING MICROSOFT NMODEL NModel is a .NET framework of small applications that can help modeling an application from its methods or scenarios’ descriptions. A model program is used to debug and improve specifications and designs. Techniques used in model-based analysis techniques include: simulation, exploration, and composition. In model based testing, a model program is used to generate test cases from and or act as an oracle. The model includes the reference for the correct results to make a judgment about the outcome of a test case. In NModel, several state variables are defined. Possible states are defined based on the number of state variables and their possible values. An initial state is defined based on the initial values of those state variables. Figure 4 below shows a simple NModel example with two state variables: TextBox, and Label, each has several possible variables. Actions which are static in NMdel ( such as FormUI in the example ) are used to change the attribute values and hence cause a transition from one state to another. 51 Int.J. of Software Engineering, IJSE Vol.4 No.2 July 2011 Figure 4 A simple NModel example. The NModel framework includes a Model Program Viewer (MPV) tool. This tool is used to draw a program state graph (Figure 5). Figure 5 A program state graph using MPV. Besides design or model visualization, NModel framework can be used in learning the system dynamic behaviors and discover any possible errors, inconsistencies, or missing information. Examples of some dynamic behaviors that can be traced through NModel includes: progress, locking, and missing states or activities problems. Our investigation of using NModel in GUI model based testing shows that NModel states and activities definition can be easily connected with the GUI components interactions. In GUI structure, each GUI component is defined by 52 Evaluate and Improve GUI Testing Coverage Automatically Alsmadi et al several possible attributes and events. Those attributes and events can be automatically converted to NModel state variables and actions respectively. The information about GUI components can be collected from the application statically or dynamically using reverse engineering processes such as .NET reflection. Microsoft could have implemented such feature in .NET 2010 (i.e. include model based testing tools such as NModel) where the user can test their GUI model as well as the code as part of the compilation process. In such case, the compiler can test the model design consistency in terms of the GUI components that the user selects and any possible interaction problems that may occur between those components. 3-5 SPEC EXPLORER AND MODEL AUTOMATON: (DETERMINISTIC AND NON DETERMINISTIC) This is another model based testing tool from Microsoft. Spec Explorer is considered a successor for NModel especially as it was developed by the same team who developed NModel earlier. Spec Explorer evolved out of the formal modeling language: ASML. It uses Spec#, which extends the C# programming language. With Spec Explorer, a test oracle can be created which is eventually linked to a specific test code implementation. “Spec Explorer is a tool for testing reactive, object-oriented software systems. The inputs and outputs of such systems can be abstractly viewed as parameterized action labels, that is, as invocations of methods with dynamically created object instances and other complex data structures as parameters and return values”[24]. In Spec Explorer, if you need to run a specific set of actions in a particular order, you can create scenarios. Spec Explorer can automate the generation and execution of test cases from an annotated model which assist in the integration or traceability between specification and implementation levels. We developed a tool to automatically define all Spec Explorer project namespaces from the GUI XML file. Each container from the XML file is represented by a namespace in the Spec explorer project. Eventually each GUI component is represented by a class. All the GUI component attributes are automatically converted to state variables. All possible event on the GUI components are transformed programmatically into Actions in the Spec Explorer project. Table 2 below shows the GUI to Spec mapping key. Table 2 GUI to Spec Explorer mapping. Table 2 GUI to Spec Explorer mapping. GUI XML GUI container GUI component GUI component attribute GUI component possible actions Spec Explorer Namespace Class Class state variable Class Actions 53 Int.J. of Software Engineering, IJSE Vol.4 No.2 July 2011 Most action preconditions and post conditions can be also formalized. For example, all GUI components actions require a precondition that their component and its container should both be visible ( and possible enabled if they will be triggered). Post conditions can be also defined formally based on the actual code in the action. Actions can change one or more properties. Those properties can be for their own component, container, or for any other component or container. Actions that do not cause a GUI model change (such as an action that simply calculate the average of some numbers) can be ignored. Actions can also change one or more attributes of those described earlier. Actions can be automatically programmed to be deterministic where each action will change only one property. Windows can be modal or modal less where a modal Window gets the focus and disable any action on all other Windows until it is closed. In order to simply the mapping process between the GUI model and Spec Explorer, Specific actions are defined based on the type of the component. Each GUI component is defined by a major action that is usually called upon that component. For example a TextBox default action is a writeText action (i.e. write/delete/modify which is the main goal for having a TextBox). Similarly, a Button main action is a Click or DoubleClick. Same thing can be applied to selection the state variables based on the GUI components. This simplifies and accelerates the transforming process as much less information will be then required from the GUI model. After the automatic conversion of the GUI model to Spec explorer, test cases can be automatically generated using Spec Explorer. Spec Explorer automatically generates test cases from a Spec# (an extension of C# programming language) as a Finite State Machine (FSM), or from ASML formal specification or scenarios to fulfill a coverage criterion based on the FSM. One problem with generating a FSM from a GUI model is the state explosion issue where a simple GUI can generate a large number of possible states. Spec Explorer can help through focusing or narrowing the FSM in each time based on defining certain preconditions or criteria that will focus the scope to a limited number of possible states. 3-6 RANDOM TESTING Random testing is a random test case generation and execution process usually accomplished through automated tools. They perform mouse clicks on the screen or keystrokes on the keyboard randomly. They are used usually to test issues such as robustness or resources’ usage, regression testing, etc. 3-7 NODE, EDGE AND PATH COVERAGE In this part, methods are developed to generate test cases automatically and evaluate coverage of the generated test cases. The work of this part is an 54 Evaluate and Improve GUI Testing Coverage Automatically Alsmadi et al extension for earlier work of the author [1, 2, and 3]. The results in this section depend on details of the GUI model (that is described in the referenced papers]. This model is generated dynamically from the application. 1. Node and Edge coverage. Several methods or algorithms are developed to automatically generate test cases. Results in Table 3 show how test cases can be generated from the model based on a particular coverage. Details on the nature and the algorithms used in test case generation and coverage evaluation can be found in previous references of the author that are listed below in the references’ section. Table33GUI GUInodes nodesororedges edgescoverage coverage Table [4][4] AUT AUT 11 11 11 22 No. of nodes nodes 43 43 43 187 187 Noofoftest testcases cases No N/E coverage N/E coverage %% 25 25 6060 33 33 7777 50 50 100 100 25 25 2323 22 187 187 50 50 38.5 38.5 22 187 187 100 100 6666 3 3 35 35 25 25 23 23 3 35 50 74 3 35 50 74 3 35 150 100 3 35 150 100 4 90 25 36 4 90 25 36 4 90 50 90 4 90 50 90 4 90 130 100 4 90 130 100 In this section, test cases are generated automatically from the GUI model. In In this section, test cases are generated automatically from the GUI model. In such approach, coverage is usually used as a target for test cases’ automatic generation. In this model, each GUI component is represented by a node. An edge in the path represents a valid event or transition in the GUI (e.g. File_Save, Edit_Copy, etc). Four AUTs are used to evaluate the developed GUI model. The number of nodes shows the user interface size of the AUT. A comparison between the number of test cases and the coverage shows the effectiveness of the test cases in their ability to test and expose the user interface structure. The value of this approach is that all framework components and activities: 55 Int.J. of Software Engineering, IJSE Vol.4 No.2 July 2011 GUI model generation, test case generation, execution and coverage evaluation are developed automatically. This facilitates the ability to repeat the process continuously with the goal of improving the overall coverage (i.e. nodes, path, etc). 3-8 DEALING WITH CONSECUTIVE TRANSITIONS, EQUIVALENT AND INTERMEDIATE STATES A program partial execution for a given specific interaction with its environment can be represented by a sequence of states, observed at discrete intervals of time, starting from an initial or startup state, then moving from one state to the next state by executing one or more atomic program steps or transitions and either ending in a final regular or erroneous state or non terminating, in which case the trace is infinite [23]. As such, equal program partial execution are those executions that start from the same initial point and end in the same final point (even if they have different transitions). Theoretically, the program can be in a large number of possible states. However, many of those states can be eliminated for several reasons. From a tester or test automation tool, many of those test cases may not be feasible, reachable or observable. Figure 6 shows a possible view of GUI states. This view divided the states on interest to 3 parts: Initial states which are usually represented by the main entry components, the final or leaf states. Both those types of states are observable are reachable. The third type is those specific intermediate states that are reachable. Figure 6 A possible GUI states’ classification [23]. As such, the following two partial program executions are equivalent: 1. MainProgram->File->Format-Copy-Paste 2. MainProgram -> Format->Copy-Paste Such viewpoint of a system focuses only on program final states and ignores 56 Evaluate and Improve GUI Testing Coverage Automatically Alsmadi et al intermediate transitions or states. In such scope, it maybe is interesting to view test case components as elements in a set. 1. Let S1 ={ MainProgram->File->Format-Copy-Paste}, and 2. S2={ MainProgram -> Format->Copy-Paste } be two ordered sets. Ordered sets are sets represented as lists with the elements ordered in a standard order. If the two ordered sets start and end with the same elements, then they are considered, from equivalent states perspective, to be equal as we only focus on the first and the last elements. 3-9 COMPARE RESULTS BETWEEN THE MODEL CHECKER AND THE TESTING In order to see the benefits of using a model checker in the application overall quality, some properties that can be tested between the model checkers and testing are evaluated in both stages for comparison and correlations. Examples of those properties are the visibility or the reachability of GUI controls. A model checker can test if there are deadlocks or progress problems in an application where there is a node that is not reachable (i.e. progress problem) or a node that once it is reached, the application is locked and cant exit from that node or state (i.e. deadlock). In testing, techniques are developed to evaluate such scenarios. However, it is usually easier to discover such problem earlier through the model. This means that the use of model checking can help indirectly improving node, path, or state coverage. However, it is usually difficult and time consuming to define a system formally. As an alternative, Formal modeling is usually accomplished for certain parts or requirements of the system rather than doing full theorem proofing. 4- CONCLUSION AND FUTURE WORK Model based testing is used to evaluate, automate and improve the overall testing coverage. In this paper, model based methods are used in GUI testing and evaluation. Different models are generated and test cases are created automatically based on those models. The major value of this approach is in automating testing activities while at the same time improving test coverage in all testing stages: test case generation, execution and verification. The proposed formal GUI model is also created automatically. The first GUI model generated is the GUI graph. This is a static representation of the GUI components, their attributes and relations. Later on, different algorithms are used to automatically generate test cases from this model, evaluate and optimize their coverage. The study used several applications, for evaluation, where those applications have considerable amount of GUI components. 57 Int.J. of Software Engineering, IJSE Vol.4 No.2 July 2011 REFERENCES [1] I. Alsmadi, and K. Magel. “GUI Path Oriented Test Generation Algorithms”. In Proceeding of IASTED (569) Human-Computer Interaction. 2007. [2] I. Alsmadi, and K. Magel. “An Object Oriented Framework for User Interface Test Automation”. MICS07. 2007. [3] I. Alsmadi, “Using genetic algorithms for test case generation and selection optimization”. CCECE2010, Calgary, Canada. [4] I. Alsmadi, S. Samarah, A. Saifan and M. AL Zamil, “Automatic Model Based Methods to Improve Test Effectiveness”, Universal Journal of Computer Science and Engineering Technology 1 (1), 41-49, Oct. 2010. [5] J. M. Wing, "Formal Methods, in Encyclopedia of Software Engineering”, J. J. Marciniak (Eds.), 1994, pp. 504-517. [6] A. Beer, S. Mohacsi , and C. Stary, “IDATG: An Open Tool for Automated Testing of Interactive Software”, Proceedings of the 22nd International Computer Software and Applications Conference, COMPSAQ 1998. Pages: 470 – 475. [7] A. Belinfante. “JTorX: a tool for on-line model-driven test derivation and execution”, TACAS 2010. [8] A. Holmes and M. Kellogg, “Automating Functional Tests Using Selenium”, Proceedings of AGILE 2006 Conference (AGILE'06). 2006. [9] Ch. McMahon, “History of a Large Test Automation Project Using Selenium”, 2009 Agile Conference. [10] N. Zhao, and M. Shum, “Technical Solution to Automate Smoke Test Using Rational Functional Tester and Virtualization Technology”, Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC'06). [11] O. Kwon and S. Hwang, “Mobile GUI Testing Tool Based on Image Flow”, Seventh IEEE/ACIS International Conference on Computer and Information Science, 2008. [12] A. Jääskeläinen, Antti Kervinen, and Mika Katara, “Creating a Test Model Library for GUI Testing of Smartphone Applications”, The Eighth International Conference on Quality Software, 2008. [13] G. Appasami and S. Joseph. “Performance analysis of various user interface test automation for Silverlight applications”, International Journal of 58 Evaluate and Improve GUI Testing Coverage Automatically Alsmadi et al Computer and Electrical Engineering, Vol. 1, No. 4, October, 2009. [14] Z. Xiaochun, B. Zhou, L. Juefeng, and G. Qiu, “A Test Automation Solution on GUI Functional Test”, Intenrational Conference on Industrial Informatics ICDN 2008. [15] Ch. Fu, M. Grechanik, and Q. Xie, “Inferring Types of References to GUI Objects in Test Scripts”, 2009 International Conference on Software Testing Verification and Validation. [16] C. Bertolini, G. Peres, M. d’Amorim, and A. Mota, “An Empirical Evaluation of Automated Black-Box Testing Techniques for Crashing GUIs”, 2009 International Conference on Software Testing Verification and Validation. [17] R. Mathew and R. Spraetz, “Test Automation on a SaaS Platform”, 2009 International Conference on Software Testing Verification and Validation. [18] M. Grechanik, Q. Xie, and Ch. Fu, “Maintaining and Evolving GUIDirected Test Scripts”, ICSE’09, May 16-24, 2009, Vancouver, Canada. [19] E. Triou, Z. Abbas, and S. Kothapalle, “Declarative Testing: A Paradigm for Testing Software Applications”, 2009 Sixth International Conference on Information Technology: New Generations, 2009. [20] S. Qian, and F. Jiang. “An Event Interaction Structure for GUI Test Case Generation”, 2nd IEEE International Conference on Computer Science and Information Technology, 2009. [21] A. Rauf, S. Anwar, M. Jaffer, and A. Shahid, “Automated GUI Test Coverage Analysis using GA”, 2010 Seventh International Conference on Information Technology. [22] F. Belli, and Ch. Budnik, “Towards Self-Testing of Component-Based Software”, Proceedings of the 29th Annual International Computer Software and Applications Conference (COMPSAC’05). [23] P. Cousot, “Abstract Interpretation Based Formal Methods and Future Challenges”, LNCS 2000. [24] M. Veanes, C. Campbell, W. Grieskamp, W. Schulte, N. Tillmann, and L. Nachmanson, “Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer”, in Formal Methods and Testing, vol. 4949, pp. 39-76, Springer Verlag, 2008. 59 Int.J. of Software Engineering, IJSE Vol.4 No.2 July 2011 60