Proceedings of the 19th international symposium on Software testing and analysis - ISSTA '10, 2010
Fault-localization techniques that apply statistical analyses to execution data gathered from mul... more Fault-localization techniques that apply statistical analyses to execution data gathered from multiple tests are quite effective when a large test suite is available. However, if no test suite is available, what is the best approach to generate one? This paper investigates the fault-localization effectiveness of test suites generated according to several test-generation techniques based on combined concrete and symbolic (concolic) execution. We evaluate these techniques by applying the Ochiai fault-localization technique to generated test suites in order to localize 35 faults in four PHP Web applications. Our results show that the test-generation techniques under consideration produce test suites with similar high fault-localization effectiveness, when given a large time budget. However, a new, "directed" test-generation technique, which aims to maximize the similarity between the path constraints of the generated tests and those of faulty executions, reaches this level of effectiveness with much smaller test suites. On average, when compared to test generation based on standard concolic execution techniques that aims to maximize code coverage, the new directed technique preserves faultlocalization effectiveness while reducing test-suite size by 86.1% and test-suite generation time by 88.6%.
Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering - SIGSOFT/FSE '11, 2011
... modules of the W3C recommendations (Level 2, plus selected parts of Level 3), the essential p... more ... modules of the W3C recommendations (Level 2, plus selected parts of Level 3), the essential parts of window8 and related nonstandard objects, and the canvas and related objects from WHATWG's HTML5 (as of Jan-uary 2011). The latter allows us to test the analysis on web ...
Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - ICSE '10, 2010
Page 1. Practical Fault Localization for Dynamic Web Applications Shay Artzi Julian Dolby Frank T... more Page 1. Practical Fault Localization for Dynamic Web Applications Shay Artzi Julian Dolby Frank Tip Marco Pistoia IBM TJ Watson Research Center PO Box 704, Yorktown Heights, NY 10598, USA {artzi,dolby,ftip,pistoia}@us.ibm.com ...
Proceedings of the 2008 international symposium on Software testing and analysis - ISSTA '08, 2008
Web script crashes and malformed dynamically-generated Web pages are common errors, and they seri... more Web script crashes and malformed dynamically-generated Web pages are common errors, and they seriously impact usability of Web applications. Current tools for Web-page validation cannot handle the dynamically-generated pages that are ubiquitous on today's Internet. In this work, we apply a dynamic test generation technique, based on combined concrete and symbolic execution, to the domain of dynamic Web applications. The technique generates tests automatically, uses the tests to detect failures, and minimizes the conditions on the inputs exposing each failure, so that the resulting bug reports are small and useful in finding and fixing the underlying faults. Our tool Apollo implements the technique for PHP. Apollo generates test inputs for the Web application, monitors the application for crashes, and validates that the output conforms to the HTML specification. This paper presents Apollo's algorithms and implementation, and an experimental evaluation that revealed 214 faults in 4 PHP Web applications.
Proceeding of the 33rd international conference on Software engineering - ICSE '11, 2011
Current practice in testing JavaScript web applications requires manual construction of test case... more Current practice in testing JavaScript web applications requires manual construction of test cases, which is difficult and tedious. We present a framework for feedback-directed automated test generation for JavaScript in which execution is monitored to collect information that directs the test generator towards inputs that yield increased coverage. We implemented several instantiations of the framework, corresponding to variations on feedback-directed random testing, in a tool called Artemis. Experiments on a suite of JavaScript applications demonstrate that a simple instantiation of the framework that uses event handler registrations as feedback information produces surprisingly good coverage if enough tests are generated. By also using coverage information and read-write sets as feedback information, a slightly better level of coverage can be achieved, and sometimes with many fewer tests. The generated tests can be used for detecting HTML validity problems and other programming errors.
Web script crashes and malformed dynamically-generated web pages are common errors, and they seri... more Web script crashes and malformed dynamically-generated web pages are common errors, and they seriously impact the usability of web applications. Current tools for web-page validation cannot handle the dynamically generated pages that are ubiquitous on today's Internet. We present a dynamic test generation technique for the domain of dynamic web applications. The technique utilizes both combined concrete and symbolic execution and explicit-state model checking. The technique generates tests automatically, runs the tests capturing logical constraints on inputs, and minimizes the conditions on the inputs to failing tests, so that the resulting bug reports are small and useful in finding and fixing the underlying faults.
We leverage combined concrete and symbolic execution and several fault-localization techniques to... more We leverage combined concrete and symbolic execution and several fault-localization techniques to create a uniquely powerful tool for localizing faults in PHP applications. The tool automatically generates tests that expose failures, and then automatically localizes the faults responsible for those failures, thus overcoming the limitation of previous fault-localization techniques that a test suite be available upfront. The fault-localization techniques we employ combine variations on the Tarantula algorithm with a technique based on maintaining a mapping between statements and the fragments of output they produce. We implemented these techniques in a tool called Apollo, and evaluated them by localizing 75 randomly selected faults that were exposed by automatically generated tests in four PHP applications. Our findings indicate that, using our best technique, 87.7% of the faults under consideration are localized to within 1% of all executed statements, which constitutes an almost five-fold improvement over the Tarantula algorithm.
... IBM Research Report Directed Test Generation for Improved Fault Localization Shay Artzi, Juli... more ... IBM Research Report Directed Test Generation for Improved Fault Localization Shay Artzi, Julian Dolby, Frank Tip, Marco Pistoia IBM Research Division ... Page 2. Directed Test Generation for Improved Fault Localization Shay Artzi Julian Dolby Frank Tip Marco Pistoia ...
... been led to the buggy code in Figure 3. The NullPointerException is thrown on line 19, but ..... more ... been led to the buggy code in Figure 3. The NullPointerException is thrown on line 19, but ... created for the method that was on the top of the stack when the exception was thrown ... For example, the program in Figure 4 [30] will crash with a null pointer exception in the toUpperCase ...
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, Nov 1, 2010
This index covers all technical items-papers, correspondence, reviews, etc.-that appeared in this... more This index covers all technical items-papers, correspondence, reviews, etc.-that appeared in this periodical during the year, and items from previous years that were commented upon or corrected in this year. Departments and other items may also be covered if they have been judged to have archival value. The Author Index contains the primary entry for each item, listed under the first author's name. The primary entry includes the coauthors' names, the title of the paper or other item, and its location, specified by the ...
Proceedings of the 19th international symposium on Software testing and analysis - ISSTA '10, 2010
Fault-localization techniques that apply statistical analyses to execution data gathered from mul... more Fault-localization techniques that apply statistical analyses to execution data gathered from multiple tests are quite effective when a large test suite is available. However, if no test suite is available, what is the best approach to generate one? This paper investigates the fault-localization effectiveness of test suites generated according to several test-generation techniques based on combined concrete and symbolic (concolic) execution. We evaluate these techniques by applying the Ochiai fault-localization technique to generated test suites in order to localize 35 faults in four PHP Web applications. Our results show that the test-generation techniques under consideration produce test suites with similar high fault-localization effectiveness, when given a large time budget. However, a new, "directed" test-generation technique, which aims to maximize the similarity between the path constraints of the generated tests and those of faulty executions, reaches this level of effectiveness with much smaller test suites. On average, when compared to test generation based on standard concolic execution techniques that aims to maximize code coverage, the new directed technique preserves faultlocalization effectiveness while reducing test-suite size by 86.1% and test-suite generation time by 88.6%.
Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering - SIGSOFT/FSE '11, 2011
... modules of the W3C recommendations (Level 2, plus selected parts of Level 3), the essential p... more ... modules of the W3C recommendations (Level 2, plus selected parts of Level 3), the essential parts of window8 and related nonstandard objects, and the canvas and related objects from WHATWG's HTML5 (as of Jan-uary 2011). The latter allows us to test the analysis on web ...
Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - ICSE '10, 2010
Page 1. Practical Fault Localization for Dynamic Web Applications Shay Artzi Julian Dolby Frank T... more Page 1. Practical Fault Localization for Dynamic Web Applications Shay Artzi Julian Dolby Frank Tip Marco Pistoia IBM TJ Watson Research Center PO Box 704, Yorktown Heights, NY 10598, USA {artzi,dolby,ftip,pistoia}@us.ibm.com ...
Proceedings of the 2008 international symposium on Software testing and analysis - ISSTA '08, 2008
Web script crashes and malformed dynamically-generated Web pages are common errors, and they seri... more Web script crashes and malformed dynamically-generated Web pages are common errors, and they seriously impact usability of Web applications. Current tools for Web-page validation cannot handle the dynamically-generated pages that are ubiquitous on today's Internet. In this work, we apply a dynamic test generation technique, based on combined concrete and symbolic execution, to the domain of dynamic Web applications. The technique generates tests automatically, uses the tests to detect failures, and minimizes the conditions on the inputs exposing each failure, so that the resulting bug reports are small and useful in finding and fixing the underlying faults. Our tool Apollo implements the technique for PHP. Apollo generates test inputs for the Web application, monitors the application for crashes, and validates that the output conforms to the HTML specification. This paper presents Apollo's algorithms and implementation, and an experimental evaluation that revealed 214 faults in 4 PHP Web applications.
Proceeding of the 33rd international conference on Software engineering - ICSE '11, 2011
Current practice in testing JavaScript web applications requires manual construction of test case... more Current practice in testing JavaScript web applications requires manual construction of test cases, which is difficult and tedious. We present a framework for feedback-directed automated test generation for JavaScript in which execution is monitored to collect information that directs the test generator towards inputs that yield increased coverage. We implemented several instantiations of the framework, corresponding to variations on feedback-directed random testing, in a tool called Artemis. Experiments on a suite of JavaScript applications demonstrate that a simple instantiation of the framework that uses event handler registrations as feedback information produces surprisingly good coverage if enough tests are generated. By also using coverage information and read-write sets as feedback information, a slightly better level of coverage can be achieved, and sometimes with many fewer tests. The generated tests can be used for detecting HTML validity problems and other programming errors.
Web script crashes and malformed dynamically-generated web pages are common errors, and they seri... more Web script crashes and malformed dynamically-generated web pages are common errors, and they seriously impact the usability of web applications. Current tools for web-page validation cannot handle the dynamically generated pages that are ubiquitous on today's Internet. We present a dynamic test generation technique for the domain of dynamic web applications. The technique utilizes both combined concrete and symbolic execution and explicit-state model checking. The technique generates tests automatically, runs the tests capturing logical constraints on inputs, and minimizes the conditions on the inputs to failing tests, so that the resulting bug reports are small and useful in finding and fixing the underlying faults.
We leverage combined concrete and symbolic execution and several fault-localization techniques to... more We leverage combined concrete and symbolic execution and several fault-localization techniques to create a uniquely powerful tool for localizing faults in PHP applications. The tool automatically generates tests that expose failures, and then automatically localizes the faults responsible for those failures, thus overcoming the limitation of previous fault-localization techniques that a test suite be available upfront. The fault-localization techniques we employ combine variations on the Tarantula algorithm with a technique based on maintaining a mapping between statements and the fragments of output they produce. We implemented these techniques in a tool called Apollo, and evaluated them by localizing 75 randomly selected faults that were exposed by automatically generated tests in four PHP applications. Our findings indicate that, using our best technique, 87.7% of the faults under consideration are localized to within 1% of all executed statements, which constitutes an almost five-fold improvement over the Tarantula algorithm.
... IBM Research Report Directed Test Generation for Improved Fault Localization Shay Artzi, Juli... more ... IBM Research Report Directed Test Generation for Improved Fault Localization Shay Artzi, Julian Dolby, Frank Tip, Marco Pistoia IBM Research Division ... Page 2. Directed Test Generation for Improved Fault Localization Shay Artzi Julian Dolby Frank Tip Marco Pistoia ...
... been led to the buggy code in Figure 3. The NullPointerException is thrown on line 19, but ..... more ... been led to the buggy code in Figure 3. The NullPointerException is thrown on line 19, but ... created for the method that was on the top of the stack when the exception was thrown ... For example, the program in Figure 4 [30] will crash with a null pointer exception in the toUpperCase ...
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, Nov 1, 2010
This index covers all technical items-papers, correspondence, reviews, etc.-that appeared in this... more This index covers all technical items-papers, correspondence, reviews, etc.-that appeared in this periodical during the year, and items from previous years that were commented upon or corrected in this year. Departments and other items may also be covered if they have been judged to have archival value. The Author Index contains the primary entry for each item, listed under the first author's name. The primary entry includes the coauthors' names, the title of the paper or other item, and its location, specified by the ...
Uploads
Papers by S. Artzi