Automation for the people: Hands-off load testing

Use Apache JMeter with Apache Ant to run load tests frequently

Paul Duvall 08 April 2008

Load testing is often relegated to late-cycle activities, but it doesn't need to be that way. In
this installment of Automation for the people, automation expert Paul Duvall describes how
you can discover and fix problems throughout the development cycle by creating a scheduled
integration build that runs JMeter tests.
View more content in this series

How many concurrent users can access your software system? How much data can be loaded
without performance degradation? What are your system's throughput requirements? How
often are these requirements tested? What if you could specify and validate that these load and
performance requirements are being met at least once a day? By running load tests as part of a
scheduled and automated build, you can more quickly determine how your system performs under
certain load conditions and quickly adapt to changes.

About this series

As developers, we work to automate processes for end-users; yet, many of us overlook
opportunities to automate our own development processes. To that end, Automation for the
people is a series of articles dedicated to exploring the practical uses of automating software
development processes and teaching you when and how to apply automation successfully.

A project I once worked on established a good set of automated tests that could load test the
application while it ran multiple transactions. The problem was that the tests required some manual
tweaking, so the development team couldn't run them without human intervention. This limited
testing to times when the tester was available (usually waking hours only). In practice, testing
occurred only every couple of days not frequently enough for timely problem detection.

In this article, I explore techniques for creating automated tests using JMeter, running the tests
as part of an automated build, and scheduling the tests to run automatically every day typically
when machine usage is lower. Running the tests as part of a scheduled build lets you:

Execute load tests at any time

Detect and resolve load and performance problems early in the development process
Monitor the latest load- and performance-test reports from your build server

Copyright IBM Corporation 2008 Trademarks

Automation for the people: Hands-off load testing Page 1 of 7
developerWorks ibm.com/developerWorks/

Reduce bottlenecks and errors that can occur when you depend on a single person to
configure and run the tests

Letting JMeter do the heavy lifting

Apache JMeter is an open source project you can use to simulate a heavy load on a server (see
Resources for more information on JMeter). JMeter's impressive documentation set describes how
to use its many features and provides a plethora of examples.

Running JMeter
After downloading and extracting the JMeter ZIP file (see Resources for a link to download
JMeter), open a command prompt to the location where you extracted JMeter and type cd bin
to change to the bin directory. From the bin directory, type jmeter to open the JMeter Swing
application, shown in Figure 1:

Figure 1. The JMeter GUI

Creating a test plan

Writing tests, by example
JMeter comes with many example test plans and scripts. Instead of creating a test plan from
scratch, use the examples in the docs directory and incrementally configure the test plans as
your project evolves. Much of the complexity is in learning to write load tests that effectively
exercise the load and performance requirements.

You can use the JMeter GUI to create test plans. The different types of test plans in JMeter
include:

Web test plan

Automation for the people: Hands-off load testing Page 2 of 7

ibm.com/developerWorks/ developerWorks

Database test plan

FTP test plan
LDAP test plan
Extended LDAP test plan
Web service test plan
JMS point-to-point test plan
JMS topic test plan
Monitor test plan
Listeners
Each of the test plans is saved in XML format in a file with a .jmx suffix. This nonbinary format
makes plans easier to edit later. Although you can create test plans by following JMeter's XML
schema, it's much easier to use the GUI. Later, you'll see an example of parameterizing JMeter
configuration values to customize ways the tests are run.

Labor-saving load testing

Using the GUI to run tests, though, requires that a person be present to execute them. This
increases the chance of process bottlenecks and knowledge silos. By running the tests through an
automated build such as an Ant build, you can configure the JMeter tests to run without opening
the JMeter application. What's more, the tests are run the same way every time and without
requiring overtime pay.

Driving JMeter tests with Ant

Once I've learned how to use a GUI software tool, I like to see if it can run certain utilities from the
command line, so that I don't need to perform the same actions repeatedly. For example, every
time I open the JMeter application, I'd like to select File > Open, open the file, and run one or more
tests. I can script this set of actions and run them the same way every time. Fortunately, an Ant
task has been written to do this very thing for JMeter: it executes load tests while providing a way
to pass in optional parameters and properties.

Example build script

The JMeter distribution's extras directory includes an example build.xml script that
demonstrates the use of the JMeter Ant task.

In Listing 1, I use Ant's taskdef task to define the JMeter task. I name it jmeter so that I can use it
elsewhere in the Ant script. To use this script you must have the ant-jmeter.jar file (see Resources
for a download link) in the Ant classpath.

Listing 1. Defining the JMeter task in Ant

<taskdef name="jmeter" classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>

The example code in Listing 2 runs a single JMeter load test, called BreweryTestPlan.jmx. To run
all of the tests in a certain directory, simply enter *.jmx in place of a specific filename. The required
attributes of the jmeter task are jmeterhome, testplan(s), and resultlog or resultlogdir. (Listing
2 doesn't show resultlogdir, because it uses resultlog.)

Automation for the people: Hands-off load testing Page 3 of 7

developerWorks ibm.com/developerWorks/

Listing 2. Running JMeter from Ant

<jmeter
jmeterhome="${jmeter.home}"
resultlog="${basedir}/target/JMeterResults.xml">
<testplans dir="${basedir}/tests/load" includes="BreweryTestPlan.jmx"/>
</jmeter>

The Ant code in Listing 2 creates an output file called JMeterResults.xml, which is used to create
HTML reports.

Rendering reports using XSLT

By providing the JMeterResults.xml file as input to the xslt Ant task in Listing 3, I can generate
an HTML report of all the JMeter tests I ran in Listing 2. The XSL stylesheet (jmeter-results-detail-
report_21.xsl) provided in the JMeter extras directory is used to transform the JMeterResults file
into HTML.

Listing 3. Creating JMeter HTML report using XSLT

<xslt in="${basedir}/target/JMeterResults.xml"
out="${basedir}/target/JMeterResults.html"
style="${jmeter.home}/extras/jmeter-results-detail-report_21.xsl"/>

JMeter also provides a less detailed XSL stylesheet file that summarizes the load tests' results.

Revealing reports in HTML

Figure 2 is an example of the HTML report that is generated using the xslt task in Listing 3. It
displays each of the load tests run, along with test status, time, and aggregate status and time of
all of the tests.

Figure 2. Generating the JMeter HTML report

Automation for the people: Hands-off load testing Page 4 of 7

ibm.com/developerWorks/ developerWorks

Later in the article, I'll show you how these reports can be displayed from a CruiseControl
Continuous Integration (CI) server (see Resources).

Passing parameters to JMeter

Depending on the types of tests you run, you may want to pass parameters and properties to
vary the way a single test or suite of tests executes. For example, Listing 4 demonstrates how to
increase the JVM memory and specify the number of threads and loops:

Listing 4. Passing optional parameters and properties to JMeter

<jmeter
jmeterhome="${jmeter.home}"
resultlog="${basedir}/target/JMeterResults.xml">
<jvmarg value="-Xincgc"/>
<jvmarg value="-Xmx128m"/>
<jvmarg value="-Dproperty=value"/>
<property name="request.threads" value="5"/>
<property name="request.loop" value="50"/>
<property name="jmeter.save.saveservice.assertion_results" value="all"/>
<property name="jmeter.save.saveservice.output_format" value="xml"/>
<testplans dir="${basedir}/tests/load" includes="BreweryTestPlan.jmx"/>
</jmeter>

Many other built-in parameters and properties are available for modifying how JMeter tests are run
(see Resources for more information).

Using parameters and properties gives you a certain level of flexibility in how to execute load tests,
but it doesn't address how to run load tests in different target environments, such as testing and
staging. To add environment-specific information to test plans, you need to put tokens in the .jmx
files that can be filtered and modified when the load tests run in an automated build script.

Just-in-time load testing

Once you get a load test running with an automated build, schedule it to run on a periodic basis,
such as nightly. You can do this using a CI or build-management server.

Scheduling CruiseControl to run load tests daily

The purpose of using a CI server is to run an automated build whenever a change is applied to
a project's version-control repository. You can also configure it to run builds at specific times.
Because load testing usually requires an elevated level of computing resources, running tests
when these resources are not at capacity such as late at night or early morning can be
beneficial.

In Listing 5, an automated build is scheduled to run at 11:00 PM (2300) with CruiseControl (see
Resources). You can modify the CruiseControl configuration file to run a delegating build with a
specific Ant target, such as one named run-load-tests.

Automation for the people: Hands-off load testing Page 5 of 7

developerWorks ibm.com/developerWorks/

Listing 5. Run a scheduled load test with CruiseControl

...
<modificationset>
<svn RepositoryLocation="${svnrepo.location}"/>
<timebuild username="admin" time="2300"/>
</modificationset>
...

By scheduling load tests to run nightly, as in Listing 5, you won't hear excuses about workload,
vacation days, or forgetting to run the tests they just run.

Displaying reports in CruiseControl

You've already seen how to display JMeter test reports using Ant. However, by itself, the JMeter
report limits the communication to one developer on a single machine. Load testing affects the
entire application, so the whole team will want to see the results. The beauty is that you can easily
configure your CI server to display these reports. Because the report is already generated with
Ant, it's just a matter of making the JMeter HTML report accessible from the CruiseControl project
dashboard. You can do this by simply adding a few lines to CruiseControl's config.xml file, as
shown in Listing 6:

Listing 6. Configuring CruiseControl to display JMeter reports

<project name="brewery">
...
<log>
<merge dir="merge dir="projects/${project.name}/reports/jmeter" />
</log>
...
</project>

Now, everyone on the team can be (literally) on the same page. Many of the other CI and build-
management servers provide similar report-integration capabilities.

Conclusion
I've demonstrated how to add automated load tests to your development toolbox. By running load
tests with the automated build and then scheduling the tests to run periodically, you can learn
about system-capacity issues long before they become a problem. This approach makes it easier
to assess the impact of architecture and data changes. When coupled with other techniques
described in this article series, it enables development teams to deliver higher-quality software
frequently.

Automation for the people: Hands-off load testing Page 6 of 7

ibm.com/developerWorks/ developerWorks

Downloadable resources
Description Name Size
Sample Ant scripts for this article j-ap04088-jmeter-example.zip 6KB

Copyright IBM Corporation 2008

(www.ibm.com/legal/copytrade.shtml)
Trademarks
(www.ibm.com/developerworks/ibm/trademarks/)

Automation for the people: Hands-off load testing Page 7 of 7