How to automate JMeter tests with Maven and Jenkins (Hudson)

So I’ve just had a great success getting my JMeter tests to run in Maven.  The next step: I want my continuous build server to run the tests automatically.  I’m using Jenkins as my continuous build server.  I used to use Hudson.  Then they forked, and I picked one.

1) Get your Jenkins running

I have a Jenkins server that is dedicated and was installed with the Ubuntu distro.  But for the purposes of this howto I’m going to run an instance locally.  This is pretty easy if you have a java environment already going.  Just download the latest jenkins war, then run it on the command line with

java -jar jenkins.war

You’ll have a fresh Jenkins instance running at http://localhost:8080.

Next you need to add the Performance Plugin.

  1. Click the Manage Jenkins link.
  2. Click the Manage Plugins link.
  3. Select the Available tab.
  4. Find the Performance Plugin, select it, then install it.

I also added the git plugin so that I can have Jenkins pull my demo project directly from GitHub.  You have to have a command line git client installed for the plugin to work, and you might have to configure the location in Manage Jenkins -> Configure the System

2) Create the Job in Jenkins

  1. Go to the Jenkins main page.
  2. Click New Job.
  3. Give the job a name.
  4. Choose free-style software project.
  5. Click OK
  6. Under source code management I chose Git.
  7. I set the repository URL to git@github.com/jribble/jmeter-demo.git
  8. I set the branch to build to Stage_1_JMeter_and_Maven 
  9. Add a Maven build step with “verify” as the goal.
  10. Down in Post-build Actions check the Publish Performance test result report checkbox.
  11. Click the Add a new report box and choose JMeter.
  12. For Report files specify **/*.jtl
  13. Also under Post-build Actions check the Archive the artifacts checkbox.
  14. Under Files to archive specify **/*jtl-report.html.  This will keep a copy of the html formatted reports for each build so you can look at them in Jenkins.
  15. Click Save
  16. Click Build Now.

After a short wait you should get a successful build.  If you click into a specific build you will see a Performance Report link as well as a link under Build Artifacts that shows you the JMeter generated html report.

Jenkins Performance Report

Once you have more than one build Jenkins will show a Performance Trend link on the project, which will show you graphs correlating the results of multiple builds.

3) Fix the project

You might notice in the Performance Trend picture above that the result file has a time stamp appended to the end.  This is the default behavior of the JMeter Maven plugin, and it’s a problem for Jenkins.  Jenkins uses the test result file name to correlate results and if the name of the file changes every day Jenkins won’t know that yesterday’s results should be reported with today’s results.

We can suppress this behavior with a simple pom file change. Simply add this configuration xml to the jmeter-maven-plugin definition:

<configuration>
	<testResultsTimestamp>false</testResultsTimestamp>
</configuration>

So the resulting pom file looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.example</groupId>
	<artifactId>jmeter-demo</artifactId>
	<packaging>jar</packaging>
	<version>1.0-SNAPSHOT</version>
	<name>jmeter-demo</name>
	<url>http://maven.apache.org</url>

	<build>
		<plugins>
			<plugin>
				<groupId>com.lazerycode.jmeter</groupId>
				<artifactId>jmeter-maven-plugin</artifactId>
				<version>1.4.1</version>
				<configuration>
					<testResultsTimestamp>false</testResultsTimestamp>
				</configuration>
				<executions>
					<execution>
						<id>jmeter-tests</id>
						<phase>verify</phase>
						<goals>
							<goal>jmeter</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
</project>

Now if you build the project again with the settings given here you will get two sets of results, because the old results aren’t cleaned up.  Might be worth changing the project configuration to start clean every time.

That’s it.  You can grab the sample project from my jmeter-demo project on github.  The Stage_2_Automate_In_Jenkins branch has the final project settings for Jenkins.

How to run JMeter tests with Maven

I just finished an effort setting up automated load tests against a GWT-RPC server interface.  It took me some time to pull together the pieces from different sources and figure out what was possible and how I wanted to organize the whole thing.  But once I was there the result is pretty simple.  So I’m going to write it down here and hopefully save someone else the effort of figuring it out like I did. So this post is the first part of the puzzle – how to create a project that will run JMeter tests with Maven.  I got a lot of help from some other people’s posts on this subject.  Jacobo Matute’s post Performance Tests with Jmeter, Maven and Hudson was a big help, but fortunately slightly out of date. My goal with this project was to have something that works the maven way.  All I should have to do is:

  • Checkout the project.
  • Run a maven goal.

I don’t want to have to install JMeter first.  I don’t want to have to go get any dependencies. That’s what Maven’s for.  This used to be a problem because there were no JMeter artifacts available, but that’s fortunately not true anymore.

1) Create a simple maven project

I use the quickstart archetype:

mvn archetype:generate -DgroupId=com.example -DartifactId=jmeter-demo -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

This creates a basic java project named jmeter-demo, with and example App class and an AppTest test class.  We don’t need either of these, so we can delete them.  Make sure you keep src/main/java and src/test/java, but you can delete the com directory in each of these. Now you should be able to

cd jmeter-demo

and

mvn install

and get a build success, but nothing was built.

2) Add the JMeter Maven plugin

All of the heavy lifting will be done for us by the JMeter Maven plugin.  Edit the pom.xml file in the jmeter-demo directory so it looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.example</groupId>
	<artifactId>jmeter-demo</artifactId>
	<packaging>jar</packaging>
	<version>1.0-SNAPSHOT</version>
	<name>jmeter-demo</name>
	<url>http://maven.apache.org</url>

	<build>
		<plugins>
			<plugin>
				<groupId>com.lazerycode.jmeter</groupId>
				<artifactId>jmeter-maven-plugin</artifactId>
				<version>1.4.1</version>
				<executions>
					<execution>
						<id>jmeter-tests</id>
						<phase>verify</phase>
						<goals>
							<goal>jmeter</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
</project>

So you’re removing the junit dependency and adding a build plugin that runs jmeter tests when you do the verify phase of the lifecycle. If you run mvn install now you will get a failure because you don’t have the jmeter test directory created.

3) Create a JMeter test

The JMeter plugin will run any test that it finds under the src/test/jmeter directory.  A JMeter  test is just xml, but it’s easiest to create this xml with the JMeter GUI.  Currently you have to install JMeter in order to run the GUI, but hopefully very soon it will be possible to run the GUI directly from the JMeter plugin. Using the GUI I created a simple test with a Thread Group that runs 10 times, an HTTP Request Sampler that requests the Google Home Page at http://www.google.com, and a View Results in Table that shows the timing results.  Then I ran it in the GUI.  I saved the test in a src/test/jmeter directory that I created.

4) Run it in Maven

Now I run

mvn install

in my project directory.  Here’s a piece of the output:

[INFO] --- jmeter-maven-plugin:1.4.1:jmeter (jmeter-tests) @ jmeter-demo ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  P E R F O R M A N C E    T E S T S
[INFO] -------------------------------------------------------
[INFO]
[WARNING] Unable to parse the 'jmeter.exit.check.pause' entry in jmeter.properties!  Falling back to a default value of '2500'.
[INFO]
[INFO] Proxy server is not being used.

[info]
[debug] JMeter is called with the following command line arguments: -n -t C:\projects\jmeter-demo\jmeter-demo\src\test\jmeter\Demo Test.jmx -l C:\projects\jmeter-demo\jmeter-demo\target\jmeter\report\Demo Test-120416.jtl -d C:\projects\jmeter-demo\jmeter-demo\target\jmeter
[info] Executing test: Demo Test.jmx
INFO    2012-04-16 14:51:59.725 [jmeter.e] (): Listeners will be started after enabling running version
INFO    2012-04-16 14:51:59.726 [jmeter.e] (): To revert to the earlier behaviour, define jmeterengine.startlistenerslater=false
[info] Completed Test: Demo Test.jmx
[info]
[info] Building JMeter Report(s)...
[info]
[info] Raw results: C:\projects\jmeter-demo\jmeter-demo\target\jmeter\report\Demo Test-120416.jtl
[info] Test report: C:\projects\jmeter-demo\jmeter-demo\target\jmeter\report\Demo Test-120416.jtl-report.html
[INFO]
[INFO] Test Results:
[INFO]
[INFO] Tests Run: 1, Failures: 0, Errors: 0
[INFO]
[INFO]

We can see that the test ran successfully, the raw results are located at target/jmeter/report/Demo Test-120416.jtl and an html report is located at target/jmeter/report/Demo Test-120416.jtl-report.html. You can grab the sample project from the Stage_1_JMeter_and_Maven branch of my jmeter-demo project. Check out my next step in the automated load tests effort:

How to automate JMeter tests with Maven and Jenkins (Hudson)