How to automate JMeter tests with Maven and Jenkins (Hudson)

So I’ve just had a great success getting my JMeter tests to run in Maven.  The next step: I want my continuous build server to run the tests automatically.  I’m using Jenkins as my continuous build server.  I used to use Hudson.  Then they forked, and I picked one.

1) Get your Jenkins running

I have a Jenkins server that is dedicated and was installed with the Ubuntu distro.  But for the purposes of this howto I’m going to run an instance locally.  This is pretty easy if you have a java environment already going.  Just download the latest jenkins war, then run it on the command line with

java -jar jenkins.war

You’ll have a fresh Jenkins instance running at http://localhost:8080.

Next you need to add the Performance Plugin.

  1. Click the Manage Jenkins link.
  2. Click the Manage Plugins link.
  3. Select the Available tab.
  4. Find the Performance Plugin, select it, then install it.

I also added the git plugin so that I can have Jenkins pull my demo project directly from GitHub.  You have to have a command line git client installed for the plugin to work, and you might have to configure the location in Manage Jenkins -> Configure the System

2) Create the Job in Jenkins

  1. Go to the Jenkins main page.
  2. Click New Job.
  3. Give the job a name.
  4. Choose free-style software project.
  5. Click OK
  6. Under source code management I chose Git.
  7. I set the repository URL to git@github.com/jribble/jmeter-demo.git
  8. I set the branch to build to Stage_1_JMeter_and_Maven 
  9. Add a Maven build step with “verify” as the goal.
  10. Down in Post-build Actions check the Publish Performance test result report checkbox.
  11. Click the Add a new report box and choose JMeter.
  12. For Report files specify **/*.jtl
  13. Also under Post-build Actions check the Archive the artifacts checkbox.
  14. Under Files to archive specify **/*jtl-report.html.  This will keep a copy of the html formatted reports for each build so you can look at them in Jenkins.
  15. Click Save
  16. Click Build Now.

After a short wait you should get a successful build.  If you click into a specific build you will see a Performance Report link as well as a link under Build Artifacts that shows you the JMeter generated html report.

Jenkins Performance Report

Once you have more than one build Jenkins will show a Performance Trend link on the project, which will show you graphs correlating the results of multiple builds.

3) Fix the project

You might notice in the Performance Trend picture above that the result file has a time stamp appended to the end.  This is the default behavior of the JMeter Maven plugin, and it’s a problem for Jenkins.  Jenkins uses the test result file name to correlate results and if the name of the file changes every day Jenkins won’t know that yesterday’s results should be reported with today’s results.

We can suppress this behavior with a simple pom file change. Simply add this configuration xml to the jmeter-maven-plugin definition:

<configuration>
	<testResultsTimestamp>false</testResultsTimestamp>
</configuration>

So the resulting pom file looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.example</groupId>
	<artifactId>jmeter-demo</artifactId>
	<packaging>jar</packaging>
	<version>1.0-SNAPSHOT</version>
	<name>jmeter-demo</name>
	<url>http://maven.apache.org</url>

	<build>
		<plugins>
			<plugin>
				<groupId>com.lazerycode.jmeter</groupId>
				<artifactId>jmeter-maven-plugin</artifactId>
				<version>1.4.1</version>
				<configuration>
					<testResultsTimestamp>false</testResultsTimestamp>
				</configuration>
				<executions>
					<execution>
						<id>jmeter-tests</id>
						<phase>verify</phase>
						<goals>
							<goal>jmeter</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
</project>

Now if you build the project again with the settings given here you will get two sets of results, because the old results aren’t cleaned up.  Might be worth changing the project configuration to start clean every time.

That’s it.  You can grab the sample project from my jmeter-demo project on github.  The Stage_2_Automate_In_Jenkins branch has the final project settings for Jenkins.

Advertisements

28 thoughts on “How to automate JMeter tests with Maven and Jenkins (Hudson)

  1. Very good demo but the repository url mentioned is different than the one in the attached picture. I could not get it to work because it keeps saying
    Fetching changes from 1 remote Git repository
    Fetching upstream changes from git@github.com:jribble/jmeter-demo.git
    ERROR: Problem fetching from origin / origin – could be unavailable. Continuing anyway
    hudson.plugins.git.GitException: Command “git.exe fetch -t git@github.com:jribble/jmeter-demo.git +refs/heads/*:refs/remotes/origin/*” returned status code 128:
    stdout:

  2. Thanks for note on testResultsTimestamp flag – I’ve commented it here:
    [https://issues.jenkins-ci.org/browse/JENKINS-9896?focusedCommentId=171503&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-171503]

  3. Hi jribble,
    In addition to my .jmx file I also have a beanshell script, some .csv files and a bunch of XML files that make up the for jmeter testplan. Where do I include these required files when executing my .jmx from Hudson?
    Thanks!

    • The key is to be able to execute the tests via maven with only two steps: 1) Check out the test project from source control and 2) execute maven. Once you have this working everything is organized for Hudson already.

      In general the maven way is to put those kinds of supporting files under src/main/resources or src/test/resources.

  4. I am getting below error. Can you please help me

    Fetching changes from the remote Git repository
    Fetching upstream changes from git@github.com:jribble/jmeter-demo.git
    ERROR: Problem fetching from origin / origin – could be unavailable. Continuing anyway
    ERROR: (Underlying report) : Error performing command: git.exe fetch -t git@github.com:jribble/jmeter-demo.git +refs/heads/*:refs/remotes/origin/*

  5. Pingback: How to automate JMeter tests with Maven and Jenkins (Hudson) | i4igeeks

  6. Hi jribble!

    Can you provide information about how many threads Jmeter engine handle in this case? You know both of these tools(Jenkins,Jmeter) runs via Java so they need more RAM than usual? I found Blazemeter Jenkins plugin which runs all test on demand. I think for distributed testing would be useful.

    • I haven’t done any testing to see what kind of load jmeter produces or how many threads it can handle. If you spend some time to find that out I would love to know what you find.

  7. Pingback: How to write JMeter tests for your GWT-RPC servlet | Ribble's Code

  8. Thank you for the post. I have one quick question. In the JMeter report, how can I setup some SLAs, for example, I have five transactions covered in the JMeter scripts. I want to setup the pass/fail critera like these:

    if response times abover 50 ms for transaction one , then it fails.
    if response times abover 100 ms for transaction two , then it fails.
    ..
    ..
    Please let me know.

  9. I have a question….i have a project develop in Magnolia…i need to test this project using JMeter with Jenkins…but i don’t know how to match my project with my script test, because when i build the job…there are a lot of errors saying “[workspace] $ mvn verify
    FATAL: command execution failed
    java.io.IOException: Cannot run program “mvn” (in directory “/var/lib/jenkins/jobs/jmeter-maven-example/workspace”): error=2, No such file or directory”.

    I have a pom file for my test and a pom file for the project. If you can help me, i will thank you too much.

    • It sounds like either maven isn’t installed or it isn’t installed in a way that the user running jenkins can run it. When I run into problems like this I ‘sudo su – jenkins’, then go to the working directory of the project and try to run the build on the command line. If you can’t do it logged in as jenkins, jenkins can’t do it either.

  10. Pingback: OCTO talks ! » Medindo o desempenho de aplicações Web – Parte 3

  11. Hi, I have a small problem.I have two tests in folder “jmeter”
    Each of test is running – it is ok.But when is running JMeter Reports (at the end)i see in inconsole:

    JMeter Report(s)…
    …..
    XML document structures must start and end within the same entity.
    ….

    It is always for the second test (first is ok)

    So the second raport in .jtl and html is not working (is empty)

  12. Hi,
    I have 2 jmx files, and it created corresponding 2 HTML files.
    Is there a way to know by looking at html doc only like what is the service fired in any header of doc or somewhere else?
    For E.X.: if i want to show the results to a third person, how can he determine for whose service call are the results shown here.

    Regards.

  13. I am getting the below error when i am trying to archive the artifacts as mentioned in your post.

    ‘**/*jtl-report.html’ doesn’t match anything: ‘**’ exists but not ‘**/*jtl-report.html’

  14. Hi Jribble!

    I am getting below error. Can you please help me

    I am integrating Jmeter with Jenkins, so while creating a new Job in Jenkins,
    in ‘Source Code Management’-> After selecting ‘Git Repositories’ it is asking for
    ‘Repository URL’.
    So I put there “https://github.com/TestingDiaries/maven-jmeter.git” this URL
    But it is showing below Error:-
    Failed to connect to repository : Error performing command: git.exe ls-remote -h https://github.com/TestingDiaries/maven-jmeter.git HEAD

    Also tell me which Repository shall I fetch??

  15. It’s a very good article.
    But we are facing one issue while executing pom.xml.
    please find below details:
    We have added one .jmx file under jmeter folder, and related .csv file under resouces folder.
    But we are getting below error :
    INFO 2016-06-28 11:54:59.898 [jmeter.e] (): Listeners will be started after enabling running version
    INFO 2016-06-28 11:54:59.898 [jmeter.e] (): To revert to the earlier behaviour, define jmeterengine.startlistenerslater=false

    And execution also not stopped.

    Could you please help us? this is the blocker for us.

  16. Hi Thanks in advance,

    can any one share me a sample POM file for running multiple tests under single test plan and how to run distributed testing using POM file of maven.

    Hope you guys understand my queries

    thanks you all

  17. In Source Code Managment section, when I put GIT repository url I am getting this error:

    HTTP ERROR 403
    Problem accessing /job/JmeterMavenJenkins/descriptorByName/hudson.plugins.git.UserRemoteConfig/checkUrl. Reason:
    Forbidden

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s