Wednesday, June 14, 2006

Shootout: Load Runner vs The Grinder vs Apache JMeter

1 INTRODUCTION

I recently needed to recommend a tool to use for a scalability testing project, and I was in the fortunate situation of having some time to survey the field, and to look into the top contenders in greater depth. From an original list of over 40 candidates, I selected three finalists in the open-source and commercial categories. I then took some time to look at them in detail, to determine which tool to recommend for the ongoing scale testing effort. Since I have seen several questions about how these tools compare to each other on various mailing lists, I'm sharing my findings here in the hopes that others will find them useful.

My three finalists were Load Runner, from Mercury Interactive; JMeter, from the Apache foundation, and The Grinder, an open-source project hosted on SourceForge.

2 SUMMARY OF RESULTS

I found that I could use any of them and get a reasonably good amount of scale test coverage. Each tool has unique things it does very well, so in that sense, there is no “wrong answer.” Conversely, each of the tools I considered have unique deficiencies that will impede or block one or more of the scenarios in our test plan. So there is no “right answer” either – any option selected will be something of a trade-off.

Based on this research, I recommended The Grinder as the tool to go forward with. It has a simple, clean UI that clearly shows what is going on without trying to do too much, and offers great power and simplicity with its unique Jython-based scripting approach. Jython allows complex scripts to be developed much more rapidly than in more formal languages like Java, yet it can access any Java library or class easily, allowing us to re-use elements of our existing work.

Mercury's Load Runner had a largely attractive feature set, but I ultimately disqualified it due to shortcomings in these make-or-break areas:

  • Very high price to license the software.

  • Generating unlimited load is not permitted. With the amount of load our license allows, I will be unable to effectively test important clustered server configurations, as well as many of our “surge” scenarios.

  • Very weak server monitoring for Solaris environments. No support for monitoring Solaris 10.

JMeter was initially seen as an attractive contender, with its easy, UI-based script development, as well as script management and deployment features. It's UI is feature-rich and this product has the Apache branding. It was ultimately brought down by the bugginess of it's UI though, as several of it's key monitors gave incorrect information or simply didn't work at all.

3 Comparison Tables

All the items in the tables below are discussed in greater detail in the following sections. These tables are to give a quick overview

3.1 Critical Items

There are several features that are key to any scale testing effort. Items in this table are key to our efforts. Not having any of these will seriously impact our ability to generate complete scale test coverage.

Item

Load Runner

JMeter

The Grinder

Solaris Monitoring

-

neutral

neutral

Unlimited Load generation

-

+

+

Supports IP spoofing

+

-

+

Large download performance

+

-

- *

* Multiple workarounds are being investigated, including calling native (libcurl) code for the most intensive downloads.

3.2 Non-Critical Items

Items in this section are not make-or-break to our test effort, but will go a long way to making the test effort more effective.

3.2.1 General

Item

Load Runner

JMeter

The Grinder

Server monitoring

mixed

neutral

neutral

Batch Mode

-

+

+

Ease - Installation

-

+

+

Ease – Script Authoring

+

+

mixed

Ease – Running Tests

neutral

+

neutral

Results Reporting

+

-

-

Agent Management

+

+

-

Cross Platform

-

+

+

Cost

-

+

+

Technical Level

+

+

-

Stability/Bugginess

neutral

-

neutral

3.2.2 Agents

Item

Load Runner

JMeter

The Grinder

Transaction power

+

neutral

+

Custom protocols

+

+

+

Out-of-the-box protocols

+

neutral

-

Transaction aggregation

+

-

+

Scalability of Agent

+

neutral

neutral

Slow sockets

+

neutral

-

External libs usable

+

+

+

Load Scheduling

+

+

-

Ease of porting JCS

-

neutral

+

3.2.3 Controller

Item

Load Runner

JMeter

The Grinder

Scalability of Controller

neutral

-

+

Real-time test monitoring

+

-

neutral

Real-time load adjustment

+

-

-

Script management

+

+

-

Script Development Environment

+

+

-

4 GENERAL

4.1 Server Monitoring -- Windows, Solaris, etc.

4.1.1 Load Runner

Mercury is extremely strong in this area for Windows testing. Unfortunately, it is very weak in unix/Solaris. For windows hosts, Load Runner uses the native performance counters available in perfmon. This allows monitoring myriad information from the OS, as well as metrics from individual applications (such as IIS) that make their information available to perfmon.

For Solaris hosts, Load Runner is restricted to the performance counters available via rpc.rstatd. This means some very basic information on CPU and memory use, but not much else. Note that Load Runner does not currently support any kind of performance monitoring on Solaris 10.

4.1.2 JMeter

JMeter has no monitoring built in. Thus, wrapper scripts are required to synchronize test data with external perf monitoring data. This is the approach I used to great effect with our previous test harness. The advantage of this method is I can monitor (and graph!) any information the OS makes available to us. Since the amount of data to us is quite large, this is a powerful technique.

4.1.3 The Grinder

The same wrapper-based approach would be required here as I detailed above for JMeter.

4.2 Can generate unlimited load

This is a make-or-break item. There are many scenarios I just can't cover if I can only open a few thousand socket connections to the server.

4.2.1 Load Runner

Load runner restricts the number of vusers you can run. Even large amounts of money only allow a licence for a modest number of users. Historically, the rate for 10,000 HTTP vusers has been $250,000. However, on a per agent basis, load is generated very efficiently so it may take less hardware to generate the same amount of load. (But for the money you spend on the load runner license, you could buy a LOT of load generation hardware!)

4.2.2 JMeter

Since this is Free/Open Source, you may run as many agents as you have hardware to put them on. You can add more and more load virtually forever, as long as you have more hardware to run additional agents on. However, in specific unicast scenarios, such as repeatedly downloading very large files (like PIPEDSCHEDULE), the ability of agents to generate load falls off abruptly due to memory issues.

4.2.3 The Grinder

In this matter the Grinder's story is the same as JMeter. The limit is only the number of Agents. The Grinder suffers the same lack of ability to effectively download large files as JMeter. A workaround that uses native code (libcurl) to send requests is being investigated.

4.3 Can run in batch (non-interactive) mode

4.3.1 Load Runner

No. Hands-free runs can be scheduled with the scheduler, but multiple specific scenarios cannot be launched from the command line. This may be adequate for single tests; it's not clear how this would work if a series of automated tests was desired.

4.3.2 JMeter

Yes, the ability to do this is supported out of the box. However, it can only be run from a single agent; the distributed testing mechanism requires the UI. So for automated nightly benchmarks it may be ok, but for push-to-failure testing where much load is required, the UI is needed. It would presumably be possible to have a wrapper script launch JMeter in batch mode at the same time on multiple agents. This would achieve arbitrary levels of load, but would not have valid data for collective statistics like total transactions per second, total transactions, etc.

4.3.3 The Grinder

As with JMeter, a single agent can be run from the command line. See JMeter comments, above.

4.4 Ease of Use

4.4.1 Load Runner

4.4.1.1 Installation

Installation takes a ton of time, a lot of disk space, and a very specific version of Windows. But it's as simple as running a windows installer, followed by 3 or 4 product updaters.

4.4.1.2 Setting up Simple tests

For HTTP tests, Load Runner is strong in this category, with it's browser recorder and icon-based scripts.

4.4.1.3 Running Tests

The UI of the controller is complex and a bit daunting. There is great power in the UI if you can find it.

4.4.2 JMeter

4.4.2.1 Installation

Be sure Sun's JRE is installed. Unpack the tar file. Simple.

4.4.2.2 Setting up Simple tests

Very quick. Start up the console, a few clicks of the mouse, and you are ready to generate load. Add thread group, add a sampler, and you have the basics. Throw in an assertion or two on your sampler to validate server responses.

4.4.2.3 Running Tests

Both distributed and local tests can be started form the UI. A menu shows the available agents, and grays out the ones that are already busy. Standalone tests can be started from the command line. JMeter wins this category hands down.

4.4.3 The Grinder

4.4.3.1 Installation

Installation is as simple as installing java, and unpacking a tar file.

4.4.3.2 Setting up Simple tests

Setting up tests, even simple tests, requires writing Jython code. So developer experience is important. A proxy script recorder is included to simplify this. In addition, there are many useful example scripts included to help you get started.

4.4.3.3 Running Tests

Involves configuring a Grinder.properties file, manually starting an agent process, manually starting the console, then telling the test to run from within the console UI.

4.5 Results Reporting

4.5.1 Auto-generated?

Having key graphs generated at the conclusion of a scale run, such as load over time, server CPU, transactions per second, etc, can save a lot of tedium, since manually generating these graphs from log files is quite time consuming.

4.5.1.1 Load Runner

Load runner has an excellent integrated analysis tool that can dynamically generate graphs on any of the myriad performance counters available to it. The downfall of this approach is that there are only a small number of performance metrics it can gather on Solaris. And while I can gather additional server metrics using sar, vmstat, dtrace, iostat, mpstat, etc., integrating this information in to the load runner framework will be difficult at best.

4.5.1.2 JMeter

JMeter does not gather any server-side performance metrics. But it can generate a limited number of client-side graphs while the test is in progress. These graphs can be saved after the tes is over. Fortunately, all the test data is written in a standard format. So it probably makes more sense to generate all the desired graphs via shell scripts during post-processing. This is the same approach I used with our previous test harness.

4.5.1.3 The Grinder

Like with JMeter, there are no graphs generated out of the box, but with the standard-format log files, scripted post-production is reasonably straightforward, giving us a powerful and flexible view on the results.

4.5.2 Analysis tools?

4.5.2.1 Load Runner

Yes very powerful tool for doing analysis after a run. An infinite number of customized graphs can be generated. These graphs can be exported into an html report.

4.5.2.2 JMeter

Nothing included. I would want to transfer over some of the scripts used in our previous test harness, or write a simple tool that dumps test data into a DB for post-analysis.

4.5.2.3 The Grinder

Nothing included. See the JMeter comments, above.

4.6 Simplicity of Agent management

4.6.1 Load Runner

This works well in Load Runner; each agent can run as a service or an application, simplifying management. Test scripts are auto-deployed to agents.

4.6.2 JMeter

JMeter is good here. Each agent is a server that the controller can connect to at will in real-time. Test scripts are automatically sent to each agent, centralizing management.

4.6.3 The Grinder

Grinder is the weakest here. The properties files that define how much load to apply, must be manually deployed to all agents. A wrapper shell script like the one used by our previous test harness could address this by always deploying the Jython scripts to the agents before each run.

4.7 Tool is cross-platform

4.7.1 Load Runner

Not really. A subset of the complete agent functionality can be had for agents running on Linux or Solaris. Non-windows agents run each vuser as a process rather than a thread, reducing the amount of load an agent can produce. The controller and VUGen both are Windows-only. And Load Runner is poor at measuring non-Windows server statistics.

4.7.2 JMeter

Yes. Java/Swing app is platform-agnostic.

4.7.3 The Grinder

Yes. This app is based on Java, Swing, and Jython. Like JMeter, it will run anywhere you can set up a JVM.

4.8 Cost

4.8.1 Load Runner

Expect to pay in the low to mid six-figures for a license allowing any kind of robust load-generation capacity. But that's not all, there are high ongoing support costs as well. For the same kind of money I could get over 100 powerful machines to use as scale agents, as well as associated network switches, cabling, etc.

4.8.2 JMeter

Free. (Apache License)

4.8.3 The Grinder

Free. (Grinder License)

4.9 Intended audience/technical level

4.9.1 Load Runner

Load Runner has the widest audience of all these tools; perhaps not surprising given its maturity as a commercial product. It's browser-recording and icon-based script development give it the lowest technical barriers to entry of any of the three products. A QA engineer with modest technical background and little to no coding skills can still be productive with tool. And it's ability to load Windows .dll's and other libraries give it a power and flexibility useful to developers and other more advanced users.

4.9.2 JMeter

JMeter does not require developer skills to perform basic tests in any of the protocols it support out of the box. A form-driven UI allows the user to design their own scenario. This scenario is then auto-deployed to all agents during test initialization.

4.9.3 The Grinder

While it's possible that a regular QA engineer could be used to run the console and perform some testing, the tool is really more aimed at developers. This is the only tool of the three that did not include any kind of icon-based or UI-based script development. At a minimum, users will need to know how to write Python/Jython code to create simple test scripts, and the ability to write custom Java classes may be required as well, depending on the scenario.

4.10 Stability/Bugginess

4.10.1 Load Runner

The controller crashes occasionally under heavy load, but this is infrequent and largely manageable. Other than this, the product seems robust enough.

4.10.2 JMeter

JMeter fares poorly in this area.

TODO

4.10.3 The Grinder

I found no issues with the Grinder, other than the previously-mentioned memory issue with large file downloads.

5 AGENTS

5.1 Power of transactions

5.1.1 how flexible on what can be passed/failed?

5.1.1.1 Load Runner

Any arbitrary criteria can be set to define if a transaction passes. This includes but is not limited to response time, contents of response body, response code, or just about anything else.

5.1.1.2 JMeter

In JMeter, samplers generate your test requests. You can add a wide variety of assertion types to any of your samplers. These will allow you assert on response code, match regular expressions against the response body, assert on the size or md5sum of the response.

5.1.1.3 The Grinder

As with Load Runner, pass/fail criteria has merely to be defined within the test script. Criteria can be whatever you want.

5.1.2 user-defined transaction/statistic types?

5.1.2.1 Load Runner

Yes – if you get away from the icon-based view in Vugen and go to the code level, you can wrap anything you want in a transaction to get timing information, pass/fail data, etc.

5.1.2.2 JMeter

Yes – done through plugins.

5.1.2.3 The Grinder

Yes – an API exists to easily wrap any Java or Jython method in a transaction.

5.2 Other Protocols

5.2.1 Which protocols are supported out of the box?

5.2.1.1 Load Runner

This varies by the type of license purchased, with each protocol having a separate cost and a separate limit for the number of allowable VUsers. The potential number of protocols is extremely high, including Java, ODBC, FTP, HTTP, and others.

5.2.1.2 JMeter

Supports several protocols out of the box:

  • jdbc

  • http

  • ftp

  • jndi

5.2.1.3 The Grinder

The Grinder only supports HTTP out of the box.

5.2.2 Can transactions wrap custom (non-http) protocols? Can transactions wrap multiple (http or other) requests to the server?

5.2.2.1 Load Runner

Yes. There are multiple ways to do this. You can implement your own protocol handler in a .dll or in Load Runner's pseudo-c. Then you can invoke this handler from any type of VUser that you have a license for. Alternately, unless your protocol is something uncommon, you can probably buy a pre-existing implementation of your protocol, and licenses for VUsers to run this protocol.

5.2.2.2 JMeter

Yes. An external Java plugin that supports your protocol must be added in to JMeter to support this.

5.2.2.3 The Grinder

Any protocol can be tested with the Grinder. An HTTP plugin is included. In other cases, you will create a separate Java class that implements a handler for your protocol. In your test script, you will wrap this Java class in a Grinder test object. Your protocol is used/invoked by calling any method you want from your java class via the test wrapper. The wrapper will pass/fail the transaction based on response time.

This default behavior can be overridden with additional code in your Jython script. For example, after invoking your protocol method, you could inspect the state of your Java object and pass/fail the transaction based on information there.

5.3 Capacity of single agent to generate load, particularly in high-bandwidth scenarios

I have typically seen libraries like Apache's HTTPClient max out the CPU to 100% when it's conducting high-bandwidth, large file downloads. The library supports high bandwidth use and many transactions per second just fine, but has issues with repeated large file downloads.

5.3.1 Load Runner

Per-agent load generation capacity is strong. Licensing constraints may limit actual load generated.

5.3.2 JMeter

With the exception of the high-bandwidth case, per agent capacity is good.

5.3.3 The Grinder

Runs out of memory when repeatedly downloading large documents in many threads. Currently, there does not seem to be a workaround inside The Grinder itself. However, with my previous test harness I was able to work around this same issue by calling native code, and there is reason to believe that approach may work here as well.

5.4 Can support IP spoofing

Assuming a large range of valid IP addresses assigned to the agent machines, does the test harness support binding outgoing requests to arbitrary IP addresses? The ability to support this is critical for out test effort. If all broker requests come in from the same IP address, the broker thrashes unrealistically as it continually updates customer settings.

5.4.1 Load Runner

Yes. (see link in appendix 1)

5.4.2 JMeter

JMeter is weak here. There is a new mechanism (not yet released but available in nightly builds) where outbound requests can round-robin on a predetermined list of local IPs. This is not good enough for Fat Client simulation.

5.4.3 The Grinder

The local IP address to bind the outbound request to can be specified in the Jython scripts. This is just what I need.

5.5 Can support variable connection speed/bandwidth throttling

5.5.1 Load Runner

Load Runner supports this out of the box.

5.5.2 JMeter

JMeter does not support this out of the box, but there is a slow socket implementation in the wild, written for the Apache HTTP Client (which JMeter uses), that should be possible to drop in fairly easily.

5.5.3 The Grinder

The Grinder does not support this. It may be possible with additional code hacking, but the path for this is not clear. Their third-party HTTP implementation means writing a custom solution may be challenging. Perhaps it would be possible using JNI and libCurl, although the author of the libCurl binding suggest there may be a memory leak in the C layer.

5.6 Can run arbitrary logic and external libraries within agent

5.6.1 Load Runner

Windows .dll's may be loaded. Home-made libraries written in Load Runner's pseudo-C libs work fine as well. Additionally, function libraries can be embedded directly in the virtual user script.

5.6.2 JMeter

External Java libraries can be accessed via the plugin architecture.

5.6.3 The Grinder

The Grinder offers lots of flexibility for loading and executing third party libraries. With Jython, any Java code may be called, and most python code may be run unchanged. And there is a decent collection of example scripts that comes with the Grinder distribution.

5.7 Scheduling

5.7.1 Load Runner

Load Runner has a powerful, UI-based scheduling tool which allows you great flexibility to schedule arbitrary amounts of load over time. Load can be incrementally stepped up and stepped down, by single threads or entire groups. There is a graphical schedule builder that can generate schedules of arbitrary complexity.

5.7.2 JMeter

JMeter has UI-based scheduling that allows per-thread startup delays, as well as runs that start in the future. JMeter tests can run forever, for a specified time interval, or for a specified number of iterations for each thread.

5.7.3 The Grinder

No per-thread ramp-in. No generic scheduling tool. Primitive per-process (instead of per-thread) scheduling is possible but use of this feature probably reduces an Agent's maximum load-generation capacity, as the overhead of running a new process is far greater than the overhead of creating a new thread.

6 CONTROLLER

6.1 Ability of Controller to handle high volume of agent data

6.1.1 Load Runner

Load runner probably handles as much or more real-time data as any product out there. But they do it effectively. If you give the controller a beefy box to run on, you should have no problems.

6.1.2 JMeter

Limited. The amount of transaction monitors you can have running is configurable. If more that one or two are going and the agents are producing a lot of transaction data, the UI takes all the CPU, bogs down and becomes unusable.

6.1.3 The Grinder

The grinder does very well here, probably better than Load Runner. By design, the agents only send a limited amount of real-time data back to the controller during a test run. And the sampling period is adjustable with a big friendly slider. This is a handy feature I didn't fully appreciate at first – if the network bandwidth numbers are updating too fast, it's hard to see how many digits are in the number before it updates again. But with the slider, you can lock that number down for enough time to really consider it.

6.2 Real-Time Monitoring (Controller)

6.2.1 Load Runner

Load Runner features very strong real-time monitoring in the controller. Client side graphs, such as total transactions per second, errors per second, can be displayed next to server side graphs like CPU use and disk activity. The user can drag and drop from a list of dozens of graph types.

6.2.2 JMeter

Basic, table-based monitoring similar to what is in our previous test harness works properly. Other monitors threw null pointer exceptions.

6.2.3 The Grinder

The Grinder is good here. It has simple, sliding performance graphs for all transactions in one tab. These graphs are similar to what you see in the Windows Task Manager, where performance metrics older than a given amount of time slide off the left side of the graph. In addition, as in our previous test harness or JMeter, there is numeric data that periodically updates in a table.

6.3 Real-Time Load Adjustment

Sometimes while a test is in progress, you want to make adjustments. Increase the load. Decrease the load. Bring another agent online.

6.3.1 Load Runner

Load Runner wrote the book on this topic, with its highly-flexible ability to start and stop load in the middle of a test, with individual agents, groups of agents, or the entire set of agents.

6.3.2 JMeter

JMeter has the ability to interactively start and stop load on an agent-by-agent basis. It cannot interactively be done at the per-thread level, but agents and thread groups can have schedulers assigned to them.

6.3.3 The Grinder

The Grinder console does not have the ability to dynamically adjust the levels of load being generated by the agents. Coupled with its lack of a scheduler, this makes the Grinder the least flexible of the three tools when it comes to interactively setting load levels.

6.4 Controller-side script management/deployment

6.4.1 Load Runner

Yes.

6.4.2 JMeter

Yes.

6.4.3 The Grinder

Yes.

6.5 Can write simple scripts in the UI?

6.5.1 Load Runner

Load Runner comes with a powerful script-development tool, VUGen. This gives the test developer the option of developing icon-based test scripts, as well as the traditional code-view development environment. In addition. Load Runner can record web browser sessions to auto-generate scripts based on the recorded actions.

6.5.2 JMeter

Scripts are based on XML. They can be written in your preferred text editor, or created in an icon-based UI in the controller window. I found this feature to be both easy to use and surprisingly flexible. There is also a recorder feature to let you interactively create your scripts.

6.5.3 The Grinder

The Grinder is the weakest of the three here. It does have a TCP Proxy feature that can record browser sessions into Jython scripts. But there is no integrated graphical environment for script development

7 CONCLUSION

I selected The Grinder due to several make-or-break issues. However, each tool has unique strengths and weaknesses. Which tool is ultimately best for you depends on a number of things, such as:

  • Does you budget allow for an expenditure ranging from several tens to hundreds of thousands of dollars?

  • Will you be testing in a windows-only environment?

  • What is the technical level of your scale testers?

Both of the open source projects have merits, but neither one is ideal. My approach will be to work with the Grinder development team to resolve the most serious offenders.

8 Appendix 1 – Additional information

Load Runner system requirements (controller must be on Windows!)

http://www.mercury.com/us/products/performance-center/loadrunner/requirements.html

Linux/Solaris server monitoring (weak)

http://www.mercury.com/us/products/performance-center/loadrunner/monitors/unix.html

JMeter home page

http://jakarta.apache.org/jmeter/

JMeter Manual

http://jakarta.apache.org/jmeter/usermanual/index.html

The Grinder home page

http://grinder.sourceforge.net/

The Grinder Manual

http://grinder.sourceforge.net/g3/getting-started.html

Windows IP address multi homing

http://support.microsoft.com/kb/q149426/

9 Appendix 2 – Distinguishing features

These are some of the distinguishing features of each product:

Cool with Load Runner

  • highly developed, mature product

  • strong support

  • It is complex, but feature-rich

Problems w/ Load Runner

  • Extreme cost, both up front and ongoing

  • Limited load generation capacity based on license/key.

  • Limited ability to monitor server stats outside of windows.

Cool w/ Grinder

  • Jython scripting means rapid script development

  • Jython simplifies coding complex tasks

  • Good real-time feedback in the UI in most tabs.

  • Sockets based agent/controller communications. Trouble-free in our testing.

Problems w/ Grinder:

(Since this original article was posted, many of these issues have been addressed. See the blog entry titled "The Grinder: Addressing the Warts.")
  • no scheduling; load is all-or-nothing

  • no slow sockets, no prospects for easily fixing this
  • Memory failures in a few large file download scenarios.

Cool w/ JMeter:

  • Less technical expertise required
  • Overall more “slick” or “polished” feel – availability of startup scripts, more utility in the UI.

Problems w/ JMeter:
  • Limited feedback in the UI when the test is running
  • Memory and CPU issues when downloading very large files
  • The UI is buggy. Big pieces, including monitors, just don't work. Many Null Pointer Exceptions in the log, etc.

23

28 comments:

Unknown said...

Very interesting review of these tools.
I was wondering why did you pick these tools in specific and did not review other tools such as WebLOAD. Obviously, I am not objective but using your criteria RadView’s WebLOAD tool is giving a good fight here and should be counted as well.

Eran Witkon
VP R&D
Radview - The Smart Choice in Web Application Testing
Mobile: +972-544-600677
mail: eranw@radview.com
http://www.radview.com

Travis Bear said...

I originally wrote this review for an internal audience at Siemens, since we needed to pick a new tool. I was already somewhat familiar with Load Runner, Silk performer, and JMeter before I started. I did a day or so on Google getting a large list of other test tool candidates, and then researching the candidates in greater detail. I started with about 40 tools which i narrowed down to 3.

Based on the white paper at your web site, it appears that Web Load is a tool for testing Microsoft web servers. While I have nothing against Microsoft, the solution we are developing at Siemens is based on Solaris and Java. All our load generation hardware is running Linux. So it appeared that your product would not be a good fit with our specific environment. True?

If I have the wrong idea, I'd be interested to know more -- how does it monitor Linux and Solaris system resources such as CPU, disk activity, etc., what kind of graphs can be generated, etc., etc.

Unknown said...

Thanks for your feedback.
1) First let’s start with the fact that Siemens in Europe are a good client of ours so you might get some inside information regarding our offering.
2) WebLOAD is a multi platform load testing tools targeted for any RIA application. I will double check our white papers to see why you got that impression.
3) WebLOAD engine run on all platforms including Linux and Solaris (the execution console and IDE runs on windows only).
4) WebLoad can test any Java/J2EE based applications, is certified by BEA WebLogic.
5) We are adding JMX support for our coming Q1 release.
6) WebLOAD support different performance measurements see table below:
Current PMM support
BEA WebLogic
IBM WebSphere
Microsoft Active Server Pages
Microsoft ASP .NET
Oracle
Microsoft SQL Server
Windows Platform
Unix platforms using SNMP
Unix platforms using RSTATD
Unix UC-Davis
Windows .NET Framework
Microsoft IIS
Sun One (former iplanet 6)
Sun One (former iplanet 4)
Netscape LDAP
General Performance Monitor
General SNMP
General Unix
7) WebLOAD includes 3 groups of statistics that provide a thorough view of the client side experience:
• The first group includes more than 30 pre-defined client side statistics (counters, timers and per-time unit statistics) which brings a very detailed view of HTTP protocol level statistics. This group includes metrics such as hits per second, connect time, process time, page time, round time and many more.
• The second group includes automatic transactions which report timers, on any page and HTTP error that was part of the session. In this group you can find the time it took to receive each page in the session, the number of times the session received specific error such as 404, etc.
• The third group includes user-defined counters, timers and transactions. In WebLOAD the user can add custom timers and counters to the test script. The user can also wrap parts of the script with a transaction block and collect statistics on that blocks (e.g. "how much time it takes to insert a new employee to the application")
In addition to the client side statistics WebLOAD can collect a wide range of server side statistics using its PMM mechanism as described above.
All WebLOAD statistics are collected on-line and can be viewed while the session is running and anytime afterwards. The statistics can be displayed as graphs or textual information. All statistics can be exported to various formats such as Excel, tabbed file, HTML.





For more information please refer to WebLOAD's user guide "WebLOAD statistcs reports".


Eran Witkon
VP R&D
Radview - The Smart Choice in Web Application Testing
Mobile: +972-544-600677
mail: eranw@radview.com
http://www.radview.com

Anonymous said...

I enjoyed reading your Shootout article but wanted to point out a potential discrepancy.

You state that LoadRunner only supports monitoring via rstatd and has no support for Solaris 10. It is my understanding that the only monitoring support JMeter and Grinder have is also through rstatd on any unix server. This leads me to a different conclusion regarding your 3 candidates and their support for monitoring. All three require rstat -d running on the Unix servers and if you want more than the basic counters supported by LR you will need to write scripts no matter which of these tools you select. I would rate LoadRunner, via the Controller or the LR Analysis application, superior to the support for either Grider or Jmeter.

I also feel that your conclusions make it sound like your evaluation disqualifies LR for some show stopping functional reasons, but then goes on to say that 2 of the reasons are licensing and licensing. Certainly, a tool that costs $100k is not going to work on a minimal budget. However, many of LR features are far superior to both Jmeter and Grinder. Most of these features are designed to save time in the preparation, execution and analysis of performance tests.

In our consulting work we are evaluating 3 Open Source tools and our preliminary findings show the monitoring and analysis side to be the most glaring area of functionality not handled by Open Source solutions. It is my experience (for the past 14 years) that LR saves countless hours on collating and analyzing results.

So for the budget conscious Open Source tools allow for a lower entry point into performance testing. However, you will spend more time creating, debugging, and much more time collecting, collating and analyzing data with the current Open Source solutions. Any analysis of commercial grade tools vs. Open Source tools should include a total cost of ownership line item to present the whole picture.

Travis Bear said...

Monitoring and analysis is one area where Load Runner is very strong, and I gave it a good rating in this area. But it's only strong if your target server runs Windows. Since our project runs on Solaris, we have to live with the limitations of rstatd, or roll own monitoring. And since rstatd is so limiting, we have had much better by logging to disk the performance metrics available through vmstat, sar, iostat, and prstat.

I hope this analysis is not seen as a dis on Load Runner -- if you've got the cash and your server runs Windows, I know from experience that it is a flexible, powerful solution. We opted for the Grinder due to licensing and due to Load Runner's weakness outside the Windows environment.

Speaking a bit more about server monitoring: JMeter and The Grinder do not do ANY performance monitoring -- either through rstatd or any other mechanism, so if you go down that road, then there is no avoiding that you will have to do the monitoring yourself.

The server monitoring solution we came up with in our shop is a little crude, but it's fast, it's general-purpose, and it works. I'll be making it available under an open-source license when I have some time. The idea is that we use a wrapper script that:

* connects to the target server(s) via ssh

*starts logging perf data (vmstat, iostat, sar, and prstat) to disk on the remote server

* starts the load generation UI, blocks until the UI exits,

* ssh back to the target server, runs awk scripts to convert the various log files into .csv files

* donwloads the .csv files

* graphs the .csv files with an automated tool such as ploticus or JFreeChart.

A similar approach to this would probably work against Windows servers as well, just replacing the solaris tools mentioned above with whatever perfmon counters you are interested in.

Boris said...

very interesting review. I am also Siemens employee in Germany and back in 2000 my department made decision to purchase LoadRunner.
Just one remark about LoadRunner test automation: Load Runner is offering OLE automation of controller and it's agents what would theoretically enable automated performance test with this tool

Matt said...

Great review, and an interesting read.

I have used LR for many years now, but have recently moved on to Grinder and JMeter.

I think LR (and the like) is a great consultants or IT Department tool. It has a low learning curve and is suited for QA's and non-developers. I think it is also very good for short-term Performance Projects.

However, the reason why we have implemented Grinder and JMeter here is not just to do with cost.

Everything we are developing is HTTP - which of course helps, but also we are implementing automated performance tests using these tools after nightly builds. And of course we can extend the functionality of these harnesses to perform scheduled, adhoc, and multi scenario based Performance Tests with ease.

It's true we have put quite a bit of effort in auto-manipulating and graphing output of these tools, as well as scripting Linux native performance tools.

But the outcome of this is more statistical information and system metrics then LR or any other tool could give us.

Another interesting listen is - http://video.google.co.uk/videoplay?docid=-6891978643577501895&q=google+benchmark+testing

Oh.. a quick grinder fix we found when wanting to increment thread ramp up:-

Add a parameter to your properties file and alter your code to:-
grinder.sleep(grinder.threadID * int(grinder.properties.getProperty("app.processIncrementInterval")))

Unknown said...

Very interesting review of these tools. Just 2 things:
- I think it would be more interesting doing a comparison viewing also other Mercury products you can use with LoadRunner like Mercury Diagnostic for J2EE
- It's important you can create a ramp in any testing product for a better testing strategies; for me this is a mustptjuh

Travis Bear said...

The Grinder does support ramping in load over time (as Matt points out, above), if that's what you're talking about.

I'm not sure about the other Mercury products. My expertise is in scale testing, so I'm probably not the best choice to review Mercury Diagnostic, or other ancillary stuff.

DJ RoLo said...

Nice work mate, you've hit some of the metrics I'm interested in and saved me some serious time, cheers!

Roland

saurabh mishra said...

Is VMSTAT Better then Rstatd ... can we rstatd only with monitoring tool...?

Oliver Erlewein said...

Hi

Good article.

We've decised on JMeter for our projects and have spent our money on hardware and scripting the distribution and charting of graphs. All in all we're still miles away from what LR would have cost us.

Nice side effect is that I can give every customer the whole installation after the project ends. No licensing problems.

Our projects are usually quite small so we don't need all the furly stuff LR offers. It would be like editing a TXT file with Word.

I find the most effort is spent in finding & fixing the issues so anything that gets me to that point is OK. I suppose grinder would have worked just as well for us.

Monitoring is a problem and I think I will take some of the stuff from the comments and have a go at it.

If my company will allow it I plan to release the number crunching parts as Open Source sometime soon so that anybody can use it. It is currently based on processing the JTL files with GnuPlot and GhostScript (PDF output). It runs on Linux and Cygwin.

Cheers
Oliver
New Zealand

Andrey said...

A good addition to The grinder would be using it together with GrinderStone: Eclipse plugins that allows to debug created scripts and thus achieve better quality of scripts. This will add some point for The Grinder to compete with LoadRunner powers.

You may visit GrinderStone home page here: http://code.google.com/p/grinderstone/

-------
Andrey Isaev
Andruschuk Borislav

Yura Kupitman said...

I agreed with the point that this review is very interesting. I had no personal experience with WebLOAD, but the impression as well feedbacks from my colleagues are really good. I'm just doubt that nobody mentioned slamd http://www.slamd.com/index.shtml project from SUN. It is very impressive framework that supports distributed load as well.

Unknown said...

In your post you have mentioned that it can run in batch more. Can you provide me a documentation using which I can automate the process everyday..

Travis Bear said...

Dime, this means running the Grinder with no console. Just start the agent (from the command line) and let it run. That's all there is to it. If you need specific details, this is covered in the grinder documentation.

-Travis

Dzmitry said...

I do not think that it is a big problem, that JMeter has no good reporting tool out-of-box. Fortunately, there are some solution, how to workaround it. One of them is to use ,
jmeter cloud
that has rather well-developed tools for analyzing results.

Innovapath said...
This comment has been removed by a blog administrator.
Unknown said...

Thanks for great comparison guide. There is a one which compares Grinder, JMeter, Gatling and Tsung as free and open source performance testing tools. Check Open Source Load Testing Tools: Which One Should You Use? if interested, it contains main features highlighted and compared, sample load test scenario and load reports.

dj said...

Thanks for sharing. Could you please post the versions of each tool that you did comparision with?

Unknown said...
This comment has been removed by a blog administrator.
Unknown said...
This comment has been removed by a blog administrator.
dwarakesh said...
This comment has been removed by a blog administrator.
Kristi said...

Great review, lots of good info! Did you check out Apica LoadTest when you did your research? It's what I use, just curious.

Ananta said...

Can I use to test android app with The Grinder use the Android Emulaotor and set the proxy?

Anonymous said...
This comment has been removed by a blog administrator.
Unknown said...

Buy Mixer Grinder At Best Price On Dealsothon
Visit:- World lowest Price Is Ready for You

Unknown said...

Buy Mixer Grinder At Best Price On Dealsothon
Visit:- World lowest Price Is Ready for You