 |
||||
|
|
|
||
|
Contents 
Introduction
Mercury Load Runner
IBM Rational Tester
Segue SilkPerformer
NeoLoad
Specifications
How we tested
Editor's choice
About RMIT |
|
||
|
|
||||
|
|
||||
We have found that most of the issues associated with an application are related to configuration rather than hardware. So by increasing the CPU speed, adding more memory, or increasing the bandwidth you're not necessarily going to shorten your response times, and by only benchmarking the actual application you're not going to be able to pinpoint the root cause of any issues. You have to use diagnostics to find the root cause of bottlenecks. Diagnostics usually come built in with the tool or you can get them as add-ons. Some even interface with third-party diagnostic tools.
To make it possible to perform end-to-end testing across every aspect of your enterprise application, you need a tool that can support your application's environment, Web services, and development frameworks. All the tools tested here support a large range of technologies but you will want to confirm with the vendor.
Scripting
Scripting used to be the most time consuming part of performance testing but now it is easy. These tools are supporting more and more protocols, which makes it simple when testing a range of different technologies, all you have to do is point and click to record a user action. Features like auto-correlation automatically detect and handle dynamic values that may have been recorded. Dynamic values such as usernames and times and dates can be parameterised with variables by a simple replace. There's no coding involved, and the tool takes snapshots of every page so you can go back and look at what you just recorded. One thing you can't do is view the script being played back just the way you recorded it -- you can only do this with functional testing tools.
Runtime
The main factors associated with running the scripts include the type of load you are going to place on the application. One can be a ramp up of load so you can see when the application is going to break. Sometimes this is not the best way to test your application as it will exhaust your hardware resources and mask the real problem.
Another approach is ramp up ramp down -- say you ramp up the number of users to 200 then sit on 200 users for 30 minutes then ramp down again. You could then separate the results into three lots -- the ramp up, the steady flat load, which ran for 30 minutes, as well as the ramp down. You would then be able to extract the information, which is most important. In most cases it's the steady load of 200 users you would be most interested in.
You can also do volume testing or run test scripts over and over with a predefined number of virtual users for a predefined time. Rational's product allows you to dynamically throw additional users onto an already running test. The tools are very flexible in how you can load up your applications. While the test scripts are running it's important that you monitor your hardware to see how it is coping. Whether the tools use agent or agentless monitors, it's essential to know the health of critical machines.
You're also going to want to be able to keep an eye on the health of the test machines which are generating the loads, just in case they are maxing out. If they are working too hard you're going to want to introduce another test machine to reduce the load.
Diagnostics
Diagnostics are crucial when you need to discover why your application is not performing and what the root cause is. It basically gives you an insight into your application, allowing you to drill down on the Web transaction that was returning the longest response time, then drill deeper to the method that the application is using to carry out the task, and then further to the SQL call that is the root cause of the delays. But beware, you will only find these capabilities in the more expensive tools.
Reporting
Once all the testing is done you will want to create detailed graphs and explanations. It's also good to be able to create your own templates so you can produce the same report without having to go through the whole process of setting up a report. Being able to overlay graphs and add annotations hence a good thing to do, though some of the graphs produced by these tools can be hard to understand. And when it comes to publishing the results your going to want reports in HTML or PDF.
From past experience, we have spent most of our time analysing the data generated from these tools. You don't want to spend too much time setting up and configuring the product and then spending hours upon hours creating scripts, you want to fast track past all this and get to what matters most -- the performance of your application.
I think you missed iMacros in your review. We use it to test our websites. It's great! It has by far the best support for AJAX and Flash applets and the scripting interface is extremely powerful.