 |
||||
|
|
|
||
|
Contents 
Introduction
Mercury Load Runner
IBM Rational Tester
Segue SilkPerformer
NeoLoad
Specifications
How we tested
Editor's choice
About RMIT |
|
||
|
|
||||
|
|
||||
Segue has been a long-time player in this area. Unfortunately we couldn't find any info on Rational to compare the two in terms of revenue but Segue recorded net revenues of an US$8.9 million in the second quarter of 2005, which was an 11 percent increase over the same time last year. If we are a bit generous and say total sales for the year nears $40 million, that's still less than six percent of Mercury's total sales last year.
We didn't have an engineer available to visit our Lab and help us with the install, because currently Segue don't have any real representation in Australia but will by the end of the year. It has a great support structure. Out of Belfast, Ireland, they provide support team between the hours 18:00 and 12:00 AEST. They also offer user forums and knowledge-base access for many resolutions as well as an online case logging service.
We were able to download SilkPerformer from the Segue Web site and pretty much ran the install without any help from Segue.
All up it took around 10 minutes to get it up and running and to run our first test script which says a lot about how easy it is to install, configure, record, and run a test. There were no proxy issues like with Rational. SilkPerformer automatically checks to see if you are going through a proxy server by using the details from your Internet Browser.
There were two main components of SilkPerformer -- the workbench and developer workbench, both similar. The Workbench is where you run your actual tests, while the developer workbench is where you can allow your developers to create and modify scripts as well as test them with up to five virtual users.
We checked the script by playing it back for one user. Then by using the true-log explorer we could analyse and customise the script. This is a very powerful area where you can find differences between your test and recorded session. From here you can also customise user data with a simple right click on the field you want to make variable. You can add verifications here to verify whether or not your application is responding with incorrect data values or error messages. This does, however, place a small overhead on the performance of your application, which is something to be wary about.
From the Workbench we configured the workload, allowing virtual users to have different connection speeds and emulate different browsers. We only saw this sort of customisation with LoadRunner. Before we ran the test with multiple users we checked the health status of the agents, which are the hosts that generate the load. In this case it was going to be the local host or the machine we installed SilkPerfomer on. By clicking on the agent's icon we were able to view a system summary for that particular agent, as well as view the maximum number of virtual users this agent can generate.
We thought this was a great way of knowing when you need to install a new agent on another machine. With most of the other vendors you have to monitor the CPU usage and memory and make a decision whether or not to employ another machine to generate the load.
There are a few ways we could distribute the workload -- by using an increasing load, steady state, dynamic, all-day, or queuing. Again this is very easy to use and the images that display each type of workload are great in helping you understand how the load really works. By running the test this opens up the performance explorer window. Here you will find a set of windows, which give you real-time information during the test.
We kept having to flick back and forth between the workbench and performance explorer window, which was a little annoying as both screens display important real-time information. It would be ideal if this could be combined into one screen. As far as diagnostics go, SilkPerformer takes a less intrusive approach, if you need to drill down to the source code, for example, you have to purchase an add-on application.
To open up a report simply click on the explore results button and export to HTML. The report is comprehensive and you can add text next to each bar graph. There are various ways of customising reports; you can do overlays, set time bound histograms, and templates.
|
| ||||||||||||||||||||||||||
I think you missed iMacros in your review. We use it to test our websites. It's great! It has by far the best support for AJAX and Flash applets and the scripting interface is extremely powerful.